{"thread":{"id":"48310","subject":"[PATCH v1 0/5] Allocate cache entries from memory pool","startedAt":"2018-04-17T16:34:47Z","lastAt":"2018-07-22T09:23:51Z","messageCount":100,"participants":["Jameson Miller","Ben Peart","Junio C Hamano","Stefan Beller","Jonathan Tan","Duy Nguyen","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"344874","messageId":"20180417163400.3875-1-jamill@microsoft.com","threadId":"48310","inReplyTo":null,"subject":"[PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:39Z","receivedAt":"2018-04-17T16:34:47Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"This patch series improves the performance of loading indexes by\nreducing the number of malloc() calls. Loading the index from disk is\npartly dominated by the time in malloc(), which is called for each\nindex entry. This patch series reduces the number of times malloc() is\ncalled as part of loading the index, and instead allocates a block of\nmemory upfront that is large enough to hold all of the cache entries,\nand chunks this memory itself. This change builds on [1], which is a\nprerequisite for this change.\n\nGit previously allocated block of memory for the index cache entries\nuntil [2].\n\nThis 5 part patch series is broken up as follows:\n\n 1/5, 2/5 - Move cache entry lifecycle methods behind an API\n\n 3/5 - Fill out memory pool API to include lifecycle and other\n \tmethods used in later patches\n\n 4/5 - Allocate cache entry structs from memory pool\n\n 5/5 - Add extra optional validation\n\nPerformance Benchmarks:\n\nTo evaluate the performance of this approach, the p0002-read-cache.sh\ntest was run with several combinations of allocators (glibc default,\ntcmalloc, jemalloc), with and without block allocation, and across\nseveral different index sized (100K, 1M, 2M entries). The details on\nhow these repositories were constructed can be found in [3].The\np0002-read-cache.sh was run with the iteration count set to 1 and\n$GIT_PERF_REPEAT_COUNT=10.\n\nThe tests were run with iteration count set to 1 because this best\napproximates the real behavior. The read_cache/discard_cache test will\nload / free the index N times, and the performance of this logic is\ndifferent between N = 1 and N > 1. As the production code does not\nread / discard the index in a loop, a better approximation is when N =\n1.\n\n100K\n\nTest baseline [4] block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.02(0.01+0.01) -33.3%\n\n1M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.17(0.07+0.09) -26.1%\n\n2M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.39(0.17+0.20) -13.3%\n\n\n100K is not a large enough sample size to show the perf impact of this\nchange, but we can see a perf improvement with 1M and 2M entries. For\ncompleteness, here is the p0002-read-cache tests for git.git and\nlinux.git:\n\ngit.git:\n\nTest baseline block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 0.30(0.26+0.03) 0.17(0.13+0.03) -43.3%\n\nlinux.git:\n\nTest baseline block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 7.05(6.01+0.84) 4.61(3.74+0.66) -34.6% \n\n\nWe also investigated the performance of just using different\nallocators. We can see that there is not a consistent performance\ngain.\n\n100K\n\nTest baseline [4] tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.03(0.01+0.01) +0.0% 0.03(0.02+0.01) +0.0% \n\n1M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.21(0.10+0.10) -8.7% 0.27(0.16+0.10) +17.4%\n\n2M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.46(0.25+0.21) +2.2% 0.57(0.36+0.21) +26.7%\n\n\n[1] https://public-inbox.org/git/20180321164152.204869-1-jamill@microsoft.com/\n\n[2] debed2a629 (read-cache.c: allocate index entries individually - 2011-10-24)\n\n[3] Constructing test repositories:\n\nThe test repositories were constructed with t/perf/repos/many_files.sh with the following parameters:\n\n100K:\t many-files.sh 4 10 9\n1M:\t many-files.sh 5 10 9\n2M:\t many-files.sh 6 8 7\n\n[4] baseline commit: 8b026eda Revert \"Merge branch 'en/rename-directory-detection'\"\n\nJameson Miller (5):\n read-cache: teach refresh_cache_entry to take istate\n Add an API creating / discarding cache_entry structs\n mem-pool: fill out functionality\n Allocate cache entries from memory pools\n Add optional memory validations around cache_entry lifecyle\n\n apply.c | 26 +++---\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 8 +-\n builtin/reset.c | 6 +-\n builtin/update-index.c | 26 +++---\n cache.h | 40 ++++++++-\n git.c | 3 +\n mem-pool.c | 136 ++++++++++++++++++++++++++++-\n mem-pool.h | 34 ++++++++\n merge-recursive.c | 4 +-\n read-cache.c | 229 +++++++++++++++++++++++++++++++++++++++----------\n resolve-undo.c | 6 +-\n split-index.c | 31 +++++--\n tree.c | 4 +-\n unpack-trees.c | 27 +++---\n 16 files changed, 476 insertions(+), 117 deletions(-)\n\n\nbase-commit: cafaccae98f749ebf33495aec42ea25060de8682\n-- \n2.14.3\n\n\n"},{"id":"344875","messageId":"20180417163400.3875-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:39Z","receivedAt":"2018-04-17T16:34:53Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"This patch series improves the performance of loading indexes by\nreducing the number of malloc() calls. Loading the index from disk is\npartly dominated by the time in malloc(), which is called for each\nindex entry. This patch series reduces the number of times malloc() is\ncalled as part of loading the index, and instead allocates a block of\nmemory upfront that is large enough to hold all of the cache entries,\nand chunks this memory itself. This change builds on [1], which is a\nprerequisite for this change.\n\nGit previously allocated block of memory for the index cache entries\nuntil [2].\n\nThis 5 part patch series is broken up as follows:\n\n 1/5, 2/5 - Move cache entry lifecycle methods behind an API\n\n 3/5 - Fill out memory pool API to include lifecycle and other\n \tmethods used in later patches\n\n 4/5 - Allocate cache entry structs from memory pool\n\n 5/5 - Add extra optional validation\n\nPerformance Benchmarks:\n\nTo evaluate the performance of this approach, the p0002-read-cache.sh\ntest was run with several combinations of allocators (glibc default,\ntcmalloc, jemalloc), with and without block allocation, and across\nseveral different index sized (100K, 1M, 2M entries). The details on\nhow these repositories were constructed can be found in [3].The\np0002-read-cache.sh was run with the iteration count set to 1 and\n$GIT_PERF_REPEAT_COUNT=10.\n\nThe tests were run with iteration count set to 1 because this best\napproximates the real behavior. The read_cache/discard_cache test will\nload / free the index N times, and the performance of this logic is\ndifferent between N = 1 and N > 1. As the production code does not\nread / discard the index in a loop, a better approximation is when N =\n1.\n\n100K\n\nTest baseline [4] block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.02(0.01+0.01) -33.3%\n\n1M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.17(0.07+0.09) -26.1%\n\n2M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.39(0.17+0.20) -13.3%\n\n\n100K is not a large enough sample size to show the perf impact of this\nchange, but we can see a perf improvement with 1M and 2M entries.\n\nFor completeness, here is the p0002-read-cache tests for git.git and\nlinux.git:\n\ngit.git:\n\nTest baseline [4] block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 0.30(0.26+0.03) 0.17(0.13+0.03) -43.3%\n\nlinux.git:\n\nTest baseline block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 7.05(6.01+0.84) 4.61(3.74+0.66) -34.6% \n\n\nWe also investigated the performance of just using different\nallocators. We can see that there is not a consistent performance\ngain.\n\n100K\n\nTest baseline [4] tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.03(0.01+0.01) +0.0% 0.03(0.02+0.01) +0.0% \n\n1M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.21(0.10+0.10) -8.7% 0.27(0.16+0.10) +17.4%\n\n2M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.46(0.25+0.21) +2.2% 0.57(0.36+0.21) +26.7%\n\n\n\n[1] https://public-inbox.org/git/20180321164152.204869-1-jamill@microsoft.com/\n\n[2] debed2a629 (read-cache.c: allocate index entries individually - 2011-10-24)\n\n[3] Constructing test repositories:\n\nThe test repositories were constructed with t/perf/repos/many_files.sh with the following parameters:\n\n100K:\t many-files.sh 4 10 9\n1M:\t many-files.sh 5 10 9\n2M:\t many-files.sh 6 8 7\n\n[4] baseline commit: 8b026eda Revert \"Merge branch 'en/rename-directory-detection'\"\n\nJameson Miller (5):\n read-cache: teach refresh_cache_entry to take istate\n Add an API creating / discarding cache_entry structs\n mem-pool: fill out functionality\n Allocate cache entries from memory pools\n Add optional memory validations around cache_entry lifecyle\n\n apply.c | 26 +++---\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 8 +-\n builtin/reset.c | 6 +-\n builtin/update-index.c | 26 +++---\n cache.h | 40 ++++++++-\n git.c | 3 +\n mem-pool.c | 136 ++++++++++++++++++++++++++++-\n mem-pool.h | 34 ++++++++\n merge-recursive.c | 4 +-\n read-cache.c | 229 +++++++++++++++++++++++++++++++++++++++----------\n resolve-undo.c | 6 +-\n split-index.c | 31 +++++--\n tree.c | 4 +-\n unpack-trees.c | 27 +++---\n 16 files changed, 476 insertions(+), 117 deletions(-)\n\n\nbase-commit: cafaccae98f749ebf33495aec42ea25060de8682\n-- \n2.14.3\n\n\n"},{"id":"344876","messageId":"20180417163400.3875-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 2/5] Add an API creating / discarding cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:41Z","receivedAt":"2018-04-17T16:34:59Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an API around managing the lifetime of cache_entry structs. Abstracting\nmemory management details behind an API will allow for alternative memory\nmanagement strategies without affecting all the call sites. This commit does\nnot change how memory is allocated / freed. A later commit in this series will\nallocate cache entries from memory pools as appropriate.\n\nMotivation:\nIt has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This\nchange is in preparation for using memory pools to reduce the number\nof malloc() calls made when loading an index.\n\nThis API makes a distinction between cache entries that are intended for use\nwith a particular to an index and cache entries that are not. This enables us\nto use the knowledge about how a cache entry will be used to make informed\ndecisions about how to handle the corresponding memory.\n---\n apply.c | 26 ++++++------\n blame.c | 5 +--\n builtin/checkout.c | 8 ++--\n builtin/difftool.c | 8 ++--\n builtin/reset.c | 6 +--\n builtin/update-index.c | 26 ++++++------\n cache.h | 29 +++++++++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 105 +++++++++++++++++++++++++++++++++++--------------\n resolve-undo.c | 6 ++-\n split-index.c | 8 ++--\n tree.c | 4 +-\n unpack-trees.c | 27 ++++++++-----\n 13 files changed, 166 insertions(+), 94 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 7e5792c996..47903f427b 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4090,12 +4090,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_index_cache_entry(&result, patch->old_mode, oid.hash, name, 0, 0);\n \t\tif (!ce)\n-\t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\treturn error(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4263,12 +4263,11 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n \tif (!state->update_index)\n \t\treturn 0;\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4278,13 +4277,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tindex_cache_entry_discard(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tindex_cache_entry_discard(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4292,13 +4291,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4422,27 +4421,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_index_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex 78c9808bd1..8067e398a1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -154,7 +154,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -252,8 +252,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b49b582071..2ae2aff81a 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, oid.hash, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\ttransient_cache_entry_discard(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex ee8dce019e..db7b256a7b 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid->hash, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\ttransient_cache_entry_discard(ce);\n \treturn ret;\n }\n \n@@ -488,8 +488,8 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n-\t\t\t\t\t\t\t dst_path, 0, 0);\n+\t\t\t\t\tmake_index_cache_entry(&wtindex, rmode, roid.hash,\n+\t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\n \t\t\t\t\t\tADD_CACHE_JUST_APPEND);\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 7f1c3f02a3..1062dab73f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,10 +134,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n-\t\t\t\t 0, 0);\n+\t\tce = make_index_cache_entry(&the_index, one->mode, one->oid.hash, one->path,\n+\t\t\t\t\t 0, 0);\n \t\tif (!ce)\n-\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\tdie(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n \t\tif (is_missing) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 10d070a76f..9adb366d6c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -403,15 +402,14 @@ static int process_path(const char *path)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -589,7 +587,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -602,8 +599,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -680,8 +676,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tindex_cache_entry_discard(ce_2);\n+\tindex_cache_entry_discard(ce_3);\n \treturn ret;\n }\n \n@@ -748,7 +744,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tindex_cache_entry_discard(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -759,7 +755,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tindex_cache_entry_discard(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex e50a847aea..eedf154815 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,34 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and freeing */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `index_cache_entry_discard`.\n+ */\n+extern struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `transient_cache_entry_discard`.\n+ */\n+extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n+extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry allocated via `make_*_index_cache_entry`.\n+ */\n+void index_cache_entry_discard(struct cache_entry *ce);\n+\n+/*\n+ * Discard cache entry allocated via `make_*_transient_cache_entry`.\n+ */\n+void transient_cache_entry_discard(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -690,7 +718,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 693f60e0a3..be118c0c6d 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -252,7 +252,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_index_cache_entry(&the_index, mode, oid ? oid->hash : null_sha1, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2cb4f53b57..04fa7e1bd0 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tindex_cache_entry_discard(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_index_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_index_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tindex_cache_entry_discard(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,12 +744,22 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode,\n+\t\t\t const unsigned char *sha1, const char *path,\n+\t\t\t int stage, unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -758,8 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, len);\n \n \thashcpy(ce->oid.hash, sha1);\n \tmemcpy(ce->name, path, len);\n@@ -769,10 +777,34 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n+\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1,\n+\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\thashcpy(ce->oid.hash, sha1);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1243,7 +1275,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1323,8 +1355,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1610,12 +1641,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1657,7 +1689,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1688,13 +1721,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1826,7 +1859,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1932,7 +1965,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tindex_cache_entry_discard(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2622,14 +2655,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_index_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2738,3 +2770,16 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+/*\n+ * Free cache entry allocated for an index.\n+ */\n+void index_cache_entry_discard(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\n+\n+void transient_cache_entry_discard(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex aed95b4b35..96ef6307a6 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,8 +146,10 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n-\t\t\t\t name, i + 1, 0);\n+\t\tnce = make_index_cache_entry(istate,\n+\t\t\t\t\t ru->mode[i],\n+\t\t\t\t\t ru->oid[i].hash,\n+\t\t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\n \t\tif (add_index_entry(istate, nce, ADD_CACHE_OK_TO_ADD)) {\ndiff --git a/split-index.c b/split-index.c\nindex 3eb8ff1b43..361c821096 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tindex_cache_entry_discard(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tindex_cache_entry_discard(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tindex_cache_entry_discard(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tindex_cache_entry_discard(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 1c68ea586b..1ba39c9374 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -16,15 +16,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e73745051e..232cdecc72 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -192,10 +192,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -205,7 +205,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -786,10 +786,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\tconst struct name_entry *n,\n+\tint stage,\n+\tstruct index_state *istate,\n+\tint is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_index_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -835,7 +842,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -844,7 +851,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\ttransient_cache_entry_discard(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1765,7 +1772,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1785,7 +1792,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tindex_cache_entry_discard(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1811,7 +1818,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tindex_cache_entry_discard(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.14.3\n\n"},{"id":"344877","messageId":"20180417163400.3875-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 1/5] read-cache: teach refresh_cache_entry to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:40Z","receivedAt":"2018-04-17T16:35:02Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactoring dependencies of make_cache_entry to work on a specific\nindex, instead of implicitly using the_index. This is in preparation\nfor making the make_cache_entry function work on a specific index.\n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 7 ++++---\n 3 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bbaf5c349a..e50a847aea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -743,7 +743,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624d..693f60e0a3 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -260,7 +260,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 10f1c6bb8a..2cb4f53b57 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1448,10 +1448,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\t struct cache_entry *ce,\n \t\t\t\t\t unsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.14.3\n\n"},{"id":"344878","messageId":"20180417163400.3875-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 3/5] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:42Z","receivedAt":"2018-04-17T16:35:05Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Adds the following functionality to memory pools:\n\n - Lifecycle management functions (init, discard)\n\n - Test whether a memory location is part of the managed pool\n\n - Function to combine 2 pools\n\nThis also adds logic to track all memory allocations made by a memory\npool.\n\nThese functions will be used in a future commit in this commit series.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n mem-pool.h | 32 +++++++++++++++++++\n 2 files changed, 131 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..09fb78d093 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,6 +5,8 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n {\n \tstruct mp_block *p;\n@@ -19,6 +21,59 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \treturn p;\n }\n \n+static void *mem_pool_alloc_custom(struct mem_pool *mem_pool, size_t block_alloc)\n+{\n+\tchar *p;\n+\tALLOC_GROW(mem_pool->custom, mem_pool->nr + 1, mem_pool->alloc);\n+\tALLOC_GROW(mem_pool->custom_end, mem_pool->nr_end + 1, mem_pool->alloc_end);\n+\n+\tp = xmalloc(block_alloc);\n+\tmem_pool->custom[mem_pool->nr++] = p;\n+\tmem_pool->custom_end[mem_pool->nr_end++] = p + block_alloc;\n+\n+\tmem_pool->pool_alloc += block_alloc;\n+\n+\treturn mem_pool->custom[mem_pool->nr - 1];\n+}\n+\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tif (!(*mem_pool))\n+\t{\n+\t\t*mem_pool = xmalloc(sizeof(struct mem_pool));\n+\t\t(*mem_pool)->pool_alloc = 0;\n+\t\t(*mem_pool)->mp_block = NULL;\n+\t\t(*mem_pool)->block_alloc = BLOCK_GROWTH_SIZE;\n+\t\t(*mem_pool)->custom = NULL;\n+\t\t(*mem_pool)->nr = 0;\n+\t\t(*mem_pool)->alloc = 0;\n+\t\t(*mem_pool)->custom_end = NULL;\n+\t\t(*mem_pool)->nr_end = 0;\n+\t\t(*mem_pool)->alloc_end = 0;\n+\n+\t\tif (initial_size > 0)\n+\t\t\tmem_pool_alloc_block(*mem_pool, initial_size);\n+\t}\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tint i;\n+\tstruct mp_block *block, *block_to_free;\n+\tfor (block = mem_pool->mp_block; block;) {\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfor (i = 0; i < mem_pool->nr; i++)\n+\t\tfree(mem_pool->custom[i]);\n+\n+\tfree(mem_pool->custom);\n+\tfree(mem_pool->custom_end);\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p;\n@@ -33,10 +88,8 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \t\t\tbreak;\n \n \tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n+\t\tif (len >= (mem_pool->block_alloc / 2))\n+\t\t\treturn mem_pool_alloc_custom(mem_pool, len);\n \n \t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n \t}\n@@ -53,3 +106,45 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tstruct mp_block *p;\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\tint i;\n+\tstruct mp_block **tail = &dst->mp_block;\n+\n+\t/* Find pointer of dst's last block (if any) */\n+\twhile (*tail)\n+\t\ttail = &(*tail)->next_block;\n+\n+\t/* Append the blocks from src to dst */\n+\t*tail = src->mp_block;\n+\n+\tALLOC_GROW(dst->custom, dst->nr + src->nr, dst->alloc);\n+\tALLOC_GROW(dst->custom_end, dst->nr_end + src->nr_end, dst->alloc_end);\n+\n+\tfor (i = 0; i < src->nr; i++) {\n+\t\tdst->custom[dst->nr++] = src->custom[i];\n+\t\tdst->custom_end[dst->nr_end++] = src->custom_end[i];\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+\tsrc->custom = NULL;\n+\tsrc->nr = 0;\n+\tsrc->alloc = 0;\n+\tsrc->custom_end = NULL;\n+\tsrc->nr_end = 0;\n+\tsrc->alloc_end = 0;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..34df4fa709 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -19,8 +19,27 @@ struct mem_pool {\n \n \t/* The total amount of memory allocated by the pool. */\n \tsize_t pool_alloc;\n+\n+\t/*\n+\t * Array of pointers to \"custom size\" memory allocations.\n+\t * This is used for \"large\" memory allocations.\n+\t * The *_end variables are used to track the range of memory\n+\t * allocated.\n+\t */\n+\tvoid **custom, **custom_end;\n+\tint nr, nr_end, alloc, alloc_end;\n };\n \n+/*\n+ * Initialize mem_pool specified initial.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n@@ -31,4 +50,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.14.3\n\n"},{"id":"344879","messageId":"20180417163400.3875-7-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 5/5] Add optional memory validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:43Z","receivedAt":"2018-04-17T16:35:09Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n---\n cache.h | 7 +++++++\n git.c | 3 +++\n mem-pool.c | 35 ++++++++++++++++++++++++++++++++++-\n mem-pool.h | 2 ++\n read-cache.c | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 90 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7c0d2343c3..f8934d8113 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -369,6 +369,13 @@ void index_cache_entry_discard(struct cache_entry *ce);\n */\n void transient_cache_entry_discard(struct cache_entry *ce);\n \n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/git.c b/git.c\nindex 3a89893712..16b6c1685b 100644\n--- a/git.c\n+++ b/git.c\n@@ -347,7 +347,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex 09fb78d093..a7e28934b0 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -60,20 +60,44 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n {\n \tint i;\n \tstruct mp_block *block, *block_to_free;\n+\tint invalidate_memory = should_validate_cache_entries();\n+\n \tfor (block = mem_pool->mp_block; block;) {\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \n-\tfor (i = 0; i < mem_pool->nr; i++)\n+\tfor (i = 0; i < mem_pool->nr; i++) {\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(mem_pool->custom[i], 0xDD, ((char *)mem_pool->custom_end[i]) - ((char *)mem_pool->custom[i]));\n+\n \t\tfree(mem_pool->custom[i]);\n+\t}\n \n \tfree(mem_pool->custom);\n \tfree(mem_pool->custom_end);\n \tfree(mem_pool);\n }\n \n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p;\n@@ -110,11 +134,20 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n {\n \tstruct mp_block *p;\n+\tint i;\n+\n+\t/* Check if memory is allocated in a block */\n \tfor (p = mem_pool->mp_block; p; p = p->next_block)\n \t\tif ((mem >= ((void *)p->space)) &&\n \t\t (mem < ((void *)p->end)))\n \t\t\treturn 1;\n \n+\t/* Check custom memory allocations */\n+\tfor (i = 0; i < mem_pool->nr; i++)\n+\t\tif (mem >= mem_pool->custom[i] &&\n+\t\t mem < mem_pool->custom_end[i])\n+\t\t\treturn 1;\n+\n \treturn 0;\n }\n \ndiff --git a/mem-pool.h b/mem-pool.h\nindex 34df4fa709..b1f9a920ba 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -63,4 +63,6 @@ void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n */\n int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n \n+int should_validate_cache_entries(void);\n+\n #endif\ndiff --git a/read-cache.c b/read-cache.c\nindex 67438bf375..d2181a0334 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1290,6 +1290,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\t\t istate->cache_nr - pos - 1);\n \tset_index_entry(istate, pos, ce);\n \tistate->cache_changed |= CE_ENTRY_ADDED;\n+\n \treturn 0;\n }\n \n@@ -2013,6 +2014,8 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n+\tvalidate_cache_entries(istate);\n+\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -2035,6 +2038,43 @@ int discard_index(struct index_state *istate)\n \treturn 0;\n }\n \n+\n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\tint validate_index_cache_entries = should_validate_cache_entries();\n+\n+\tif (!validate_index_cache_entries)\n+\t\treturn;\n+\n+\tif (!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2828,6 +2868,10 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n */\n void index_cache_entry_discard(struct cache_entry *ce)\n {\n+\tint invalidate_cache_entry = should_validate_cache_entries();\n+\n+\tif (ce && invalidate_cache_entry)\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n }\n \n void transient_cache_entry_discard(struct cache_entry *ce)\n-- \n2.14.3\n\n"},{"id":"344880","messageId":"20180417163400.3875-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v1 4/5] Allocate cache entries from memory pools","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-17T16:34:42Z","receivedAt":"2018-04-17T16:35:14Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Improve performance of reading a large index by reducing the number of\nmalloc() calls. When reading an index with a large number of entries,\na portion of the time is dominated in malloc() calls. This can be\nmitigated by allocating a single large block of memory up front into a\nmemory pool and have git hand out chunks of time.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\n\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entry structs will be allocated from. When reading in\nthe index from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool and cache entries from the memory pool will be\ndiscarded as well. This means the lifetime of cache_entry structs are\ntied to the lifetime of the index_state they were allocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n---\n cache.h | 2 ++\n read-cache.c | 95 +++++++++++++++++++++++++++++++++++++++++++++--------------\n split-index.c | 23 +++++++++++++--\n 3 files changed, 95 insertions(+), 25 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex eedf154815..7c0d2343c3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -328,6 +329,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 04fa7e1bd0..67438bf375 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,42 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is an estimate of the average pathname length in the index. We use\n+ * this for V4 index files to guess the un-deltafied size of the index in\n+ * memory because of pathname deltafication. This is not required for V2/V3\n+ * index formats because their pathnames are not compressed. If the initial\n+ * amount of memory set aside is not sufficient, the mem pool will allocate\n+ * extra memory.\n+ */\n+#define CACHE_ENTRY_AVG_PATH_LENGTH_ESTIMATE 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\treturn mem_pool_alloc(mem_pool, cache_entry_size(len));\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\treturn mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +782,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1641,13 +1677,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1689,7 +1725,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1721,13 +1757,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1801,6 +1837,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_AVG_PATH_LENGTH_ESTIMATE);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1847,10 +1899,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1859,7 +1916,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1956,17 +2013,6 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n-\n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tindex_cache_entry_discard(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -1980,6 +2026,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2772,11 +2824,10 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n }\n \n /*\n- * Free cache entry allocated for an index.\n+ * Indicate that a cache entry is no longer is use\n */\n void index_cache_entry_discard(struct cache_entry *ce)\n {\n-\tfree(ce);\n }\n \n void transient_cache_entry_discard(struct cache_entry *ce)\ndiff --git a/split-index.c b/split-index.c\nindex 361c821096..a920c0ad07 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -330,6 +345,8 @@ void add_split_index(struct index_state *istate)\n void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n \t\t/*\n \t\t * can't discard_split_index(&the_index); because that\n \t\t * will destroy split_index->base->cache[], which may\n-- \n2.14.3\n\n"},{"id":"344920","messageId":"d2743302-43dd-da3e-f0ba-55dcb824c0f4@gmail.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-17T18:39:41Z","receivedAt":"2018-04-17T18:39:54Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/17/2018 12:34 PM, Jameson Miller wrote:\n\n> 100K\n> \n> Test baseline [4] block_allocation\n> ------------------------------------------------------------------------------------\n> 0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.02(0.01+0.01) -33.3%\n> \n> 1M:\n> \n> Test baseline block_allocation\n> ------------------------------------------------------------------------------------\n> 0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.17(0.07+0.09) -26.1%\n> \n> 2M:\n> \n> Test baseline block_allocation\n> ------------------------------------------------------------------------------------\n> 0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.39(0.17+0.20) -13.3%\n> \n> \n> 100K is not a large enough sample size to show the perf impact of this\n> change, but we can see a perf improvement with 1M and 2M entries.\n\nI see a 33% change with 100K files which is a substantial improvement \neven in the 100K case. I do see that the actual wall clock savings \naren't nearly as much with a small repo as it is with the larger repos \nwhich makes sense.\n"},{"id":"344924","messageId":"abb3ad34-94ff-4778-3e44-eaf1ccac1849@gmail.com","threadId":"48310","inReplyTo":"20180417163400.3875-3-jamill@microsoft.com","subject":"Re: [PATCH v1 1/5] read-cache: teach refresh_cache_entry to take istate","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-17T19:00:07Z","receivedAt":"2018-04-17T19:00:13Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/17/2018 12:34 PM, Jameson Miller wrote:\n> Refactoring dependencies of make_cache_entry to work on a specific\n\nRefactoring/Refactor\n\nIt's helpful to refer to functions with parens i.e. make_cache_entry().\n\nIn addition, it appears you only needed to update refresh_cache_entry() \nso perhaps something like:\n\n\nRefactor refresh_cache_entry() to work on a specific index, instead of \nimplicitly using the_index. This is in preparation for making the \nmake_cache_entry() function work on a specific index.\n\n\nAlso, be sure to certify your work by adding your \"Signed-off-by: \" line \nto each commit. Details at:\n\nhttps://github.com/git/git/blob/master/Documentation/SubmittingPatches\n\n\n> index, instead of implicitly using the_index. This is in preparation\n> for making the make_cache_entry function work on a specific index.\n"},{"id":"344952","messageId":"921a8d5c-b118-c5b1-35dd-459130faeed9@gmail.com","threadId":"48310","inReplyTo":"20180417163400.3875-4-jamill@microsoft.com","subject":"Re: [PATCH v1 2/5] Add an API creating / discarding cache_entry structs","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-04-17T23:11:59Z","receivedAt":"2018-04-17T23:12:07Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 4/17/2018 12:34 PM, Jameson Miller wrote:\n> Add an API around managing the lifetime of cache_entry structs. Abstracting\n> memory management details behind an API will allow for alternative memory\n> management strategies without affecting all the call sites. This commit does\n> not change how memory is allocated / freed. A later commit in this series will\n> allocate cache entries from memory pools as appropriate.\n> \n> Motivation:\n> It has been observed that the time spent loading an index with a large\n> number of entries is partly dominated by malloc() calls. This\n> change is in preparation for using memory pools to reduce the number\n> of malloc() calls made when loading an index.\n> \n> This API makes a distinction between cache entries that are intended for use\n> with a particular to an index and cache entries that are not. \n\nThe wording here is awkward. Did you mean \"intended for use with a \nparticular index?\"\n\nThis enables us\n> to use the knowledge about how a cache entry will be used to make informed\n> decisions about how to handle the corresponding memory.\n> ---\n> apply.c | 26 ++++++------\n> blame.c | 5 +--\n> builtin/checkout.c | 8 ++--\n> builtin/difftool.c | 8 ++--\n> builtin/reset.c | 6 +--\n> builtin/update-index.c | 26 ++++++------\n> cache.h | 29 +++++++++++++-\n> merge-recursive.c | 2 +-\n> read-cache.c | 105 +++++++++++++++++++++++++++++++++++--------------\n> resolve-undo.c | 6 ++-\n> split-index.c | 8 ++--\n> tree.c | 4 +-\n> unpack-trees.c | 27 ++++++++-----\n> 13 files changed, 166 insertions(+), 94 deletions(-)\n> \n> diff --git a/apply.c b/apply.c\n> index 7e5792c996..47903f427b 100644\n> --- a/apply.c\n> +++ b/apply.c\n> @@ -4090,12 +4090,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n> \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n> \t\t\t\t \"(%s).\"), name);\n> \n> -\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n> +\t\tce = make_index_cache_entry(&result, patch->old_mode, oid.hash, name, 0, 0);\n> \t\tif (!ce)\n> -\t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n> +\t\t\treturn error(_(\"make_index_cache_entry failed for path '%s'\"),\n> \t\t\t\t name);\n> \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n> -\t\t\tfree(ce);\n> +\t\t\tindex_cache_entry_discard(ce);\n\nI personally prefer name symmetry. To me, make_index_cache_entry() \nshould be paired with discard_index_cache_entry().\n\nThe rest of this patch looks like a fairly mechanical refactoring with \nthe biggest exception being the difference between the the \n*_index_cache_entry() APIs and the *_transient_cache_entry() APIs.\n\nThere are quite a few changes but I didn't see any instances that were \nmissed or any errors. I see that later patches will put verification \ncode in place to detect if any were done incorrectly and to prevent \nregressions moving forward.\n\nOverall, it looks correct and reasonable.\n\n\n\n> \n> -struct cache_entry *make_cache_entry(unsigned int mode,\n> -\t\tconst unsigned char *sha1, const char *path, int stage,\n> -\t\tunsigned int refresh_options)\n> +struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n> +{\n> +\treturn xcalloc(1, cache_entry_size(len));\n> +}\n> +\n> +struct cache_entry *make_empty_transient_cache_entry(size_t len)\n> +{\n> +\treturn xcalloc(1, cache_entry_size(len));\n> +}\n> +\n> +struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode,\n> +\t\t\t const unsigned char *sha1, const char *path,\n> +\t\t\t int stage, unsigned int refresh_options)\n> {\n> -\tint size, len;\n> \tstruct cache_entry *ce, *ret;\n> +\tint len;\n> \n> \tif (!verify_path(path)) {\n> \t\terror(\"Invalid path '%s'\", path);\n> @@ -758,8 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n> \t}\n> \n> \tlen = strlen(path);\n> -\tsize = cache_entry_size(len);\n> -\tce = xcalloc(1, size);\n> +\tce = make_empty_index_cache_entry(istate, len);\n> \n> \thashcpy(ce->oid.hash, sha1);\n> \tmemcpy(ce->name, path, len);\n> @@ -769,10 +777,34 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n> \n> \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n> \tif (ret != ce)\n> -\t\tfree(ce);\n> +\t\tindex_cache_entry_discard(ce);\n> +\n> \treturn ret;\n> }\n> \n> +struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1,\n> +\t\t\t const char *path, int stage)\n> +{\n> +\tstruct cache_entry *ce;\n> +\tint len;\n> +\n> +\tif (!verify_path(path)) {\n> +\t\terror(\"Invalid path '%s'\", path);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tlen = strlen(path);\n> +\tce = make_empty_transient_cache_entry(len);\n> +\n> +\thashcpy(ce->oid.hash, sha1);\n> +\tmemcpy(ce->name, path, len);\n> +\tce->ce_flags = create_ce_flags(stage);\n> +\tce->ce_namelen = len;\n> +\tce->ce_mode = create_ce_mode(mode);\n> +\n\nNit, feel free to ignore. There isn't a lot of initialization here but I \nwonder if it makes sense to have an internal helper function to ensure \nthese stay the same.\n\n> +\treturn ce;\n> +}\n> +\n> /*\n> * Chmod an index entry with either +x or -x.\n> *\n> @@ -1243,7 +1275,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n> {\n> \tstruct stat st;\n> \tstruct cache_entry *updated;\n> -\tint changed, size;\n> +\tint changed;\n> \tint refresh = options & CE_MATCH_REFRESH;\n> \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n> \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n> @@ -1323,8 +1355,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n> \t\treturn NULL;\n> \t}\n> \n> -\tsize = ce_size(ce);\n> -\tupdated = xmalloc(size);\n> +\tupdated = make_empty_index_cache_entry(istate, ce_namelen(ce));\n> \tcopy_cache_entry(updated, ce);\n> \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n> \tfill_stat_cache_info(updated, &st);\n> @@ -1610,12 +1641,13 @@ int read_index(struct index_state *istate)\n> \treturn read_index_from(istate, get_index_file(), get_git_dir());\n> }\n> \n> -static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n> +static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n> +\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n> \t\t\t\t\t\t unsigned int flags,\n> \t\t\t\t\t\t const char *name,\n> \t\t\t\t\t\t size_t len)\n> {\n> -\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n> +\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n> \n> \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n> \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n> @@ -1657,7 +1689,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n> \treturn (const char *)ep + 1 - cp_;\n> }\n> \n> -static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n> +static struct cache_entry *create_from_disk(struct index_state *istate,\n> +\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n> \t\t\t\t\t unsigned long *ent_size,\n> \t\t\t\t\t struct strbuf *previous_name)\n\nNit. Just wondering why you pulled refresh_cache_entry() out into an \nearlier/separate commit but then did create_from_disk() as part of the \nlarge refactoring focused on adding the create/discard APIs?\n\n\n\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index e73745051e..232cdecc72 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -192,10 +192,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n> \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n> }\n> \n> -static struct cache_entry *dup_entry(const struct cache_entry *ce)\n> +static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n\nDitto with the dup_entry() and create_ce_entry() functions refactoring.\n\n> -static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n> +static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n> +\tconst struct name_entry *n,\n> +\tint stage,\n> +\tstruct index_state *istate,\n> +\tint is_transient)\n"},{"id":"344968","messageId":"xmqqwox5i0f7.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-04-18T04:49:32Z","receivedAt":"2018-04-18T04:49:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> This patch series improves the performance of loading indexes by\n> reducing the number of malloc() calls. ...\n>\n> Jameson Miller (5):\n> read-cache: teach refresh_cache_entry to take istate\n> Add an API creating / discarding cache_entry structs\n> mem-pool: fill out functionality\n> Allocate cache entries from memory pools\n> Add optional memory validations around cache_entry lifecyle\n>\n> apply.c | 26 +++---\n> blame.c | 5 +-\n> builtin/checkout.c | 8 +-\n> builtin/difftool.c | 8 +-\n> builtin/reset.c | 6 +-\n> builtin/update-index.c | 26 +++---\n> cache.h | 40 ++++++++-\n> git.c | 3 +\n> mem-pool.c | 136 ++++++++++++++++++++++++++++-\n> mem-pool.h | 34 ++++++++\n> merge-recursive.c | 4 +-\n> read-cache.c | 229 +++++++++++++++++++++++++++++++++++++++----------\n> resolve-undo.c | 6 +-\n> split-index.c | 31 +++++--\n> tree.c | 4 +-\n> unpack-trees.c | 27 +++---\n> 16 files changed, 476 insertions(+), 117 deletions(-)\n>\n>\n> base-commit: cafaccae98f749ebf33495aec42ea25060de8682\n\nI couldn't quite figure out what these five patches were based on,\neven with this line. Basing on and referring to a commit that is\nnot part of our published history with \"base-commit\" is not all that\nuseful to others.\n\nOffhand without applying these patches and viewing the changes in\nwider contexts, one thing that makes me wonder is how the two\nallocation schemes can be used with two implementations of free().\nUpon add_index_entry() that replaces an index entry for an existing\npath, we'd discard an entry that was originally read as part of\nread_cache(). If we do that again, the second add_index_entry()\nwill be discading, in its call to replace_index_entry(), the entry\nthat was allocated by the caller of the first add_index_entry()\ncall. And replace_index_entry() would not have a way to know from\nwhich allocation the entry's memory came from.\n\nPerhaps it is not how you are using the \"transient\" stuff, and from\nthe comment in 2/5, it is for \"entries that are not meant to go into\nthe index\", but then higher stage index entries in unpack_trees seem\nto be allocated via the \"transient\" stuff, so I am not sure what the\nplans are for things like merge_recursive() that uses unpack_trees()\nto prepare the index and then later \"fix it up\" by further futzing\naround the index to adjust for renames it finds, etc.\n\nLet me read it fully once we know where these patches are to be\napplied, but before that I cannot say much about them X-<.\n\nThanks.\n\n"},{"id":"345242","messageId":"CAGZ79kbOQLkNZxHbR0bDwVHbT0x47v8qp81E2WFm72HdZJ4ZRA@mail.gmail.com","threadId":"48310","inReplyTo":"xmqqwox5i0f7.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-20T17:49:30Z","receivedAt":"2018-04-20T17:49:35Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> base-commit: cafaccae98f749ebf33495aec42ea25060de8682\n>\n> I couldn't quite figure out what these five patches were based on,\n> even with this line. Basing on and referring to a commit that is\n> not part of our published history with \"base-commit\" is not all that\n> useful to others.\n\nI'd like to second this. In the object store refactoring, I am at a point where\nI'd want to migrate the memory management of {object, tree, commit, tag}.c\nwhich currently is done in alloc.c to a memory pool, that has a dedicated\npointer to it.\n\nSo I'd either have to refactor alloc.c to take the_repository[1] or\nI'd play around with the mem_pool to manage memory in the\nobject layer. I guess this playing around can happen with\nwhat is at origin/jm/mem-pool, however the life cycle management\npart of the third patch[2] would allow for stopping memleaks there.\nSo I am interested in this series as well.\n\n[1] proof of concept in patches nearby\nhttps://public-inbox.org/git/20180206001749.218943-31-sbeller@google.com/\n\n[2] https://public-inbox.org/git/20180417163400.3875-5-jamill@microsoft.com/\n\nThanks,\nStefan\n"},{"id":"345291","messageId":"20180420162136.144ac5529072f22067abb3b9@google.com","threadId":"48310","inReplyTo":"20180417163400.3875-5-jamill@microsoft.com","subject":"Re: [PATCH v1 3/5] mem-pool: fill out functionality","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-20T23:21:36Z","receivedAt":"2018-04-20T23:21:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 17 Apr 2018 16:34:42 +0000\nJameson Miller wrote:\n\n> @@ -19,8 +19,27 @@ struct mem_pool {\n> \n> \t/* The total amount of memory allocated by the pool. */\n> \tsize_t pool_alloc;\n> +\n> +\t/*\n> +\t * Array of pointers to \"custom size\" memory allocations.\n> +\t * This is used for \"large\" memory allocations.\n> +\t * The *_end variables are used to track the range of memory\n> +\t * allocated.\n> +\t */\n> +\tvoid **custom, **custom_end;\n> +\tint nr, nr_end, alloc, alloc_end;\n\nThis seems overly complicated - the struct mem_pool already has a linked\nlist of pages, so couldn't you create a custom page and insert it behind\nthe current front page instead whenever you needed a large-size page?\n\nAlso, when combining, there could be some wasted space on one of the\npages. I'm not sure if that's worth calling out, though.\n"},{"id":"345293","messageId":"20180420163441.208644d772a25ddbbdbb1616@google.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-20T23:34:41Z","receivedAt":"2018-04-20T23:34:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 17 Apr 2018 16:34:39 +0000\nJameson Miller wrote:\n\n> Jameson Miller (5):\n> read-cache: teach refresh_cache_entry to take istate\n> Add an API creating / discarding cache_entry structs\n> mem-pool: fill out functionality\n> Allocate cache entries from memory pools\n> Add optional memory validations around cache_entry lifecyle\n\nIn this patch set, there is no enforcement that the cache entry created\nby make_index_cache_entry() goes into the correct index when\nadd_index_entry() is invoked. (Junio described similar things, I\nbelieve, in [1].) This might be an issue when we bring up and drop\nmultiple indexes, and dropping one index causes a cache entry in another\nto become invalidated.\n\nOne solution is to store the index for which the cache entry was created\nin the cache entry itself, but that does increase its size. Another is\nto change the API such that a cache entry is created and added in the\nsame function, and then have some rollback if the cache entry turns out\nto be invalid (to support add-empty-entry -> fill -> verify), but I\ndon't know if this is feasible. Anyway, all these alternatives should be\nat least discussed in the commit message, I think.\n\nThe make_transient_cache_entry() function might be poorly named, since\nas far as I can tell, the entries produced by that function are actually\nthe longest lasting, since they are never freed.\n\nAlong those lines, I was slightly surprised to find out in patch 4 that\ncache entry freeing is a no-op. That's fine, but in that case, it would\nbe better to delete all the calls to the \"discard\" function, and\ndocument in the others that the entries they create will only be freed\nwhen the memory pool itself is discarded.\n\n[1] https://public-inbox.org/git/xmqqwox5i0f7.fsf@gitster-ct.c.googlers.com/\n"},{"id":"345465","messageId":"b11211ab-4133-84b4-d27b-5dabcd2a6612@gmail.com","threadId":"48310","inReplyTo":"d2743302-43dd-da3e-f0ba-55dcb824c0f4@gmail.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T14:09:07Z","receivedAt":"2018-04-23T14:09:28Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/17/2018 02:39 PM, Ben Peart wrote:\n>\n>\n> On 4/17/2018 12:34 PM, Jameson Miller wrote:\n>\n>> 100K\n>>\n>> Test                                       baseline [4] block_allocation\n>> ------------------------------------------------------------------------------------\n>> 0002.1: read_cache/discard_cache 1 times   0.03(0.01+0.01) \n>> 0.02(0.01+0.01) -33.3%\n>>\n>> 1M:\n>>\n>> Test                                       baseline block_allocation\n>> ------------------------------------------------------------------------------------\n>> 0002.1: read_cache/discard_cache 1 times   0.23(0.12+0.11) \n>> 0.17(0.07+0.09) -26.1%\n>>\n>> 2M:\n>>\n>> Test                                       baseline block_allocation\n>> ------------------------------------------------------------------------------------\n>> 0002.1: read_cache/discard_cache 1 times   0.45(0.26+0.19) \n>> 0.39(0.17+0.20) -13.3%\n>>\n>>\n>> 100K is not a large enough sample size to show the perf impact of this\n>> change, but we can see a perf improvement with 1M and 2M entries.\n>\n> I see a 33% change with 100K files which is a substantial improvement \n> even in the 100K case.  I do see that the actual wall clock savings \n> aren't nearly as much with a small repo as it is with the larger repos \n> which makes sense.\n\nYou are correct - I should have been more careful in my wording. What I \nmeant is that the wall time savings with 100K is not large, because this \noperation is already very fast.\n\n"},{"id":"345475","messageId":"cb1c1c86-48f7-5cf6-16ff-0d54550fe958@gmail.com","threadId":"48310","inReplyTo":"xmqqwox5i0f7.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T16:19:05Z","receivedAt":"2018-04-23T16:19:22Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/18/2018 12:49 AM, Junio C Hamano wrote:\n> Jameson Miller writes:\n>\n>> This patch series improves the performance of loading indexes by\n>> reducing the number of malloc() calls. ...\n>>\n>> Jameson Miller (5):\n>> read-cache: teach refresh_cache_entry to take istate\n>> Add an API creating / discarding cache_entry structs\n>> mem-pool: fill out functionality\n>> Allocate cache entries from memory pools\n>> Add optional memory validations around cache_entry lifecyle\n>>\n>> apply.c | 26 +++---\n>> blame.c | 5 +-\n>> builtin/checkout.c | 8 +-\n>> builtin/difftool.c | 8 +-\n>> builtin/reset.c | 6 +-\n>> builtin/update-index.c | 26 +++---\n>> cache.h | 40 ++++++++-\n>> git.c | 3 +\n>> mem-pool.c | 136 ++++++++++++++++++++++++++++-\n>> mem-pool.h | 34 ++++++++\n>> merge-recursive.c | 4 +-\n>> read-cache.c | 229 +++++++++++++++++++++++++++++++++++++++----------\n>> resolve-undo.c | 6 +-\n>> split-index.c | 31 +++++--\n>> tree.c | 4 +-\n>> unpack-trees.c | 27 +++---\n>> 16 files changed, 476 insertions(+), 117 deletions(-)\n>>\n>>\n>> base-commit: cafaccae98f749ebf33495aec42ea25060de8682\n> I couldn't quite figure out what these five patches were based on,\n> even with this line. Basing on and referring to a commit that is\n> not part of our published history with \"base-commit\" is not all that\n> useful to others.\nMy apologies - this patch series should be applied to the 'next'\nbranch.  It applies cleanly on top of\nb46fe60e1d (\"merge-fix/ps/test-chmtime-get\", 2018-04-20), which\nis a commit in the 'next' branch.\n> Offhand without applying these patches and viewing the changes in\n> wider contexts, one thing that makes me wonder is how the two\n> allocation schemes can be used with two implementations of free().\n> Upon add_index_entry() that replaces an index entry for an existing\n> path, we'd discard an entry that was originally read as part of\n> read_cache(). If we do that again, the second add_index_entry()\n> will be discading, in its call to replace_index_entry(), the entry\n> that was allocated by the caller of the first add_index_entry()\n> call. And replace_index_entry() would not have a way to know from\n> which allocation the entry's memory came from.\n>\n> Perhaps it is not how you are using the \"transient\" stuff, and from\n> the comment in 2/5, it is for \"entries that are not meant to go into\n> the index\", but then higher stage index entries in unpack_trees seem\n> to be allocated via the \"transient\" stuff, so I am not sure what the\n> plans are for things like merge_recursive() that uses unpack_trees()\n> to prepare the index and then later \"fix it up\" by further futzing\n> around the index to adjust for renames it finds, etc.\n\nGood points. The intention with this logic is that any entries\nthat *could* go into an index are allocated from the memory\npool. The \"transient\" entries only exist for a short period of\ntime. These have a defined lifetime and we can always trace the\ncorresponding \"free\" call. make_transient_cache_entry() is only\nused to construct a temporary cache_entry to pass to the\ncheckout_entry() / write_entry(). There is a note in checkout.c\nindicating that write_entry() needs to be re-factored to not take\na cache_entry.\n\nThe cache_entry type could gain knowledge about where it was\nallocated from. This would allow us to only have a\nsingle \"free()\" function, which could inspect the cache_entry to\nsee if it was allocated from a mem_pool (and possibly which\nmem_pool) and take the appropriate action. The downside of this\napproach is that cache_entry would need to gain a field to track\nthis information, and I *think* all of the bit field spots are\ntaken.\n\n> Let me read it fully once we know where these patches are to be\n> applied, but before that I cannot say much about them X-<.\n>\n> Thanks.\n>\n\n"},{"id":"345479","messageId":"3ec5999c-f81d-eafb-1597-4fa93f943a6d@gmail.com","threadId":"48310","inReplyTo":"CAGZ79kbOQLkNZxHbR0bDwVHbT0x47v8qp81E2WFm72HdZJ4ZRA@mail.gmail.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T16:44:50Z","receivedAt":"2018-04-23T16:45:04Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/20/2018 01:49 PM, Stefan Beller wrote:\n>>> base-commit: cafaccae98f749ebf33495aec42ea25060de8682\n>>\n>> I couldn't quite figure out what these five patches were based on,\n>> even with this line. Basing on and referring to a commit that is\n>> not part of our published history with \"base-commit\" is not all that\n>> useful to others.\n> \n> I'd like to second this. In the object store refactoring, I am at a point where\n> I'd want to migrate the memory management of {object, tree, commit, tag}.c\n> which currently is done in alloc.c to a memory pool, that has a dedicated\n> pointer to it.\n> \n> So I'd either have to refactor alloc.c to take the_repository[1] or\n> I'd play around with the mem_pool to manage memory in the\n> object layer. I guess this playing around can happen with\n> what is at origin/jm/mem-pool, however the life cycle management\n> part of the third patch[2] would allow for stopping memleaks there.\n> So I am interested in this series as well.\n>\n\nSorry about the confusion here. This patch series can be applied to the \nnext branch. They apply cleanly on [3]. The lifecycle functions are \nre-introduced in [4], which we could incorporate sooner if useful. I \ndidn't have a consumer for the calls in the original patch series, and \nso deferred them until there was a caller. I would be interested to \nunderstand how the mem_pool would fit your needs, and if it is \nsufficient or needs modification for your use cases.\n\n> [1] proof of concept in patches nearby\n> https://public-inbox.org/git/20180206001749.218943-31-sbeller@google.com/\n> \n> [2] https://public-inbox.org/git/20180417163400.3875-5-jamill@microsoft.com/\n> \n> Thanks,\n> Stefan\n> \n\n[3] b46fe60e1d (\"merge-fix/ps/test-chmtime-get\", 2018-04-20)\n\n[4]\nhttps://public-inbox.org/git/20180417163400.3875-5-jamill@microsoft.com/\n"},{"id":"345482","messageId":"8866ddf7-399c-c596-5fb5-c78bc040f4ac@gmail.com","threadId":"48310","inReplyTo":"20180420163441.208644d772a25ddbbdbb1616@google.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T17:14:35Z","receivedAt":"2018-04-23T17:14:52Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/20/2018 07:34 PM, Jonathan Tan wrote:\n> On Tue, 17 Apr 2018 16:34:39 +0000\n> Jameson Miller wrote:\n> \n>> Jameson Miller (5):\n>> read-cache: teach refresh_cache_entry to take istate\n>> Add an API creating / discarding cache_entry structs\n>> mem-pool: fill out functionality\n>> Allocate cache entries from memory pools\n>> Add optional memory validations around cache_entry lifecyle\n> \n> In this patch set, there is no enforcement that the cache entry created\n> by make_index_cache_entry() goes into the correct index when\n> add_index_entry() is invoked. (Junio described similar things, I\n> believe, in [1].) This might be an issue when we bring up and drop\n> multiple indexes, and dropping one index causes a cache entry in another\n> to become invalidated.\n\nCorrect - it is up to the caller here to coordinate this. The code \nshould be set up so this is not a problem here. In the case of a \nsplit-index, the cache entries should be allocated from the memory pool \nassociated with the \"most common\" / base index. If you found a place I \nmissed or seems questionable, or have suggestions, I would be glad to \nlook into it.\n\n> \n> One solution is to store the index for which the cache entry was created\n> in the cache entry itself, but that does increase its size. Another is\n\nYes, this is an option. For this initial patch series, I decided to not \nadd extra fields to the cache_entry type, but I think incorporating this \nin cache_entry is a viable option, and has some positive properties.\n\n> to change the API such that a cache entry is created and added in the\n> same function, and then have some rollback if the cache entry turns out\n> to be invalid (to support add-empty-entry -> fill -> verify), but I\n> don't know if this is feasible. Anyway, all these alternatives should be\n> at least discussed in the commit message, I think.\n\nI can include a discussion of these in the commit message. Thanks.\n\n> \n> The make_transient_cache_entry() function might be poorly named, since\n> as far as I can tell, the entries produced by that function are actually\n> the longest lasting, since they are never freed.\n\nThey should always be freed (and are usually freed close to where they \nare allocated, or by the calling function). If you see an instance where \nthis is not the case, please point it out, because that is not the \nintention.\n\n> \n> Along those lines, I was slightly surprised to find out in patch 4 that\n> cache entry freeing is a no-op. That's fine, but in that case, it would\n> be better to delete all the calls to the \"discard\" function, and\n> document in the others that the entries they create will only be freed\n> when the memory pool itself is discarded.\n\nI can add a comment inside the function body. In the next commit, I do \nadd logic to perform extra (optional) verification in the discard \nfunction. I did wrestle with this fact, but I feel there is value in \nhaving the locations where it is appropriate to free these entries in \ncode, even if this particular implementation is not utilizing it right \nnow. Hopefully the verification logic added in 5/5 is enough to justify \nkeeping this function around.\n\n> \n> [1] https://public-inbox.org/git/xmqqwox5i0f7.fsf@gitster-ct.c.googlers.com/\n> \n"},{"id":"345483","messageId":"CAGZ79kYeY2yNzUZpzE+L8nxku0RdTsrKc6=VEFQwZtWMKEafdg@mail.gmail.com","threadId":"48310","inReplyTo":"3ec5999c-f81d-eafb-1597-4fa93f943a6d@gmail.com","subject":"Re: [PATCH v1 0/5] Allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-23T17:18:53Z","receivedAt":"2018-04-23T17:18:57Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Apr 23, 2018 at 9:44 AM, Jameson Miller\n wrote:\n> I would be interested to understand how the\n> mem_pool would fit your needs, and if it is sufficient or needs modification\n> for your use cases.\n>\n>> [1] proof of concept in patches nearby\n>> https://public-inbox.org/git/20180206001749.218943-31-sbeller@google.com/\n>>\n\nCurrenlty the parsed objects are loaded into memory and never freed.\nSee alloc.c which implements a specialized memory allocator for this\nobject loading.\nWhen working with submodules, their objects are also just put into this\nglobally-namespaced object store. (See struct object **obj_hash in\nobject.c)\n\nI want to make the object store a per-repository object, such that\nwhen working with submodules, you can free up all submodule objects\nonce you are done with a given submodule.\n\nTo do so, the memory allocation needs to manage the whole life cycle,\nwhile preserving the efficiency of alloc.c. See 855419f764\n(Add specialized object allocator, 2006-06-19). The mem-pool that\nyou propose can allocate large slabs of memory and we can put\nobjects in there without alignment overhead, such that we preserve\nthe memory efficiency while being able to track all the memory.\n\nSo I would think it is sufficient as-is with this series, maybe we need\na little tweaking there, but nothing large IMHO.\n\nThanks,\nStefan\n"},{"id":"345484","messageId":"898c0b43-b090-fb1f-45d5-ea4041843cdb@gmail.com","threadId":"48310","inReplyTo":"20180420162136.144ac5529072f22067abb3b9@google.com","subject":"Re: [PATCH v1 3/5] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T17:27:09Z","receivedAt":"2018-04-23T17:27:23Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/20/2018 07:21 PM, Jonathan Tan wrote:\n> On Tue, 17 Apr 2018 16:34:42 +0000\n> Jameson Miller wrote:\n> \n>> @@ -19,8 +19,27 @@ struct mem_pool {\n>> \n>> \t/* The total amount of memory allocated by the pool. */\n>> \tsize_t pool_alloc;\n>> +\n>> +\t/*\n>> +\t * Array of pointers to \"custom size\" memory allocations.\n>> +\t * This is used for \"large\" memory allocations.\n>> +\t * The *_end variables are used to track the range of memory\n>> +\t * allocated.\n>> +\t */\n>> +\tvoid **custom, **custom_end;\n>> +\tint nr, nr_end, alloc, alloc_end;\n> \n> This seems overly complicated - the struct mem_pool already has a linked\n> list of pages, so couldn't you create a custom page and insert it behind\n> the current front page instead whenever you needed a large-size page?\n\nYes - that is another option. However, the linked list of pages includes \nmemory that *could* have space for an allocation, while the \"custom\" \nregion will never have left over memory that can be used for other \nallocations. When searching pages for memory to satisfy a request, there \nis no reason to search through the \"custom\" pages. There is a trade-off \nbetween complexity and implementation, so I am open to suggestions.\n\nThis was discussed in [1], where it originally was implemented closer to \nwhat you describe here.\n\n> \n> Also, when combining, there could be some wasted space on one of the\n> pages. I'm not sure if that's worth calling out, though.\n> \n\nYes, we bring over the whole page. However, these pages are now \navailable for new allocations.\n\n[1]\nhttps://public-inbox.org/git/xmqqk1u2k91l.fsf@gitster-ct.c.googlers.com/\n"},{"id":"345490","messageId":"20180423104931.c465b6a260219c96feba41ff@google.com","threadId":"48310","inReplyTo":"898c0b43-b090-fb1f-45d5-ea4041843cdb@gmail.com","subject":"Re: [PATCH v1 3/5] mem-pool: fill out functionality","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-04-23T17:49:31Z","receivedAt":"2018-04-23T17:49:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 23 Apr 2018 13:27:09 -0400\nJameson Miller wrote:\n\n> > This seems overly complicated - the struct mem_pool already has a linked\n> > list of pages, so couldn't you create a custom page and insert it behind\n> > the current front page instead whenever you needed a large-size page?\n> \n> Yes - that is another option. However, the linked list of pages includes \n> memory that *could* have space for an allocation, while the \"custom\" \n> region will never have left over memory that can be used for other \n> allocations. When searching pages for memory to satisfy a request, there \n> is no reason to search through the \"custom\" pages. There is a trade-off \n> between complexity and implementation, so I am open to suggestions.\n> \n> This was discussed in [1], where it originally was implemented closer to \n> what you describe here.\n> \n> > Also, when combining, there could be some wasted space on one of the\n> > pages. I'm not sure if that's worth calling out, though.\n> \n> Yes, we bring over the whole page. However, these pages are now \n> available for new allocations.\n> \n> [1]\n> https://public-inbox.org/git/xmqqk1u2k91l.fsf@gitster-ct.c.googlers.com/\n\nAh, I didn't realize that the plan was to search over all pages when\nallocating memory from the pool, instead of only searching the last\npage. This seems like a departure from the fast-import.c way, where as\nfar as I can tell, new_object() searches only one page. If we do plan to\ndo this, searching all pages doesn't seem like a good idea to me,\nespecially since the objects we're storing in the pool are of similar\nsize.\n\nIf we decide to go ahead with searching all the pages, though, the\n\"custom\" pages should probably be another linked list instead of an\narray.\n"},{"id":"345494","messageId":"5584faf6-23a0-366a-bd87-cf9e420f212c@gmail.com","threadId":"48310","inReplyTo":"20180423104931.c465b6a260219c96feba41ff@google.com","subject":"Re: [PATCH v1 3/5] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jameson.miller81@gmail.com","sentAt":"2018-04-23T18:20:14Z","receivedAt":"2018-04-23T18:20:36Z","isPatch":true,"sender":{"key":"jameson.miller81@gmail.com","avatar":null},"body":"\n\nOn 04/23/2018 01:49 PM, Jonathan Tan wrote:\n> On Mon, 23 Apr 2018 13:27:09 -0400\n> Jameson Miller wrote:\n> \n>>> This seems overly complicated - the struct mem_pool already has a linked\n>>> list of pages, so couldn't you create a custom page and insert it behind\n>>> the current front page instead whenever you needed a large-size page?\n>>\n>> Yes - that is another option. However, the linked list of pages includes\n>> memory that *could* have space for an allocation, while the \"custom\"\n>> region will never have left over memory that can be used for other\n>> allocations. When searching pages for memory to satisfy a request, there\n>> is no reason to search through the \"custom\" pages. There is a trade-off\n>> between complexity and implementation, so I am open to suggestions.\n>>\n>> This was discussed in [1], where it originally was implemented closer to\n>> what you describe here.\n>>\n>>> Also, when combining, there could be some wasted space on one of the\n>>> pages. I'm not sure if that's worth calling out, though.\n>>\n>> Yes, we bring over the whole page. However, these pages are now\n>> available for new allocations.\n>>\n>> [1]\n>> https://public-inbox.org/git/xmqqk1u2k91l.fsf@gitster-ct.c.googlers.com/\n> \n> Ah, I didn't realize that the plan was to search over all pages when\n> allocating memory from the pool, instead of only searching the last\n> page. This seems like a departure from the fast-import.c way, where as\n> far as I can tell, new_object() searches only one page. If we do plan to\n> do this, searching all pages doesn't seem like a good idea to me,\n> especially since the objects we're storing in the pool are of similar\n> size.\n\nI see. However, the new_object() logic in fast-import is a different \nthan the logic mem_pool was abstracting, and is not covered by the \nmem_pool functionality. The behavior of searching over all pages for one \nto satisfy the request existed previously and was not changed in the \nmem_pool implementation.\n\n> \n> If we decide to go ahead with searching all the pages, though, the\n> \"custom\" pages should probably be another linked list instead of an\n> array.\n> \n\nThis is an option - I went with the current design because we only need \npointers to a block of memory (as well tracking how large the allocation \nis for verification purposes). We don't necessarily need the extra \noverhead of a structure to track the linked list nodes, when it is \nprovided by the existing array manipulation functions. I am open to \nfeedback on this point, however.\n\n"},{"id":"346155","messageId":"20180430153122.243976-1-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:40Z","receivedAt":"2018-04-30T15:31:46Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Changes from V1:\n\n - Based patch series off of commit in master\n\n - Minor updates based on initial code review feedback\n\nSummary:\n\nThis patch series improves the performance of loading indexes by\nreducing the number of malloc() calls. Loading the index from disk is\npartly dominated by the time in malloc(), which is called for each\nindex entry. This patch series reduces the number of times malloc() is\ncalled as part of loading the index, and instead allocates a block of\nmemory upfront that is large enough to hold all of the cache entries,\nand chunks this memory itself. This change builds on [1].\n\n\nGit previously allocated block of memory for the index cache entries\nuntil [2].\n\nThis 5 part patch series is broken up as follows:\n\n 1/5, 2/5 - Move cache entry lifecycle methods behind an API\n\n 3/5 - Fill out memory pool API to include lifecycle and other\n \tmethods used in later patches\n\n 4/5 - Allocate cache entry structs from memory pool\n\n 5/5 - Add extra optional validation\n\nPerformance Benchmarks:\n\nTo evaluate the performance of this approach, the p0002-read-cache.sh\ntest was run with several combinations of allocators (glibc default,\ntcmalloc, jemalloc), with and without block allocation, and across\nseveral different index sized (100K, 1M, 2M entries). The details on\nhow these repositories were constructed can be found in [3].The\np0002-read-cache.sh was run with the iteration count set to 1 and\n$GIT_PERF_REPEAT_COUNT=10.\n\nThe tests were run with iteration count set to 1 because this best\napproximates the real behavior. The read_cache/discard_cache test will\nload / free the index N times, and the performance of this logic is\ndifferent between N = 1 and N > 1. As the production code does not\nread / discard the index in a loop, a better approximation is when N =\n1.\n\n100K\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.02(0.01+0.01) -33.3%\n\n1M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.17(0.07+0.09) -26.1%\n\n2M:\n\nTest baseline block_allocation\n ------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.39(0.17+0.20) -13.3%\n\nWith 100K entries, it only takes 0.3 seconds to read the entries even\nwithout using block allocation, so there is only a small change in the\nwall clock time. We can see a larger wall clack improvement with 1M\nand 2M entries.\n\nFor completeness, here is the p0002-read-cache tests for git.git and\nlinux.git:\n\ngit.git:\n\nTest baseline block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 0.30(0.26+0.03) 0.17(0.13+0.03) -43.3%\n\nlinux.git:\n\nTest baseline block_allocation\n ---------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1000 times 7.05(6.01+0.84) 4.61(3.74+0.66) -34.6% \n\n\nWe also investigated the performance of just using different\nallocators. We can see that there is not a consistent performance\ngain.\n\n100K\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.03(0.01+0.01) 0.03(0.01+0.01) +0.0% 0.03(0.02+0.01) +0.0% \n\n1M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.23(0.12+0.11) 0.21(0.10+0.10) -8.7% 0.27(0.16+0.10) +17.4%\n\n2M:\n\nTest baseline tcmalloc jemalloc\n ------------------------------------------------------------------------------------------------------------------\n0002.1: read_cache/discard_cache 1 times 0.45(0.26+0.19) 0.46(0.25+0.21) +2.2% 0.57(0.36+0.21) +26.7%\n\n\n\n[1] https://public-inbox.org/git/20180321164152.204869-1-jamill@microsoft.com/\n\n[2] debed2a629 (read-cache.c: allocate index entries individually - 2011-10-24)\n\n[3] Constructing test repositories:\n\nThe test repositories were constructed with t/perf/repos/many_files.sh with the following parameters:\n\n100K:\t many-files.sh 4 10 9\n1M:\t many-files.sh 5 10 9\n2M:\t many-files.sh 6 8 7\n\nBase Ref: master\nWeb-Diff: git@github.com:jamill/git.git/commit/2152d28016\nCheckout: git fetch git@github.com:jamill/git.git users/jamill/block_allocation-v2 && git checkout 2152d28016\n\nJameson Miller (5):\n read-cache: teach refresh_cache_entry() to take istate\n block alloc: add lifecycle APIs for cache_entry structs\n mem-pool: fill out functionality\n block alloc: allocate cache entries from mem_pool\n block alloc: add validations around cache_entry lifecyle\n\n apply.c | 26 +++---\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 8 +-\n builtin/reset.c | 6 +-\n builtin/update-index.c | 26 +++---\n cache.h | 40 ++++++++-\n git.c | 3 +\n mem-pool.c | 136 ++++++++++++++++++++++++++++-\n mem-pool.h | 34 ++++++++\n merge-recursive.c | 4 +-\n read-cache.c | 232 +++++++++++++++++++++++++++++++++++++++----------\n resolve-undo.c | 6 +-\n split-index.c | 31 +++++--\n tree.c | 4 +-\n unpack-trees.c | 27 +++---\n 16 files changed, 479 insertions(+), 117 deletions(-)\n\n\nbase-commit: 1f1cddd558b54bb0ce19c8ace353fd07b758510d\n-- \n2.14.3\n\n\n"},{"id":"346156","messageId":"20180430153122.243976-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"[PATCH v2 1/5] read-cache: teach refresh_cache_entry() to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:41Z","receivedAt":"2018-04-30T15:31:49Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactor refresh_cache_entry() to work on a specific index, instead of\nimplicitly using the_index. This is in preparation for making the\nmake_cache_entry function work on a specific index.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 7 ++++---\n 3 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 77b7acebb6..31f8f0420a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -743,7 +743,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624d..693f60e0a3 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -260,7 +260,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 10f1c6bb8a..2cb4f53b57 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1448,10 +1448,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\t struct cache_entry *ce,\n \t\t\t\t\t unsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.14.3\n\n"},{"id":"346157","messageId":"20180430153122.243976-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"[PATCH v2 3/5] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:43Z","receivedAt":"2018-04-30T15:31:51Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Adds the following functionality to memory pools:\n\n - Lifecycle management functions (init, discard)\n\n - Test whether a memory location is part of the managed pool\n\n - Function to combine 2 pools\n\nThis also adds logic to track all memory allocations made by a memory\npool.\n\nThese functions will be used in a future commit in this commit series.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n mem-pool.h | 32 +++++++++++++++++\n 2 files changed, 142 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..a495885c4b 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,6 +5,8 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n {\n \tstruct mp_block *p;\n@@ -19,6 +21,60 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \treturn p;\n }\n \n+static void *mem_pool_alloc_custom(struct mem_pool *mem_pool, size_t block_alloc)\n+{\n+\tchar *p;\n+\tALLOC_GROW(mem_pool->custom, mem_pool->nr + 1, mem_pool->alloc);\n+\tALLOC_GROW(mem_pool->custom_end, mem_pool->nr_end + 1, mem_pool->alloc_end);\n+\n+\tp = xmalloc(block_alloc);\n+\tmem_pool->custom[mem_pool->nr++] = p;\n+\tmem_pool->custom_end[mem_pool->nr_end++] = p + block_alloc;\n+\n+\tmem_pool->pool_alloc += block_alloc;\n+\n+\treturn mem_pool->custom[mem_pool->nr];\n+}\n+\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tif (!(*mem_pool))\n+\t{\n+\t\t*mem_pool = xmalloc(sizeof(struct mem_pool));\n+\t\t(*mem_pool)->pool_alloc = 0;\n+\t\t(*mem_pool)->mp_block = NULL;\n+\t\t(*mem_pool)->block_alloc = BLOCK_GROWTH_SIZE;\n+\t\t(*mem_pool)->custom = NULL;\n+\t\t(*mem_pool)->nr = 0;\n+\t\t(*mem_pool)->alloc = 0;\n+\t\t(*mem_pool)->custom_end = NULL;\n+\t\t(*mem_pool)->nr_end = 0;\n+\t\t(*mem_pool)->alloc_end = 0;\n+\n+\t\tif (initial_size > 0)\n+\t\t\tmem_pool_alloc_block(*mem_pool, initial_size);\n+\t}\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tint i;\n+\tstruct mp_block *block, *block_to_free;\n+\tfor (block = mem_pool->mp_block; block;)\n+\t{\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfor (i = 0; i < mem_pool->nr; i++)\n+\t\tfree(mem_pool->custom[i]);\n+\n+\tfree(mem_pool->custom);\n+\tfree(mem_pool->custom_end);\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p;\n@@ -33,10 +89,8 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \t\t\tbreak;\n \n \tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n+\t\tif (len >= (mem_pool->block_alloc / 2))\n+\t\t\treturn mem_pool_alloc_custom(mem_pool, len);\n \n \t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n \t}\n@@ -53,3 +107,55 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tint i;\n+\tstruct mp_block *p;\n+\n+\t/* Check if memory is allocated in a block */\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\t/* Check if memory is allocated in custom block */\n+\tfor (i = 0; i < mem_pool->nr; i++)\n+\t\tif ((mem >= mem_pool->custom[i]) &&\n+\t\t (mem < mem_pool->custom_end[i]))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\tint i;\n+\tstruct mp_block **tail = &dst->mp_block;\n+\n+\t/* Find pointer of dst's last block (if any) */\n+\twhile (*tail)\n+\t\ttail = &(*tail)->next_block;\n+\n+\t/* Append the blocks from src to dst */\n+\t*tail = src->mp_block;\n+\n+\t/* Combine custom allocations */\n+\tALLOC_GROW(dst->custom, dst->nr + src->nr, dst->alloc);\n+\tALLOC_GROW(dst->custom_end, dst->nr_end + src->nr_end, dst->alloc_end);\n+\n+\tfor (i = 0; i < src->nr; i++) {\n+\t\tdst->custom[dst->nr++] = src->custom[i];\n+\t\tdst->custom_end[dst->nr_end++] = src->custom_end[i];\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+\tsrc->custom = NULL;\n+\tsrc->nr = 0;\n+\tsrc->alloc = 0;\n+\tsrc->custom_end = NULL;\n+\tsrc->nr_end = 0;\n+\tsrc->alloc_end = 0;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..34df4fa709 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -19,8 +19,27 @@ struct mem_pool {\n \n \t/* The total amount of memory allocated by the pool. */\n \tsize_t pool_alloc;\n+\n+\t/*\n+\t * Array of pointers to \"custom size\" memory allocations.\n+\t * This is used for \"large\" memory allocations.\n+\t * The *_end variables are used to track the range of memory\n+\t * allocated.\n+\t */\n+\tvoid **custom, **custom_end;\n+\tint nr, nr_end, alloc, alloc_end;\n };\n \n+/*\n+ * Initialize mem_pool specified initial.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n@@ -31,4 +50,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.14.3\n\n"},{"id":"346158","messageId":"20180430153122.243976-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"[PATCH v2 5/5] block alloc: add validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:45Z","receivedAt":"2018-04-30T15:31:56Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 7 +++++++\n git.c | 3 +++\n mem-pool.c | 24 +++++++++++++++++++++++-\n mem-pool.h | 2 ++\n read-cache.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7ed68f28e0..8f10f0649b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -369,6 +369,13 @@ void discard_index_cache_entry(struct cache_entry *ce);\n */\n void discard_transient_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/git.c b/git.c\nindex f598fae7b7..28ec7a6c4f 100644\n--- a/git.c\n+++ b/git.c\n@@ -347,7 +347,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex a495885c4b..77adb5d5b9 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -60,21 +60,43 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n {\n \tint i;\n \tstruct mp_block *block, *block_to_free;\n+\tint invalidate_memory = should_validate_cache_entries();\n+\n \tfor (block = mem_pool->mp_block; block;)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \n-\tfor (i = 0; i < mem_pool->nr; i++)\n+\tfor (i = 0; i < mem_pool->nr; i++) {\n+\t\tmemset(mem_pool->custom[i], 0xDD, ((char *)mem_pool->custom_end[i]) - ((char *)mem_pool->custom[i]));\n \t\tfree(mem_pool->custom[i]);\n+\t}\n \n \tfree(mem_pool->custom);\n \tfree(mem_pool->custom_end);\n \tfree(mem_pool);\n }\n \n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p;\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 34df4fa709..b1f9a920ba 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -63,4 +63,6 @@ void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n */\n int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n \n+int should_validate_cache_entries(void);\n+\n #endif\ndiff --git a/read-cache.c b/read-cache.c\nindex 01cd7fea41..e1dc9f7f33 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1270,6 +1270,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n {\n \tint pos;\n \n+\tvalidate_cache_entries(istate);\n+\n \tif (option & ADD_CACHE_JUST_APPEND)\n \t\tpos = istate->cache_nr;\n \telse {\n@@ -1290,6 +1292,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\t\t istate->cache_nr - pos - 1);\n \tset_index_entry(istate, pos, ce);\n \tistate->cache_changed |= CE_ENTRY_ADDED;\n+\n+\tvalidate_cache_entries(istate);\n \treturn 0;\n }\n \n@@ -2013,6 +2017,8 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n+\tvalidate_cache_entries(istate);\n+\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -2035,6 +2041,43 @@ int discard_index(struct index_state *istate)\n \treturn 0;\n }\n \n+\n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\tint validate_index_cache_entries = should_validate_cache_entries();\n+\n+\tif (!validate_index_cache_entries)\n+\t\treturn;\n+\n+\tif (!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2828,6 +2871,10 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n */\n void discard_index_cache_entry(struct cache_entry *ce)\n {\n+\tint invalidate_cache_entry = should_validate_cache_entries();\n+\n+\tif (ce && invalidate_cache_entry)\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n }\n \n void discard_transient_cache_entry(struct cache_entry *ce)\n-- \n2.14.3\n\n"},{"id":"346159","messageId":"20180430153122.243976-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"[PATCH v2 2/5] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:42Z","receivedAt":"2018-04-30T15:31:57Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an API around managing the lifetime of cache_entry\nstructs. Abstracting memory management details behind an API will\nallow for alternative memory management strategies without affecting\nall the call sites. This commit does not change how memory is\nallocated / freed. A later commit in this series will allocate cache\nentries from memory pools as appropriate.\n\nMotivation:\nIt has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This change\nis in preparation for using memory pools to reduce the number of\nmalloc() calls made when loading an index.\n\nThis API makes a distinction between cache entries that are intended\nfor use with a particular index and cache entries that are not. This\nenables us to use the knowledge about how a cache entry will be used\nto make informed decisions about how to handle the corresponding\nmemory.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 26 ++++++------\n blame.c | 5 +--\n builtin/checkout.c | 8 ++--\n builtin/difftool.c | 8 ++--\n builtin/reset.c | 6 +--\n builtin/update-index.c | 26 ++++++------\n cache.h | 29 +++++++++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 105 +++++++++++++++++++++++++++++++++++--------------\n resolve-undo.c | 6 ++-\n split-index.c | 8 ++--\n tree.c | 4 +-\n unpack-trees.c | 27 ++++++++-----\n 13 files changed, 166 insertions(+), 94 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 7e5792c996..123646e1aa 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4090,12 +4090,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_index_cache_entry(&result, patch->old_mode, oid.hash, name, 0, 0);\n \t\tif (!ce)\n-\t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\treturn error(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4263,12 +4263,11 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n \tif (!state->update_index)\n \t\treturn 0;\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4278,13 +4277,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tdiscard_index_cache_entry(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_index_cache_entry(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4292,13 +4291,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4422,27 +4421,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_index_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex 78c9808bd1..8067e398a1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -154,7 +154,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -252,8 +252,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b49b582071..a8af98e14c 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, oid.hash, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\tdiscard_transient_cache_entry(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex aad0e073ee..8edcb1a8b6 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid->hash, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\tdiscard_transient_cache_entry(ce);\n \treturn ret;\n }\n \n@@ -488,8 +488,8 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n-\t\t\t\t\t\t\t dst_path, 0, 0);\n+\t\t\t\t\tmake_index_cache_entry(&wtindex, rmode, roid.hash,\n+\t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\n \t\t\t\t\t\tADD_CACHE_JUST_APPEND);\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 7f1c3f02a3..1062dab73f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,10 +134,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n-\t\t\t\t 0, 0);\n+\t\tce = make_index_cache_entry(&the_index, one->mode, one->oid.hash, one->path,\n+\t\t\t\t\t 0, 0);\n \t\tif (!ce)\n-\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\tdie(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n \t\tif (is_missing) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 10d070a76f..2a7ab3cd7a 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -403,15 +402,14 @@ static int process_path(const char *path)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -589,7 +587,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -602,8 +599,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -680,8 +676,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tdiscard_index_cache_entry(ce_2);\n+\tdiscard_index_cache_entry(ce_3);\n \treturn ret;\n }\n \n@@ -748,7 +744,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tdiscard_index_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -759,7 +755,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tdiscard_index_cache_entry(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex 31f8f0420a..3760adbe25 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,34 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and freeing */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `discard_index_cache_entry`.\n+ */\n+extern struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `discard_transient_cache_entry`.\n+ */\n+extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n+extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry allocated via `make_*_index_cache_entry`.\n+ */\n+void discard_index_cache_entry(struct cache_entry *ce);\n+\n+/*\n+ * Discard cache entry allocated via `make_*_transient_cache_entry`.\n+ */\n+void discard_transient_cache_entry(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -690,7 +718,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 693f60e0a3..be118c0c6d 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -252,7 +252,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_index_cache_entry(&the_index, mode, oid ? oid->hash : null_sha1, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2cb4f53b57..2a61cee130 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tdiscard_index_cache_entry(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_index_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_index_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_index_cache_entry(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,12 +744,22 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode,\n+\t\t\t const unsigned char *sha1, const char *path,\n+\t\t\t int stage, unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -758,8 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, len);\n \n \thashcpy(ce->oid.hash, sha1);\n \tmemcpy(ce->name, path, len);\n@@ -769,10 +777,34 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n+\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1,\n+\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\thashcpy(ce->oid.hash, sha1);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1243,7 +1275,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1323,8 +1355,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1610,12 +1641,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1657,7 +1689,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1688,13 +1721,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1826,7 +1859,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1932,7 +1965,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tdiscard_index_cache_entry(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2622,14 +2655,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_index_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2738,3 +2770,16 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+/*\n+ * Free cache entry allocated for an index.\n+ */\n+void discard_index_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\n+\n+void discard_transient_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex aed95b4b35..96ef6307a6 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,8 +146,10 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n-\t\t\t\t name, i + 1, 0);\n+\t\tnce = make_index_cache_entry(istate,\n+\t\t\t\t\t ru->mode[i],\n+\t\t\t\t\t ru->oid[i].hash,\n+\t\t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\n \t\tif (add_index_entry(istate, nce, ADD_CACHE_OK_TO_ADD)) {\ndiff --git a/split-index.c b/split-index.c\nindex 3eb8ff1b43..d3326d2645 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tdiscard_index_cache_entry(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tdiscard_index_cache_entry(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tdiscard_index_cache_entry(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tdiscard_index_cache_entry(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 1c68ea586b..1ba39c9374 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -16,15 +16,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e73745051e..1ce6242f3e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -192,10 +192,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -205,7 +205,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -786,10 +786,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\tconst struct name_entry *n,\n+\tint stage,\n+\tstruct index_state *istate,\n+\tint is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_index_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -835,7 +842,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -844,7 +851,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_transient_cache_entry(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1765,7 +1772,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1785,7 +1792,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tdiscard_index_cache_entry(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1811,7 +1818,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tdiscard_index_cache_entry(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.14.3\n\n"},{"id":"346160","messageId":"20180430153122.243976-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"[PATCH v2 4/5] block alloc: allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-04-30T15:31:44Z","receivedAt":"2018-04-30T15:32:01Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"When reading large indexes from disk, a portion of the time is\ndominated in malloc() calls. This can be mitigated by allocating a\nlarge block of memory and manage it ourselves via memory pools.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entries will be allocated from. When reading in the\nindex from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool will be discarded as well - so the lifetime of\na cache_entry is tied to the lifetime of the index_state that it was\nallocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n\nAlternatives:\nThe current design does not track whether a cache_entry is allocated\nfrom a pool or not. Instead, it relies on the caller to know how the\ncache_entry will be used and to handle its lifecyle appropriately,\nincluding calling the correct free() method. Instead, we could include\na bit of information in the cache_entry (either a bit indicating\nwhether cache_entry was allocated from a pool or not, or possibly even\na pointer back to the allocating pool), which can then be used to make\ninformed decisions about these objects. The downside of this approach\nis that the cache_entry type would need to grow to incorporate this\ninformation.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 ++\n read-cache.c | 95 +++++++++++++++++++++++++++++++++++++++++++++--------------\n split-index.c | 23 +++++++++++++--\n 3 files changed, 95 insertions(+), 25 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 3760adbe25..7ed68f28e0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -328,6 +329,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 2a61cee130..01cd7fea41 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,42 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is a guess of an pathname in the index. We use this for V4\n+ * index files to guess the un-deltafied size of the index in memory\n+ * because of pathname deltafication. This is not required for V2/V3\n+ * index formats because their pathnames are not compressed. If the\n+ * initial amount of memory set aside is not sufficient, the mem pool\n+ * will allocate extra memory.\n+ */\n+#define CACHE_ENTRY_PATH_LENGTH 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\treturn mem_pool_alloc(mem_pool, cache_entry_size(len));\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\treturn mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +782,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1641,13 +1677,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1689,7 +1725,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1721,13 +1757,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1801,6 +1837,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_PATH_LENGTH);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1847,10 +1899,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1859,7 +1916,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1956,17 +2013,6 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n-\n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tdiscard_index_cache_entry(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -1980,6 +2026,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2772,11 +2824,10 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n }\n \n /*\n- * Free cache entry allocated for an index.\n+ * Indicate that a cache entry is no longer is use\n */\n void discard_index_cache_entry(struct cache_entry *ce)\n {\n-\tfree(ce);\n }\n \n void discard_transient_cache_entry(struct cache_entry *ce)\ndiff --git a/split-index.c b/split-index.c\nindex d3326d2645..6d82c4e148 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -330,6 +345,8 @@ void add_split_index(struct index_state *istate)\n void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n \t\t/*\n \t\t * can't discard_split_index(&the_index); because that\n \t\t * will destroy split_index->base->cache[], which may\n-- \n2.14.3\n\n"},{"id":"346195","messageId":"CAGZ79katqnSvQzqiXdQ_SjGWUUF19_kwFBrq+Z9Vihv7xR792w@mail.gmail.com","threadId":"48310","inReplyTo":"20180430153122.243976-4-jamill@microsoft.com","subject":"Re: [PATCH v2 3/5] mem-pool: fill out functionality","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-30T21:42:09Z","receivedAt":"2018-04-30T21:42:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Apr 30, 2018 at 8:31 AM, Jameson Miller wrote:\n> Adds the following functionality to memory pools:\n>\n> - Lifecycle management functions (init, discard)\n>\n> - Test whether a memory location is part of the managed pool\n>\n> - Function to combine 2 pools\n>\n> This also adds logic to track all memory allocations made by a memory\n> pool.\n>\n> These functions will be used in a future commit in this commit series.\n>\n> Signed-off-by: Jameson Miller \n> ---\n> mem-pool.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n> mem-pool.h | 32 +++++++++++++++++\n\n> diff --git a/mem-pool.h b/mem-pool.h\n> index 829ad58ecf..34df4fa709 100644\n> --- a/mem-pool.h\n> +++ b/mem-pool.h\n> @@ -19,8 +19,27 @@ struct mem_pool {\n>\n> /* The total amount of memory allocated by the pool. */\n> size_t pool_alloc;\n> +\n> + /*\n> + * Array of pointers to \"custom size\" memory allocations.\n> + * This is used for \"large\" memory allocations.\n> + * The *_end variables are used to track the range of memory\n> + * allocated.\n> + */\n> + void **custom, **custom_end;\n> + int nr, nr_end, alloc, alloc_end;\n> };\n\nWhat is the design goal of this mem pool?\nWhat is it really good at, which patterns of use should we avoid?\n\nIt looks like internally the mem-pool can either use mp_blocks\nthat are stored as a linked list, or it can have custom allocations\nstored in an array.\n\nIs the linked list or the custom part sorted by some key?\nDoes it need to be sorted?\n\nI am currently looking at alloc.c, which is really good for\nallocating memory for equally sized parts, i.e. it is very efficient\nat providing memory for fixed sized structs. And on top of that\nit is not tracking any memory as it relies on program termination\nfor cleanup.\n\nThis memory pool seems to be optimized for allocations of\nvarying sizes, some of them huge (to be stored in the custom\npart) and most of them rather small as they go into the mp_blocks?\n\nThanks,\nStefan\n"},{"id":"346294","messageId":"BL0PR2101MB1106AC4280E98A83A1A9BB6DCE810@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79katqnSvQzqiXdQ_SjGWUUF19_kwFBrq+Z9Vihv7xR792w@mail.gmail.com","subject":"RE: [PATCH v2 3/5] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-01T15:43:19Z","receivedAt":"2018-05-01T15:43:25Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Thank you for taking a look - I think these are good questions. Please let me know if you have further questions.\n\n> -----Original Message-----\n> From: Stefan Beller \n> Sent: Monday, April 30, 2018 5:42 PM\n> To: Jameson Miller \n> Cc: git@vger.kernel.org; gitster@pobox.com; pclouds@gmail.com;\n> jonathantanmy@google.com\n> Subject: Re: [PATCH v2 3/5] mem-pool: fill out functionality\n> \n> On Mon, Apr 30, 2018 at 8:31 AM, Jameson Miller \n> wrote:\n> > Adds the following functionality to memory pools:\n> >\n> > - Lifecycle management functions (init, discard)\n> >\n> > - Test whether a memory location is part of the managed pool\n> >\n> > - Function to combine 2 pools\n> >\n> > This also adds logic to track all memory allocations made by a memory\n> > pool.\n> >\n> > These functions will be used in a future commit in this commit series.\n> >\n> > Signed-off-by: Jameson Miller \n> > ---\n> > mem-pool.c | 114\n> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n> > mem-pool.h | 32 +++++++++++++++++\n> \n> > diff --git a/mem-pool.h b/mem-pool.h\n> > index 829ad58ecf..34df4fa709 100644\n> > --- a/mem-pool.h\n> > +++ b/mem-pool.h\n> > @@ -19,8 +19,27 @@ struct mem_pool {\n> >\n> > /* The total amount of memory allocated by the pool. */\n> > size_t pool_alloc;\n> > +\n> > + /*\n> > + * Array of pointers to \"custom size\" memory allocations.\n> > + * This is used for \"large\" memory allocations.\n> > + * The *_end variables are used to track the range of memory\n> > + * allocated.\n> > + */\n> > + void **custom, **custom_end;\n> > + int nr, nr_end, alloc, alloc_end;\n> > };\n> \n> What is the design goal of this mem pool?\n> What is it really good at, which patterns of use should we avoid?\n> \n\nThis memory pool is designed to provide many small (small\ncompared to the memory pool block size) chunks of memory from a\nlarger block of allocated memory. This reduces the overhead of\nperforming many small memory allocations from the heap. In the\nideal case, we know the total amount of memory required, and the\npool can make a single allocation to satisfy that requirement,\nand hand it out in chunks to consumers.\n\nWe should avoid making many large memory requests (large compared\nto the memory pool block size), as these requests will be\nfulfilled from individual memory allocations (i.e. the \"custom\"\nallocations). While there is not a correctness issue here, it\nwill not perform as well when requests are fulfilled from the\ninternal memory blocks.\n\n> It looks like internally the mem-pool can either use mp_blocks that are stored as\n> a linked list, or it can have custom allocations stored in an array.\n> \n> Is the linked list or the custom part sorted by some key?\n> Does it need to be sorted?\n> \n> I am currently looking at alloc.c, which is really good for allocating memory for\n> equally sized parts, i.e. it is very efficient at providing memory for fixed sized\n> structs. And on top of that it is not tracking any memory as it relies on program\n> termination for cleanup.\n\nThe linked list is ordered in the order the blocks were allocated\nin. The last allocated block will be the head of the linked\nlist. This means that most memory requests should be fulfilled by\nthe head block, reducing the need to iterate through the list to\nfind available memory.\n\nThe custom memory blocks are in their own list because they will\nnever contain any free memory. There is no need to include these\nallocations in the list of blocks that could potentially have\nfree memory.\n\nI expect this code would be efficient at allocating many equal\nsized parts, as long as those parts are comparitively small\ncompared to the block size. In this case, you would never\nallocate \"custom\" blocks, and the overwhelming majority of\nallocations would come from the head block. If you know the total\namount of memory you will need, then you can size the memory pool\nso all allocations come from the head block.\n\n> \n> This memory pool seems to be optimized for allocations of varying sizes, some\n> of them huge (to be stored in the custom\n> part) and most of them rather small as they go into the mp_blocks?\n\nI would say this memory pool is optimized for allocations of\nvarying sizes (although it should be pretty efficient when the\nallocations are of the same size), but can handle the edge case\nwhen there happens to be a need for a \"huge allocation\".\n> \n> Thanks,\n> Stefan\n"},{"id":"346539","messageId":"CACsJy8DR4wNoucfL0++-k73EPnJL1YSsoB5prjG6YtDPzu9qRQ@mail.gmail.com","threadId":"48310","inReplyTo":"20180430153122.243976-4-jamill@microsoft.com","subject":"Re: [PATCH v2 3/5] mem-pool: fill out functionality","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T16:18:16Z","receivedAt":"2018-05-03T16:18:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Another I noticed in the jm/mem-pool series is this loop in mem_pool_alloc()\n\nfor (p = mem_pool->mp_block; p; p = p->next_block)\n if (p->end - p->next_free >= len)\n break;\n\nYou always go from the start (mp_block) but at some point those first\nblocks are filled up and we don't really need to walk from the start\nanymore. If we allow the mem-pool user to set a \"minimum alloc\" limit,\nthen we can determine if the remaining space in a block is not useful\nfor any future allocation, we can just skip it and start looking for\nan available from a new pointer, avail_block or something.\n\nI'm writing this with alloc.c in mind because we have a lot more\nblocks to allocate there. Unlike read-cache, you can't really estimate\nhow many mp_blocks you're going to need. This linked list could become\nlong. And because alloc.c does fixed size allocation, we use up one\nblock after another and will never find free space in previous blocks.\n\nOn Mon, Apr 30, 2018 at 5:31 PM, Jameson Miller wrote:\n> diff --git a/mem-pool.c b/mem-pool.c\n> index 389d7af447..a495885c4b 100644\n> --- a/mem-pool.c\n> +++ b/mem-pool.c\n> @@ -5,6 +5,8 @@\n> #include \"cache.h\"\n> #include \"mem-pool.h\"\n>\n> +#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n\n#define BLOCK_GROWTH_SIZE (1024*1024 - sizeof(struct mp_block))\n\n(wrapped in brackets and no trailing semicolon)\n\n> +\n> static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n> {\n> struct mp_block *p;\n> @@ -19,6 +21,60 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n> return p;\n> }\n>\n> +static void *mem_pool_alloc_custom(struct mem_pool *mem_pool, size_t block_alloc)\n> +{\n> + char *p;\n\nAn empty line between variable declaration and function body would be nice.\n\n> + ALLOC_GROW(mem_pool->custom, mem_pool->nr + 1, mem_pool->alloc);\n> + ALLOC_GROW(mem_pool->custom_end, mem_pool->nr_end + 1, mem_pool->alloc_end);\n> +\n\nIf you put both custom and custom_end in a struct, then you can grow\njust one array (of the new struct) and have fewer support variables\nlike nr_end and alloc_end\n\nThe only downside that I can see is the potential padding between\nstruct increasing memory consumption here. but since you don't care\nabout reallocation cost (i.e. large memory allocations should be\nrare), it probably does not matter either.\n\nBut wait, can we just reuse struct mp_block for this? You allocate a\nnew mp_block (for just one allocation) as usual, then you can can\nmaintain a linked list of custom alloc in \"struct mp_block\n*mp_custom_block\" or something. This way we can walk both bulk and\ncustom allocation the same way.\n\n> + p = xmalloc(block_alloc);\n> + mem_pool->custom[mem_pool->nr++] = p;\n> + mem_pool->custom_end[mem_pool->nr_end++] = p + block_alloc;\n> +\n> + mem_pool->pool_alloc += block_alloc;\n> +\n> + return mem_pool->custom[mem_pool->nr];\n> +}\n> +\n> +void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n> +{\n> + if (!(*mem_pool))\n\nI think (!*mem_pool) should be enough, although you could avoid the\noperator precedence headache by doing\n\nif (*mem_pool)\n return;\n\n> + {\n> + *mem_pool = xmalloc(sizeof(struct mem_pool));\n\nI think we tend to do *mem_pool = xmalloc(sizeof(**mem_pool));\n\n> + (*mem_pool)->pool_alloc = 0;\n\nEghh.. perhaps just declare\n\nstruct mem_pool *pool;\n\nthen allocate a new memory to this, initialize everything and only do\n\n*mem_pool = pool;\n\nat the end? It's much less of this (*mem_pool)->\n\n> + (*mem_pool)->mp_block = NULL;\n\nJust memset() it once (or use xcallo) and only initialize\nnon-zero/null fields afterwards.\n\n> + (*mem_pool)->block_alloc = BLOCK_GROWTH_SIZE;\n> + (*mem_pool)->custom = NULL;\n> + (*mem_pool)->nr = 0;\n> + (*mem_pool)->alloc = 0;\n> + (*mem_pool)->custom_end = NULL;\n> + (*mem_pool)->nr_end = 0;\n> + (*mem_pool)->alloc_end = 0;\n> +\n> + if (initial_size > 0)\n> + mem_pool_alloc_block(*mem_pool, initial_size);\n> + }\n> +}\n> +\n> +void mem_pool_discard(struct mem_pool *mem_pool)\n> +{\n> + int i;\n> + struct mp_block *block, *block_to_free;\n> + for (block = mem_pool->mp_block; block;)\n> + {\n> + block_to_free = block;\n> + block = block->next_block;\n> + free(block_to_free);\n> + }\n> +\n> + for (i = 0; i < mem_pool->nr; i++)\n> + free(mem_pool->custom[i]);\n> +\n> + free(mem_pool->custom);\n> + free(mem_pool->custom_end);\n> + free(mem_pool);\n> +}\n> +\n> void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n> {\n> struct mp_block *p;\n> @@ -33,10 +89,8 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n> break;\n>\n> if (!p) {\n> - if (len >= (mem_pool->block_alloc / 2)) {\n> - mem_pool->pool_alloc += len;\n> - return xmalloc(len);\n> - }\n> + if (len >= (mem_pool->block_alloc / 2))\n> + return mem_pool_alloc_custom(mem_pool, len);\n>\n> p = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n> }\n> @@ -53,3 +107,55 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n> memset(r, 0, len);\n> return r;\n> }\n> +\n> +int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n> +{\n> + int i;\n> + struct mp_block *p;\n> +\n> + /* Check if memory is allocated in a block */\n> + for (p = mem_pool->mp_block; p; p = p->next_block)\n> + if ((mem >= ((void *)p->space)) &&\n> + (mem < ((void *)p->end)))\n> + return 1;\n> +\n> + /* Check if memory is allocated in custom block */\n> + for (i = 0; i < mem_pool->nr; i++)\n> + if ((mem >= mem_pool->custom[i]) &&\n> + (mem < mem_pool->custom_end[i]))\n> + return 1;\n> +\n> + return 0;\n> +}\n> +\n> +void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n> +{\n> + int i;\n> + struct mp_block **tail = &dst->mp_block;\n> +\n> + /* Find pointer of dst's last block (if any) */\n> + while (*tail)\n> + tail = &(*tail)->next_block;\n> +\n> + /* Append the blocks from src to dst */\n> + *tail = src->mp_block;\n> +\n> + /* Combine custom allocations */\n> + ALLOC_GROW(dst->custom, dst->nr + src->nr, dst->alloc);\n> + ALLOC_GROW(dst->custom_end, dst->nr_end + src->nr_end, dst->alloc_end);\n> +\n> + for (i = 0; i < src->nr; i++) {\n> + dst->custom[dst->nr++] = src->custom[i];\n> + dst->custom_end[dst->nr_end++] = src->custom_end[i];\n> + }\n> +\n> + dst->pool_alloc += src->pool_alloc;\n> + src->pool_alloc = 0;\n> + src->mp_block = NULL;\n> + src->custom = NULL;\n> + src->nr = 0;\n> + src->alloc = 0;\n> + src->custom_end = NULL;\n> + src->nr_end = 0;\n> + src->alloc_end = 0;\n> +}\n> diff --git a/mem-pool.h b/mem-pool.h\n> index 829ad58ecf..34df4fa709 100644\n> --- a/mem-pool.h\n> +++ b/mem-pool.h\n> @@ -19,8 +19,27 @@ struct mem_pool {\n>\n> /* The total amount of memory allocated by the pool. */\n> size_t pool_alloc;\n> +\n> + /*\n> + * Array of pointers to \"custom size\" memory allocations.\n> + * This is used for \"large\" memory allocations.\n> + * The *_end variables are used to track the range of memory\n> + * allocated.\n> + */\n> + void **custom, **custom_end;\n> + int nr, nr_end, alloc, alloc_end;\n> };\n>\n> +/*\n> + * Initialize mem_pool specified initial.\n> + */\n> +void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n> +\n> +/*\n> + * Discard a memory pool and free all the memory it is responsible for.\n> + */\n> +void mem_pool_discard(struct mem_pool *mem_pool);\n> +\n> /*\n> * Alloc memory from the mem_pool.\n> */\n> @@ -31,4 +50,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n> */\n> void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n>\n> +/*\n> + * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n> + * pool will be empty and not contain any memory. It still needs to be free'd\n> + * with a call to `mem_pool_discard`.\n> + */\n> +void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n> +\n> +/*\n> + * Check if a memory pointed at by 'mem' is part of the range of\n> + * memory managed by the specified mem_pool.\n> + */\n> +int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n> +\n> #endif\n> --\n> 2.14.3\n>\n\n\n\n-- \nDuy\n"},{"id":"346540","messageId":"CACsJy8Dxz8dph=Ouy2C9PXQaJUJw86FHdobvJxcR-Dx+w5iO0Q@mail.gmail.com","threadId":"48310","inReplyTo":"20180430153122.243976-6-jamill@microsoft.com","subject":"Re: [PATCH v2 5/5] block alloc: add validations around cache_entry lifecyle","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T16:28:41Z","receivedAt":"2018-05-03T16:29:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 30, 2018 at 5:31 PM, Jameson Miller wrote:\n> Add an option (controlled by an environment variable) perform extra\n> validations on mem_pool allocated cache entries. When set:\n>\n> 1) Invalidate cache_entry memory when discarding cache_entry.\n>\n> 2) When discarding index_state struct, verify that all cache_entries\n> were allocated from expected mem_pool.\n>\n> 3) When discarding mem_pools, invalidate mem_pool memory.\n\nOn linux step 3 could be better achieved by allocating blocks with\nmmap() and freeing them with munmap(). Access to already munmap()'d\nblocks will result in segmentation fault regardless of values. I\nguess Windows also has something similar (I vaguely remember something\nabout \"locking memory\" and stuff, but my win32 knowledge is decade old\nat this point)\n\n(Actually with glibc on linux, i'm pretty sure mmap is already used\nfor large allocation so step 3 is achieved without doing anything; not\nsure about other libc implementations)\n\n> This should provide extra checks that mem_pools and their allocated\n> cache_entries are being used as expected.\n>\n> Signed-off-by: Jameson Miller \n> ---\n> cache.h | 7 +++++++\n> git.c | 3 +++\n> mem-pool.c | 24 +++++++++++++++++++++++-\n> mem-pool.h | 2 ++\n> read-cache.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++\n> 5 files changed, 82 insertions(+), 1 deletion(-)\n>\n> diff --git a/cache.h b/cache.h\n> index 7ed68f28e0..8f10f0649b 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -369,6 +369,13 @@ void discard_index_cache_entry(struct cache_entry *ce);\n> */\n> void discard_transient_cache_entry(struct cache_entry *ce);\n>\n> +/*\n> + * Validate the cache entries in the index. This is an internal\n> + * consistency check that the cache_entry structs are allocated from\n> + * the expected memory pool.\n> + */\n> +void validate_cache_entries(const struct index_state *istate);\n> +\n> #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n> #define active_cache (the_index.cache)\n> #define active_nr (the_index.cache_nr)\n> diff --git a/git.c b/git.c\n> index f598fae7b7..28ec7a6c4f 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -347,7 +347,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n>\n> trace_argv_printf(argv, \"trace: built-in: git\");\n>\n> + validate_cache_entries(&the_index);\n> status = p->fn(argc, argv, prefix);\n> + validate_cache_entries(&the_index);\n> +\n> if (status)\n> return status;\n>\n> diff --git a/mem-pool.c b/mem-pool.c\n> index a495885c4b..77adb5d5b9 100644\n> --- a/mem-pool.c\n> +++ b/mem-pool.c\n> @@ -60,21 +60,43 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n> {\n> int i;\n> struct mp_block *block, *block_to_free;\n> + int invalidate_memory = should_validate_cache_entries();\n\nHeh.. cache-entries logic should not enter mem-pool.c.\n\n> +\n> for (block = mem_pool->mp_block; block;)\n> {\n> block_to_free = block;\n> block = block->next_block;\n> +\n> + if (invalidate_memory)\n> + memset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n> +\n> free(block_to_free);\n> }\n>\n> - for (i = 0; i < mem_pool->nr; i++)\n> + for (i = 0; i < mem_pool->nr; i++) {\n> + memset(mem_pool->custom[i], 0xDD, ((char *)mem_pool->custom_end[i]) - ((char *)mem_pool->custom[i]));\n\n\"if (invalidate_memory)\" missing\n\n> free(mem_pool->custom[i]);\n> + }\n>\n> free(mem_pool->custom);\n> free(mem_pool->custom_end);\n> free(mem_pool);\n> }\n>\n> +int should_validate_cache_entries(void)\n> +{\n> + static int validate_index_cache_entries = -1;\n> +\n> + if (validate_index_cache_entries < 0) {\n> + if (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n\nThere's a safer version that you can use, get_env_bool()\n\n> + validate_index_cache_entries = 1;\n> + else\n> + validate_index_cache_entries = 0;\n> + }\n> +\n> + return validate_index_cache_entries;\n> +}\n> +\n> void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n> {\n> struct mp_block *p;\n> diff --git a/mem-pool.h b/mem-pool.h\n> index 34df4fa709..b1f9a920ba 100644\n> --- a/mem-pool.h\n> +++ b/mem-pool.h\n> @@ -63,4 +63,6 @@ void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n> */\n> int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n>\n> +int should_validate_cache_entries(void);\n> +\n> #endif\n> diff --git a/read-cache.c b/read-cache.c\n> index 01cd7fea41..e1dc9f7f33 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1270,6 +1270,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n> {\n> int pos;\n>\n> + validate_cache_entries(istate);\n\nValidating _all_ entries every time an entry is added sounds way too expensive.\n\n> +\n> if (option & ADD_CACHE_JUST_APPEND)\n> pos = istate->cache_nr;\n> else {\n> @@ -1290,6 +1292,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n> istate->cache_nr - pos - 1);\n> set_index_entry(istate, pos, ce);\n> istate->cache_changed |= CE_ENTRY_ADDED;\n> +\n> + validate_cache_entries(istate);\n> return 0;\n> }\n>\n> @@ -2013,6 +2017,8 @@ int is_index_unborn(struct index_state *istate)\n>\n> int discard_index(struct index_state *istate)\n> {\n> + validate_cache_entries(istate);\n> +\n> resolve_undo_clear_index(istate);\n> istate->cache_nr = 0;\n> istate->cache_changed = 0;\n> @@ -2035,6 +2041,43 @@ int discard_index(struct index_state *istate)\n> return 0;\n> }\n>\n> +\n> +/*\n> + * Validate the cache entries of this index.\n> + * All cache entries associated with this index\n> + * should have been allocated by the memory pool\n> + * associated with this index, or by a referenced\n> + * split index.\n> + */\n> +void validate_cache_entries(const struct index_state *istate)\n> +{\n> + int i;\n> + int validate_index_cache_entries = should_validate_cache_entries();\n> +\n> + if (!validate_index_cache_entries)\n> + return;\n> +\n> + if (!istate || !istate->initialized)\n> + return;\n> +\n> + for (i = 0; i < istate->cache_nr; i++) {\n> + if (!istate) {\n> + die(\"internal error: cache entry is not allocated from expected memory pool\");\n> + } else if (!istate->ce_mem_pool ||\n> + !mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n> + if (!istate->split_index ||\n> + !istate->split_index->base ||\n> + !istate->split_index->base->ce_mem_pool ||\n> + !mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n> + die(\"internal error: cache entry is not allocated from expected memory pool\");\n> + }\n> + }\n> + }\n> +\n> + if (istate->split_index)\n> + validate_cache_entries(istate->split_index->base);\n> +}\n> +\n> int unmerged_index(const struct index_state *istate)\n> {\n> int i;\n> @@ -2828,6 +2871,10 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n> */\n> void discard_index_cache_entry(struct cache_entry *ce)\n> {\n> + int invalidate_cache_entry = should_validate_cache_entries();\n> +\n> + if (ce && invalidate_cache_entry)\n> + memset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n> }\n>\n> void discard_transient_cache_entry(struct cache_entry *ce)\n> --\n> 2.14.3\n>\n\n\n\n-- \nDuy\n"},{"id":"346542","messageId":"CACsJy8C3yZ6bSZGw1d5DjwcPgHMDE7iGGUOn9SvyLmshzs9DvQ@mail.gmail.com","threadId":"48310","inReplyTo":"20180430153122.243976-1-jamill@microsoft.com","subject":"Re: [PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T16:35:48Z","receivedAt":"2018-05-03T16:36:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 30, 2018 at 5:31 PM, Jameson Miller wrote:\n> This patch series improves the performance of loading indexes by\n> reducing the number of malloc() calls. Loading the index from disk is\n> partly dominated by the time in malloc(), which is called for each\n> index entry. This patch series reduces the number of times malloc() is\n> called as part of loading the index, and instead allocates a block of\n> memory upfront that is large enough to hold all of the cache entries,\n> and chunks this memory itself. This change builds on [1].\n\nI have only looked at the mem-pool related patches to see if\nmem-pool.c is good enough to replace alloc.c. To me, it's a \"yes\"\nafter we optimize mem_pool_alloc() a bit (not that performance really\nmatters in alloc.c case, but that may be because it's already\nblazingly fast that we never noticed about it).\n\nI probably should look at read-cache.c changes too. Maybe later.\nAlthough after the change to use xmalloc() per entry a few years(?)\nago, it should be straight forward to use a different memory\nallocator.\n-- \nDuy\n"},{"id":"346550","messageId":"CAGZ79kbyPya0igC9v5JwNTsi0TSp-sj6v5SB4MOt625+yVB-Gg@mail.gmail.com","threadId":"48310","inReplyTo":"CACsJy8C3yZ6bSZGw1d5DjwcPgHMDE7iGGUOn9SvyLmshzs9DvQ@mail.gmail.com","subject":"Re: [PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T17:21:42Z","receivedAt":"2018-05-03T17:21:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 3, 2018 at 9:35 AM, Duy Nguyen wrote:\n> On Mon, Apr 30, 2018 at 5:31 PM, Jameson Miller wrote:\n>> This patch series improves the performance of loading indexes by\n>> reducing the number of malloc() calls. Loading the index from disk is\n>> partly dominated by the time in malloc(), which is called for each\n>> index entry. This patch series reduces the number of times malloc() is\n>> called as part of loading the index, and instead allocates a block of\n>> memory upfront that is large enough to hold all of the cache entries,\n>> and chunks this memory itself. This change builds on [1].\n>\n> I have only looked at the mem-pool related patches to see if\n> mem-pool.c is good enough to replace alloc.c. To me, it's a \"yes\"\n> after we optimize mem_pool_alloc() a bit (not that performance really\n> matters in alloc.c case, but that may be because it's already\n> blazingly fast that we never noticed about it).\n\nalloc.c was not just about speed, but mostly about dense packing?\n855419f764a (Add specialized object allocator, 2006-06-19)\n\nTo me it is also a clear yes when it comes to combining these\ntwo memory pools.\n"},{"id":"346571","messageId":"CACsJy8DF40o3gesuNbHACK+NiE=iZuXxLacyEQjECYewDy_icw@mail.gmail.com","threadId":"48310","inReplyTo":"CAGZ79kbyPya0igC9v5JwNTsi0TSp-sj6v5SB4MOt625+yVB-Gg@mail.gmail.com","subject":"Re: [PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T19:17:23Z","receivedAt":"2018-05-03T19:17:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, May 3, 2018 at 7:21 PM, Stefan Beller wrote:\n> On Thu, May 3, 2018 at 9:35 AM, Duy Nguyen wrote:\n>> On Mon, Apr 30, 2018 at 5:31 PM, Jameson Miller wrote:\n>>> This patch series improves the performance of loading indexes by\n>>> reducing the number of malloc() calls. Loading the index from disk is\n>>> partly dominated by the time in malloc(), which is called for each\n>>> index entry. This patch series reduces the number of times malloc() is\n>>> called as part of loading the index, and instead allocates a block of\n>>> memory upfront that is large enough to hold all of the cache entries,\n>>> and chunks this memory itself. This change builds on [1].\n>>\n>> I have only looked at the mem-pool related patches to see if\n>> mem-pool.c is good enough to replace alloc.c. To me, it's a \"yes\"\n>> after we optimize mem_pool_alloc() a bit (not that performance really\n>> matters in alloc.c case, but that may be because it's already\n>> blazingly fast that we never noticed about it).\n>\n> alloc.c was not just about speed, but mostly about dense packing?\n> 855419f764a (Add specialized object allocator, 2006-06-19)\n\nI know. I vaguely remembered Linus made that change but did not really\nlook it up :) That reference should be included when/if you switch\nfrom alloc.c to mem-pool.c.\n\n> To me it is also a clear yes when it comes to combining these\n> two memory pools.\n\nI also did not notice that jm/mem-pool already landed in master. Have\nyou tried measure (both memory usage and allocation speed) of it and\nalloc.c? Just take some big repo as an example and do count-objects -v\nto see how many blobs/trees/commits it has, then allocate the same\namount with both alloc.c and mem-pool.c and measure both speed/mem.\nI'm pretty sure you're right that mem-pool.c is a clear yes. I was\njust being more conservative because we do (slightly) change\nallocator's behavior when we make the switch. But it's also very\nlikely that any performance difference will be insignificant.\n\nI'm asking this because if mem-pool.c is a clear winner, you can start\nto update you series to use it now and kill alloc.c in the process.\n\nPS. Is Jeff back yet? I'm sure Junio is listening and all but I'm\nafraid he's too busy being a maintainer so Jeff's opinion in this area\nis really valuable. He has all the fun and weird use cases to play\nwith at github.\n-- \nDuy\n"},{"id":"346579","messageId":"CAGZ79kb1K9iysibzmn2SUfyTayXOn97wTsoL4s557j+_ievkBA@mail.gmail.com","threadId":"48310","inReplyTo":"CACsJy8DF40o3gesuNbHACK+NiE=iZuXxLacyEQjECYewDy_icw@mail.gmail.com","subject":"Re: [PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T20:58:38Z","receivedAt":"2018-05-03T20:58:43Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 3, 2018 at 12:17 PM, Duy Nguyen wrote:\n>\n>> To me it is also a clear yes when it comes to combining these\n>> two memory pools.\n>\n> I also did not notice that jm/mem-pool already landed in master.\n\nOh, thanks for telling! Now that I look at it, I am doubting it;\n\nThe reason for my doubt is the potential quadratic behavior for\nnew allocations, in mem_pool_alloc() we walk all mp_blocks to\nsee if we can fit the requested allocation in one of the later blocks.\nSo if we call mem_pool_alloc a million times, we get a O(n)\nmp_blocks which we'd have to walk in each call.\n\nHowever in alloc.c we do know that a slab is full as soon as we\nlook take the next slab. That is the beauty of knowing 'len' at\nconstruction time of the allocator.\n\nSo I guess I'll just re-use the mp_block and introduce another\nstruct fixed_sized_mem_pool, which will not look into other mp_blocks\nbut the current.\n\n\n> Have\n> you tried measure (both memory usage and allocation speed) of it and\n> alloc.c?\n\nNo, I was about to, but then started reading the code in an attempt to replace\nalloc.c by a mempool and saw the quadratic behavior.\n\n> Just take some big repo as an example and do count-objects -v\n> to see how many blobs/trees/commits it has, then allocate the same\n> amount with both alloc.c and mem-pool.c and measure both speed/mem.\n> I'm pretty sure you're right that mem-pool.c is a clear yes. I was\n> just being more conservative because we do (slightly) change\n> allocator's behavior when we make the switch. But it's also very\n> likely that any performance difference will be insignificant.\n>\n> I'm asking this because if mem-pool.c is a clear winner, you can start\n> to update you series to use it now and kill alloc.c in the process.\n\nI'll implement the fixed_sized_mem_pool and take some measurements.\n\n>\n> PS. Is Jeff back yet?\n\nHis last email on the public list is Apr 10th, stating that he'll be offline for\n\"a few weeks\", in <20180406175349.GB32228@sigill.intra.peff.net> he\nsaid the vacation part is 3 weeks. So I think he is done with vacation and\nis just hiding to figure out a nice comeback. ;-)\n\n> I'm sure Junio is listening and all but I'm\n> afraid he's too busy being a maintainer so Jeff's opinion in this area\n> is really valuable. He has all the fun and weird use cases to play\n> with at github.\n\nok. I'll cc him for these patches.\n\nThanks,\nStefan\n"},{"id":"346585","messageId":"BL0PR2101MB1106BA184260609DA69988A6CE870@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79kb1K9iysibzmn2SUfyTayXOn97wTsoL4s557j+_ievkBA@mail.gmail.com","subject":"RE: [PATCH v2 0/5] Allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-03T21:13:16Z","receivedAt":"2018-05-03T21:13:22Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"\n\n> -----Original Message-----\n> From: git-owner@vger.kernel.org On Behalf Of\n> Stefan Beller\n> Sent: Thursday, May 3, 2018 4:59 PM\n> To: Duy Nguyen \n> Cc: Jameson Miller ; git@vger.kernel.org;\n> gitster@pobox.com; jonathantanmy@google.com\n> Subject: Re: [PATCH v2 0/5] Allocate cache entries from memory pool\n> \n> On Thu, May 3, 2018 at 12:17 PM, Duy Nguyen wrote:\n> >\n> >> To me it is also a clear yes when it comes to combining these two\n> >> memory pools.\n> >\n> > I also did not notice that jm/mem-pool already landed in master.\n> \n> Oh, thanks for telling! Now that I look at it, I am doubting it;\n> \n> The reason for my doubt is the potential quadratic behavior for new allocations,\n> in mem_pool_alloc() we walk all mp_blocks to see if we can fit the requested\n> allocation in one of the later blocks.\n> So if we call mem_pool_alloc a million times, we get a O(n) mp_blocks which\n> we'd have to walk in each call.\n\nWith the current design, when a new mp_block is allocated, it is\nplaced at the head of the linked list. This means that the most\nrecently allocated mp_block is the 1st block that is\nsearched. The *vast* majority of allocations should be fulfilled\nfrom this 1st block. It is only when the block is full that we\nsearch other mp_blocks in the list. If this is a concern, I think\nwe have a couple low cost options to mitigate it (maybe a flag to\ncontrol whether we search past the 1st mp_block for space, or\nlogic to move blocks out of the search queue when they are\nfull or fall below a threshold for available space).\n\nIf this is of interest, I could contribute a patch to enable one\nof these behaviors?\n\n> \n> However in alloc.c we do know that a slab is full as soon as we look take the\n> next slab. That is the beauty of knowing 'len' at construction time of the\n> allocator.\n> \n> So I guess I'll just re-use the mp_block and introduce another struct\n> fixed_sized_mem_pool, which will not look into other mp_blocks but the\n> current.\n> \n> \n> > Have\n> > you tried measure (both memory usage and allocation speed) of it and\n> > alloc.c?\n> \n> No, I was about to, but then started reading the code in an attempt to replace\n> alloc.c by a mempool and saw the quadratic behavior.\n> \n> > Just take some big repo as an example and do count-objects -v to see\n> > how many blobs/trees/commits it has, then allocate the same amount\n> > with both alloc.c and mem-pool.c and measure both speed/mem.\n> > I'm pretty sure you're right that mem-pool.c is a clear yes. I was\n> > just being more conservative because we do (slightly) change\n> > allocator's behavior when we make the switch. But it's also very\n> > likely that any performance difference will be insignificant.\n> >\n> > I'm asking this because if mem-pool.c is a clear winner, you can start\n> > to update you series to use it now and kill alloc.c in the process.\n> \n> I'll implement the fixed_sized_mem_pool and take some measurements.\n> \n> >\n> > PS. Is Jeff back yet?\n> \n> His last email on the public list is Apr 10th, stating that he'll be offline for \"a few\n> weeks\", in <20180406175349.GB32228@sigill.intra.peff.net> he said the\n> vacation part is 3 weeks. So I think he is done with vacation and is just hiding to\n> figure out a nice comeback. ;-)\n> \n> > I'm sure Junio is listening and all but I'm afraid he's too busy being\n> > a maintainer so Jeff's opinion in this area is really valuable. He has\n> > all the fun and weird use cases to play with at github.\n> \n> ok. I'll cc him for these patches.\n> \n> Thanks,\n> Stefan\n"},{"id":"346592","messageId":"20180503221802.61110-1-sbeller@google.com","threadId":"48310","inReplyTo":"BL0PR2101MB1106BA184260609DA69988A6CE870@BL0PR2101MB1106.namprd21.prod.outlook.com","subject":"[PATCH] alloc.c: replace alloc by mempool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T22:18:02Z","receivedAt":"2018-05-03T22:18:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n\n>> The reason for my doubt is the potential quadratic behavior for new allocations,\n>> in mem_pool_alloc() we walk all mp_blocks to see if we can fit the requested\n>> allocation in one of the later blocks.\n>> So if we call mem_pool_alloc a million times, we get a O(n) mp_blocks which\n>> we'd have to walk in each call.\n>\n> With the current design, when a new mp_block is allocated, it is\n> placed at the head of the linked list. This means that the most\n> recently allocated mp_block is the 1st block that is\n> searched. The *vast* majority of allocations should be fulfilled\n> from this 1st block. It is only when the block is full that we\n> search other mp_blocks in the list.\n\nI just measured on git.git and linux.git (both of which are not *huge* by\nany standard, but should give a good indication. linux has 6M objects,\nand when allocating 1024 at a time, we run into the new block allocation\n~6000 times).\n\nI could not measure any meaningful difference.\n\nlinux.git $ git count-objects -v\ncount: 0\nsize: 0\nin-pack: 6036543\npacks: 2\nsize-pack: 2492985\nprune-packable: 0\ngarbage: 0\nsize-garbage: 0\n\n(with this patch)\n Performance counter stats for '/u/git/git count-objects -v' (30 runs):\n\n 2.123683 task-clock:u (msec) # 0.831 CPUs utilized ( +- 2.32% )\n 0 context-switches:u # 0.000 K/sec \n 0 cpu-migrations:u # 0.000 K/sec \n 126 page-faults:u # 0.059 M/sec ( +- 0.22% )\n 895,900 cycles:u # 0.422 GHz ( +- 1.40% )\n 976,596 instructions:u # 1.09 insn per cycle ( +- 0.01% )\n 218,256 branches:u # 102.772 M/sec ( +- 0.01% )\n 8,331 branch-misses:u # 3.82% of all branches ( +- 0.61% )\n\n 0.002556203 seconds time elapsed ( +- 2.20% )\n\n Performance counter stats for 'git count-objects -v' (30 runs):\n\n 2.410352 task-clock:u (msec) # 0.801 CPUs utilized ( +- 2.79% )\n 0 context-switches:u # 0.000 K/sec \n 0 cpu-migrations:u # 0.000 K/sec \n 131 page-faults:u # 0.054 M/sec ( +- 0.16% )\n 993,301 cycles:u # 0.412 GHz ( +- 1.99% )\n 1,087,428 instructions:u # 1.09 insn per cycle ( +- 0.02% )\n 244,292 branches:u # 101.351 M/sec ( +- 0.02% )\n 9,264 branch-misses:u # 3.79% of all branches ( +- 0.57% )\n\n 0.003010854 seconds time elapsed ( +- 2.54% )\n\nSo I think we could just replace it for now and optimize again later, if it\nturns out to be a problem. I think the easiest optimisation is to increase\nthe allocation size of having a lot more objects per mp_block.\n\n> If this is a concern, I think\n> we have a couple low cost options to mitigate it (maybe a flag to\n> control whether we search past the 1st mp_block for space, or\n> logic to move blocks out of the search queue when they are\n> full or fall below a threshold for available space).\n\nInstead of a flag I thought of its own struct with its own functions,\nwhich would not just have a different searching behavior, but also\nstore the size in the struct such that you can just call\nfixed_size_mem_pool_alloc(void) to get another pointer.\nA flag might be more elegant.\n\n>\n> If this is of interest, I could contribute a patch to enable one\n> of these behaviors?\n\nI am tempted to just do away with alloc.c for now and use the mem-pool.\n\nThanks,\nStefan\n\n\n\n alloc.c | 60 +++++++++++----------------------------------------------\n 1 file changed, 11 insertions(+), 49 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacdd..bf003e161be 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -15,6 +15,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"mem-pool.h\"\n \n #define BLOCKING 1024\n \n@@ -26,61 +27,39 @@ union any_object {\n \tstruct tag tag;\n };\n \n-struct alloc_state {\n-\tint count; /* total number of nodes allocated */\n-\tint nr; /* number of nodes left in current allocation */\n-\tvoid *p; /* first free node in current allocation */\n-};\n-\n-static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n-{\n-\tvoid *ret;\n-\n-\tif (!s->nr) {\n-\t\ts->nr = BLOCKING;\n-\t\ts->p = xmalloc(BLOCKING * node_size);\n-\t}\n-\ts->nr--;\n-\ts->count++;\n-\tret = s->p;\n-\ts->p = (char *)s->p + node_size;\n-\tmemset(ret, 0, node_size);\n-\treturn ret;\n-}\n-\n-static struct alloc_state blob_state;\n+static struct mem_pool blob_state = {NULL, sizeof(struct blob)*1024 - sizeof(struct mp_block), 0 };\n void *alloc_blob_node(void)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = mem_pool_alloc(&blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n+static struct mem_pool tree_state = {NULL, sizeof(struct tree)*1024 - sizeof(struct mp_block), 0 };\n void *alloc_tree_node(void)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = mem_pool_alloc(&tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n+static struct mem_pool tag_state = {NULL, sizeof(struct tag)*1024 - sizeof(struct mp_block), 0 };\n void *alloc_tag_node(void)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = mem_pool_alloc(&tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n+static struct mem_pool object_state = {NULL, sizeof(union any_object)*1024 - sizeof(struct mp_block), 0 };\n void *alloc_object_node(void)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = mem_pool_alloc(&object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n+static struct mem_pool commit_state = {NULL, sizeof(struct commit)*1024 - sizeof(struct mp_block), 0 };\n \n unsigned int alloc_commit_index(void)\n {\n@@ -90,26 +69,9 @@ unsigned int alloc_commit_index(void)\n \n void *alloc_commit_node(void)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = mem_pool_alloc(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n \treturn c;\n }\n \n-static void report(const char *name, unsigned int count, size_t size)\n-{\n-\tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\n-\t\t\tname, count, (uintmax_t) size);\n-}\n-\n-#define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n-\n-void alloc_report(void)\n-{\n-\tREPORT(blob, struct blob);\n-\tREPORT(tree, struct tree);\n-\tREPORT(commit, struct commit);\n-\tREPORT(tag, struct tag);\n-\tREPORT(object, union any_object);\n-}\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346683","messageId":"CACsJy8De2SUGj9hgq3h7pODm=9Wn+TkCKfJjrm1c=UxS9BZvTA@mail.gmail.com","threadId":"48310","inReplyTo":"20180503221802.61110-1-sbeller@google.com","subject":"Re: [PATCH] alloc.c: replace alloc by mempool","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-04T16:33:52Z","receivedAt":"2018-05-04T16:34:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, May 4, 2018 at 12:18 AM, Stefan Beller wrote:\n> I just measured on git.git and linux.git (both of which are not *huge* by\n> any standard, but should give a good indication. linux has 6M objects,\n> and when allocating 1024 at a time, we run into the new block allocation\n> ~6000 times).\n>\n> I could not measure any meaningful difference.\n>\n> linux.git $ git count-objects -v\n> count: 0\n> size: 0\n> in-pack: 6036543\n> packs: 2\n> size-pack: 2492985\n> prune-packable: 0\n> garbage: 0\n> size-garbage: 0\n>\n> (with this patch)\n> Performance counter stats for '/u/git/git count-objects -v' (30 runs):\n>\n> 2.123683 task-clock:u (msec) # 0.831 CPUs utilized ( +- 2.32% )\n> 0 context-switches:u # 0.000 K/sec\n> 0 cpu-migrations:u # 0.000 K/sec\n> 126 page-faults:u # 0.059 M/sec ( +- 0.22% )\n> 895,900 cycles:u # 0.422 GHz ( +- 1.40% )\n> 976,596 instructions:u # 1.09 insn per cycle ( +- 0.01% )\n> 218,256 branches:u # 102.772 M/sec ( +- 0.01% )\n> 8,331 branch-misses:u # 3.82% of all branches ( +- 0.61% )\n>\n> 0.002556203 seconds time elapsed ( +- 2.20% )\n>\n> Performance counter stats for 'git count-objects -v' (30 runs):\n>\n> 2.410352 task-clock:u (msec) # 0.801 CPUs utilized ( +- 2.79% )\n> 0 context-switches:u # 0.000 K/sec\n> 0 cpu-migrations:u # 0.000 K/sec\n> 131 page-faults:u # 0.054 M/sec ( +- 0.16% )\n> 993,301 cycles:u # 0.412 GHz ( +- 1.99% )\n> 1,087,428 instructions:u # 1.09 insn per cycle ( +- 0.02% )\n> 244,292 branches:u # 101.351 M/sec ( +- 0.02% )\n> 9,264 branch-misses:u # 3.79% of all branches ( +- 0.57% )\n>\n> 0.003010854 seconds time elapsed ( +- 2.54% )\n>\n> So I think we could just replace it for now and optimize again later, if it\n> turns out to be a problem. I think the easiest optimisation is to increase\n> the allocation size of having a lot more objects per mp_block.\n\nYeah. I also tested this from a different angle: memory overhead. For\n2M objects with one mp_block containing 1024 objects (same setting as\nalloc.c), the overhead (not counting malloc() internal overhead) is\n46KB and we don't have any extra overhead due to padding between\nobjects. This is true for all struct blob, commit, tree and tag. This\nis really good. alloc.c has zero overhead when measured this way but\n46KB is practically zero to me.\n-- \nDuy\n"},{"id":"346940","messageId":"xmqqsh73nfta.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"CACsJy8De2SUGj9hgq3h7pODm=9Wn+TkCKfJjrm1c=UxS9BZvTA@mail.gmail.com","subject":"Re: [PATCH] alloc.c: replace alloc by mempool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-08T00:37:05Z","receivedAt":"2018-05-08T00:37:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n>> So I think we could just replace it for now and optimize again later, if it\n>> turns out to be a problem. I think the easiest optimisation is to increase\n>> the allocation size of having a lot more objects per mp_block.\n>\n> Yeah. I also tested this from a different angle: memory overhead. For\n> 2M objects with one mp_block containing 1024 objects (same setting as\n> alloc.c), the overhead (not counting malloc() internal overhead) is\n> 46KB and we don't have any extra overhead due to padding between\n> objects. This is true for all struct blob, commit, tree and tag. This\n> is really good. alloc.c has zero overhead when measured this way but\n> 46KB is practically zero to me.\n\nThanks.\n\nThe above in short sounds like arguing \"replacing alloc.c internal\nwith mempool incurs negligible memory overhead and performance\ndegradation, but that can be optimized later\". It was unclear to me\nwhy such a replacement needs to happen in the first place, though.\nWithout such a code churn, there won't be extra overhead or need to\nfix it up later, no?\n"},{"id":"346941","messageId":"CAGZ79kbC1f856Pbf-Uucdr-FnNemjTTb07RiECBMDW2ZTA2xDA@mail.gmail.com","threadId":"48310","inReplyTo":"xmqqsh73nfta.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH] alloc.c: replace alloc by mempool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T00:44:36Z","receivedAt":"2018-05-08T00:44:40Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, May 7, 2018 at 5:37 PM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>>> So I think we could just replace it for now and optimize again later, if it\n>>> turns out to be a problem. I think the easiest optimisation is to increase\n>>> the allocation size of having a lot more objects per mp_block.\n>>\n>> Yeah. I also tested this from a different angle: memory overhead. For\n>> 2M objects with one mp_block containing 1024 objects (same setting as\n>> alloc.c), the overhead (not counting malloc() internal overhead) is\n>> 46KB and we don't have any extra overhead due to padding between\n>> objects. This is true for all struct blob, commit, tree and tag. This\n>> is really good. alloc.c has zero overhead when measured this way but\n>> 46KB is practically zero to me.\n>\n> Thanks.\n>\n> The above in short sounds like arguing \"replacing alloc.c internal\n> with mempool incurs negligible memory overhead and performance\n> degradation, but that can be optimized later\". It was unclear to me\n> why such a replacement needs to happen in the first place, though.\n\nThe replacement with mem-pool might be easier than making sure\nthat alloc.c has no globals and handles allocations per repository\ncorrectly. It would make the sb/object-store-alloc series shorter than\nit currently is, and maybe easier to review the code.\n\nHowever now that sb/object-store-alloc is rerolled with keeping\nthe logic of alloc.c and not replacing it with mem-pool, the only\nreason would be ease of maintainability by less code\nOn the other hand we have different implementations of\nlists, vectors and hashmaps, so having different memory\nallocators doesn't hurt.\n\n> Without such a code churn, there won't be extra overhead or need to\n> fix it up later, no?\n\nThe original motivation is to get the object store on a per-repository\nbasis. It might be cheaper to use an existing 'objectified' code instead\nof doing that again.\n"},{"id":"346944","messageId":"xmqqd0y7nedx.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"CAGZ79kbC1f856Pbf-Uucdr-FnNemjTTb07RiECBMDW2ZTA2xDA@mail.gmail.com","subject":"Re: [PATCH] alloc.c: replace alloc by mempool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-08T01:07:54Z","receivedAt":"2018-05-08T01:07:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> The replacement with mem-pool might be easier than making sure\n> that alloc.c has no globals and handles allocations per repository\n> correctly. It would make the sb/object-store-alloc series shorter than\n> it currently is, and maybe easier to review the code.\n>\n> However now that sb/object-store-alloc is rerolled with keeping\n> the logic of alloc.c and not replacing it with mem-pool, the only\n> reason would be ease of maintainability by less code\n\nThat is sensible considertation; the patch should be sold as such,\ninstead of with an empty log message as if the reasoning is obvious\nto everybody without being told ;-)\n"},{"id":"348369","messageId":"20180523144637.153551-1-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:29Z","receivedAt":"2018-05-23T14:47:34Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Changes from V2:\n\n\t- Tweak logic of finding available memory block for memory\n allocation\n\t\n\t - Only search head block\n\t \n\t- Tweaked handling of large memory allocations.\n\t\n\t - Large blocks now tracked in same manner as \"regular\"\n blocks\n\t \n\t - Large blocks are placed at end of linked list of memory\n blocks\n\n\t- Cache_entry type gains notion of whether it was allocated\n from memory pool or not\n\t\n\t - Collapsed cache_entry discard logic into single\n function. This should make code easier to maintain\n\n\t- Small tweaks based on V1 feedback\n\nBase Ref: master\nWeb-Diff: git@github.com:jamill/git.git/commit/d608515f9e\nCheckout: git fetch git@github.com:jamill/git.git users/jamill/block_allocation-v3 && git checkout d608515f9e\n\n\nJameson Miller (7):\n read-cache: teach refresh_cache_entry() to take istate\n block alloc: add lifecycle APIs for cache_entry structs\n mem-pool: only search head block for available space\n mem-pool: add lifecycle management functions\n mem-pool: fill out functionality\n block alloc: allocate cache entries from mem_pool\n block alloc: add validations around cache_entry lifecyle\n\n apply.c | 26 +++--\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 8 +-\n builtin/reset.c | 6 +-\n builtin/update-index.c | 26 +++--\n cache.h | 53 +++++++++-\n fast-import.c | 2 +-\n git.c | 3 +\n mem-pool.c | 116 ++++++++++++++++++++--\n mem-pool.h | 25 ++++-\n merge-recursive.c | 4 +-\n read-cache.c | 261 ++++++++++++++++++++++++++++++++++++++++---------\n resolve-undo.c | 6 +-\n split-index.c | 58 ++++++++---\n tree.c | 4 +-\n unpack-trees.c | 38 +++----\n 17 files changed, 514 insertions(+), 135 deletions(-)\n\n\nbase-commit: ccdcbd54c4475c2238b310f7113ab3075b5abc9c\n-- \n2.14.3\n\n\n"},{"id":"348370","messageId":"20180523144637.153551-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 1/7] read-cache: teach refresh_cache_entry() to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:35Z","receivedAt":"2018-05-23T14:47:42Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactor refresh_cache_entry() to work on a specific index, instead of\nimplicitly using the_index. This is in preparation for making the\nmake_cache_entry function work on a specific index.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 7 ++++---\n 3 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 0c1fb9fbcc..f0a407602c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -744,7 +744,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624d..693f60e0a3 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -260,7 +260,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 10f1c6bb8a..2cb4f53b57 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1448,10 +1448,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\t struct cache_entry *ce,\n \t\t\t\t\t unsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.14.3\n\n"},{"id":"348371","messageId":"20180523144637.153551-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 3/7] mem-pool: only search head block for available space","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:37Z","receivedAt":"2018-05-23T14:47:45Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Instead of searching all memory blocks for available space to fulfill\na memory request, only search the head block. If the head block does\nnot have space, assume that previous block would most likely not be\nable to fulfill request either. This could potentially lead to more\nmemory fragmentation, but also avoids searching memory blocks that\nprobably will not be able to fulfill request.\n\nThis pattern will benefit consumers that are able to generate a good\nestimate for how much memory will be needed, or if they are performing\nfixed sized allocations, so that once a block is exhausted it will\nnever be able to fulfill a future request.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..c80124f1fe 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -21,16 +21,16 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n-\tstruct mp_block *p;\n+\tstruct mp_block *p = NULL;\n \tvoid *r;\n \n \t/* round up to a 'uintmax_t' alignment */\n \tif (len & (sizeof(uintmax_t) - 1))\n \t\tlen += sizeof(uintmax_t) - (len & (sizeof(uintmax_t) - 1));\n \n-\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n-\t\tif (p->end - p->next_free >= len)\n-\t\t\tbreak;\n+\tif (mem_pool->mp_block &&\n+\t mem_pool->mp_block->end - mem_pool->mp_block->next_free >= len)\n+\t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n \t\tif (len >= (mem_pool->block_alloc / 2)) {\n-- \n2.14.3\n\n"},{"id":"348372","messageId":"20180523144637.153551-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 2/7] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:36Z","receivedAt":"2018-05-23T14:47:48Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an API around managing the lifetime of cache_entry\nstructs. Abstracting memory management details behind an API will\nallow for alternative memory management strategies without affecting\nall the call sites. This commit does not change how memory is\nallocated / freed. A later commit in this series will allocate cache\nentries from memory pools as appropriate.\n\nMotivation:\nIt has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This change\nis in preparation for using memory pools to reduce the number of\nmalloc() calls made when loading an index.\n\nThis API makes a distinction between cache entries that are intended\nfor use with a particular index and cache entries that are not. This\nenables us to use the knowledge about how a cache entry will be used\nto make informed decisions about how to handle the corresponding\nmemory.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 26 ++++++-------\n blame.c | 5 +--\n builtin/checkout.c | 8 ++--\n builtin/difftool.c | 8 ++--\n builtin/reset.c | 6 +--\n builtin/update-index.c | 26 ++++++-------\n cache.h | 24 +++++++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 100 ++++++++++++++++++++++++++++++++++---------------\n resolve-undo.c | 6 ++-\n split-index.c | 8 ++--\n tree.c | 4 +-\n unpack-trees.c | 33 +++++++++++-----\n 13 files changed, 162 insertions(+), 94 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 7e5792c996..b769fe0d15 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4090,12 +4090,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_index_cache_entry(&result, patch->old_mode, oid.hash, name, 0, 0);\n \t\tif (!ce)\n-\t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\treturn error(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4263,12 +4263,11 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n \tif (!state->update_index)\n \t\treturn 0;\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4278,13 +4277,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tdiscard_cache_entry(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4292,13 +4291,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4422,27 +4421,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_index_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex 78c9808bd1..8067e398a1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -154,7 +154,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -252,8 +252,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex b49b582071..1b7c90b418 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, oid.hash, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex aad0e073ee..0289b9c62b 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid->hash, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n@@ -488,8 +488,8 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n-\t\t\t\t\t\t\t dst_path, 0, 0);\n+\t\t\t\t\tmake_index_cache_entry(&wtindex, rmode, roid.hash,\n+\t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\n \t\t\t\t\t\tADD_CACHE_JUST_APPEND);\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 7f1c3f02a3..1062dab73f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,10 +134,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n-\t\t\t\t 0, 0);\n+\t\tce = make_index_cache_entry(&the_index, one->mode, one->oid.hash, one->path,\n+\t\t\t\t\t 0, 0);\n \t\tif (!ce)\n-\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\tdie(_(\"make_index_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n \t\tif (is_missing) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 10d070a76f..7ea8aeecc4 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -403,15 +402,14 @@ static int process_path(const char *path)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -589,7 +587,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -602,8 +599,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -680,8 +676,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tdiscard_cache_entry(ce_2);\n+\tdiscard_cache_entry(ce_3);\n \treturn ret;\n }\n \n@@ -748,7 +744,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -759,7 +755,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tdiscard_cache_entry(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex f0a407602c..204f788438 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,29 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and freeing */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `discard_cache_entry`.\n+ */\n+extern struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `discard_cache_entry`.\n+ */\n+extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n+extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry.\n+ */\n+void discard_cache_entry(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -691,7 +714,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 693f60e0a3..be118c0c6d 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -252,7 +252,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_index_cache_entry(&the_index, mode, oid ? oid->hash : null_sha1, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"addinfo_cache failed for path '%s'\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2cb4f53b57..d51cc83312 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tdiscard_cache_entry(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_index_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_index_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,12 +744,22 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode,\n+\t\t\t const unsigned char *sha1, const char *path,\n+\t\t\t int stage, unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -758,8 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, len);\n \n \thashcpy(ce->oid.hash, sha1);\n \tmemcpy(ce->name, path, len);\n@@ -769,10 +777,34 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n+\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1,\n+\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\thashcpy(ce->oid.hash, sha1);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1243,7 +1275,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1323,8 +1355,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1610,12 +1641,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1657,7 +1689,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1688,13 +1721,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1826,7 +1859,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1932,7 +1965,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tdiscard_cache_entry(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2622,14 +2655,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_index_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2738,3 +2770,11 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+/*\n+ * Free cache entry.\n+ */\n+void discard_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex aed95b4b35..96ef6307a6 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,8 +146,10 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n-\t\t\t\t name, i + 1, 0);\n+\t\tnce = make_index_cache_entry(istate,\n+\t\t\t\t\t ru->mode[i],\n+\t\t\t\t\t ru->oid[i].hash,\n+\t\t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\n \t\tif (add_index_entry(istate, nce, ADD_CACHE_OK_TO_ADD)) {\ndiff --git a/split-index.c b/split-index.c\nindex 3eb8ff1b43..ab638b844d 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tdiscard_cache_entry(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tdiscard_cache_entry(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tdiscard_cache_entry(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 1c68ea586b..1ba39c9374 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -16,15 +16,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_index_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dec37ad1e7..dd4d12ba45 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -192,10 +192,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_index_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -205,7 +205,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -786,10 +786,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\tconst struct name_entry *n,\n+\tint stage,\n+\tstruct index_state *istate,\n+\tint is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_index_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -835,7 +842,13 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\n+\t\t/*\n+\t\t * If this is a merge operation, then the cache\n+\t\t * entries are only temporary and discarded several\n+\t\t * lines below.\n+\t\t */\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -844,7 +857,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1765,7 +1778,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1785,7 +1798,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tdiscard_cache_entry(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1811,7 +1824,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tdiscard_cache_entry(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.14.3\n\n"},{"id":"348373","messageId":"20180523144637.153551-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 4/7] mem-pool: add lifecycle management functions","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:38Z","receivedAt":"2018-05-23T14:47:53Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add initialization and discard functions to mem-pool type. As part of\nthis, we now also track \"large\" allocations of memory so that these\ncan also be cleaned up when discarding the memory pool.\n\nThese changes are in preparation for a future commit that will utilize\ncreating and discarding memory pool.\n\nSigned-off-by: Jameson Miller \n---\n fast-import.c | 2 +-\n mem-pool.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++----\n mem-pool.h | 12 +++++++++++-\n 3 files changed, 71 insertions(+), 6 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 34edf3fb8f..571898e5db 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -300,7 +300,7 @@ static int global_argc;\n static const char **global_argv;\n \n /* Memory pools */\n-static struct mem_pool fi_mem_pool = {NULL, 2*1024*1024 -\n+static struct mem_pool fi_mem_pool = {NULL, NULL, 2*1024*1024 -\n \t\t\t\t sizeof(struct mp_block), 0 };\n \n /* Atom management */\ndiff --git a/mem-pool.c b/mem-pool.c\nindex c80124f1fe..01595bcca5 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,20 +5,77 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n {\n \tstruct mp_block *p;\n \n \tmem_pool->pool_alloc += sizeof(struct mp_block) + block_alloc;\n \tp = xmalloc(st_add(sizeof(struct mp_block), block_alloc));\n+\n \tp->next_block = mem_pool->mp_block;\n \tp->next_free = (char *)p->space;\n \tp->end = p->next_free + block_alloc;\n+\n \tmem_pool->mp_block = p;\n \n+\tif (!mem_pool->mp_block_tail)\n+\t\tmem_pool->mp_block_tail = p;\n+\n+\treturn p;\n+}\n+\n+static void *mem_pool_alloc_custom(struct mem_pool *mem_pool, size_t block_alloc)\n+{\n+\tstruct mp_block *p;\n+\n+\tmem_pool->pool_alloc += sizeof(struct mp_block) + block_alloc;\n+\tp = xmalloc(st_add(sizeof(struct mp_block), block_alloc));\n+\n+\tp->next_block = NULL;\n+\tp->next_free = (char *)p->space;\n+\tp->end = p->next_free + block_alloc;\n+\n+\tif (mem_pool->mp_block_tail)\n+\t\tmem_pool->mp_block_tail->next_block = p;\n+\telse\n+\t\tmem_pool->mp_block = p;\n+\n+\tmem_pool->mp_block_tail = p;\n \treturn p;\n }\n \n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tstruct mem_pool *pool;\n+\n+\tif (*mem_pool)\n+\t\treturn;\n+\n+\tpool = xcalloc(1, sizeof(*pool));\n+\n+\tpool->block_alloc = BLOCK_GROWTH_SIZE;\n+\n+\tif (initial_size > 0)\n+\t\tmem_pool_alloc_block(pool, initial_size);\n+\n+\t*mem_pool = pool;\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tstruct mp_block *block, *block_to_free;\n+\tfor (block = mem_pool->mp_block; block;)\n+\t{\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p = NULL;\n@@ -33,10 +90,8 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n+\t\tif (len >= (mem_pool->block_alloc / 2))\n+\t\t\treturn mem_pool_alloc_custom(mem_pool, len);\n \n \t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n \t}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..5d3e6a367a 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -9,7 +9,7 @@ struct mp_block {\n };\n \n struct mem_pool {\n-\tstruct mp_block *mp_block;\n+\tstruct mp_block *mp_block, *mp_block_tail;\n \n \t/*\n \t * The amount of available memory to grow the pool by.\n@@ -21,6 +21,16 @@ struct mem_pool {\n \tsize_t pool_alloc;\n };\n \n+/*\n+ * Initialize mem_pool with specified initial size.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n-- \n2.14.3\n\n"},{"id":"348374","messageId":"20180523144637.153551-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 5/7] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:40Z","receivedAt":"2018-05-23T14:47:56Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add functions for:\n\n - combining two memory pools\n\n - determining if a memory address is within the range managed by a\n memory pool\n\nThese functions will be used by future commits.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 40 ++++++++++++++++++++++++++++++++++++++++\n mem-pool.h | 13 +++++++++++++\n 2 files changed, 53 insertions(+)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 01595bcca5..cc7d3a7ab1 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -108,3 +108,43 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Check if memory is allocated in a block */\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\t/* Append the blocks from src to dst */\n+\tif (dst->mp_block && src->mp_block) {\n+\t\t/*\n+\t\t * src and dst have blocks, append\n+\t\t * blocks from src to dst.\n+\t\t */\n+\t\tdst->mp_block_tail->next_block = src->mp_block;\n+\t\tdst->mp_block_tail = src->mp_block_tail;\n+\t} else if (src->mp_block) {\n+\t\t/*\n+\t\t * src has blocks, dst is empty\n+\t\t * use pointers from src to set up dst.\n+\t\t */\n+\t\tdst->mp_block = src->mp_block;\n+\t\tdst->mp_block_tail = src->mp_block_tail;\n+\t} else {\n+\t\t// src is empty, nothing to do.\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+\tsrc->mp_block_tail = NULL;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 5d3e6a367a..5c892d3bdb 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -41,4 +41,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.14.3\n\n"},{"id":"348375","messageId":"20180523144637.153551-7-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 6/7] block alloc: allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:41Z","receivedAt":"2018-05-23T14:47:59Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"When reading large indexes from disk, a portion of the time is\ndominated in malloc() calls. This can be mitigated by allocating a\nlarge block of memory and manage it ourselves via memory pools.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\n\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entries will be allocated from. When reading in the\nindex from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool will be discarded as well - so the lifetime of\na cache_entry is tied to the lifetime of the index_state that it was\nallocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 21 ++++++++++\n read-cache.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++---------\n split-index.c | 50 ++++++++++++++++++++----\n unpack-trees.c | 13 +------\n 4 files changed, 165 insertions(+), 38 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 204f788438..7aae9c8db0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -156,6 +157,7 @@ struct cache_entry {\n \tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n \tunsigned int ce_flags;\n+\tunsigned int mem_pool_allocated;\n \tunsigned int ce_namelen;\n \tunsigned int index;\t/* for link extension */\n \tstruct object_id oid;\n@@ -227,6 +229,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\t\t const struct cache_entry *src)\n {\n \tunsigned int state = dst->ce_flags & CE_HASHED;\n+\tint mem_pool_allocated = dst->mem_pool_allocated;\n \n \t/* Don't copy hash chain and name */\n \tmemcpy(&dst->ce_stat_data, &src->ce_stat_data,\n@@ -235,6 +238,9 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \n \t/* Restore the hash state */\n \tdst->ce_flags = (dst->ce_flags & ~CE_HASHED) | state;\n+\n+\t/* Restore the mem_pool_allocated flag */\n+\tdst->mem_pool_allocated = mem_pool_allocated;\n }\n \n static inline unsigned create_ce_flags(unsigned stage)\n@@ -328,6 +334,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\n@@ -362,6 +369,20 @@ extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Duplicate a cache_entry. Allocate memory for the new entry from a\n+ * memory_pool. Takes into account cache_entry fields that are meant\n+ * for managing the underlying memory allocation of the cache_entry.\n+ */\n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce, struct index_state *istate);\n+\n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/read-cache.c b/read-cache.c\nindex d51cc83312..02fe5d333c 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,48 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is an estimate of the pathname length in the index. We use\n+ * this for V4 index files to guess the un-deltafied size of the index\n+ * in memory because of pathname deltafication. This is not required\n+ * for V2/V3 index formats because their pathnames are not compressed.\n+ * If the initial amount of memory set aside is not sufficient, the\n+ * mem pool will allocate extra memory.\n+ */\n+#define CACHE_ENTRY_PATH_LENGTH 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry *ce;\n+\tce = mem_pool_alloc(mem_pool, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry * ce;\n+\tce = mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +788,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1641,13 +1683,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_index_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1689,7 +1731,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1721,13 +1763,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1801,6 +1843,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_PATH_LENGTH);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1847,10 +1905,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1859,7 +1922,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1956,17 +2019,13 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n+\t/*\n+\t * Cache entries in istate->cache[] should have been allocated\n+\t * from the memory pool associated with this index, or from an\n+\t * associated split_index. There is no need to free individual\n+\t * cache entries.\n+\t */\n \n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tdiscard_cache_entry(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -1980,6 +2039,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2771,10 +2836,26 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tsrc->untracked = NULL;\n }\n \n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n+\t\t struct index_state *istate)\n+{\n+\tunsigned int size = ce_size(ce);\n+\tint mem_pool_allocated;\n+\tstruct cache_entry *new_entry = make_empty_index_cache_entry(istate, ce_namelen(ce));\n+\tmem_pool_allocated = new_entry->mem_pool_allocated;\n+\n+\tmemcpy(new_entry, ce, size);\n+\tnew_entry->mem_pool_allocated = mem_pool_allocated;\n+\treturn new_entry;\n+}\n+\n /*\n * Free cache entry.\n */\n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && ce->mem_pool_allocated)\n+\t\treturn;\n+\n \tfree(ce);\n }\ndiff --git a/split-index.c b/split-index.c\nindex ab638b844d..004d785d2f 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -331,12 +346,31 @@ void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n \t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n+\t\t * When removing the split index, we need to move\n+\t\t * ownership of the mem_pool associated with the\n+\t\t * base index to the main index. There may be cache entries\n+\t\t * allocated from the base's memory pool that are shared with\n+\t\t * the_index.cache[].\n \t\t */\n-\t\tistate->split_index = NULL;\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n+\t\t/*\n+\t\t * The split index no longer owns the mem_pool backing\n+\t\t * its cache array. As we are discarding this index,\n+\t\t * mark the index as having no cache entries, so it\n+\t\t * will not attempt to clean up the cache entries or\n+\t\t * validate them.\n+\t\t */\n+\t\tif (istate->split_index->base)\n+\t\t\tistate->split_index->base->cache_nr = 0;\n+\n+\t\t/*\n+\t\t * We can discard the split index because its\n+\t\t * memory pool has been incorporated into the\n+\t\t * memory pool associated with the the_index.\n+\t\t */\n+\t\tdiscard_split_index(istate);\n+\n \t\tistate->cache_changed |= SOMETHING_CHANGED;\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dd4d12ba45..dc53f811ae 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -192,20 +192,11 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n-{\n-\tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = make_empty_index_cache_entry(istate, ce_namelen(ce));\n-\n-\tmemcpy(new_entry, ce, size);\n-\treturn new_entry;\n-}\n-\n static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n+\tdo_add_entry(o, dup_cache_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -1778,7 +1769,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n+\tstruct cache_entry *merge = dup_cache_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n-- \n2.14.3\n\n"},{"id":"348376","messageId":"20180523144637.153551-8-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"[PATCH v3 7/7] block alloc: add validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-23T14:47:42Z","receivedAt":"2018-05-23T14:48:01Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 6 ++++++\n git.c | 3 +++\n mem-pool.c | 7 ++++++-\n mem-pool.h | 2 +-\n read-cache.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 5 files changed, 69 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7aae9c8db0..2916e953ad 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -369,6 +369,12 @@ extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Check configuration if we should perform extra validation on cache\n+ * entries.\n+ */\n+int should_validate_cache_entries(void);\n+\n /*\n * Duplicate a cache_entry. Allocate memory for the new entry from a\n * memory_pool. Takes into account cache_entry fields that are meant\ndiff --git a/git.c b/git.c\nindex bab6bbfded..7ce65eab78 100644\n--- a/git.c\n+++ b/git.c\n@@ -347,7 +347,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex cc7d3a7ab1..6770b4f740 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -63,13 +63,18 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n \t*mem_pool = pool;\n }\n \n-void mem_pool_discard(struct mem_pool *mem_pool)\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory)\n {\n \tstruct mp_block *block, *block_to_free;\n+\n \tfor (block = mem_pool->mp_block; block;)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \ndiff --git a/mem-pool.h b/mem-pool.h\nindex 5c892d3bdb..68d8428902 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -29,7 +29,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n /*\n * Discard a memory pool and free all the memory it is responsible for.\n */\n-void mem_pool_discard(struct mem_pool *mem_pool);\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory);\n \n /*\n * Alloc memory from the mem_pool.\ndiff --git a/read-cache.c b/read-cache.c\nindex 02fe5d333c..fb2cec6ac6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2023,8 +2023,10 @@ int discard_index(struct index_state *istate)\n \t * Cache entries in istate->cache[] should have been allocated\n \t * from the memory pool associated with this index, or from an\n \t * associated split_index. There is no need to free individual\n-\t * cache entries.\n+\t * cache entries. validate_cache_entries can detect when this\n+\t * assertion does not hold.\n \t */\n+\tvalidate_cache_entries(istate);\n \n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2041,13 +2043,45 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \n \tif (istate->ce_mem_pool) {\n-\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tmem_pool_discard(istate->ce_mem_pool, should_validate_cache_entries());\n \t\tistate->ce_mem_pool = NULL;\n \t}\n \n \treturn 0;\n }\n \n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!should_validate_cache_entries() ||!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2854,8 +2888,25 @@ struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n */\n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && should_validate_cache_entries())\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n+\n \tif (ce && ce->mem_pool_allocated)\n \t\treturn;\n \n \tfree(ce);\n }\n+\n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n-- \n2.14.3\n\n"},{"id":"348403","messageId":"xmqqefi1sliw.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180523144637.153551-3-jamill@microsoft.com","subject":"Re: [PATCH v3 2/7] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-24T04:52:23Z","receivedAt":"2018-05-24T04:52:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> Add an API around managing the lifetime of cache_entry\n> structs. Abstracting memory management details behind an API will\n> allow for alternative memory management strategies without affecting\n> all the call sites. This commit does not change how memory is\n> allocated / freed. A later commit in this series will allocate cache\n> entries from memory pools as appropriate.\n>\n> Motivation:\n> It has been observed that the time spent loading an index with a large\n> number of entries is partly dominated by malloc() calls. This change\n> is in preparation for using memory pools to reduce the number of\n> malloc() calls made when loading an index.\n>\n> This API makes a distinction between cache entries that are intended\n> for use with a particular index and cache entries that are not. This\n> enables us to use the knowledge about how a cache entry will be used\n> to make informed decisions about how to handle the corresponding\n> memory.\n\nYuck. make_index_cache_entry()?\n\nGenerally we use \"cache\" when working on the_index without passing\nistate, and otherwise \"index\", which means that readers can assume\nthat distim_cache_entry(...)\" is a shorter and more limited way to\nsay \"distim_index_entry(&the_index, ...)\". Having both index and\ncache in the same name smells crazy.\n\nIf most of the alocations are for permanent kind, give it a shorter\nname call it make_cache_entry(&the_index, ...), and call the other\nnon-permanent one with a longer and more cumbersome name, perhaps\nmake_transient_cache_entry(...). Avoid saying \"index\" in the former\nname, as the design decision this series is making to allocate\nmemory for a cache-entry from a pool associated to an index_state is\nalready seen by what its first parameter is.\n\n> diff --git a/cache.h b/cache.h\n> index f0a407602c..204f788438 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -339,6 +339,29 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n> extern void free_name_hash(struct index_state *istate);\n> \n> \n> +/* Cache entry creation and freeing */\n> +\n> +/*\n> + * Create cache_entry intended for use in the specified index. Caller\n> + * is responsible for discarding the cache_entry with\n> + * `discard_cache_entry`.\n> + */\n> +extern struct cache_entry *make_index_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n> +extern struct cache_entry *make_empty_index_cache_entry(struct index_state *istate, size_t name_len);\n> +\n> +/*\n> + * Create a cache_entry that is not intended to be added to an index.\n> + * Caller is responsible for discarding the cache_entry\n> + * with `discard_cache_entry`.\n> + */\n> +extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n> +extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n> +\n> +/*\n> + * Discard cache entry.\n> + */\n> +void discard_cache_entry(struct cache_entry *ce);\n\nI am not yet convinced that it is a good idea to require each istate\nto hold a separate pool. Anything that uses unpack_trees() can do\n\"starting from this src_index, perform various mergy operations and\ndeposit the result in dst_index\". Sometimes the two logical indices\npoint at the same istate, sometimes different. When src and dst are\ndifferent istates, the code that used to simply add another pointer\nto the same ce to the dst index now needs to duplicate it out of the\npool associated with dst?\n\nIn any case, perhaps it will become clearer why it is a good idea as\nwe read on, so let's do so.\n"},{"id":"348404","messageId":"xmqqa7spsle1.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"Re: [PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-24T04:55:18Z","receivedAt":"2018-05-24T04:55:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> Changes from V2:\n>\n> \t- Tweak logic of finding available memory block for memory\n> allocation\n> \t\n> \t - Only search head block\n\nHmph. Is that because we generally do not free() a lot so once a\nblock is filled, there is not much chance that we have reclaimed\nspace in the block later?\n\n> \t- Tweaked handling of large memory allocations.\n> \t\n> \t - Large blocks now tracked in same manner as \"regular\"\n> blocks\n> \t \n> \t - Large blocks are placed at end of linked list of memory\n> blocks\n\nIf we are only carving out of the most recently allocated block, it\nseems that there is no point looking for \"the end\", no?\n\n\n> \t- Cache_entry type gains notion of whether it was allocated\n> from memory pool or not\n> \t\n> \t - Collapsed cache_entry discard logic into single\n> function. This should make code easier to maintain\n\nThat certainly should be safer to have a back-pointer pointing to\nwhich pool each entry came from, but doesn't it result in memory\nbloat?\n"},{"id":"348430","messageId":"BL0PR2101MB11068051AD589871D4D707CBCE6A0@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"xmqqa7spsle1.fsf@gitster-ct.c.googlers.com","subject":"RE: [PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-24T14:44:14Z","receivedAt":"2018-05-24T14:44:19Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"\n\n> -----Original Message-----\n> From: Junio C Hamano On Behalf Of Junio C Hamano\n> Sent: Thursday, May 24, 2018 12:55 AM\n> To: Jameson Miller \n> Cc: git@vger.kernel.org; pclouds@gmail.com; jonathantanmy@google.com;\n> sbeller@google.com; peartben@gmail.com\n> Subject: Re: [PATCH v3 0/7] allocate cache entries from memory pool\n> \n> Jameson Miller writes:\n> \n> > Changes from V2:\n> >\n> > \t- Tweak logic of finding available memory block for memory\n> > allocation\n> >\n> > \t - Only search head block\n> \n> Hmph. Is that because we generally do not free() a lot so once a block is filled,\n> there is not much chance that we have reclaimed space in the block later?\n> \n\nThe design of the memory pool is that once the memory is\nclaimed from the pool, it is not reused until the\ncontaining pool is discarded. Individual entries are not\nfreed, only the entire memory pool is freed, and only after we\nare sure that there are no references to any of the entries in the\npool.\n\nThe memory pool design makes some tradeoffs. It is not meant to\nbe completely replace malloc / free as a general purpose\nallocator, but rather used in scenarios where the benefit (faster\nallocations, lower bookkeeping overhead) is worth the\ntradeoffs (not able to free individual allocations). The access\npatterns around cache entries are well matched with the memory\npool to get the benefits - the majority of cache entries are\nallocated up front when reading the index from disk, and are then\ndiscarded in bulk when the index is freed (if the index is freed\nat all (rather than just existing)).\n\n> > \t- Tweaked handling of large memory allocations.\n> >\n> > \t - Large blocks now tracked in same manner as \"regular\"\n> > blocks\n> >\n> > \t - Large blocks are placed at end of linked list of memory\n> > blocks\n> \n> If we are only carving out of the most recently allocated block, it seems that\n> there is no point looking for \"the end\", no?\n\nRight. If we are not searching the list, then there isn't any point in\nAppending odd large items to the end vs sticking it immediately past\nthe head block. I will remove the usage of the tail pointer in the\nnext version.\n\nYes, this is true. I can remove the usage of the tail pointer here,\nas it is not really leveraged. I will make this change in the next version.\n\n> \n> \n> > \t- Cache_entry type gains notion of whether it was allocated\n> > from memory pool or not\n> >\n> > \t - Collapsed cache_entry discard logic into single\n> > function. This should make code easier to maintain\n> \n> That certainly should be safer to have a back-pointer pointing to which pool\n> each entry came from, but doesn't it result in memory bloat?\n\nCurrently, entries claimed from a memory pool are not freed, so we only need\nto know whether the entry came from a memory pool or not. This has less memory \nimpact than a full pointer but is also a bit more restrictive.\n\nWe debated several approaches for what to do here and landed on using a simple bit\nfor this rather than the full pointer. In the current code we use a full integer field for this, but\nwe can convert this into a bit or bit field. The current flags word is full, so this would require\na second flags field.\n\n"},{"id":"348431","messageId":"BL0PR2101MB11060377AB5095E79EA9EA06CE6A0@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"xmqqefi1sliw.fsf@gitster-ct.c.googlers.com","subject":"RE: [PATCH v3 2/7] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-24T14:47:58Z","receivedAt":"2018-05-24T14:48:03Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"\n\n> -----Original Message-----\n> From: Junio C Hamano On Behalf Of Junio C Hamano\n> Sent: Thursday, May 24, 2018 12:52 AM\n> To: Jameson Miller \n> Cc: git@vger.kernel.org; pclouds@gmail.com; jonathantanmy@google.com;\n> sbeller@google.com; peartben@gmail.com\n> Subject: Re: [PATCH v3 2/7] block alloc: add lifecycle APIs for cache_entry\n> structs\n> \n> Jameson Miller writes:\n> \n> > Add an API around managing the lifetime of cache_entry structs.\n> > Abstracting memory management details behind an API will allow for\n> > alternative memory management strategies without affecting all the\n> > call sites. This commit does not change how memory is allocated /\n> > freed. A later commit in this series will allocate cache entries from\n> > memory pools as appropriate.\n> >\n> > Motivation:\n> > It has been observed that the time spent loading an index with a large\n> > number of entries is partly dominated by malloc() calls. This change\n> > is in preparation for using memory pools to reduce the number of\n> > malloc() calls made when loading an index.\n> >\n> > This API makes a distinction between cache entries that are intended\n> > for use with a particular index and cache entries that are not. This\n> > enables us to use the knowledge about how a cache entry will be used\n> > to make informed decisions about how to handle the corresponding\n> > memory.\n> \n> Yuck. make_index_cache_entry()?\n> \n> Generally we use \"cache\" when working on the_index without passing istate,\n> and otherwise \"index\", which means that readers can assume that\n> distim_cache_entry(...)\" is a shorter and more limited way to say\n> \"distim_index_entry(&the_index, ...)\". Having both index and cache in the same\n> name smells crazy.\n> \n> If most of the alocations are for permanent kind, give it a shorter name call it\n> make_cache_entry(&the_index, ...), and call the other non-permanent one with\n> a longer and more cumbersome name, perhaps\n> make_transient_cache_entry(...). Avoid saying \"index\" in the former name, as\n> the design decision this series is making to allocate memory for a cache-entry\n> from a pool associated to an index_state is already seen by what its first\n> parameter is.\n\nI like this suggestion - I will make this change in the next version of this series.\n\n> \n> > diff --git a/cache.h b/cache.h\n> > index f0a407602c..204f788438 100644\n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -339,6 +339,29 @@ extern void remove_name_hash(struct index_state\n> > *istate, struct cache_entry *ce) extern void free_name_hash(struct\n> > index_state *istate);\n> >\n> >\n> > +/* Cache entry creation and freeing */\n> > +\n> > +/*\n> > + * Create cache_entry intended for use in the specified index. Caller\n> > + * is responsible for discarding the cache_entry with\n> > + * `discard_cache_entry`.\n> > + */\n> > +extern struct cache_entry *make_index_cache_entry(struct index_state\n> > +*istate, unsigned int mode, const unsigned char *sha1, const char\n> > +*path, int stage, unsigned int refresh_options); extern struct\n> > +cache_entry *make_empty_index_cache_entry(struct index_state *istate,\n> > +size_t name_len);\n> > +\n> > +/*\n> > + * Create a cache_entry that is not intended to be added to an index.\n> > + * Caller is responsible for discarding the cache_entry\n> > + * with `discard_cache_entry`.\n> > + */\n> > +extern struct cache_entry *make_transient_cache_entry(unsigned int\n> > +mode, const unsigned char *sha1, const char *path, int stage); extern\n> > +struct cache_entry *make_empty_transient_cache_entry(size_t\n> > +name_len);\n> > +\n> > +/*\n> > + * Discard cache entry.\n> > + */\n> > +void discard_cache_entry(struct cache_entry *ce);\n> \n> I am not yet convinced that it is a good idea to require each istate to hold a\n> separate pool. Anything that uses unpack_trees() can do \"starting from this\n> src_index, perform various mergy operations and deposit the result in\n> dst_index\". Sometimes the two logical indices point at the same istate,\n> sometimes different. When src and dst are different istates, the code that used\n> to simply add another pointer to the same ce to the dst index now needs to\n> duplicate it out of the pool associated with dst?\n\nI did not see any instances in unpack_trees() where it copied\njust the cache_entry pointer from src to dst, but I will check\nagain.\n\nYou are correct, all the cache_entries need to be duplicated\nbefore being added to the destination index, which is what I\nthink the code already does. We tried to make this more\nexplicity by converting the inline xcalloc/memcpy instances to an\nactual function.\n\nIn the existing code (before this patch series), the index\nimplicitly \"owns\" its cache_entry instances. This can be seen in\nthe discard_index function, where the index will discard any\ncache entries that it has a reference to (that are not contained\nin the split index). If there is code that just copies the\npointer to an unrelated index, then this cache entry would be\nfreed when the source index is freed (unless the cache entry is\nremoved from the src index).\n\nWith memory pools, the same pattern is followed, but this\nrelationship is a bit more explicit.\n\n> \n> In any case, perhaps it will become clearer why it is a good idea as we read on,\n> so let's do so.\n\nThank you for your review so far. I look forward to your thoughts on\nthe overall change and if it is a change you are interested in.\n"},{"id":"348566","messageId":"CAGZ79kZ6B84Upx2HBzUTPcYnhiV=GkDU53Z6fC2rJGdsZb2-oA@mail.gmail.com","threadId":"48310","inReplyTo":"20180523144637.153551-1-jamill@microsoft.com","subject":"Re: [PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-25T22:41:46Z","receivedAt":"2018-05-25T22:41:51Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 23, 2018 at 7:47 AM, Jameson Miller wrote:\n> Changes from V2:\n>\n> - Tweak logic of finding available memory block for memory\n> allocation\n>\n> - Only search head block\n>\n> - Tweaked handling of large memory allocations.\n>\n> - Large blocks now tracked in same manner as \"regular\"\n> blocks\n>\n> - Large blocks are placed at end of linked list of memory\n> blocks\n>\n> - Cache_entry type gains notion of whether it was allocated\n> from memory pool or not\n>\n> - Collapsed cache_entry discard logic into single\n> function. This should make code easier to maintain\n>\n> - Small tweaks based on V1 feedback\n>\n> Base Ref: master\n> Web-Diff: git@github.com:jamill/git.git/commit/d608515f9e\n\nUnrelated to this series, but to the tool you use to generate the cover-letter:\nI think the Web-Diff only works when you give the http[s] address, git@\nwon't work here?\n"},{"id":"348567","messageId":"CAGZ79kbZcqzRAqip_CbEmxxhXr_UcAfO-siRgoVjqdd7XsH8sg@mail.gmail.com","threadId":"48310","inReplyTo":"BL0PR2101MB11068051AD589871D4D707CBCE6A0@BL0PR2101MB1106.namprd21.prod.outlook.com","subject":"Re: [PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-25T22:53:42Z","receivedAt":"2018-05-25T22:53:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">\n> The memory pool design makes some tradeoffs. It is not meant to\n> be completely replace malloc / free as a general purpose\n> allocator, but rather used in scenarios where the benefit (faster\n> allocations, lower bookkeeping overhead) is worth the\n> tradeoffs (not able to free individual allocations).\n\nSo this is the actual stated design goal of this memory pool?\nFast&cheap allocation with little overhead for giving up individual frees?\n\n> We debated several approaches for what to do here\n\nit would be awesome if the list could participate in the discussion\neven if only read-only.\n"},{"id":"348568","messageId":"CAGZ79kYxEVWeCqgVjFXAbnK8RkCfZuUvr1z+LJ1DOx=Ai1DTgg@mail.gmail.com","threadId":"48310","inReplyTo":"20180523144637.153551-2-jamill@microsoft.com","subject":"Re: [PATCH v3 1/7] read-cache: teach refresh_cache_entry() to take istate","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-25T22:54:56Z","receivedAt":"2018-05-25T22:55:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 23, 2018 at 7:47 AM, Jameson Miller wrote:\n> Refactor refresh_cache_entry() to work on a specific index, instead of\n> implicitly using the_index. This is in preparation for making the\n> make_cache_entry function work on a specific index.\n>\n> Signed-off-by: Jameson Miller \n\nReviewed-by: Stefan Beller \n"},{"id":"349011","messageId":"CAGZ79kZ-1jwnhzZrCeoZTZrNSwmnO=6asWQgWXRj7tjfYVr=-A@mail.gmail.com","threadId":"48310","inReplyTo":"20180523144637.153551-6-jamill@microsoft.com","subject":"Re: [PATCH v3 5/7] mem-pool: fill out functionality","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-01T19:28:04Z","receivedAt":"2018-06-01T19:28:09Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> @@ -108,3 +108,43 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n> memset(r, 0, len);\n> return r;\n> }\n> +\n> +int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n> +{\n> + struct mp_block *p;\n> +\n> + /* Check if memory is allocated in a block */\n> + for (p = mem_pool->mp_block; p; p = p->next_block)\n> + if ((mem >= ((void *)p->space)) &&\n> + (mem < ((void *)p->end)))\n> + return 1;\n> +\n> + return 0;\n> +}\n> +\n> +void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n> +{\n> + /* Append the blocks from src to dst */\n> + if (dst->mp_block && src->mp_block) {\n> + /*\n> + * src and dst have blocks, append\n> + * blocks from src to dst.\n> + */\n> + dst->mp_block_tail->next_block = src->mp_block;\n> + dst->mp_block_tail = src->mp_block_tail;\n> + } else if (src->mp_block) {\n> + /*\n> + * src has blocks, dst is empty\n> + * use pointers from src to set up dst.\n> + */\n> + dst->mp_block = src->mp_block;\n> + dst->mp_block_tail = src->mp_block_tail;\n> + } else {\n> + // src is empty, nothing to do.\n\ncomment style.\n"},{"id":"350588","messageId":"20180620201557.77155-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 1/8] read-cache: teach refresh_cache_entry() to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:24Z","receivedAt":"2018-06-20T20:17:30Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactor refresh_cache_entry() to work on a specific index, instead of\nimplicitly using the_index. This is in preparation for making the\nmake_cache_entry function apply to a specific index.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 9 +++++----\n 3 files changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 89a107a7f7..9538511d9f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -751,7 +751,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex f110e1c5ec..11a767cc72 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -323,7 +323,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"add_cacheinfo failed to refresh for path '%s'; merge aborting.\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 372588260e..fa8366ecab 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1473,10 +1473,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n-\t\t\t\t\t unsigned int options)\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\tstruct cache_entry *ce,\n+\t\t\t\t\tunsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.14.3\n\n"},{"id":"350589","messageId":"20180620201557.77155-1-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180417163400.3875-1-jamill@microsoft.com","subject":"[PATCH v4 0/8] Allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:23Z","receivedAt":"2018-06-20T20:17:33Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Changes from V3:\n\nMainly changes from last round of feedback:\n\n - Rename make_index_cache_entry -> make_cache_entry\n\n - Rename make_empty_index_cache_entry -> make_empty-cache_entry\n\n - Remove tail pointer in mem_pool\n\n - Small code tweaks\n\n - More accurately calculate mp_block size for platforms that do not\n support flexible arrays\n\n\nOne thing that came up with my testing is that the current automated\ntests do not fully cover the code path of \"large\" allocations from a\nmemory pool. I was able to force this condition by manually tweaking\nsome variables and then running the automated tests, but this is not\nideal for preventing regressions in the future.\n\nOne way I can think of testing this is to add a test-helper and\ndirectly test the memory pool struct. This will allow me to control\nthe parameters and different conditions. I was hoping for some\nguidance before I actually implemented these tests.\n\nEither way, I would like to do the additional tests in a separate\npatch series to have a more focused discussion. I am not sure if these\ntests would prevent inclusion of this patch series - I am open to\nguidance here.\n\nBase Ref: master\nWeb-Diff: https://github.com/jamill/git/compare/242ba98e44...667b8de06c\n\nJameson Miller (8):\n read-cache: teach refresh_cache_entry() to take istate\n block alloc: add lifecycle APIs for cache_entry structs\n mem-pool: only search head block for available space\n mem-pool: tweak math on mp_block allocation size\n mem-pool: add lifecycle management functions\n mem-pool: fill out functionality\n block alloc: allocate cache entries from mem_pool\n block alloc: add validations around cache_entry lifecyle\n\n apply.c | 24 +++--\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 6 +-\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 +++--\n cache.h | 53 +++++++++-\n git.c | 3 +\n mem-pool.c | 124 +++++++++++++++++++----\n mem-pool.h | 23 +++++\n merge-recursive.c | 4 +-\n read-cache.c | 259 ++++++++++++++++++++++++++++++++++++++++---------\n resolve-undo.c | 4 +-\n split-index.c | 58 ++++++++---\n tree.c | 4 +-\n unpack-trees.c | 40 ++++----\n 16 files changed, 504 insertions(+), 139 deletions(-)\n\n\nbase-commit: 242ba98e44d8314fb184d240939614a3c9b424db\n-- \n2.14.3\n\n\n"},{"id":"350590","messageId":"20180620201557.77155-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 3/8] mem-pool: only search head block for available space","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:27Z","receivedAt":"2018-06-20T20:17:33Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Instead of searching all memory blocks for available space to fulfill\na memory request, only search the head block. If the head block does\nnot have space, assume that previous block would most likely not be\nable to fulfill request either. This could potentially lead to more\nmemory fragmentation, but also avoids searching memory blocks that\nprobably will not be able to fulfill request.\n\nThis pattern will benefit consumers that are able to generate a good\nestimate for how much memory will be needed, or if they are performing\nfixed sized allocations, so that once a block is exhausted it will\nnever be able to fulfill a future request.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..c80124f1fe 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -21,16 +21,16 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n-\tstruct mp_block *p;\n+\tstruct mp_block *p = NULL;\n \tvoid *r;\n \n \t/* round up to a 'uintmax_t' alignment */\n \tif (len & (sizeof(uintmax_t) - 1))\n \t\tlen += sizeof(uintmax_t) - (len & (sizeof(uintmax_t) - 1));\n \n-\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n-\t\tif (p->end - p->next_free >= len)\n-\t\t\tbreak;\n+\tif (mem_pool->mp_block &&\n+\t mem_pool->mp_block->end - mem_pool->mp_block->next_free >= len)\n+\t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n \t\tif (len >= (mem_pool->block_alloc / 2)) {\n-- \n2.14.3\n\n"},{"id":"350591","messageId":"20180620201557.77155-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 5/8] mem-pool: add lifecycle management functions","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:30Z","receivedAt":"2018-06-20T20:17:37Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add initialization and discard functions to mem_pool type. As the\nmemory allocated by mem_pool can now be freed, we also track the large\nmemory alllocations so they can be freed as well.\n\nIf the there are existing mp_blocks in the mem_pool's linked list of\nmp_blocks, then the mp_block for a large allocation is inserted\nafter the head block. This is because only the head mp_block is considered\nwhen searching for availble space. This results in the following\ndesirable properties:\n\n1) The mp_block allocated for the large request will not be included\nin the search for available space in future requests, as the large\nmp_block is sized for the specific request and does not contain any\nspare space.\n\n2) The head mp_block will not bumped from considation for future\nmemory requests just because a request for a large chunk of memory\ncame in.\n\nThese changes are in preparation for a future commit that will utilize\ncreating and discarding memory pool.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++------------\n mem-pool.h | 10 ++++++++++\n 2 files changed, 61 insertions(+), 12 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex a5d5eed923..4e544459e9 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,7 +5,14 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n-static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n+/*\n+ * Allocate a new mp_block and insert it after the block specified in\n+ * `insert_after`. If `insert_after` is NULL, then insert block at the\n+ * head of the linked list.\n+ */\n+static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc, struct mp_block *insert_after)\n {\n \tstruct mp_block *p;\n \tsize_t total_alloc = st_add(offsetof(struct mp_block, space), block_alloc);\n@@ -13,14 +20,51 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \tmem_pool->pool_alloc = st_add(mem_pool->pool_alloc, total_alloc);\n \tp = xmalloc(total_alloc);\n \n-\tp->next_block = mem_pool->mp_block;\n \tp->next_free = (char *)p->space;\n \tp->end = p->next_free + block_alloc;\n-\tmem_pool->mp_block = p;\n+\n+\tif (insert_after) {\n+\t\tp->next_block = insert_after->next_block;\n+\t\tinsert_after->next_block = p;\n+\t} else {\n+\t\tp->next_block = mem_pool->mp_block;\n+\t\tmem_pool->mp_block = p;\n+\t}\n \n \treturn p;\n }\n \n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tstruct mem_pool *pool;\n+\n+\tif (*mem_pool)\n+\t\treturn;\n+\n+\tpool = xcalloc(1, sizeof(*pool));\n+\n+\tpool->block_alloc = BLOCK_GROWTH_SIZE;\n+\n+\tif (initial_size > 0)\n+\t\tmem_pool_alloc_block(pool, initial_size, NULL);\n+\n+\t*mem_pool = pool;\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tstruct mp_block *block, *block_to_free;\n+\n+\twhile ((block = mem_pool->mp_block))\n+\t{\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p = NULL;\n@@ -33,15 +77,10 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \tif (mem_pool->mp_block &&\n \t mem_pool->mp_block->end - mem_pool->mp_block->next_free >= len)\n \t\tp = mem_pool->mp_block;\n-\n-\tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n-\n-\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n-\t}\n+\telse if (len >= (mem_pool->block_alloc / 2))\n+\t\tp = mem_pool_alloc_block(mem_pool, len, mem_pool->mp_block);\n+\telse\n+\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc, NULL);\n \n \tr = p->next_free;\n \tp->next_free += len;\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..f75b3365d5 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -21,6 +21,16 @@ struct mem_pool {\n \tsize_t pool_alloc;\n };\n \n+/*\n+ * Initialize mem_pool with specified initial size.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n-- \n2.14.3\n\n"},{"id":"350592","messageId":"20180620201557.77155-9-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 8/8] block alloc: add validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:34Z","receivedAt":"2018-06-20T20:17:42Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 6 ++++++\n git.c | 3 +++\n mem-pool.c | 6 +++++-\n mem-pool.h | 2 +-\n read-cache.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 5 files changed, 68 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 74d3ebebc2..c0d6b976f5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -369,6 +369,12 @@ extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Check configuration if we should perform extra validation on cache\n+ * entries.\n+ */\n+int should_validate_cache_entries(void);\n+\n /*\n * Duplicate a cache_entry. Allocate memory for the new entry from a\n * memory_pool. Takes into account cache_entry fields that are meant\ndiff --git a/git.c b/git.c\nindex c2f48d53dd..010898ba6d 100644\n--- a/git.c\n+++ b/git.c\n@@ -414,7 +414,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex 0f19cc01a9..92d106a637 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -51,7 +51,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n \t*mem_pool = pool;\n }\n \n-void mem_pool_discard(struct mem_pool *mem_pool)\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory)\n {\n \tstruct mp_block *block, *block_to_free;\n \n@@ -60,6 +60,10 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \ndiff --git a/mem-pool.h b/mem-pool.h\nindex adeefdcb28..999d3c3a52 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -29,7 +29,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n /*\n * Discard a memory pool and free all the memory it is responsible for.\n */\n-void mem_pool_discard(struct mem_pool *mem_pool);\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory);\n \n /*\n * Alloc memory from the mem_pool.\ndiff --git a/read-cache.c b/read-cache.c\nindex a8932ce2a6..2652f2aeb0 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2047,8 +2047,10 @@ int discard_index(struct index_state *istate)\n \t * Cache entries in istate->cache[] should have been allocated\n \t * from the memory pool associated with this index, or from an\n \t * associated split_index. There is no need to free individual\n-\t * cache entries.\n+\t * cache entries. validate_cache_entries can detect when this\n+\t * assertion does not hold.\n \t */\n+\tvalidate_cache_entries(istate);\n \n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2065,13 +2067,45 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \n \tif (istate->ce_mem_pool) {\n-\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tmem_pool_discard(istate->ce_mem_pool, should_validate_cache_entries());\n \t\tistate->ce_mem_pool = NULL;\n \t}\n \n \treturn 0;\n }\n \n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!should_validate_cache_entries() ||!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2875,8 +2909,25 @@ struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n \n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && should_validate_cache_entries())\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n+\n \tif (ce && ce->mem_pool_allocated)\n \t\treturn;\n \n \tfree(ce);\n }\n+\n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n-- \n2.14.3\n\n"},{"id":"350593","messageId":"20180620201557.77155-8-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 7/8] block alloc: allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:32Z","receivedAt":"2018-06-20T20:17:44Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"When reading large indexes from disk, a portion of the time is\ndominated in malloc() calls. This can be mitigated by allocating a\nlarge block of memory and manage it ourselves via memory pools.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\n\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entries will be allocated from. When reading in the\nindex from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool will be discarded as well - so the lifetime of\na cache_entry is tied to the lifetime of the index_state that it was\nallocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 21 ++++++++++\n mem-pool.c | 3 +-\n read-cache.c | 119 ++++++++++++++++++++++++++++++++++++++++++++++++---------\n split-index.c | 50 ++++++++++++++++++++----\n unpack-trees.c | 13 +------\n 5 files changed, 167 insertions(+), 39 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex abcc27ff87..74d3ebebc2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -156,6 +157,7 @@ struct cache_entry {\n \tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n \tunsigned int ce_flags;\n+\tunsigned int mem_pool_allocated;\n \tunsigned int ce_namelen;\n \tunsigned int index;\t/* for link extension */\n \tstruct object_id oid;\n@@ -227,6 +229,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\t\t const struct cache_entry *src)\n {\n \tunsigned int state = dst->ce_flags & CE_HASHED;\n+\tint mem_pool_allocated = dst->mem_pool_allocated;\n \n \t/* Don't copy hash chain and name */\n \tmemcpy(&dst->ce_stat_data, &src->ce_stat_data,\n@@ -235,6 +238,9 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \n \t/* Restore the hash state */\n \tdst->ce_flags = (dst->ce_flags & ~CE_HASHED) | state;\n+\n+\t/* Restore the mem_pool_allocated flag */\n+\tdst->mem_pool_allocated = mem_pool_allocated;\n }\n \n static inline unsigned create_ce_flags(unsigned stage)\n@@ -328,6 +334,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\n@@ -362,6 +369,20 @@ extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Duplicate a cache_entry. Allocate memory for the new entry from a\n+ * memory_pool. Takes into account cache_entry fields that are meant\n+ * for managing the underlying memory allocation of the cache_entry.\n+ */\n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce, struct index_state *istate);\n+\n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 81fda969d3..0f19cc01a9 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -55,7 +55,8 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n {\n \tstruct mp_block *block, *block_to_free;\n \n-\twhile ((block = mem_pool->mp_block))\n+\tblock = mem_pool->mp_block;\n+\twhile (block)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\ndiff --git a/read-cache.c b/read-cache.c\nindex 6396e04e45..a8932ce2a6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,48 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is an estimate of the pathname length in the index. We use\n+ * this for V4 index files to guess the un-deltafied size of the index\n+ * in memory because of pathname deltafication. This is not required\n+ * for V2/V3 index formats because their pathnames are not compressed.\n+ * If the initial amount of memory set aside is not sufficient, the\n+ * mem pool will allocate extra memory.\n+ */\n+#define CACHE_ENTRY_PATH_LENGTH 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry *ce;\n+\tce = mem_pool_alloc(mem_pool, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry * ce;\n+\tce = mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +788,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1665,13 +1707,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1713,7 +1755,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1745,13 +1787,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1825,6 +1867,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_PATH_LENGTH);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1871,10 +1929,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1883,7 +1946,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1980,17 +2043,13 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n+\t/*\n+\t * Cache entries in istate->cache[] should have been allocated\n+\t * from the memory pool associated with this index, or from an\n+\t * associated split_index. There is no need to free individual\n+\t * cache entries.\n+\t */\n \n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tdiscard_cache_entry(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -2004,6 +2063,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2795,7 +2860,23 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tsrc->untracked = NULL;\n }\n \n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n+\t\t\t\t struct index_state *istate)\n+{\n+\tunsigned int size = ce_size(ce);\n+\tint mem_pool_allocated;\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n+\tmem_pool_allocated = new_entry->mem_pool_allocated;\n+\n+\tmemcpy(new_entry, ce, size);\n+\tnew_entry->mem_pool_allocated = mem_pool_allocated;\n+\treturn new_entry;\n+}\n+\n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && ce->mem_pool_allocated)\n+\t\treturn;\n+\n \tfree(ce);\n }\ndiff --git a/split-index.c b/split-index.c\nindex 317900db8b..84f067e10d 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -331,12 +346,31 @@ void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n \t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n+\t\t * When removing the split index, we need to move\n+\t\t * ownership of the mem_pool associated with the\n+\t\t * base index to the main index. There may be cache entries\n+\t\t * allocated from the base's memory pool that are shared with\n+\t\t * the_index.cache[].\n \t\t */\n-\t\tistate->split_index = NULL;\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n+\t\t/*\n+\t\t * The split index no longer owns the mem_pool backing\n+\t\t * its cache array. As we are discarding this index,\n+\t\t * mark the index as having no cache entries, so it\n+\t\t * will not attempt to clean up the cache entries or\n+\t\t * validate them.\n+\t\t */\n+\t\tif (istate->split_index->base)\n+\t\t\tistate->split_index->base->cache_nr = 0;\n+\n+\t\t/*\n+\t\t * We can discard the split index because its\n+\t\t * memory pool has been incorporated into the\n+\t\t * memory pool associated with the the_index.\n+\t\t */\n+\t\tdiscard_split_index(istate);\n+\n \t\tistate->cache_changed |= SOMETHING_CHANGED;\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 15f10f0055..907126dff4 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,20 +203,11 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n-{\n-\tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n-\n-\tmemcpy(new_entry, ce, size);\n-\treturn new_entry;\n-}\n-\n static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n+\tdo_add_entry(o, dup_cache_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -1802,7 +1793,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n+\tstruct cache_entry *merge = dup_cache_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n-- \n2.14.3\n\n"},{"id":"350594","messageId":"20180620201557.77155-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 4/8] mem-pool: tweak math on mp_block allocation size","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:28Z","receivedAt":"2018-06-20T20:17:49Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"The amount of memory to allocate for mp_blocks was off by a\nsizeof(uintmax_t) on platforms that do not support flexible arrays. To\naccount for this, use the offset of the space field when calculating\nthe total amount of memory to allocate for a mp_block.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 6 ++++--\n 1 file changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex c80124f1fe..a5d5eed923 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -8,9 +8,11 @@\n static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n {\n \tstruct mp_block *p;\n+\tsize_t total_alloc = st_add(offsetof(struct mp_block, space), block_alloc);\n+\n+\tmem_pool->pool_alloc = st_add(mem_pool->pool_alloc, total_alloc);\n+\tp = xmalloc(total_alloc);\n \n-\tmem_pool->pool_alloc += sizeof(struct mp_block) + block_alloc;\n-\tp = xmalloc(st_add(sizeof(struct mp_block), block_alloc));\n \tp->next_block = mem_pool->mp_block;\n \tp->next_free = (char *)p->space;\n \tp->end = p->next_free + block_alloc;\n-- \n2.14.3\n\n"},{"id":"350595","messageId":"20180620201557.77155-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 2/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:26Z","receivedAt":"2018-06-20T20:17:51Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an API around managing the lifetime of cache_entry\nstructs. Abstracting memory management details behind this API will\nallow for alternative memory management strategies without affecting\nall the call sites. This commit does not change how memory is\nallocated or freed. A later commit in this series will allocate cache\nentries from memory pools as appropriate.\n\nMotivation:\nIt has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This change\nis in preparation for using memory pools to reduce the number of\nmalloc() calls made when loading an index.\n\nThis API makes a distinction between cache entries that are intended\nfor use with a particular index and cache entries that are not. This\nenables us to use the knowledge about how a cache entry will be used\nto make informed decisions about how to handle the corresponding\nmemory.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 24 ++++++-------\n blame.c | 5 ++-\n builtin/checkout.c | 8 ++---\n builtin/difftool.c | 6 ++--\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 ++++++--------\n cache.h | 24 ++++++++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 96 ++++++++++++++++++++++++++++++++++----------------\n resolve-undo.c | 4 ++-\n split-index.c | 8 ++---\n tree.c | 4 +--\n unpack-trees.c | 35 ++++++++++++------\n 13 files changed, 155 insertions(+), 89 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex d79e61591b..1b5d923f4e 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4090,12 +4090,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_cache_entry(&result, patch->old_mode, oid.hash, name, 0, 0);\n \t\tif (!ce)\n \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4263,12 +4263,11 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n \tif (!state->update_index)\n \t\treturn 0;\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4278,13 +4277,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tdiscard_cache_entry(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4292,13 +4291,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4422,27 +4421,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex 14d0e0b575..4c6668d0e1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -154,7 +154,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -252,8 +252,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 2e1d2376d2..e44755c371 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, oid.hash, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex bc97d4aef2..a35d2d4e08 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid->hash, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n@@ -488,7 +488,7 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n+\t\t\t\t\tmake_cache_entry(&wtindex, rmode, roid.hash,\n \t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex a862c70fab..0ea0a19d5e 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,7 +134,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n+\t\tce = make_cache_entry(&the_index, one->mode, one->oid.hash, one->path,\n \t\t\t\t 0, 0);\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex a8709a26ec..ea2f2a476c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -402,15 +401,14 @@ static int process_path(const char *path, struct stat *st, int stat_errno)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path, mode))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -599,7 +597,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -612,8 +609,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -690,8 +686,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tdiscard_cache_entry(ce_2);\n+\tdiscard_cache_entry(ce_3);\n \treturn ret;\n }\n \n@@ -758,7 +754,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -769,7 +765,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tdiscard_cache_entry(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex 9538511d9f..abcc27ff87 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,29 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and cleanup */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `discard_cache_entry`.\n+ */\n+extern struct cache_entry *make_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `discard_cache_entry`.\n+ */\n+extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n+extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry.\n+ */\n+void discard_cache_entry(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -698,7 +721,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 11a767cc72..330958eb68 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -315,7 +315,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_cache_entry(&the_index, mode, oid ? oid->hash : null_sha1, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"add_cacheinfo failed for path '%s'; merge aborting.\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex fa8366ecab..6396e04e45 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tdiscard_cache_entry(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,12 +744,22 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_cache_entry(struct index_state *istate, unsigned int mode,\n+\t\t\t\t const unsigned char *sha1, const char *path,\n+\t\t\t\t int stage, unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path, mode)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -758,8 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, len);\n \n \thashcpy(ce->oid.hash, sha1);\n \tmemcpy(ce->name, path, len);\n@@ -769,10 +777,33 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1,\n+\t\t\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path, mode)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\thashcpy(ce->oid.hash, sha1);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1268,7 +1299,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1348,8 +1379,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1635,12 +1665,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1682,7 +1713,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1713,13 +1745,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1851,7 +1883,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1957,7 +1989,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tdiscard_cache_entry(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2647,14 +2679,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2763,3 +2794,8 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+void discard_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex fc5b3b83d9..966c5b5f84 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,7 +146,9 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n+\t\tnce = make_cache_entry(istate,\n+\t\t\t\t ru->mode[i],\n+\t\t\t\t ru->oid[i].hash,\n \t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\ndiff --git a/split-index.c b/split-index.c\nindex 660c75f31f..317900db8b 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tdiscard_cache_entry(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tdiscard_cache_entry(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tdiscard_cache_entry(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 244eb5e665..5111ce8376 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -16,15 +16,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3a85a02a77..15f10f0055 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,10 +203,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -216,7 +216,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -797,10 +797,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\t\t\t\t\t const struct name_entry *n,\n+\t\t\t\t\t int stage,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t int is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -846,7 +853,15 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\n+\t\t/*\n+\t\t * If the merge bit is set, then the cache entries are\n+\t\t * discarded in the following block. In this case,\n+\t\t * construct \"transient\" cache_entries, as they are\n+\t\t * not stored in the index. otherwise construct the\n+\t\t * cache entry from the index aware logic.\n+\t\t */\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -855,7 +870,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1787,7 +1802,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1807,7 +1822,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tdiscard_cache_entry(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1833,7 +1848,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tdiscard_cache_entry(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.14.3\n\n"},{"id":"350596","messageId":"20180620201557.77155-7-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v4 6/8] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:17:31Z","receivedAt":"2018-06-20T20:17:55Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add functions for:\n\n - combining two memory pools\n\n - determining if a memory address is within the range managed by a\n memory pool\n\nThese functions will be used by future commits.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n mem-pool.h | 13 +++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 4e544459e9..81fda969d3 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -94,3 +94,45 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Check if memory is allocated in a block */\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Append the blocks from src to dst */\n+\tif (dst->mp_block && src->mp_block) {\n+\t\t/*\n+\t\t * src and dst have blocks, append\n+\t\t * blocks from src to dst.\n+\t\t */\n+\t\tp = dst->mp_block;\n+\t\twhile (p->next_block)\n+\t\t\tp = p->next_block;\n+\n+\t\tp->next_block = src->mp_block;\n+\t} else if (src->mp_block) {\n+\t\t/*\n+\t\t * src has blocks, dst is empty.\n+\t\t */\n+\t\tdst->mp_block = src->mp_block;\n+\t} else {\n+\t\t/* src is empty, nothing to do. */\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex f75b3365d5..adeefdcb28 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -41,4 +41,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.14.3\n\n"},{"id":"350597","messageId":"DM5PR21MB07803E8D2627676788659E63CE770@DM5PR21MB0780.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79kbZcqzRAqip_CbEmxxhXr_UcAfO-siRgoVjqdd7XsH8sg@mail.gmail.com","subject":"RE: [PATCH v3 0/7] allocate cache entries from memory pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-20T20:41:05Z","receivedAt":"2018-06-20T20:41:10Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"> \n> > We debated several approaches for what to do here\n> \n> it would be awesome if the list could participate in the discussion even if only\n> read-only.\n\nA bit of delay in my response here, but I like the suggestion. here is a summary of\nsome approaches I considered:\n\n1) Do not include any information about where the cache_entry was allocated.\n Pros: No extra memory overhead\n Cons: Caller is responsible for freeing the cache entry correctly\n\nThis was our initial approach. We were hoping that all cache\nentries could be allocated from a memory pool, and we would not\nhave to worry about non-pool allocated entries. There are still a\ncouple of places that need \"temporary\" cache entries at the\nmoment, so we couldn't move completely to only memory pool\nallocated cache_entries. This would have resulted in the code\nallocating many \"temporary\" cache_entries from a pool, and the\nmemory would not be eligible to be reclaimed until the entire\nmemory pool was freed - and this was a tradeoff we didn't want to\nmake.\n\n2) Include an extra field encoding whether the cache_entry was\n allocated from a memory pool\n\n Pro: the discard function can now make a decision regarding how\n to free the cache_entry\n Con: each cache entry grows by an extra int field.\n\nThe bits in the existing ce_flags field are all claimed, so we\nneed an extra field to track this bit of information. We could\nclaim just a bit in the field now, which would result in the\ncache_entry still growing by the same amount, but any future bits\nwould not require extra space. This pushes off the work for an\nactual bit field off into future work.\n\n3) Encode whether the cache_entry was allocated from a memory\n pool as a bit in a field.\n\n Pro: only a single bit is required to encode this information.\n Con: All the existings bits in the existing ce_flags field are\n claimed. We need to add an extra bit field and take the same\n memory growth.\n\nI considered this approach (and am still open to it), but I went\nfor a simpler initial approach to make sure the overall change is\nacceptable. There is no difference in the memory footprint with\nthis change, so it is only to enable future changes more easily.\n\n4) Include pointer to the memory pool from which this cache_entry\n was created from\n\n Pro: Could (potentially) do some extra bookkeeping, such as\n automatically cleaning up the memory_pool when all\n allocated cache_entries are freed.\n Con: extra complexity, larger growth to cache_entry struct to\n accommodate mem_pool pointer\n\nIn the end, we didn't see a tangible benefit to this option at this point.\n\nGiven the tradeoffs, I went with option #2 for now.\n"},{"id":"350679","messageId":"CAGZ79katH2Kk2NDDyZjYzC9fEKS6cDdHK1Cw2JLjjSfmYOPapw@mail.gmail.com","threadId":"48310","inReplyTo":"20180620201557.77155-3-jamill@microsoft.com","subject":"Re: [PATCH v4 2/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-21T21:14:54Z","receivedAt":"2018-06-21T21:15:10Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> --- a/cache.h\n> +++ b/cache.h\n> @@ -339,6 +339,29 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n> extern void free_name_hash(struct index_state *istate);\n>\n>\n> +/* Cache entry creation and cleanup */\n> +\n> +/*\n> + * Create cache_entry intended for use in the specified index. Caller\n> + * is responsible for discarding the cache_entry with\n> + * `discard_cache_entry`.\n> + */\n> +extern struct cache_entry *make_cache_entry(struct index_state *istate, unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n\nHow much work is it to convert these to object_id while at it instead\nof char *sha1?\n(I think we really dislike introducing new sha1 based code; If it\ndoesn't sound easy\nmaybe we can have a static inline wrapper here that just converts oid\nto sha1 and\nthen calls this function?)\n\nIs it possible to line break these functions (c.f. refs.h which I\nthink has one of\nthe best styles in the git project. It has long parameter lists, but\nstill manages\nto stay below a reasonable line length) ?\n\n> +extern struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t name_len);\n\n> +\n> +/*\n> + * Create a cache_entry that is not intended to be added to an index.\n> + * Caller is responsible for discarding the cache_entry\n> + * with `discard_cache_entry`.\n> + */\n> +extern struct cache_entry *make_transient_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage);\n> +extern struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n> +\n> +/*\n> + * Discard cache entry.\n> + */\n> +void discard_cache_entry(struct cache_entry *ce);\n\nPlease be consistent in the use of the extern keyword and omit them at all\ntimes here and above.\n"},{"id":"350689","messageId":"CAGZ79kbL5wjJvKj-=gtGgtMFCPs+zTwSAV-Aeo_2GNC_h=1nNA@mail.gmail.com","threadId":"48310","inReplyTo":"20180620201557.77155-4-jamill@microsoft.com","subject":"Re: [PATCH v4 3/8] mem-pool: only search head block for available space","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-06-21T21:33:53Z","receivedAt":"2018-06-21T21:34:08Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jun 20, 2018 at 1:17 PM Jameson Miller wrote:\n>\n> Instead of searching all memory blocks for available space to fulfill\n> a memory request, only search the head block. If the head block does\n> not have space, assume that previous block would most likely not be\n> able to fulfill request either. This could potentially lead to more\n> memory fragmentation, but also avoids searching memory blocks that\n> probably will not be able to fulfill request.\n\nDo we have any numbers on performance or memory pressure here?\n(I would think benchmarking fast-import would suffice as that is where\nthe mem pool originated)\n\n> This pattern will benefit consumers that are able to generate a good\n> estimate for how much memory will be needed, or if they are performing\n> fixed sized allocations, so that once a block is exhausted it will\n> never be able to fulfill a future request.\n\nWould this be a good candidate to contain parts of\nhttps://public-inbox.org/git/DM5PR21MB07803E8D2627676788659E63CE770@DM5PR21MB0780.namprd21.prod.outlook.com/\n?\n"},{"id":"351209","messageId":"20180628135932.225288-1-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180620201557.77155-1-jamill@microsoft.com","subject":"[PATCH v5 0/8] Allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:02Z","receivedAt":"2018-06-28T14:00:07Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Changes from v4 are minor code review feedback items:\n\n - Remove extern keyword from new function definitions in cache.h\n - Make_cache_entry(..) functions work with object_id instead of sha\n - Add details to commit message for \"block alloc: allocate cache entries from mem_pool\"\n - Add details to commit message for \"mem-pool: only search head block for available space\"\n\nThe largest change is the new commit \"read-cache: make_cache_entry should take object_id struct\"\n\nBase Ref: master\nWeb-Diff: https://github.com/jamill/git/compare/ed843436dd...55c9b9008f\n\nJameson Miller (8):\n read-cache: teach refresh_cache_entry() to take istate\n read-cache: make_cache_entry should take object_id struct\n block alloc: add lifecycle APIs for cache_entry structs\n mem-pool: only search head block for available space\n mem-pool: add life cycle management functions\n mem-pool: fill out functionality\n block alloc: allocate cache entries from mem-pool\n block alloc: add validations around cache_entry lifecyle\n\n apply.c | 24 ++--\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 6 +-\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 ++--\n cache.h | 64 +++++++++-\n git.c | 3 +\n mem-pool.c | 114 ++++++++++++++++--\n mem-pool.h | 23 ++++\n merge-recursive.c | 4 +-\n read-cache.c | 264 ++++++++++++++++++++++++++++++++++-------\n resolve-undo.c | 4 +-\n split-index.c | 58 +++++++--\n tree.c | 4 +-\n unpack-trees.c | 40 ++++---\n 16 files changed, 515 insertions(+), 134 deletions(-)\n\n\nbase-commit: ed843436dd4924c10669820cc73daf50f0b4dabd\n-- \n2.17.1\n\n\n"},{"id":"351210","messageId":"20180628135932.225288-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 2/8] read-cache: make_cache_entry should take object_id struct","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:07Z","receivedAt":"2018-06-28T14:00:13Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"The make_cache_entry function should take an object_id struct instead\nof sha.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 2 +-\n builtin/checkout.c | 2 +-\n builtin/difftool.c | 4 ++--\n builtin/reset.c | 2 +-\n cache.h | 7 ++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 8 +++++---\n resolve-undo.c | 2 +-\n 8 files changed, 18 insertions(+), 11 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 959c457910..8ef975a32d 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4092,7 +4092,7 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_cache_entry(patch->old_mode, &oid, name, 0, 0);\n \t\tif (!ce)\n \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 2e1d2376d2..548bf40f25 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -230,7 +230,7 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_cache_entry(mode, &oid, path, 2, 0);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex bc97d4aef2..873a06f0d9 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,7 +321,7 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_cache_entry(mode, oid, path, 0, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n \tfree(ce);\n@@ -488,7 +488,7 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n+\t\t\t\t\tmake_cache_entry(rmode, &roid,\n \t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex a862c70fab..00109b041f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,7 +134,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n+\t\tce = make_cache_entry(one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\ndiff --git a/cache.h b/cache.h\nindex 93af25f586..3fbf24771a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -698,7 +698,12 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int stage,\n+\t\t\t\t\t unsigned int refresh_options);\n+\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 8b3d6781c7..873321e5c2 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -318,7 +318,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_cache_entry(mode, oid ? oid : &null_oid, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"add_cacheinfo failed for path '%s'; merge aborting.\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex fa8366ecab..9624ce1784 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -746,8 +746,10 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n }\n \n struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t int stage,\n+\t\t\t\t unsigned int refresh_options)\n {\n \tint size, len;\n \tstruct cache_entry *ce, *ret;\n@@ -761,7 +763,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tsize = cache_entry_size(len);\n \tce = xcalloc(1, size);\n \n-\thashcpy(ce->oid.hash, sha1);\n+\thashcpy(ce->oid.hash, oid->hash);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(stage);\n \tce->ce_namelen = len;\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex fc5b3b83d9..4d4e5cb6bf 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,7 +146,7 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n+\t\tnce = make_cache_entry(ru->mode[i], &ru->oid[i],\n \t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\n-- \n2.17.1\n\n"},{"id":"351211","messageId":"20180628135932.225288-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 3/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:09Z","receivedAt":"2018-06-28T14:00:17Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an API around managing the lifetime of cache_entry\nstructs. Abstracting memory management details behind this API will\nallow for alternative memory management strategies without affecting\nall the call sites. This commit does not change how memory is\nallocated or freed. A later commit in this series will allocate cache\nentries from memory pools as appropriate.\n\nMotivation:\nIt has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This change\nis in preparation for using memory pools to reduce the number of\nmalloc() calls made when loading an index.\n\nThis API makes a distinction between cache entries that are intended\nfor use with a particular index and cache entries that are not. This\nenables us to use the knowledge about how a cache entry will be used\nto make informed decisions about how to handle the corresponding\nmemory.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 24 +++++------\n blame.c | 5 +--\n builtin/checkout.c | 8 ++--\n builtin/difftool.c | 6 +--\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 +++++-------\n cache.h | 40 +++++++++++++++---\n merge-recursive.c | 2 +-\n read-cache.c | 93 +++++++++++++++++++++++++++++-------------\n resolve-undo.c | 4 +-\n split-index.c | 8 ++--\n tree.c | 4 +-\n unpack-trees.c | 35 +++++++++++-----\n 13 files changed, 165 insertions(+), 92 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 8ef975a32d..8a4a4439bc 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4092,12 +4092,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, &oid, name, 0, 0);\n+\t\tce = make_cache_entry(&result, patch->old_mode, &oid, name, 0, 0);\n \t\tif (!ce)\n \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4265,9 +4265,8 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4280,13 +4279,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tdiscard_cache_entry(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4294,13 +4293,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4424,27 +4423,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex a5c9bd78ab..87dade745c 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -173,7 +173,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -271,8 +271,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 548bf40f25..56d1e1a28d 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, &oid, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 873a06f0d9..4593f0c2cd 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n@@ -488,7 +488,7 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, &roid,\n+\t\t\t\t\tmake_cache_entry(&wtindex, rmode, &roid,\n \t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 00109b041f..c3f0cfa1e8 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,7 +134,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, &one->oid, one->path,\n+\t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex a8709a26ec..ea2f2a476c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -402,15 +401,14 @@ static int process_path(const char *path, struct stat *st, int stat_errno)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path, mode))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -599,7 +597,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -612,8 +609,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -690,8 +686,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tdiscard_cache_entry(ce_2);\n+\tdiscard_cache_entry(ce_3);\n \treturn ret;\n }\n \n@@ -758,7 +754,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -769,7 +765,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tdiscard_cache_entry(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex 3fbf24771a..035a627bea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,40 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and cleanup */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `discard_cache_entry`.\n+ */\n+struct cache_entry *make_cache_entry(struct index_state *istate,\n+\t\t\t\t unsigned int mode,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t int stage,\n+\t\t\t\t unsigned int refresh_options);\n+\n+struct cache_entry *make_empty_cache_entry(struct index_state *istate,\n+\t\t\t\t\t size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `discard_cache_entry`.\n+ */\n+struct cache_entry *make_transient_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int stage);\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry.\n+ */\n+void discard_cache_entry(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -698,12 +732,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t int stage,\n-\t\t\t\t\t unsigned int refresh_options);\n-\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 873321e5c2..06d77afbeb 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -318,7 +318,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid : &null_oid, path, stage, 0);\n+\tce = make_cache_entry(&the_index, mode, oid ? oid : &null_oid, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"add_cacheinfo failed for path '%s'; merge aborting.\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 9624ce1784..116fd51680 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tdiscard_cache_entry(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,14 +744,25 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n+struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_cache_entry(struct index_state *istate,\n+\t\t\t\t unsigned int mode,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t const char *path,\n \t\t\t\t int stage,\n \t\t\t\t unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path, mode)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -760,8 +770,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, len);\n \n \thashcpy(ce->oid.hash, oid->hash);\n \tmemcpy(ce->name, path, len);\n@@ -771,10 +780,33 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n+\t\t\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path, mode)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\thashcpy(ce->oid.hash, oid->hash);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1270,7 +1302,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1350,8 +1382,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1637,12 +1668,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1684,7 +1716,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1715,13 +1748,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1853,7 +1886,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1959,7 +1992,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tdiscard_cache_entry(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2649,14 +2682,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2765,3 +2797,8 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+void discard_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 4d4e5cb6bf..c30ae5cf49 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,7 +146,9 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], &ru->oid[i],\n+\t\tnce = make_cache_entry(istate,\n+\t\t\t\t ru->mode[i],\n+\t\t\t\t &ru->oid[i],\n \t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\ndiff --git a/split-index.c b/split-index.c\nindex 660c75f31f..317900db8b 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tdiscard_cache_entry(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tdiscard_cache_entry(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tdiscard_cache_entry(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 2c9c49725c..7bbf304f9f 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -17,15 +17,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3a85a02a77..33cba550b0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,10 +203,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -216,7 +216,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -797,10 +797,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\tconst struct name_entry *n,\n+\tint stage,\n+\tstruct index_state *istate,\n+\tint is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -846,7 +853,15 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\n+\t\t/*\n+\t\t * If the merge bit is set, then the cache entries are\n+\t\t * discarded in the following block. In this case,\n+\t\t * construct \"transient\" cache_entries, as they are\n+\t\t * not stored in the index. otherwise construct the\n+\t\t * cache entry from the index aware logic.\n+\t\t */\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -855,7 +870,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1787,7 +1802,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1807,7 +1822,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tdiscard_cache_entry(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1833,7 +1848,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tdiscard_cache_entry(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.17.1\n\n"},{"id":"351212","messageId":"20180628135932.225288-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 5/8] mem-pool: add life cycle management functions","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:12Z","receivedAt":"2018-06-28T14:00:24Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add initialization and discard functions to mem_pool type. As the\nmemory allocated by mem_pool can now be freed, we also track the large\nallocations.\n\nIf the there are existing mp_blocks in the mem_poo's linked list of\nmp_blocksl, then the mp_block for a large allocation is inserted\nbehind the head block. This is because only the head mp_block is considered\nwhen searching for availble space. This results in the following\ndesirable properties:\n\n1) The mp_block allocated for the large request will not be included\nnot included in the search for available in future requests, the large\nmp_block is sized for the specific request and does not contain any\nspare space.\n\n2) The head mp_block will not bumped from considation for future\nmemory requests just because a request for a large chunk of memory\ncame in.\n\nThese changes are in preparation for a future commit that will utilize\ncreating and discarding memory pool.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++--------\n mem-pool.h | 10 +++++++++\n 2 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex c80124f1fe..1769400d2d 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,20 +5,65 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n-static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n+/*\n+ * Allocate a new mp_block and insert it after the block specified in\n+ * `insert_after`. If `insert_after` is NULL, then insert block at the\n+ * head of the linked list.\n+ */\n+static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc, struct mp_block *insert_after)\n {\n \tstruct mp_block *p;\n \n \tmem_pool->pool_alloc += sizeof(struct mp_block) + block_alloc;\n \tp = xmalloc(st_add(sizeof(struct mp_block), block_alloc));\n-\tp->next_block = mem_pool->mp_block;\n+\n \tp->next_free = (char *)p->space;\n \tp->end = p->next_free + block_alloc;\n-\tmem_pool->mp_block = p;\n+\n+\tif (insert_after) {\n+\t\tp->next_block = insert_after->next_block;\n+\t\tinsert_after->next_block = p;\n+\t} else {\n+\t\tp->next_block = mem_pool->mp_block;\n+\t\tmem_pool->mp_block = p;\n+\t}\n \n \treturn p;\n }\n \n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tstruct mem_pool *pool;\n+\n+\tif (*mem_pool)\n+\t\treturn;\n+\n+\tpool = xcalloc(1, sizeof(*pool));\n+\n+\tpool->block_alloc = BLOCK_GROWTH_SIZE;\n+\n+\tif (initial_size > 0)\n+\t\tmem_pool_alloc_block(pool, initial_size, NULL);\n+\n+\t*mem_pool = pool;\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tstruct mp_block *block, *block_to_free;\n+\n+\twhile ((block = mem_pool->mp_block))\n+\t{\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p = NULL;\n@@ -33,12 +78,10 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n+\t\tif (len >= (mem_pool->block_alloc / 2))\n+\t\t\treturn mem_pool_alloc_block(mem_pool, len, mem_pool->mp_block);\n \n-\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n+\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc, NULL);\n \t}\n \n \tr = p->next_free;\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..f75b3365d5 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -21,6 +21,16 @@ struct mem_pool {\n \tsize_t pool_alloc;\n };\n \n+/*\n+ * Initialize mem_pool with specified initial size.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n-- \n2.17.1\n\n"},{"id":"351213","messageId":"20180628135932.225288-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 4/8] mem-pool: only search head block for available space","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:10Z","receivedAt":"2018-06-28T14:00:26Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Instead of searching all memory blocks for available space to fulfill\na memory request, only search the head block. If the head block does\nnot have space, assume that previous block would most likely not be\nable to fulfill request either. This could potentially lead to more\nmemory fragmentation, but also avoids searching memory blocks that\nprobably will not be able to fulfill request.\n\nThis pattern will benefit consumers that are able to generate a good\nestimate for how much memory will be needed, or if they are performing\nfixed sized allocations, so that once a block is exhausted it will\nnever be able to fulfill a future request.\n\nThe impact of this change on memory was tested by running fast-import on\nthe git.git and linux.git repositories. The total amount of memory\nallocated to the mem-pools did not change with this approach, indicating\nthat there is not much memory benefit to searching all nodes in the\nlinked list.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..c80124f1fe 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -21,16 +21,16 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n-\tstruct mp_block *p;\n+\tstruct mp_block *p = NULL;\n \tvoid *r;\n \n \t/* round up to a 'uintmax_t' alignment */\n \tif (len & (sizeof(uintmax_t) - 1))\n \t\tlen += sizeof(uintmax_t) - (len & (sizeof(uintmax_t) - 1));\n \n-\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n-\t\tif (p->end - p->next_free >= len)\n-\t\t\tbreak;\n+\tif (mem_pool->mp_block &&\n+\t mem_pool->mp_block->end - mem_pool->mp_block->next_free >= len)\n+\t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n \t\tif (len >= (mem_pool->block_alloc / 2)) {\n-- \n2.17.1\n\n"},{"id":"351214","messageId":"20180628135932.225288-7-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 6/8] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:13Z","receivedAt":"2018-06-28T14:00:29Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add functions for:\n\n - combining two memory pools\n\n - determining if a memory address is within the range managed by a\n memory pool\n\nThese functions will be used by future commits.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n mem-pool.h | 13 +++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 1769400d2d..b250a5fe40 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -96,3 +96,45 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Check if memory is allocated in a block */\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Append the blocks from src to dst */\n+\tif (dst->mp_block && src->mp_block) {\n+\t\t/*\n+\t\t * src and dst have blocks, append\n+\t\t * blocks from src to dst.\n+\t\t */\n+\t\tp = dst->mp_block;\n+\t\twhile (p->next_block)\n+\t\t\tp = p->next_block;\n+\n+\t\tp->next_block = src->mp_block;\n+\t} else if (src->mp_block) {\n+\t\t/*\n+\t\t * src has blocks, dst is empty.\n+\t\t */\n+\t\tdst->mp_block = src->mp_block;\n+\t} else {\n+\t\t/* src is empty, nothing to do. */\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex f75b3365d5..adeefdcb28 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -41,4 +41,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.17.1\n\n"},{"id":"351215","messageId":"20180628135932.225288-9-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 8/8] block alloc: add validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:16Z","receivedAt":"2018-06-28T14:00:31Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 6 ++++++\n git.c | 3 +++\n mem-pool.c | 6 +++++-\n mem-pool.h | 2 +-\n read-cache.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 5 files changed, 68 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8fd89ae51f..b3faef3efc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -380,6 +380,12 @@ struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Check configuration if we should perform extra validation on cache\n+ * entries.\n+ */\n+int should_validate_cache_entries(void);\n+\n /*\n * Duplicate a cache_entry. Allocate memory for the new entry from a\n * memory_pool. Takes into account cache_entry fields that are meant\ndiff --git a/git.c b/git.c\nindex 9dbe6ffaa7..c7e4f0351a 100644\n--- a/git.c\n+++ b/git.c\n@@ -414,7 +414,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex 139617cb23..a2841a4a9a 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -50,7 +50,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n \t*mem_pool = pool;\n }\n \n-void mem_pool_discard(struct mem_pool *mem_pool)\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory)\n {\n \tstruct mp_block *block, *block_to_free;\n \n@@ -59,6 +59,10 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \ndiff --git a/mem-pool.h b/mem-pool.h\nindex adeefdcb28..999d3c3a52 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -29,7 +29,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n /*\n * Discard a memory pool and free all the memory it is responsible for.\n */\n-void mem_pool_discard(struct mem_pool *mem_pool);\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory);\n \n /*\n * Alloc memory from the mem_pool.\ndiff --git a/read-cache.c b/read-cache.c\nindex f346437800..ba31e929e8 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2050,8 +2050,10 @@ int discard_index(struct index_state *istate)\n \t * Cache entries in istate->cache[] should have been allocated\n \t * from the memory pool associated with this index, or from an\n \t * associated split_index. There is no need to free individual\n-\t * cache entries.\n+\t * cache entries. validate_cache_entries can detect when this\n+\t * assertion does not hold.\n \t */\n+\tvalidate_cache_entries(istate);\n \n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2068,13 +2070,45 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \n \tif (istate->ce_mem_pool) {\n-\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tmem_pool_discard(istate->ce_mem_pool, should_validate_cache_entries());\n \t\tistate->ce_mem_pool = NULL;\n \t}\n \n \treturn 0;\n }\n \n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!should_validate_cache_entries() ||!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2878,8 +2912,25 @@ struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n \n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && should_validate_cache_entries())\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n+\n \tif (ce && ce->mem_pool_allocated)\n \t\treturn;\n \n \tfree(ce);\n }\n+\n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n-- \n2.17.1\n\n"},{"id":"351216","messageId":"20180628135932.225288-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 1/8] read-cache: teach refresh_cache_entry() to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:06Z","receivedAt":"2018-06-28T14:00:34Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactor refresh_cache_entry() to work on a specific index, instead of\nimplicitly using the_index. This is in preparation for making the\nmake_cache_entry function apply to a specific index.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 9 +++++----\n 3 files changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d49092d94d..93af25f586 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -751,7 +751,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex bed4a5be02..8b3d6781c7 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -326,7 +326,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"add_cacheinfo failed to refresh for path '%s'; merge aborting.\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 372588260e..fa8366ecab 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1473,10 +1473,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n-\t\t\t\t\t unsigned int options)\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\tstruct cache_entry *ce,\n+\t\t\t\t\tunsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.17.1\n\n"},{"id":"351217","messageId":"20180628135932.225288-8-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v5 7/8] block alloc: allocate cache entries from mem-pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:00:15Z","receivedAt":"2018-06-28T14:00:36Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"When reading large indexes from disk, a portion of the time is\ndominated in malloc() calls. This can be mitigated by allocating a\nlarge block of memory and manage it ourselves via memory pools.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\n\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entries will be allocated from. When reading in the\nindex from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool will be discarded as well - so the lifetime of\na cache_entry is tied to the lifetime of the index_state that it was\nallocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n\nManaging transient cache_entry structs:\nCache entries are usually allocated for an index, but this is not always\nthe case. Cache entries are sometimes allocated because this is the\ntype that the existing checkout_entry function works with. Because of\nthis, the existing code needs to handle cache entries associated with an\nindex / memory pool, and those that only exist transiently. Several\nstrategies were contemplated around how to handle this:\n\nChosen approach:\nAn extra field was added to the cache_entry type to track whether the\ncache_entry was allocated from a memory pool or not. This is currently\nan int field, as there are no more available bits in the existing\nce_flags bit field. If / when more bits are needed, this new field can\nbe turned into a proper bit field.\n\nAlternatives:\n\n1) Do not include any information about how the cache_entry was\nallocated. Calling code would be responsible for tracking whether the\ncache_entry needed to be freed or not.\n Pro: No extra memory overhead to track this state\n Con: Extra complexity in callers to handle this correctly.\n\nThe extra complexity and burden to not regress this behavior in the\nfuture was more than we wanted.\n\n2) cache_entry would gain knowledge about which mem_pool allocated it\n Pro: Could (potentially) do extra logic to know when a mem_pool no\n longer had references to any cache_entry\n Con: cache_entry would grow heavier by a pointer, instead of int\n\nWe didn't see a tangible benefit to this approach\n\n3) Do not add any extra information to a cache_entry, but when freeing a\n cache entry, check if the memory exists in a region managed by existing\n mem_pools.\n Pro: No extra memory overhead to track state\n Con: Extra computation is performed when freeing cache entries\n\nWe decided tracking and iterating over known memory pool regions was\nless desirable than adding an extra field to track this stae.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 21 +++++++++\n mem-pool.c | 3 +-\n read-cache.c | 119 +++++++++++++++++++++++++++++++++++++++++--------\n split-index.c | 50 +++++++++++++++++----\n unpack-trees.c | 13 +-----\n 5 files changed, 167 insertions(+), 39 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 035a627bea..8fd89ae51f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -156,6 +157,7 @@ struct cache_entry {\n \tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n \tunsigned int ce_flags;\n+\tunsigned int mem_pool_allocated;\n \tunsigned int ce_namelen;\n \tunsigned int index;\t/* for link extension */\n \tstruct object_id oid;\n@@ -227,6 +229,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\t\t const struct cache_entry *src)\n {\n \tunsigned int state = dst->ce_flags & CE_HASHED;\n+\tint mem_pool_allocated = dst->mem_pool_allocated;\n \n \t/* Don't copy hash chain and name */\n \tmemcpy(&dst->ce_stat_data, &src->ce_stat_data,\n@@ -235,6 +238,9 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \n \t/* Restore the hash state */\n \tdst->ce_flags = (dst->ce_flags & ~CE_HASHED) | state;\n+\n+\t/* Restore the mem_pool_allocated flag */\n+\tdst->mem_pool_allocated = mem_pool_allocated;\n }\n \n static inline unsigned create_ce_flags(unsigned stage)\n@@ -328,6 +334,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\n@@ -373,6 +380,20 @@ struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Duplicate a cache_entry. Allocate memory for the new entry from a\n+ * memory_pool. Takes into account cache_entry fields that are meant\n+ * for managing the underlying memory allocation of the cache_entry.\n+ */\n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce, struct index_state *istate);\n+\n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/mem-pool.c b/mem-pool.c\nindex b250a5fe40..139617cb23 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -54,7 +54,8 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n {\n \tstruct mp_block *block, *block_to_free;\n \n-\twhile ((block = mem_pool->mp_block))\n+\tblock = mem_pool->mp_block;\n+\twhile (block)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\ndiff --git a/read-cache.c b/read-cache.c\nindex 116fd51680..f346437800 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,48 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is an estimate of the pathname length in the index. We use\n+ * this for V4 index files to guess the un-deltafied size of the index\n+ * in memory because of pathname deltafication. This is not required\n+ * for V2/V3 index formats because their pathnames are not compressed.\n+ * If the initial amount of memory set aside is not sufficient, the\n+ * mem pool will allocate extra memory.\n+ */\n+#define CACHE_ENTRY_PATH_LENGTH 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry *ce;\n+\tce = mem_pool_alloc(mem_pool, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry * ce;\n+\tce = mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +788,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1668,13 +1710,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1716,7 +1758,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1748,13 +1790,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1828,6 +1870,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_PATH_LENGTH);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1874,10 +1932,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1886,7 +1949,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1983,17 +2046,13 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n+\t/*\n+\t * Cache entries in istate->cache[] should have been allocated\n+\t * from the memory pool associated with this index, or from an\n+\t * associated split_index. There is no need to free individual\n+\t * cache entries.\n+\t */\n \n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tdiscard_cache_entry(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -2007,6 +2066,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2798,7 +2863,23 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tsrc->untracked = NULL;\n }\n \n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n+\t\t\t\t struct index_state *istate)\n+{\n+\tunsigned int size = ce_size(ce);\n+\tint mem_pool_allocated;\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n+\tmem_pool_allocated = new_entry->mem_pool_allocated;\n+\n+\tmemcpy(new_entry, ce, size);\n+\tnew_entry->mem_pool_allocated = mem_pool_allocated;\n+\treturn new_entry;\n+}\n+\n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && ce->mem_pool_allocated)\n+\t\treturn;\n+\n \tfree(ce);\n }\ndiff --git a/split-index.c b/split-index.c\nindex 317900db8b..84f067e10d 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -331,12 +346,31 @@ void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n \t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n+\t\t * When removing the split index, we need to move\n+\t\t * ownership of the mem_pool associated with the\n+\t\t * base index to the main index. There may be cache entries\n+\t\t * allocated from the base's memory pool that are shared with\n+\t\t * the_index.cache[].\n \t\t */\n-\t\tistate->split_index = NULL;\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n+\t\t/*\n+\t\t * The split index no longer owns the mem_pool backing\n+\t\t * its cache array. As we are discarding this index,\n+\t\t * mark the index as having no cache entries, so it\n+\t\t * will not attempt to clean up the cache entries or\n+\t\t * validate them.\n+\t\t */\n+\t\tif (istate->split_index->base)\n+\t\t\tistate->split_index->base->cache_nr = 0;\n+\n+\t\t/*\n+\t\t * We can discard the split index because its\n+\t\t * memory pool has been incorporated into the\n+\t\t * memory pool associated with the the_index.\n+\t\t */\n+\t\tdiscard_split_index(istate);\n+\n \t\tistate->cache_changed |= SOMETHING_CHANGED;\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 33cba550b0..a3b5131732 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,20 +203,11 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n-{\n-\tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n-\n-\tmemcpy(new_entry, ce, size);\n-\treturn new_entry;\n-}\n-\n static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n+\tdo_add_entry(o, dup_cache_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -1802,7 +1793,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n+\tstruct cache_entry *merge = dup_cache_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n-- \n2.17.1\n\n"},{"id":"351218","messageId":"DM5PR21MB07808CBB6B7E7AF816CC435BCE4F0@DM5PR21MB0780.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79katH2Kk2NDDyZjYzC9fEKS6cDdHK1Cw2JLjjSfmYOPapw@mail.gmail.com","subject":"RE: [PATCH v4 2/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:07:09Z","receivedAt":"2018-06-28T14:07:14Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"> How much work is it to convert these to object_id while at it instead of char\n> *sha1?\n> (I think we really dislike introducing new sha1 based code; If it doesn't sound\n> easy maybe we can have a static inline wrapper here that just converts oid to\n> sha1 and then calls this function?)\n\nI have made this change in my latest patch series with 0ac48f3af7 (\"read-cache: make_cache_entry should take object_id struct\", 2018-06-22)\n\n> \n> Is it possible to line break these functions (c.f. refs.h which I think has one of the\n> best styles in the git project. It has long parameter lists, but still manages to stay\n> below a reasonable line length) ?\n> \n> > +extern struct cache_entry *make_empty_cache_entry(struct index_state\n> > +*istate, size_t name_len);\n> \n\nI have formatted the function declarations to break them up\n\n> > +\n> > +/*\n> > + * Create a cache_entry that is not intended to be added to an index.\n> > + * Caller is responsible for discarding the cache_entry\n> > + * with `discard_cache_entry`.\n> > + */\n> > +extern struct cache_entry *make_transient_cache_entry(unsigned int\n> > +mode, const unsigned char *sha1, const char *path, int stage); extern\n> > +struct cache_entry *make_empty_transient_cache_entry(size_t\n> > +name_len);\n> > +\n> > +/*\n> > + * Discard cache entry.\n> > + */\n> > +void discard_cache_entry(struct cache_entry *ce);\n> \n> Please be consistent in the use of the extern keyword and omit them at all times\n> here and above.\n\nThe header file is a bit inconsistent overall, but I switched my declarations to not use the extern keyword.\n\n\n\n"},{"id":"351219","messageId":"DM5PR21MB07803918120F4CC5B340230ACE4F0@DM5PR21MB0780.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79kbL5wjJvKj-=gtGgtMFCPs+zTwSAV-Aeo_2GNC_h=1nNA@mail.gmail.com","subject":"RE: [PATCH v4 3/8] mem-pool: only search head block for available space","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-06-28T14:12:12Z","receivedAt":"2018-06-28T14:12:17Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"> Do we have any numbers on performance or memory pressure here?\n> (I would think benchmarking fast-import would suffice as that is where the mem\n> pool originated)\n\nI ran fast-import on the git.git and linux.git repositories - this action reports the \noverall memory allocated to pools. The overall memory used by pools did not change\nin my tests (git.git used 2 blocks, and linux.git used 12 blocks). I included this information\nin the corresponding commit message.\n\n> \n> > This pattern will benefit consumers that are able to generate a good\n> > estimate for how much memory will be needed, or if they are performing\n> > fixed sized allocations, so that once a block is exhausted it will\n> > never be able to fulfill a future request.\n> \n> Would this be a good candidate to contain parts of\n\nI included these details in the commit message that adds the field indicating the\ncache_entry was allocated from a mem-pool.\n"},{"id":"351240","messageId":"xmqqin62ygsp.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180628135932.225288-3-jamill@microsoft.com","subject":"Re: [PATCH v5 2/8] read-cache: make_cache_entry should take object_id struct","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-28T17:14:30Z","receivedAt":"2018-06-28T17:14:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> The make_cache_entry function should take an object_id struct instead\n> of sha.\n\nThe name of the hash is SHA-1, not sha ;-)\n\nIt is not worth a reroll, but I do not think \"should\" is a\nparticularly good thing to say in the title or justification in the\nlog message in this case. It is more like you (or somebody else who\ncommented) _want_ to make it take an oid for _some_ reason. \"teach\nmake_cache_entry() to take object_id\" is probably a better title\nthat admits that we do not explicitly say _why_ we are doing so,\nthan saying \"it should\", which equally is not explicit ;-)\n\n> diff --git a/read-cache.c b/read-cache.c\n> index fa8366ecab..9624ce1784 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -746,8 +746,10 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n> }\n> \n> struct cache_entry *make_cache_entry(unsigned int mode,\n> -\t\tconst unsigned char *sha1, const char *path, int stage,\n> -\t\tunsigned int refresh_options)\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t const char *path,\n> +\t\t\t\t int stage,\n> +\t\t\t\t unsigned int refresh_options)\n> {\n> \tint size, len;\n> \tstruct cache_entry *ce, *ret;\n> @@ -761,7 +763,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n> \tsize = cache_entry_size(len);\n> \tce = xcalloc(1, size);\n> \n> -\thashcpy(ce->oid.hash, sha1);\n> +\thashcpy(ce->oid.hash, oid->hash);\n> \tmemcpy(ce->name, path, len);\n> \tce->ce_flags = create_ce_flags(stage);\n> \tce->ce_namelen = len;\n\nThe patch itself looks good.\n\nThanks.\n"},{"id":"351241","messageId":"xmqqefgqygqf.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180628135932.225288-6-jamill@microsoft.com","subject":"Re: [PATCH v5 5/8] mem-pool: add life cycle management functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-28T17:15:52Z","receivedAt":"2018-06-28T17:15:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> Add initialization and discard functions to mem_pool type. As the\n> memory allocated by mem_pool can now be freed, we also track the large\n> allocations.\n>\n> If the there are existing mp_blocks in the mem_poo's linked list of\n\nmem_poo?\n\n> mp_blocksl, then the mp_block for a large allocation is inserted\n\nmp_blocksl?\n\n"},{"id":"351254","messageId":"xmqqsh56wy4l.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180628135932.225288-4-jamill@microsoft.com","subject":"Re: [PATCH v5 3/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-28T18:43:06Z","receivedAt":"2018-06-28T18:43:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> Add an API around managing the lifetime of cache_entry\n> structs. Abstracting memory management details behind this API will\n> allow for alternative memory management strategies without affecting\n> all the call sites. This commit does not change how memory is\n> allocated or freed. A later commit in this series will allocate cache\n> entries from memory pools as appropriate.\n>\n> Motivation:\n> It has been observed that the time spent loading an index with a large\n> number of entries is partly dominated by malloc() calls. This change\n> is in preparation for using memory pools to reduce the number of\n> malloc() calls made when loading an index.\n\nNot worth a reroll, but having these four lines at the very\nbeginning, dropping the line \"Motivation:\", and then following that\nwith \"Add an API around ...\" as the second paragraph, would make the\nabove easier to read, without stutter-causing \"Motivation:\" in the\nmiddle.\n\n> diff --git a/apply.c b/apply.c\n> index 8ef975a32d..8a4a4439bc 100644\n> --- a/apply.c\n> +++ b/apply.c\n> @@ -4092,12 +4092,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n> \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n> \t\t\t\t \"(%s).\"), name);\n> \n> -\t\tce = make_cache_entry(patch->old_mode, &oid, name, 0, 0);\n> +\t\tce = make_cache_entry(&result, patch->old_mode, &oid, name, 0, 0);\n> \t\tif (!ce)\n> \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n> \t\t\t\t name);\n> \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n> -\t\t\tfree(ce);\n> +\t\t\tdiscard_cache_entry(ce);\n> \t\t\treturn error(_(\"could not add %s to temporary index\"),\n> \t\t\t\t name);\n> \t\t}\n\nSo..., even though it wasn't clear in the proposed log message, two\nlarge part of the lifecycle management API is (1) make_cache_entry()\nknows for which istate it is creating the entry and (2) discarding\nthe entry may not be just a simple matter of free()ing. Both of\nwhich makes perfect sense, but if the changes are that easily\nenumeratable, it would have been nicer for readers if the commit did\nso in the proposed log message.\n\n> @@ -4424,27 +4423,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n> \t\t\t\t struct patch *patch)\n> {\n> \tint stage, namelen;\n> -\tunsigned ce_size, mode;\n> +\tunsigned mode;\n> \tstruct cache_entry *ce;\n> \n> \tif (!state->update_index)\n> \t\treturn 0;\n> \tnamelen = strlen(patch->new_name);\n> -\tce_size = cache_entry_size(namelen);\n> \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n> \n> \tremove_file_from_cache(patch->new_name);\n> \tfor (stage = 1; stage < 4; stage++) {\n> \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n> \t\t\tcontinue;\n> -\t\tce = xcalloc(1, ce_size);\n> +\t\tce = make_empty_cache_entry(&the_index, namelen);\n\n... and another one in the enumeration is make_empty_cache_entry()\nwhich is somehow different. And the readers are forced to read its\nimplementation to find out how it is different, but the use of the\nsame discard_cache_entry() suggests that the liftime rule of an\nentry allcoated by it may be similar to those created by\nmake_cache_entry().\n\n> ...\n> \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n> -\t\t\tfree(ce);\n> +\t\t\tdiscard_cache_entry(ce);\n> \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n> \t\t\t\t patch->new_name);\n> \t\t}\n\n> @@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n> \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n> \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n> \tfree(result_buf.ptr);\n> -\tce = make_cache_entry(mode, &oid, path, 2, 0);\n> +\tce = make_transient_cache_entry(mode, &oid, path, 2);\n\n... and then yet another, which is \"transient\". An intelligent\nreader can guess from the lack of istate parameter (and from the\nword \"transient\") that the resulting one would not belong to any\nin-core index.\n\n> \tif (!ce)\n> \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n> \tstatus = checkout_entry(ce, state, NULL);\n> -\tfree(ce);\n> +\tdiscard_cache_entry(ce);\n\n... but discovers that it is discarded the same way, realizes that\nce knows how it was allocated to allow discard() different way to\ndiscard it, and his/her earlier conjecture about make_empty() does\nnot hold at all and gets somewhat disappointed.\n\n> diff --git a/cache.h b/cache.h\n> index 3fbf24771a..035a627bea 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -339,6 +339,40 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n> extern void free_name_hash(struct index_state *istate);\n> \n> \n> +/* Cache entry creation and cleanup */\n> +\n> +/*\n> + * Create cache_entry intended for use in the specified index. Caller\n> + * is responsible for discarding the cache_entry with\n> + * `discard_cache_entry`.\n> + */\n> +struct cache_entry *make_cache_entry(struct index_state *istate,\n> +\t\t\t\t unsigned int mode,\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t const char *path,\n> +\t\t\t\t int stage,\n> +\t\t\t\t unsigned int refresh_options);\n> +\n> +struct cache_entry *make_empty_cache_entry(struct index_state *istate,\n> +\t\t\t\t\t size_t name_len);\n> +\n> +/*\n> + * Create a cache_entry that is not intended to be added to an index.\n> + * Caller is responsible for discarding the cache_entry\n> + * with `discard_cache_entry`.\n> + */\n> +struct cache_entry *make_transient_cache_entry(unsigned int mode,\n> +\t\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t\t const char *path,\n> +\t\t\t\t\t int stage);\n> +\n> +struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n\nOK, finally it becomes clear that we have per-istate and transient\nsets of two (i.e. one that takes the path, stage and mode pfront,\nand the other that only takes the length of the name), and ...\n\n> +/*\n> + * Discard cache entry.\n> + */\n> +void discard_cache_entry(struct cache_entry *ce);\n\n... a single function that knows how to discard each kind. It would\nhave really helped the reader to talk about them in the proposed log\nmessage, as they are only 5 functions. Another way to make it easier\nfor readers would have been to show the diff for cache.h first before\ndiffs for all the others.\n"},{"id":"351257","messageId":"xmqqo9fuwwvy.fsf@gitster-ct.c.googlers.com","threadId":"48310","inReplyTo":"20180628135932.225288-7-jamill@microsoft.com","subject":"Re: [PATCH v5 6/8] mem-pool: fill out functionality","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-28T19:09:53Z","receivedAt":"2018-06-28T19:10:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jameson Miller writes:\n\n> +void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n> +{\n> +\tstruct mp_block *p;\n> +\n> +\t/* Append the blocks from src to dst */\n> +\tif (dst->mp_block && src->mp_block) {\n> +\t\t/*\n> +\t\t * src and dst have blocks, append\n> +\t\t * blocks from src to dst.\n> +\t\t */\n> +\t\tp = dst->mp_block;\n> +\t\twhile (p->next_block)\n> +\t\t\tp = p->next_block;\n> +\n> +\t\tp->next_block = src->mp_block;\n\nJust being curious, but does this interact with the \"we carve out\nonly from the first block\" done in step 4/8? The remaining unused\nspace in the first block in the src pool would be wasted, which may\nnot be such a big deal and may not even be worth comparing the\navailable space in two blocks and picking a larger one. But we do\nwant to decide _after_ thinking things through nevertheless.\n"},{"id":"351271","messageId":"20180628222744.10496-1-szeder.dev@gmail.com","threadId":"48310","inReplyTo":"20180628135932.225288-3-jamill@microsoft.com","subject":"Re: [PATCH v5 2/8] read-cache: make_cache_entry should take object_id struct","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-06-28T22:27:44Z","receivedAt":"2018-06-28T22:27:58Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> The make_cache_entry function should take an object_id struct instead\n> of sha.\n\n\n> diff --git a/read-cache.c b/read-cache.c\n> index fa8366ecab..9624ce1784 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -746,8 +746,10 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n> }\n> \n> struct cache_entry *make_cache_entry(unsigned int mode,\n> -\t\tconst unsigned char *sha1, const char *path, int stage,\n> -\t\tunsigned int refresh_options)\n> +\t\t\t\t const struct object_id *oid,\n> +\t\t\t\t const char *path,\n> +\t\t\t\t int stage,\n> +\t\t\t\t unsigned int refresh_options)\n> {\n> \tint size, len;\n> \tstruct cache_entry *ce, *ret;\n> @@ -761,7 +763,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n> \tsize = cache_entry_size(len);\n> \tce = xcalloc(1, size);\n> \n> -\thashcpy(ce->oid.hash, sha1);\n> +\thashcpy(ce->oid.hash, oid->hash);\n\nSpeaking of using struct object_id instead of sha, please use oidcpy()\nhere.\n\n"},{"id":"351272","messageId":"20180628222800.10570-1-szeder.dev@gmail.com","threadId":"48310","inReplyTo":"20180628135932.225288-4-jamill@microsoft.com","subject":"Re: [PATCH v5 3/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-06-28T22:28:00Z","receivedAt":"2018-06-28T22:28:15Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> diff --git a/read-cache.c b/read-cache.c\n> index 9624ce1784..116fd51680 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n\n> +struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n> +\t\t\t\t\t const char *path, int stage)\n> +{\n> +\tstruct cache_entry *ce;\n> +\tint len;\n> +\n> +\tif (!verify_path(path, mode)) {\n> +\t\terror(\"Invalid path '%s'\", path);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\tlen = strlen(path);\n> +\tce = make_empty_transient_cache_entry(len);\n> +\n> +\thashcpy(ce->oid.hash, oid->hash);\n\nPlease use oidcpy() here, too.\n\n"},{"id":"351540","messageId":"DM5PR21MB0780A5E4F75D3BB1384DB9C3CE430@DM5PR21MB0780.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"xmqqo9fuwwvy.fsf@gitster-ct.c.googlers.com","subject":"RE: [PATCH v5 6/8] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T18:28:20Z","receivedAt":"2018-07-02T18:28:25Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"> > +void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src) {\n> > +\tstruct mp_block *p;\n> > +\n> > +\t/* Append the blocks from src to dst */\n> > +\tif (dst->mp_block && src->mp_block) {\n> > +\t\t/*\n> > +\t\t * src and dst have blocks, append\n> > +\t\t * blocks from src to dst.\n> > +\t\t */\n> > +\t\tp = dst->mp_block;\n> > +\t\twhile (p->next_block)\n> > +\t\t\tp = p->next_block;\n> > +\n> > +\t\tp->next_block = src->mp_block;\n> \n> Just being curious, but does this interact with the \"we carve out only from the\n> first block\" done in step 4/8? The remaining unused space in the first block in\n> the src pool would be wasted, which may not be such a big deal and may not\n> even be worth comparing the available space in two blocks and picking a larger\n> one. But we do want to decide _after_ thinking things through nevertheless.\n\nGood question - and this is something I had thought about.\n\nIn the context of current usage, this function is currently not\nused frequently. It is only used in split index with\nmove_cache_to_base_index and remove_split_index. In either case,\nthe cache_entries should already be loaded into memory, and I\ndon't expect a big enough difference in the amount of left over\nspace to make a meaningful difference.\n\nIn the general case I could see this being a bigger case.\n\nFor now, I thought the logic as is was an appropriate tradeoff. I\ndon't think it would be complicated to do something smarter here,\nbut I also didn't see much benefit in current usage. If we prefer\nsomething else here, I can update this logic.\n"},{"id":"351544","messageId":"20180702193745.8940-3-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 2/8] read-cache: teach make_cache_entry to take object_id","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:30Z","receivedAt":"2018-07-02T19:49:36Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Teach make_cache_entry function to take object_id instead of a SHA-1.\n---\n apply.c | 2 +-\n builtin/checkout.c | 2 +-\n builtin/difftool.c | 4 ++--\n builtin/reset.c | 2 +-\n cache.h | 7 ++++++-\n merge-recursive.c | 2 +-\n read-cache.c | 8 +++++---\n resolve-undo.c | 2 +-\n 8 files changed, 18 insertions(+), 11 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 959c457910..8ef975a32d 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4092,7 +4092,7 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, oid.hash, name, 0, 0);\n+\t\tce = make_cache_entry(patch->old_mode, &oid, name, 0, 0);\n \t\tif (!ce)\n \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 2e1d2376d2..548bf40f25 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -230,7 +230,7 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, oid.hash, path, 2, 0);\n+\tce = make_cache_entry(mode, &oid, path, 2, 0);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex bc97d4aef2..873a06f0d9 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,7 +321,7 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid->hash, path, 0, 0);\n+\tce = make_cache_entry(mode, oid, path, 0, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n \tfree(ce);\n@@ -488,7 +488,7 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, roid.hash,\n+\t\t\t\t\tmake_cache_entry(rmode, &roid,\n \t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex a862c70fab..00109b041f 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,7 +134,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, one->oid.hash, one->path,\n+\t\tce = make_cache_entry(one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\ndiff --git a/cache.h b/cache.h\nindex 93af25f586..3fbf24771a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -698,7 +698,12 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode, const unsigned char *sha1, const char *path, int stage, unsigned int refresh_options);\n+extern struct cache_entry *make_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int stage,\n+\t\t\t\t\t unsigned int refresh_options);\n+\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 8b3d6781c7..873321e5c2 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -318,7 +318,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid->hash : null_sha1, path, stage, 0);\n+\tce = make_cache_entry(mode, oid ? oid : &null_oid, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"add_cacheinfo failed for path '%s'; merge aborting.\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex fa8366ecab..c12664c789 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -746,8 +746,10 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n }\n \n struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\tconst unsigned char *sha1, const char *path, int stage,\n-\t\tunsigned int refresh_options)\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t int stage,\n+\t\t\t\t unsigned int refresh_options)\n {\n \tint size, len;\n \tstruct cache_entry *ce, *ret;\n@@ -761,7 +763,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tsize = cache_entry_size(len);\n \tce = xcalloc(1, size);\n \n-\thashcpy(ce->oid.hash, sha1);\n+\toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(stage);\n \tce->ce_namelen = len;\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex fc5b3b83d9..4d4e5cb6bf 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,7 +146,7 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], ru->oid[i].hash,\n+\t\tnce = make_cache_entry(ru->mode[i], &ru->oid[i],\n \t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\n-- \n2.17.1\n\n"},{"id":"351545","messageId":"20180702193745.8940-4-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 3/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:31Z","receivedAt":"2018-07-02T19:49:40Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"It has been observed that the time spent loading an index with a large\nnumber of entries is partly dominated by malloc() calls. This change\nis in preparation for using memory pools to reduce the number of\nmalloc() calls made to allocate cahce entries when loading an index.\n\nAdd an API to allocate and discard cache entries, abstracting the\ndetails of managing the memory backing the cache entries. This commit\ndoes actually change how memory is managed - this will be done in a\nlater commit in the series.\n\nThis change makes the distinction between cache entries that are\nassociated with an index and cache entries that are not associated with\nan index. A main use of cache entries is with an index, and we can\noptimize the memory management around this. We still have other cases\nwhere a cache entry is not persisted with an index, and so we need to\nhandle the \"transient\" use case as well.\n\nTo keep the congnitive overhead of managing the cache entries, there\nwill only be a single discard function. This means there must be enough\ninformation kept with the cache entry so that we know how to discard\nthem.\n\nA summary of the main functions in the API is:\n\nmake_cache_entry: create cache entry for use in an index. Uses specified\n parameters to populate cache_entry fields.\n\nmake_empty_cache_entry: Create an empty cache entry for use in an index.\n Returns cache entry with empty fields.\n\nmake_transient_cache_entry: create cache entry that is not used in an\n index. Uses specified parameters to populate\n cache_entry fields.\n\nmake_empty_transient_cache_entry: create cache entry that is not used in\n an index. Returns cache entry with\n empty fields.\n\ndiscard_cache_entry: A single function that knows how to discard a cache\n entry regardless of how it was allocated.\n\nSigned-off-by: Jameson Miller \n---\n apply.c | 24 +++++------\n blame.c | 5 +--\n builtin/checkout.c | 8 ++--\n builtin/difftool.c | 6 +--\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 +++++-------\n cache.h | 40 +++++++++++++++---\n merge-recursive.c | 2 +-\n read-cache.c | 93 +++++++++++++++++++++++++++++-------------\n resolve-undo.c | 4 +-\n split-index.c | 8 ++--\n tree.c | 4 +-\n unpack-trees.c | 35 +++++++++++-----\n 13 files changed, 165 insertions(+), 92 deletions(-)\n\ndiff --git a/apply.c b/apply.c\nindex 8ef975a32d..8a4a4439bc 100644\n--- a/apply.c\n+++ b/apply.c\n@@ -4092,12 +4092,12 @@ static int build_fake_ancestor(struct apply_state *state, struct patch *list)\n \t\t\treturn error(_(\"sha1 information is lacking or useless \"\n \t\t\t\t \"(%s).\"), name);\n \n-\t\tce = make_cache_entry(patch->old_mode, &oid, name, 0, 0);\n+\t\tce = make_cache_entry(&result, patch->old_mode, &oid, name, 0, 0);\n \t\tif (!ce)\n \t\t\treturn error(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t\t name);\n \t\tif (add_index_entry(&result, ce, ADD_CACHE_OK_TO_ADD)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"could not add %s to temporary index\"),\n \t\t\t\t name);\n \t\t}\n@@ -4265,9 +4265,8 @@ static int add_index_file(struct apply_state *state,\n \tstruct stat st;\n \tstruct cache_entry *ce;\n \tint namelen = strlen(path);\n-\tunsigned ce_size = cache_entry_size(namelen);\n \n-\tce = xcalloc(1, ce_size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(0);\n@@ -4280,13 +4279,13 @@ static int add_index_file(struct apply_state *state,\n \n \t\tif (!skip_prefix(buf, \"Subproject commit \", &s) ||\n \t\t get_oid_hex(s, &ce->oid)) {\n-\t\t\tfree(ce);\n-\t\t return error(_(\"corrupt patch for submodule %s\"), path);\n+\t\t\tdiscard_cache_entry(ce);\n+\t\t\treturn error(_(\"corrupt patch for submodule %s\"), path);\n \t\t}\n \t} else {\n \t\tif (!state->cached) {\n \t\t\tif (lstat(path, &st) < 0) {\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t\t\treturn error_errno(_(\"unable to stat newly \"\n \t\t\t\t\t\t \"created file '%s'\"),\n \t\t\t\t\t\t path);\n@@ -4294,13 +4293,13 @@ static int add_index_file(struct apply_state *state,\n \t\t\tfill_stat_cache_info(ce, &st);\n \t\t}\n \t\tif (write_object_file(buf, size, blob_type, &ce->oid) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to create backing store \"\n \t\t\t\t \"for newly created file %s\"), path);\n \t\t}\n \t}\n \tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(_(\"unable to add cache entry for %s\"), path);\n \t}\n \n@@ -4424,27 +4423,26 @@ static int add_conflicted_stages_file(struct apply_state *state,\n \t\t\t\t struct patch *patch)\n {\n \tint stage, namelen;\n-\tunsigned ce_size, mode;\n+\tunsigned mode;\n \tstruct cache_entry *ce;\n \n \tif (!state->update_index)\n \t\treturn 0;\n \tnamelen = strlen(patch->new_name);\n-\tce_size = cache_entry_size(namelen);\n \tmode = patch->new_mode ? patch->new_mode : (S_IFREG | 0644);\n \n \tremove_file_from_cache(patch->new_name);\n \tfor (stage = 1; stage < 4; stage++) {\n \t\tif (is_null_oid(&patch->threeway_stage[stage - 1]))\n \t\t\tcontinue;\n-\t\tce = xcalloc(1, ce_size);\n+\t\tce = make_empty_cache_entry(&the_index, namelen);\n \t\tmemcpy(ce->name, patch->new_name, namelen);\n \t\tce->ce_mode = create_ce_mode(mode);\n \t\tce->ce_flags = create_ce_flags(stage);\n \t\tce->ce_namelen = namelen;\n \t\toidcpy(&ce->oid, &patch->threeway_stage[stage - 1]);\n \t\tif (add_cache_entry(ce, ADD_CACHE_OK_TO_ADD) < 0) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(_(\"unable to add cache entry for %s\"),\n \t\t\t\t patch->new_name);\n \t\t}\ndiff --git a/blame.c b/blame.c\nindex a5c9bd78ab..87dade745c 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -173,7 +173,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \tconst char *ident;\n \ttime_t now;\n-\tint size, len;\n+\tint len;\n \tstruct cache_entry *ce;\n \tunsigned mode;\n \tstruct strbuf msg = STRBUF_INIT;\n@@ -271,8 +271,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t\t\t/* Let's not bother reading from HEAD tree */\n \t\t\tmode = S_IFREG | 0644;\n \t}\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, &origin->blob_oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 548bf40f25..56d1e1a28d 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -77,7 +77,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = base->len + strlen(pathname);\n-\tce = xcalloc(1, cache_entry_size(len));\n+\tce = make_empty_cache_entry(&the_index, len);\n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, base->buf, base->len);\n \tmemcpy(ce->name + base->len, pathname, len - base->len);\n@@ -96,7 +96,7 @@ static int update_some(const struct object_id *oid, struct strbuf *base,\n \t\tif (ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n \t\t\told->ce_flags |= CE_UPDATE;\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n@@ -230,11 +230,11 @@ static int checkout_merged(int pos, const struct checkout *state)\n \tif (write_object_file(result_buf.ptr, result_buf.size, blob_type, &oid))\n \t\tdie(_(\"Unable to add merge result for '%s'\"), path);\n \tfree(result_buf.ptr);\n-\tce = make_cache_entry(mode, &oid, path, 2, 0);\n+\tce = make_transient_cache_entry(mode, &oid, path, 2);\n \tif (!ce)\n \t\tdie(_(\"make_cache_entry failed for path '%s'\"), path);\n \tstatus = checkout_entry(ce, state, NULL);\n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn status;\n }\n \ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 873a06f0d9..4593f0c2cd 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -321,10 +321,10 @@ static int checkout_path(unsigned mode, struct object_id *oid,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid, path, 0, 0);\n+\tce = make_transient_cache_entry(mode, oid, path, 0);\n \tret = checkout_entry(ce, state, NULL);\n \n-\tfree(ce);\n+\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n@@ -488,7 +488,7 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\t\t\t * index.\n \t\t\t\t */\n \t\t\t\tstruct cache_entry *ce2 =\n-\t\t\t\t\tmake_cache_entry(rmode, &roid,\n+\t\t\t\t\tmake_cache_entry(&wtindex, rmode, &roid,\n \t\t\t\t\t\t\t dst_path, 0, 0);\n \n \t\t\t\tadd_index_entry(&wtindex, ce2,\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 00109b041f..c3f0cfa1e8 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -134,7 +134,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t\tcontinue;\n \t\t}\n \n-\t\tce = make_cache_entry(one->mode, &one->oid, one->path,\n+\t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex a8709a26ec..ea2f2a476c 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -268,15 +268,14 @@ static int process_lstat_error(const char *path, int err)\n \n static int add_one_path(const struct cache_entry *old, const char *path, int len, struct stat *st)\n {\n-\tint option, size;\n+\tint option;\n \tstruct cache_entry *ce;\n \n \t/* Was the old index entry already up-to-date? */\n \tif (old && !ce_stage(old) && !ce_match_stat(old, st, 0))\n \t\treturn 0;\n \n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(0);\n \tce->ce_namelen = len;\n@@ -285,13 +284,13 @@ static int add_one_path(const struct cache_entry *old, const char *path, int len\n \n \tif (index_path(&ce->oid, path, st,\n \t\t info_only ? 0 : HASH_WRITE_OBJECT)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn -1;\n \t}\n \toption = allow_add ? ADD_CACHE_OK_TO_ADD : 0;\n \toption |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;\n \tif (add_cache_entry(ce, option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"%s: cannot add to the index - missing --add option?\", path);\n \t}\n \treturn 0;\n@@ -402,15 +401,14 @@ static int process_path(const char *path, struct stat *st, int stat_errno)\n static int add_cacheinfo(unsigned int mode, const struct object_id *oid,\n \t\t\t const char *path, int stage)\n {\n-\tint size, len, option;\n+\tint len, option;\n \tstruct cache_entry *ce;\n \n \tif (!verify_path(path, mode))\n \t\treturn error(\"Invalid path '%s'\", path);\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -599,7 +597,6 @@ static struct cache_entry *read_one_ent(const char *which,\n {\n \tunsigned mode;\n \tstruct object_id oid;\n-\tint size;\n \tstruct cache_entry *ce;\n \n \tif (get_tree_entry(ent, path, &oid, &mode)) {\n@@ -612,8 +609,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(&the_index, namelen);\n \n \toidcpy(&ce->oid, &oid);\n \tmemcpy(ce->name, path, namelen);\n@@ -690,8 +686,8 @@ static int unresolve_one(const char *path)\n \terror(\"%s: cannot add their version to the index.\", path);\n \tret = -1;\n free_return:\n-\tfree(ce_2);\n-\tfree(ce_3);\n+\tdiscard_cache_entry(ce_2);\n+\tdiscard_cache_entry(ce_3);\n \treturn ret;\n }\n \n@@ -758,7 +754,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t\t\t\t ce->name, ce_namelen(ce), 0);\n \t\tif (old && ce->ce_mode == old->ce_mode &&\n \t\t !oidcmp(&ce->oid, &old->oid)) {\n-\t\t\tfree(old);\n+\t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n \t\t/* Be careful. The working tree may not have the\n@@ -769,7 +765,7 @@ static int do_reupdate(int ac, const char **av,\n \t\tpath = xstrdup(ce->name);\n \t\tupdate_one(path);\n \t\tfree(path);\n-\t\tfree(old);\n+\t\tdiscard_cache_entry(old);\n \t\tif (save_nr != active_nr)\n \t\t\tgoto redo;\n \t}\ndiff --git a/cache.h b/cache.h\nindex 3fbf24771a..035a627bea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -339,6 +339,40 @@ extern void remove_name_hash(struct index_state *istate, struct cache_entry *ce)\n extern void free_name_hash(struct index_state *istate);\n \n \n+/* Cache entry creation and cleanup */\n+\n+/*\n+ * Create cache_entry intended for use in the specified index. Caller\n+ * is responsible for discarding the cache_entry with\n+ * `discard_cache_entry`.\n+ */\n+struct cache_entry *make_cache_entry(struct index_state *istate,\n+\t\t\t\t unsigned int mode,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t int stage,\n+\t\t\t\t unsigned int refresh_options);\n+\n+struct cache_entry *make_empty_cache_entry(struct index_state *istate,\n+\t\t\t\t\t size_t name_len);\n+\n+/*\n+ * Create a cache_entry that is not intended to be added to an index.\n+ * Caller is responsible for discarding the cache_entry\n+ * with `discard_cache_entry`.\n+ */\n+struct cache_entry *make_transient_cache_entry(unsigned int mode,\n+\t\t\t\t\t const struct object_id *oid,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int stage);\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n+\n+/*\n+ * Discard cache entry.\n+ */\n+void discard_cache_entry(struct cache_entry *ce);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\n@@ -698,12 +732,6 @@ extern int remove_file_from_index(struct index_state *, const char *path);\n extern int add_to_index(struct index_state *, const char *path, struct stat *, int flags);\n extern int add_file_to_index(struct index_state *, const char *path, int flags);\n \n-extern struct cache_entry *make_cache_entry(unsigned int mode,\n-\t\t\t\t\t const struct object_id *oid,\n-\t\t\t\t\t const char *path,\n-\t\t\t\t\t int stage,\n-\t\t\t\t\t unsigned int refresh_options);\n-\n extern int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n extern int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n extern void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 873321e5c2..06d77afbeb 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -318,7 +318,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tstruct cache_entry *ce;\n \tint ret;\n \n-\tce = make_cache_entry(mode, oid ? oid : &null_oid, path, stage, 0);\n+\tce = make_cache_entry(&the_index, mode, oid ? oid : &null_oid, path, stage, 0);\n \tif (!ce)\n \t\treturn err(o, _(\"add_cacheinfo failed for path '%s'; merge aborting.\"), path);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex c12664c789..41e4d0e67a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tdiscard_cache_entry(old);\n \tce->ce_flags &= ~CE_HASHED;\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n@@ -74,7 +74,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tstruct cache_entry *old_entry = istate->cache[nr], *new_entry;\n \tint namelen = strlen(new_name);\n \n-\tnew_entry = xmalloc(cache_entry_size(namelen));\n+\tnew_entry = make_empty_cache_entry(istate, namelen);\n \tcopy_cache_entry(new_entry, old_entry);\n \tnew_entry->ce_flags &= ~CE_HASHED;\n \tnew_entry->ce_namelen = namelen;\n@@ -623,7 +623,7 @@ static struct cache_entry *create_alias_ce(struct index_state *istate,\n \n \t/* Ok, create the new entry using the name of the existing alias */\n \tlen = ce_namelen(alias);\n-\tnew_entry = xcalloc(1, cache_entry_size(len));\n+\tnew_entry = make_empty_cache_entry(istate, len);\n \tmemcpy(new_entry->name, alias->name, len);\n \tcopy_cache_entry(new_entry, ce);\n \tsave_or_free_index_entry(istate, ce);\n@@ -640,7 +640,7 @@ void set_object_name_for_intent_to_add_entry(struct cache_entry *ce)\n \n int add_to_index(struct index_state *istate, const char *path, struct stat *st, int flags)\n {\n-\tint size, namelen, was_same;\n+\tint namelen, was_same;\n \tmode_t st_mode = st->st_mode;\n \tstruct cache_entry *ce, *alias = NULL;\n \tunsigned ce_option = CE_MATCH_IGNORE_VALID|CE_MATCH_IGNORE_SKIP_WORKTREE|CE_MATCH_RACY_IS_DIRTY;\n@@ -662,8 +662,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\twhile (namelen && path[namelen-1] == '/')\n \t\t\tnamelen--;\n \t}\n-\tsize = cache_entry_size(namelen);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, namelen);\n \tmemcpy(ce->name, path, namelen);\n \tce->ce_namelen = namelen;\n \tif (!intent_only)\n@@ -704,13 +703,13 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t\t\tce_mark_uptodate(alias);\n \t\t\talias->ce_flags |= CE_ADDED;\n \n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn 0;\n \t\t}\n \t}\n \tif (!intent_only) {\n \t\tif (index_path(&ce->oid, path, st, newflags)) {\n-\t\t\tfree(ce);\n+\t\t\tdiscard_cache_entry(ce);\n \t\t\treturn error(\"unable to index file %s\", path);\n \t\t}\n \t} else\n@@ -727,9 +726,9 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\t ce->ce_mode == alias->ce_mode);\n \n \tif (pretend)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \telse if (add_index_entry(istate, ce, add_option)) {\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \t\treturn error(\"unable to add %s to index\", path);\n \t}\n \tif (verbose && !was_same)\n@@ -745,14 +744,25 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \treturn add_to_index(istate, path, &st, flags);\n }\n \n-struct cache_entry *make_cache_entry(unsigned int mode,\n+struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_empty_transient_cache_entry(size_t len)\n+{\n+\treturn xcalloc(1, cache_entry_size(len));\n+}\n+\n+struct cache_entry *make_cache_entry(struct index_state *istate,\n+\t\t\t\t unsigned int mode,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t const char *path,\n \t\t\t\t int stage,\n \t\t\t\t unsigned int refresh_options)\n {\n-\tint size, len;\n \tstruct cache_entry *ce, *ret;\n+\tint len;\n \n \tif (!verify_path(path, mode)) {\n \t\terror(\"Invalid path '%s'\", path);\n@@ -760,8 +770,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \t}\n \n \tlen = strlen(path);\n-\tsize = cache_entry_size(len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, len);\n \n \toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n@@ -771,10 +780,33 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \n \tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n \treturn ret;\n }\n \n+struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n+\t\t\t\t\t const char *path, int stage)\n+{\n+\tstruct cache_entry *ce;\n+\tint len;\n+\n+\tif (!verify_path(path, mode)) {\n+\t\terror(\"Invalid path '%s'\", path);\n+\t\treturn NULL;\n+\t}\n+\n+\tlen = strlen(path);\n+\tce = make_empty_transient_cache_entry(len);\n+\n+\toidcpy(&ce->oid, oid);\n+\tmemcpy(ce->name, path, len);\n+\tce->ce_flags = create_ce_flags(stage);\n+\tce->ce_namelen = len;\n+\tce->ce_mode = create_ce_mode(mode);\n+\n+\treturn ce;\n+}\n+\n /*\n * Chmod an index entry with either +x or -x.\n *\n@@ -1270,7 +1302,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n {\n \tstruct stat st;\n \tstruct cache_entry *updated;\n-\tint changed, size;\n+\tint changed;\n \tint refresh = options & CE_MATCH_REFRESH;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n \tint ignore_skip_worktree = options & CE_MATCH_IGNORE_SKIP_WORKTREE;\n@@ -1350,8 +1382,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t\treturn NULL;\n \t}\n \n-\tsize = ce_size(ce);\n-\tupdated = xmalloc(size);\n+\tupdated = make_empty_cache_entry(istate, ce_namelen(ce));\n \tcopy_cache_entry(updated, ce);\n \tmemcpy(updated->name, ce->name, ce->ce_namelen + 1);\n \tfill_stat_cache_info(updated, &st);\n@@ -1637,12 +1668,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+\t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1684,7 +1716,8 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+static struct cache_entry *create_from_disk(struct index_state *istate,\n+\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n {\n@@ -1715,13 +1748,13 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1853,7 +1886,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1959,7 +1992,7 @@ int discard_index(struct index_state *istate)\n \t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n \t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n \t\t\tcontinue;\n-\t\tfree(istate->cache[i]);\n+\t\tdiscard_cache_entry(istate->cache[i]);\n \t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2649,14 +2682,13 @@ int read_index_unmerged(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct cache_entry *new_ce;\n-\t\tint size, len;\n+\t\tint len;\n \n \t\tif (!ce_stage(ce))\n \t\t\tcontinue;\n \t\tunmerged = 1;\n \t\tlen = ce_namelen(ce);\n-\t\tsize = cache_entry_size(len);\n-\t\tnew_ce = xcalloc(1, size);\n+\t\tnew_ce = make_empty_cache_entry(istate, len);\n \t\tmemcpy(new_ce->name, ce->name, len);\n \t\tnew_ce->ce_flags = create_ce_flags(0) | CE_CONFLICTED;\n \t\tnew_ce->ce_namelen = len;\n@@ -2765,3 +2797,8 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tdst->untracked = src->untracked;\n \tsrc->untracked = NULL;\n }\n+\n+void discard_cache_entry(struct cache_entry *ce)\n+{\n+\tfree(ce);\n+}\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 4d4e5cb6bf..c30ae5cf49 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -146,7 +146,9 @@ int unmerge_index_entry_at(struct index_state *istate, int pos)\n \t\tstruct cache_entry *nce;\n \t\tif (!ru->mode[i])\n \t\t\tcontinue;\n-\t\tnce = make_cache_entry(ru->mode[i], &ru->oid[i],\n+\t\tnce = make_cache_entry(istate,\n+\t\t\t\t ru->mode[i],\n+\t\t\t\t &ru->oid[i],\n \t\t\t\t name, i + 1, 0);\n \t\tif (matched)\n \t\t\tnce->ce_flags |= CE_MATCHED;\ndiff --git a/split-index.c b/split-index.c\nindex 660c75f31f..317900db8b 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -123,7 +123,7 @@ static void replace_entry(size_t pos, void *data)\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n \tsrc->ce_namelen = dst->ce_namelen;\n \tcopy_cache_entry(dst, src);\n-\tfree(src);\n+\tdiscard_cache_entry(src);\n \tsi->nr_replacements++;\n }\n \n@@ -224,7 +224,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\tbase->ce_flags = base_flags;\n \t\t\tif (ret)\n \t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n-\t\t\tfree(base);\n+\t\t\tdiscard_cache_entry(base);\n \t\t\tsi->base->cache[ce->index - 1] = ce;\n \t\t}\n \t\tfor (i = 0; i < si->base->cache_nr; i++) {\n@@ -301,7 +301,7 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \t ce == istate->split_index->base->cache[ce->index - 1])\n \t\tce->ce_flags |= CE_REMOVE;\n \telse\n-\t\tfree(ce);\n+\t\tdiscard_cache_entry(ce);\n }\n \n void replace_index_entry_in_base(struct index_state *istate,\n@@ -314,7 +314,7 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t old_entry->index <= istate->split_index->base->cache_nr) {\n \t\tnew_entry->index = old_entry->index;\n \t\tif (old_entry != istate->split_index->base->cache[new_entry->index - 1])\n-\t\t\tfree(istate->split_index->base->cache[new_entry->index - 1]);\n+\t\t\tdiscard_cache_entry(istate->split_index->base->cache[new_entry->index - 1]);\n \t\tistate->split_index->base->cache[new_entry->index - 1] = new_entry;\n \t}\n }\ndiff --git a/tree.c b/tree.c\nindex 2c9c49725c..7bbf304f9f 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -17,15 +17,13 @@ static int read_one_entry_opt(struct index_state *istate,\n \t\t\t unsigned mode, int stage, int opt)\n {\n \tint len;\n-\tunsigned int size;\n \tstruct cache_entry *ce;\n \n \tif (S_ISDIR(mode))\n \t\treturn READ_TREE_RECURSIVE;\n \n \tlen = strlen(pathname);\n-\tsize = cache_entry_size(baselen + len);\n-\tce = xcalloc(1, size);\n+\tce = make_empty_cache_entry(istate, baselen + len);\n \n \tce->ce_mode = create_ce_mode(mode);\n \tce->ce_flags = create_ce_flags(stage);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3a85a02a77..33cba550b0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,10 +203,10 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce)\n+static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n {\n \tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = xmalloc(size);\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n \n \tmemcpy(new_entry, ce, size);\n \treturn new_entry;\n@@ -216,7 +216,7 @@ static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce), set, clear);\n+\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -797,10 +797,17 @@ static int ce_in_traverse_path(const struct cache_entry *ce,\n \treturn (info->pathlen < ce_namelen(ce));\n }\n \n-static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage)\n+static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n+\tconst struct name_entry *n,\n+\tint stage,\n+\tstruct index_state *istate,\n+\tint is_transient)\n {\n \tint len = traverse_path_len(info, n);\n-\tstruct cache_entry *ce = xcalloc(1, cache_entry_size(len));\n+\tstruct cache_entry *ce =\n+\t\tis_transient ?\n+\t\tmake_empty_transient_cache_entry(len) :\n+\t\tmake_empty_cache_entry(istate, len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -846,7 +853,15 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\tstage = 3;\n \t\telse\n \t\t\tstage = 2;\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage);\n+\n+\t\t/*\n+\t\t * If the merge bit is set, then the cache entries are\n+\t\t * discarded in the following block. In this case,\n+\t\t * construct \"transient\" cache_entries, as they are\n+\t\t * not stored in the index. otherwise construct the\n+\t\t * cache entry from the index aware logic.\n+\t\t */\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n \t}\n \n \tif (o->merge) {\n@@ -855,7 +870,7 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\tfor (i = 0; i < n; i++) {\n \t\t\tstruct cache_entry *ce = src[i + o->merge];\n \t\t\tif (ce != o->df_conflict_entry)\n-\t\t\t\tfree(ce);\n+\t\t\t\tdiscard_cache_entry(ce);\n \t\t}\n \t\treturn rc;\n \t}\n@@ -1787,7 +1802,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce);\n+\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n@@ -1807,7 +1822,7 @@ static int merged_entry(const struct cache_entry *ce,\n \n \t\tif (verify_absent(merge,\n \t\t\t\t ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\tfree(merge);\n+\t\t\tdiscard_cache_entry(merge);\n \t\t\treturn -1;\n \t\t}\n \t\tinvalidate_ce_path(merge, o);\n@@ -1833,7 +1848,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tupdate = 0;\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o)) {\n-\t\t\t\tfree(merge);\n+\t\t\t\tdiscard_cache_entry(merge);\n \t\t\t\treturn -1;\n \t\t\t}\n \t\t\t/* Migrate old flags over */\n-- \n2.17.1\n\n"},{"id":"351546","messageId":"20180702193745.8940-5-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 4/8] mem-pool: only search head block for available space","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:33Z","receivedAt":"2018-07-02T19:49:43Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Instead of searching all memory blocks for available space to fulfill\na memory request, only search the head block. If the head block does\nnot have space, assume that previous block would most likely not be\nable to fulfill request either. This could potentially lead to more\nmemory fragmentation, but also avoids searching memory blocks that\nprobably will not be able to fulfill request.\n\nThis pattern will benefit consumers that are able to generate a good\nestimate for how much memory will be needed, or if they are performing\nfixed sized allocations, so that once a block is exhausted it will\nnever be able to fulfill a future request.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 389d7af447..c80124f1fe 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -21,16 +21,16 @@ static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t b\n \n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n-\tstruct mp_block *p;\n+\tstruct mp_block *p = NULL;\n \tvoid *r;\n \n \t/* round up to a 'uintmax_t' alignment */\n \tif (len & (sizeof(uintmax_t) - 1))\n \t\tlen += sizeof(uintmax_t) - (len & (sizeof(uintmax_t) - 1));\n \n-\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n-\t\tif (p->end - p->next_free >= len)\n-\t\t\tbreak;\n+\tif (mem_pool->mp_block &&\n+\t mem_pool->mp_block->end - mem_pool->mp_block->next_free >= len)\n+\t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n \t\tif (len >= (mem_pool->block_alloc / 2)) {\n-- \n2.17.1\n\n"},{"id":"351547","messageId":"20180702193745.8940-6-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 5/8] mem-pool: add life cycle management functions","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:34Z","receivedAt":"2018-07-02T19:49:45Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add initialization and discard functions to mem_pool type. As the\nmemory allocated by mem_pool can now be freed, we also track the large\nallocations.\n\nIf the there are existing mp_blocks in the mem_poo's linked list of\nmp_blocksl, then the mp_block for a large allocation is inserted\nbehind the head block. This is because only the head mp_block is considered\nwhen searching for availble space. This results in the following\ndesirable properties:\n\n1) The mp_block allocated for the large request will not be included\nnot included in the search for available in future requests, the large\nmp_block is sized for the specific request and does not contain any\nspare space.\n\n2) The head mp_block will not bumped from considation for future\nmemory requests just because a request for a large chunk of memory\ncame in.\n\nThese changes are in preparation for a future commit that will utilize\ncreating and discarding memory pool.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++--------\n mem-pool.h | 10 +++++++++\n 2 files changed, 61 insertions(+), 8 deletions(-)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex c80124f1fe..1769400d2d 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -5,20 +5,65 @@\n #include \"cache.h\"\n #include \"mem-pool.h\"\n \n-static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc)\n+#define BLOCK_GROWTH_SIZE 1024*1024 - sizeof(struct mp_block);\n+\n+/*\n+ * Allocate a new mp_block and insert it after the block specified in\n+ * `insert_after`. If `insert_after` is NULL, then insert block at the\n+ * head of the linked list.\n+ */\n+static struct mp_block *mem_pool_alloc_block(struct mem_pool *mem_pool, size_t block_alloc, struct mp_block *insert_after)\n {\n \tstruct mp_block *p;\n \n \tmem_pool->pool_alloc += sizeof(struct mp_block) + block_alloc;\n \tp = xmalloc(st_add(sizeof(struct mp_block), block_alloc));\n-\tp->next_block = mem_pool->mp_block;\n+\n \tp->next_free = (char *)p->space;\n \tp->end = p->next_free + block_alloc;\n-\tmem_pool->mp_block = p;\n+\n+\tif (insert_after) {\n+\t\tp->next_block = insert_after->next_block;\n+\t\tinsert_after->next_block = p;\n+\t} else {\n+\t\tp->next_block = mem_pool->mp_block;\n+\t\tmem_pool->mp_block = p;\n+\t}\n \n \treturn p;\n }\n \n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n+{\n+\tstruct mem_pool *pool;\n+\n+\tif (*mem_pool)\n+\t\treturn;\n+\n+\tpool = xcalloc(1, sizeof(*pool));\n+\n+\tpool->block_alloc = BLOCK_GROWTH_SIZE;\n+\n+\tif (initial_size > 0)\n+\t\tmem_pool_alloc_block(pool, initial_size, NULL);\n+\n+\t*mem_pool = pool;\n+}\n+\n+void mem_pool_discard(struct mem_pool *mem_pool)\n+{\n+\tstruct mp_block *block, *block_to_free;\n+\n+\twhile ((block = mem_pool->mp_block))\n+\t{\n+\t\tblock_to_free = block;\n+\t\tblock = block->next_block;\n+\t\tfree(block_to_free);\n+\t}\n+\n+\tfree(mem_pool);\n+}\n+\n void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct mp_block *p = NULL;\n@@ -33,12 +78,10 @@ void *mem_pool_alloc(struct mem_pool *mem_pool, size_t len)\n \t\tp = mem_pool->mp_block;\n \n \tif (!p) {\n-\t\tif (len >= (mem_pool->block_alloc / 2)) {\n-\t\t\tmem_pool->pool_alloc += len;\n-\t\t\treturn xmalloc(len);\n-\t\t}\n+\t\tif (len >= (mem_pool->block_alloc / 2))\n+\t\t\treturn mem_pool_alloc_block(mem_pool, len, mem_pool->mp_block);\n \n-\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc);\n+\t\tp = mem_pool_alloc_block(mem_pool, mem_pool->block_alloc, NULL);\n \t}\n \n \tr = p->next_free;\ndiff --git a/mem-pool.h b/mem-pool.h\nindex 829ad58ecf..f75b3365d5 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -21,6 +21,16 @@ struct mem_pool {\n \tsize_t pool_alloc;\n };\n \n+/*\n+ * Initialize mem_pool with specified initial size.\n+ */\n+void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n+\n+/*\n+ * Discard a memory pool and free all the memory it is responsible for.\n+ */\n+void mem_pool_discard(struct mem_pool *mem_pool);\n+\n /*\n * Alloc memory from the mem_pool.\n */\n-- \n2.17.1\n\n"},{"id":"351548","messageId":"20180702193745.8940-8-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 7/8] block alloc: allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:37Z","receivedAt":"2018-07-02T19:49:48Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"When reading large indexes from disk, a portion of the time is\ndominated in malloc() calls. This can be mitigated by allocating a\nlarge block of memory and manage it ourselves via memory pools.\n\nThis change moves the cache entry allocation to be on top of memory\npools.\n\nDesign:\n\nThe index_state struct will gain a notion of an associated memory_pool\nfrom which cache_entries will be allocated from. When reading in the\nindex from disk, we have information on the number of entries and\ntheir size, which can guide us in deciding how large our initial\nmemory allocation should be. When an index is discarded, the\nassociated memory_pool will be discarded as well - so the lifetime of\na cache_entry is tied to the lifetime of the index_state that it was\nallocated for.\n\nIn the case of a Split Index, the following rules are followed. 1st,\nsome terminology is defined:\n\nTerminology:\n - 'the_index': represents the logical view of the index\n\n - 'split_index': represents the \"base\" cache entries. Read from the\n split index file.\n\n'the_index' can reference a single split_index, as well as\ncache_entries from the split_index. `the_index` will be discarded\nbefore the `split_index` is. This means that when we are allocating\ncache_entries in the presence of a split index, we need to allocate\nthe entries from the `split_index`'s memory pool. This allows us to\nfollow the pattern that `the_index` can reference cache_entries from\nthe `split_index`, and that the cache_entries will not be freed while\nthey are still being referenced.\n\nManaging transient cache_entry structs:\nCache entries are usually allocated for an index, but this is not always\nthe case. Cache entries are sometimes allocated because this is the\ntype that the existing checkout_entry function works with. Because of\nthis, the existing code needs to handle cache entries associated with an\nindex / memory pool, and those that only exist transiently. Several\nstrategies were contemplated around how to handle this:\n\nChosen approach:\nAn extra field was added to the cache_entry type to track whether the\ncache_entry was allocated from a memory pool or not. This is currently\nan int field, as there are no more available bits in the existing\nce_flags bit field. If / when more bits are needed, this new field can\nbe turned into a proper bit field.\n\nAlternatives:\n\n1) Do not include any information about how the cache_entry was\nallocated. Calling code would be responsible for tracking whether the\ncache_entry needed to be freed or not.\n Pro: No extra memory overhead to track this state\n Con: Extra complexity in callers to handle this correctly.\n\nThe extra complexity and burden to not regress this behavior in the\nfuture was more than we wanted.\n\n2) cache_entry would gain knowledge about which mem_pool allocated it\n Pro: Could (potentially) do extra logic to know when a mem_pool no\n longer had references to any cache_entry\n Con: cache_entry would grow heavier by a pointer, instead of int\n\nWe didn't see a tangible benefit to this approach\n\n3) Do not add any extra information to a cache_entry, but when freeing a\n cache entry, check if the memory exists in a region managed by existing\n mem_pools.\n Pro: No extra memory overhead to track state\n Con: Extra computation is performed when freeing cache entries\n\nWe decided tracking and iterating over known memory pool regions was\nless desirable than adding an extra field to track this stae.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 21 +++++++++\n mem-pool.c | 3 +-\n read-cache.c | 119 +++++++++++++++++++++++++++++++++++++++++--------\n split-index.c | 50 +++++++++++++++++----\n unpack-trees.c | 13 +-----\n 5 files changed, 167 insertions(+), 39 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 035a627bea..8fd89ae51f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -15,6 +15,7 @@\n #include \"path.h\"\n #include \"sha1-array.h\"\n #include \"repository.h\"\n+#include \"mem-pool.h\"\n \n #include \n typedef struct git_zstream {\n@@ -156,6 +157,7 @@ struct cache_entry {\n \tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n \tunsigned int ce_flags;\n+\tunsigned int mem_pool_allocated;\n \tunsigned int ce_namelen;\n \tunsigned int index;\t/* for link extension */\n \tstruct object_id oid;\n@@ -227,6 +229,7 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \t\t\t\t const struct cache_entry *src)\n {\n \tunsigned int state = dst->ce_flags & CE_HASHED;\n+\tint mem_pool_allocated = dst->mem_pool_allocated;\n \n \t/* Don't copy hash chain and name */\n \tmemcpy(&dst->ce_stat_data, &src->ce_stat_data,\n@@ -235,6 +238,9 @@ static inline void copy_cache_entry(struct cache_entry *dst,\n \n \t/* Restore the hash state */\n \tdst->ce_flags = (dst->ce_flags & ~CE_HASHED) | state;\n+\n+\t/* Restore the mem_pool_allocated flag */\n+\tdst->mem_pool_allocated = mem_pool_allocated;\n }\n \n static inline unsigned create_ce_flags(unsigned stage)\n@@ -328,6 +334,7 @@ struct index_state {\n \tstruct untracked_cache *untracked;\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n+\tstruct mem_pool *ce_mem_pool;\n };\n \n extern struct index_state the_index;\n@@ -373,6 +380,20 @@ struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Duplicate a cache_entry. Allocate memory for the new entry from a\n+ * memory_pool. Takes into account cache_entry fields that are meant\n+ * for managing the underlying memory allocation of the cache_entry.\n+ */\n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce, struct index_state *istate);\n+\n+/*\n+ * Validate the cache entries in the index. This is an internal\n+ * consistency check that the cache_entry structs are allocated from\n+ * the expected memory pool.\n+ */\n+void validate_cache_entries(const struct index_state *istate);\n+\n #ifndef NO_THE_INDEX_COMPATIBILITY_MACROS\n #define active_cache (the_index.cache)\n #define active_nr (the_index.cache_nr)\ndiff --git a/mem-pool.c b/mem-pool.c\nindex b250a5fe40..139617cb23 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -54,7 +54,8 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n {\n \tstruct mp_block *block, *block_to_free;\n \n-\twhile ((block = mem_pool->mp_block))\n+\tblock = mem_pool->mp_block;\n+\twhile (block)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\ndiff --git a/read-cache.c b/read-cache.c\nindex 41e4d0e67a..b07369660b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -46,6 +46,48 @@\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n \t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED | FSMONITOR_CHANGED)\n \n+\n+/*\n+ * This is an estimate of the pathname length in the index. We use\n+ * this for V4 index files to guess the un-deltafied size of the index\n+ * in memory because of pathname deltafication. This is not required\n+ * for V2/V3 index formats because their pathnames are not compressed.\n+ * If the initial amount of memory set aside is not sufficient, the\n+ * mem pool will allocate extra memory.\n+ */\n+#define CACHE_ENTRY_PATH_LENGTH 80\n+\n+static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry *ce;\n+\tce = mem_pool_alloc(mem_pool, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static inline struct cache_entry *mem_pool__ce_calloc(struct mem_pool *mem_pool, size_t len)\n+{\n+\tstruct cache_entry * ce;\n+\tce = mem_pool_calloc(mem_pool, 1, cache_entry_size(len));\n+\tce->mem_pool_allocated = 1;\n+\treturn ce;\n+}\n+\n+static struct mem_pool *find_mem_pool(struct index_state *istate)\n+{\n+\tstruct mem_pool **pool_ptr;\n+\n+\tif (istate->split_index && istate->split_index->base)\n+\t\tpool_ptr = &istate->split_index->base->ce_mem_pool;\n+\telse\n+\t\tpool_ptr = &istate->ce_mem_pool;\n+\n+\tif (!*pool_ptr)\n+\t\tmem_pool_init(pool_ptr, 0);\n+\n+\treturn *pool_ptr;\n+}\n+\n struct index_state the_index;\n static const char *alternate_index_output;\n \n@@ -746,7 +788,7 @@ int add_file_to_index(struct index_state *istate, const char *path, int flags)\n \n struct cache_entry *make_empty_cache_entry(struct index_state *istate, size_t len)\n {\n-\treturn xcalloc(1, cache_entry_size(len));\n+\treturn mem_pool__ce_calloc(find_mem_pool(istate), len);\n }\n \n struct cache_entry *make_empty_transient_cache_entry(size_t len)\n@@ -1668,13 +1710,13 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n-static struct cache_entry *cache_entry_from_ondisk(struct index_state *istate,\n+static struct cache_entry *cache_entry_from_ondisk(struct mem_pool *mem_pool,\n \t\t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t\t unsigned int flags,\n \t\t\t\t\t\t const char *name,\n \t\t\t\t\t\t size_t len)\n {\n-\tstruct cache_entry *ce = make_empty_cache_entry(istate, len);\n+\tstruct cache_entry *ce = mem_pool__ce_alloc(mem_pool, len);\n \n \tce->ce_stat_data.sd_ctime.sec = get_be32(&ondisk->ctime.sec);\n \tce->ce_stat_data.sd_mtime.sec = get_be32(&ondisk->mtime.sec);\n@@ -1716,7 +1758,7 @@ static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n \treturn (const char *)ep + 1 - cp_;\n }\n \n-static struct cache_entry *create_from_disk(struct index_state *istate,\n+static struct cache_entry *create_from_disk(struct mem_pool *mem_pool,\n \t\t\t\t\t struct ondisk_cache_entry *ondisk,\n \t\t\t\t\t unsigned long *ent_size,\n \t\t\t\t\t struct strbuf *previous_name)\n@@ -1748,13 +1790,13 @@ static struct cache_entry *create_from_disk(struct index_state *istate,\n \t\t/* v3 and earlier */\n \t\tif (len == CE_NAMEMASK)\n \t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags, name, len);\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags, name, len);\n \n \t\t*ent_size = ondisk_ce_size(ce);\n \t} else {\n \t\tunsigned long consumed;\n \t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(istate, ondisk, flags,\n+\t\tce = cache_entry_from_ondisk(mem_pool, ondisk, flags,\n \t\t\t\t\t previous_name->buf,\n \t\t\t\t\t previous_name->len);\n \n@@ -1828,6 +1870,22 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_fsmonitor(istate);\n }\n \n+static size_t estimate_cache_size_from_compressed(unsigned int entries)\n+{\n+\treturn entries * (sizeof(struct cache_entry) + CACHE_ENTRY_PATH_LENGTH);\n+}\n+\n+static size_t estimate_cache_size(size_t ondisk_size, unsigned int entries)\n+{\n+\tlong per_entry = sizeof(struct cache_entry) - sizeof(struct ondisk_cache_entry);\n+\n+\t/*\n+\t * Account for potential alignment differences.\n+\t */\n+\tper_entry += align_padding_size(sizeof(struct cache_entry), -sizeof(struct ondisk_cache_entry));\n+\treturn ondisk_size + entries * per_entry;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1874,10 +1932,15 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 4) {\n \t\tprevious_name = &previous_name_buf;\n-\telse\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size_from_compressed(istate->cache_nr));\n+\t} else {\n \t\tprevious_name = NULL;\n+\t\tmem_pool_init(&istate->ce_mem_pool,\n+\t\t\t estimate_cache_size(mmap_size, istate->cache_nr));\n+\t}\n \n \tsrc_offset = sizeof(*hdr);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n@@ -1886,7 +1949,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(istate, disk_ce, &consumed, previous_name);\n+\t\tce = create_from_disk(istate->ce_mem_pool, disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1983,17 +2046,13 @@ int is_index_unborn(struct index_state *istate)\n \n int discard_index(struct index_state *istate)\n {\n-\tint i;\n+\t/*\n+\t * Cache entries in istate->cache[] should have been allocated\n+\t * from the memory pool associated with this index, or from an\n+\t * associated split_index. There is no need to free individual\n+\t * cache entries.\n+\t */\n \n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tif (istate->cache[i]->index &&\n-\t\t istate->split_index &&\n-\t\t istate->split_index->base &&\n-\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n-\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n-\t\t\tcontinue;\n-\t\tdiscard_cache_entry(istate->cache[i]);\n-\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -2007,6 +2066,12 @@ int discard_index(struct index_state *istate)\n \tdiscard_split_index(istate);\n \tfree_untracked_cache(istate->untracked);\n \tistate->untracked = NULL;\n+\n+\tif (istate->ce_mem_pool) {\n+\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tistate->ce_mem_pool = NULL;\n+\t}\n+\n \treturn 0;\n }\n \n@@ -2798,7 +2863,23 @@ void move_index_extensions(struct index_state *dst, struct index_state *src)\n \tsrc->untracked = NULL;\n }\n \n+struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n+\t\t\t\t struct index_state *istate)\n+{\n+\tunsigned int size = ce_size(ce);\n+\tint mem_pool_allocated;\n+\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n+\tmem_pool_allocated = new_entry->mem_pool_allocated;\n+\n+\tmemcpy(new_entry, ce, size);\n+\tnew_entry->mem_pool_allocated = mem_pool_allocated;\n+\treturn new_entry;\n+}\n+\n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && ce->mem_pool_allocated)\n+\t\treturn;\n+\n \tfree(ce);\n }\ndiff --git a/split-index.c b/split-index.c\nindex 317900db8b..84f067e10d 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -73,16 +73,31 @@ void move_cache_to_base_index(struct index_state *istate)\n \tint i;\n \n \t/*\n-\t * do not delete old si->base, its index entries may be shared\n-\t * with istate->cache[]. Accept a bit of leaking here because\n-\t * this code is only used by short-lived update-index.\n+\t * If there was a previous base index, then transfer ownership of allocated\n+\t * entries to the parent index.\n \t */\n+\tif (si->base &&\n+\t\tsi->base->ce_mem_pool) {\n+\n+\t\tif (!istate->ce_mem_pool)\n+\t\t\tmem_pool_init(&istate->ce_mem_pool, 0);\n+\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\t}\n+\n \tsi->base = xcalloc(1, sizeof(*si->base));\n \tsi->base->version = istate->version;\n \t/* zero timestamp disables racy test in ce_write_index() */\n \tsi->base->timestamp = istate->timestamp;\n \tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n \tsi->base->cache_nr = istate->cache_nr;\n+\n+\t/*\n+\t * The mem_pool needs to move with the allocated entries.\n+\t */\n+\tsi->base->ce_mem_pool = istate->ce_mem_pool;\n+\tistate->ce_mem_pool = NULL;\n+\n \tCOPY_ARRAY(si->base->cache, istate->cache, istate->cache_nr);\n \tmark_base_index_entries(si->base);\n \tfor (i = 0; i < si->base->cache_nr; i++)\n@@ -331,12 +346,31 @@ void remove_split_index(struct index_state *istate)\n {\n \tif (istate->split_index) {\n \t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n+\t\t * When removing the split index, we need to move\n+\t\t * ownership of the mem_pool associated with the\n+\t\t * base index to the main index. There may be cache entries\n+\t\t * allocated from the base's memory pool that are shared with\n+\t\t * the_index.cache[].\n \t\t */\n-\t\tistate->split_index = NULL;\n+\t\tmem_pool_combine(istate->ce_mem_pool, istate->split_index->base->ce_mem_pool);\n+\n+\t\t/*\n+\t\t * The split index no longer owns the mem_pool backing\n+\t\t * its cache array. As we are discarding this index,\n+\t\t * mark the index as having no cache entries, so it\n+\t\t * will not attempt to clean up the cache entries or\n+\t\t * validate them.\n+\t\t */\n+\t\tif (istate->split_index->base)\n+\t\t\tistate->split_index->base->cache_nr = 0;\n+\n+\t\t/*\n+\t\t * We can discard the split index because its\n+\t\t * memory pool has been incorporated into the\n+\t\t * memory pool associated with the the_index.\n+\t\t */\n+\t\tdiscard_split_index(istate);\n+\n \t\tistate->cache_changed |= SOMETHING_CHANGED;\n \t}\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 33cba550b0..a3b5131732 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -203,20 +203,11 @@ static int do_add_entry(struct unpack_trees_options *o, struct cache_entry *ce,\n \t\t\t ADD_CACHE_OK_TO_ADD | ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static struct cache_entry *dup_entry(const struct cache_entry *ce, struct index_state *istate)\n-{\n-\tunsigned int size = ce_size(ce);\n-\tstruct cache_entry *new_entry = make_empty_cache_entry(istate, ce_namelen(ce));\n-\n-\tmemcpy(new_entry, ce, size);\n-\treturn new_entry;\n-}\n-\n static void add_entry(struct unpack_trees_options *o,\n \t\t const struct cache_entry *ce,\n \t\t unsigned int set, unsigned int clear)\n {\n-\tdo_add_entry(o, dup_entry(ce, &o->result), set, clear);\n+\tdo_add_entry(o, dup_cache_entry(ce, &o->result), set, clear);\n }\n \n /*\n@@ -1802,7 +1793,7 @@ static int merged_entry(const struct cache_entry *ce,\n \t\t\tstruct unpack_trees_options *o)\n {\n \tint update = CE_UPDATE;\n-\tstruct cache_entry *merge = dup_entry(ce, &o->result);\n+\tstruct cache_entry *merge = dup_cache_entry(ce, &o->result);\n \n \tif (!old) {\n \t\t/*\n-- \n2.17.1\n\n"},{"id":"351549","messageId":"20180702193745.8940-9-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 8/8] block alloc: add validations around cache_entry lifecyle","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:39Z","receivedAt":"2018-07-02T19:49:51Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add an option (controlled by an environment variable) perform extra\nvalidations on mem_pool allocated cache entries. When set:\n\n 1) Invalidate cache_entry memory when discarding cache_entry.\n\n 2) When discarding index_state struct, verify that all cache_entries\n were allocated from expected mem_pool.\n\n 3) When discarding mem_pools, invalidate mem_pool memory.\n\nThis should provide extra checks that mem_pools and their allocated\ncache_entries are being used as expected.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 6 ++++++\n git.c | 3 +++\n mem-pool.c | 6 +++++-\n mem-pool.h | 2 +-\n read-cache.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++--\n 5 files changed, 68 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8fd89ae51f..b3faef3efc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -380,6 +380,12 @@ struct cache_entry *make_empty_transient_cache_entry(size_t name_len);\n */\n void discard_cache_entry(struct cache_entry *ce);\n \n+/*\n+ * Check configuration if we should perform extra validation on cache\n+ * entries.\n+ */\n+int should_validate_cache_entries(void);\n+\n /*\n * Duplicate a cache_entry. Allocate memory for the new entry from a\n * memory_pool. Takes into account cache_entry fields that are meant\ndiff --git a/git.c b/git.c\nindex 9dbe6ffaa7..c7e4f0351a 100644\n--- a/git.c\n+++ b/git.c\n@@ -414,7 +414,10 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \n \ttrace_argv_printf(argv, \"trace: built-in: git\");\n \n+\tvalidate_cache_entries(&the_index);\n \tstatus = p->fn(argc, argv, prefix);\n+\tvalidate_cache_entries(&the_index);\n+\n \tif (status)\n \t\treturn status;\n \ndiff --git a/mem-pool.c b/mem-pool.c\nindex 139617cb23..a2841a4a9a 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -50,7 +50,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size)\n \t*mem_pool = pool;\n }\n \n-void mem_pool_discard(struct mem_pool *mem_pool)\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory)\n {\n \tstruct mp_block *block, *block_to_free;\n \n@@ -59,6 +59,10 @@ void mem_pool_discard(struct mem_pool *mem_pool)\n \t{\n \t\tblock_to_free = block;\n \t\tblock = block->next_block;\n+\n+\t\tif (invalidate_memory)\n+\t\t\tmemset(block_to_free->space, 0xDD, ((char *)block_to_free->end) - ((char *)block_to_free->space));\n+\n \t\tfree(block_to_free);\n \t}\n \ndiff --git a/mem-pool.h b/mem-pool.h\nindex adeefdcb28..999d3c3a52 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -29,7 +29,7 @@ void mem_pool_init(struct mem_pool **mem_pool, size_t initial_size);\n /*\n * Discard a memory pool and free all the memory it is responsible for.\n */\n-void mem_pool_discard(struct mem_pool *mem_pool);\n+void mem_pool_discard(struct mem_pool *mem_pool, int invalidate_memory);\n \n /*\n * Alloc memory from the mem_pool.\ndiff --git a/read-cache.c b/read-cache.c\nindex b07369660b..fd67e2e8a4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2050,8 +2050,10 @@ int discard_index(struct index_state *istate)\n \t * Cache entries in istate->cache[] should have been allocated\n \t * from the memory pool associated with this index, or from an\n \t * associated split_index. There is no need to free individual\n-\t * cache entries.\n+\t * cache entries. validate_cache_entries can detect when this\n+\t * assertion does not hold.\n \t */\n+\tvalidate_cache_entries(istate);\n \n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n@@ -2068,13 +2070,45 @@ int discard_index(struct index_state *istate)\n \tistate->untracked = NULL;\n \n \tif (istate->ce_mem_pool) {\n-\t\tmem_pool_discard(istate->ce_mem_pool);\n+\t\tmem_pool_discard(istate->ce_mem_pool, should_validate_cache_entries());\n \t\tistate->ce_mem_pool = NULL;\n \t}\n \n \treturn 0;\n }\n \n+/*\n+ * Validate the cache entries of this index.\n+ * All cache entries associated with this index\n+ * should have been allocated by the memory pool\n+ * associated with this index, or by a referenced\n+ * split index.\n+ */\n+void validate_cache_entries(const struct index_state *istate)\n+{\n+\tint i;\n+\n+\tif (!should_validate_cache_entries() ||!istate || !istate->initialized)\n+\t\treturn;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (!istate) {\n+\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t} else if (!istate->ce_mem_pool ||\n+\t\t\t!mem_pool_contains(istate->ce_mem_pool, istate->cache[i])) {\n+\t\t\tif (!istate->split_index ||\n+\t\t\t\t!istate->split_index->base ||\n+\t\t\t\t!istate->split_index->base->ce_mem_pool ||\n+\t\t\t\t!mem_pool_contains(istate->split_index->base->ce_mem_pool, istate->cache[i])) {\n+\t\t\t\tdie(\"internal error: cache entry is not allocated from expected memory pool\");\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (istate->split_index)\n+\t\tvalidate_cache_entries(istate->split_index->base);\n+}\n+\n int unmerged_index(const struct index_state *istate)\n {\n \tint i;\n@@ -2878,8 +2912,25 @@ struct cache_entry *dup_cache_entry(const struct cache_entry *ce,\n \n void discard_cache_entry(struct cache_entry *ce)\n {\n+\tif (ce && should_validate_cache_entries())\n+\t\tmemset(ce, 0xCD, cache_entry_size(ce->ce_namelen));\n+\n \tif (ce && ce->mem_pool_allocated)\n \t\treturn;\n \n \tfree(ce);\n }\n+\n+int should_validate_cache_entries(void)\n+{\n+\tstatic int validate_index_cache_entries = -1;\n+\n+\tif (validate_index_cache_entries < 0) {\n+\t\tif (getenv(\"GIT_TEST_VALIDATE_INDEX_CACHE_ENTRIES\"))\n+\t\t\tvalidate_index_cache_entries = 1;\n+\t\telse\n+\t\t\tvalidate_index_cache_entries = 0;\n+\t}\n+\n+\treturn validate_index_cache_entries;\n+}\n-- \n2.17.1\n\n"},{"id":"351551","messageId":"20180702193745.8940-7-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 6/8] mem-pool: fill out functionality","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:35Z","receivedAt":"2018-07-02T19:50:00Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Add functions for:\n\n - combining two memory pools\n\n - determining if a memory address is within the range managed by a\n memory pool\n\nThese functions will be used by future commits.\n\nSigned-off-by: Jameson Miller \n---\n mem-pool.c | 42 ++++++++++++++++++++++++++++++++++++++++++\n mem-pool.h | 13 +++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/mem-pool.c b/mem-pool.c\nindex 1769400d2d..b250a5fe40 100644\n--- a/mem-pool.c\n+++ b/mem-pool.c\n@@ -96,3 +96,45 @@ void *mem_pool_calloc(struct mem_pool *mem_pool, size_t count, size_t size)\n \tmemset(r, 0, len);\n \treturn r;\n }\n+\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Check if memory is allocated in a block */\n+\tfor (p = mem_pool->mp_block; p; p = p->next_block)\n+\t\tif ((mem >= ((void *)p->space)) &&\n+\t\t (mem < ((void *)p->end)))\n+\t\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src)\n+{\n+\tstruct mp_block *p;\n+\n+\t/* Append the blocks from src to dst */\n+\tif (dst->mp_block && src->mp_block) {\n+\t\t/*\n+\t\t * src and dst have blocks, append\n+\t\t * blocks from src to dst.\n+\t\t */\n+\t\tp = dst->mp_block;\n+\t\twhile (p->next_block)\n+\t\t\tp = p->next_block;\n+\n+\t\tp->next_block = src->mp_block;\n+\t} else if (src->mp_block) {\n+\t\t/*\n+\t\t * src has blocks, dst is empty.\n+\t\t */\n+\t\tdst->mp_block = src->mp_block;\n+\t} else {\n+\t\t/* src is empty, nothing to do. */\n+\t}\n+\n+\tdst->pool_alloc += src->pool_alloc;\n+\tsrc->pool_alloc = 0;\n+\tsrc->mp_block = NULL;\n+}\ndiff --git a/mem-pool.h b/mem-pool.h\nindex f75b3365d5..adeefdcb28 100644\n--- a/mem-pool.h\n+++ b/mem-pool.h\n@@ -41,4 +41,17 @@ void *mem_pool_alloc(struct mem_pool *pool, size_t len);\n */\n void *mem_pool_calloc(struct mem_pool *pool, size_t count, size_t size);\n \n+/*\n+ * Move the memory associated with the 'src' pool to the 'dst' pool. The 'src'\n+ * pool will be empty and not contain any memory. It still needs to be free'd\n+ * with a call to `mem_pool_discard`.\n+ */\n+void mem_pool_combine(struct mem_pool *dst, struct mem_pool *src);\n+\n+/*\n+ * Check if a memory pointed at by 'mem' is part of the range of\n+ * memory managed by the specified mem_pool.\n+ */\n+int mem_pool_contains(struct mem_pool *mem_pool, void *mem);\n+\n #endif\n-- \n2.17.1\n\n"},{"id":"351552","messageId":"20180702193745.8940-2-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180702193745.8940-1-jamill@microsoft.com","subject":"[PATCH v6 1/8] read-cache: teach refresh_cache_entry to take istate","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:29Z","receivedAt":"2018-07-02T19:50:02Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Refactor refresh_cache_entry() to work on a specific index, instead of\nimplicitly using the_index. This is in preparation for making the\nmake_cache_entry function apply to a specific index.\n\nSigned-off-by: Jameson Miller \n---\n cache.h | 2 +-\n merge-recursive.c | 2 +-\n read-cache.c | 9 +++++----\n 3 files changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d49092d94d..93af25f586 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -751,7 +751,7 @@ extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n #define REFRESH_IGNORE_SUBMODULES\t0x0010\t/* ignore submodules */\n #define REFRESH_IN_PORCELAIN\t0x0020\t/* user friendly output, not \"needs update\" */\n extern int refresh_index(struct index_state *, unsigned int flags, const struct pathspec *pathspec, char *seen, const char *header_msg);\n-extern struct cache_entry *refresh_cache_entry(struct cache_entry *, unsigned int);\n+extern struct cache_entry *refresh_cache_entry(struct index_state *, struct cache_entry *, unsigned int);\n \n /*\n * Opportunistically update the index but do not complain if we can't.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex bed4a5be02..8b3d6781c7 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -326,7 +326,7 @@ static int add_cacheinfo(struct merge_options *o,\n \tif (refresh) {\n \t\tstruct cache_entry *nce;\n \n-\t\tnce = refresh_cache_entry(ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n+\t\tnce = refresh_cache_entry(&the_index, ce, CE_MATCH_REFRESH | CE_MATCH_IGNORE_MISSING);\n \t\tif (!nce)\n \t\t\treturn err(o, _(\"add_cacheinfo failed to refresh for path '%s'; merge aborting.\"), path);\n \t\tif (nce != ce)\ndiff --git a/read-cache.c b/read-cache.c\nindex 372588260e..fa8366ecab 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -767,7 +767,7 @@ struct cache_entry *make_cache_entry(unsigned int mode,\n \tce->ce_namelen = len;\n \tce->ce_mode = create_ce_mode(mode);\n \n-\tret = refresh_cache_entry(ce, refresh_options);\n+\tret = refresh_cache_entry(&the_index, ce, refresh_options);\n \tif (ret != ce)\n \t\tfree(ce);\n \treturn ret;\n@@ -1473,10 +1473,11 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \treturn has_errors;\n }\n \n-struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n-\t\t\t\t\t unsigned int options)\n+struct cache_entry *refresh_cache_entry(struct index_state *istate,\n+\t\t\t\t\tstruct cache_entry *ce,\n+\t\t\t\t\tunsigned int options)\n {\n-\treturn refresh_cache_ent(&the_index, ce, options, NULL, NULL);\n+\treturn refresh_cache_ent(istate, ce, options, NULL, NULL);\n }\n \n \n-- \n2.17.1\n\n"},{"id":"351553","messageId":"20180702193745.8940-1-jamill@microsoft.com","threadId":"48310","inReplyTo":"20180628135932.225288-1-jamill@microsoft.com","subject":"[PATCH v6 0/8] Allocate cache entries from mem_pool","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-02T19:49:27Z","receivedAt":"2018-07-02T19:50:02Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"Changes since v5:\n - Updated commit messages for:\n 7581156e8a read-cache: teach refresh_cache_entry to take istate\n 2c962aa0cd block alloc: add lifecycle APIs for cache_entry structs\n\n - Use oidcpy function instead of hashcpy function\n\n### Interdiff (v5..v6):\n\ndiff --git a/read-cache.c b/read-cache.c\nindex ba31e929e8..fd67e2e8a4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -814,7 +814,7 @@ struct cache_entry *make_cache_entry(struct index_state *istate,\n \tlen = strlen(path);\n \tce = make_empty_cache_entry(istate, len);\n \n-\thashcpy(ce->oid.hash, oid->hash);\n+\toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(stage);\n \tce->ce_namelen = len;\n@@ -840,7 +840,7 @@ struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct o\n \tlen = strlen(path);\n \tce = make_empty_transient_cache_entry(len);\n \n-\thashcpy(ce->oid.hash, oid->hash);\n+\toidcpy(&ce->oid, oid);\n \tmemcpy(ce->name, path, len);\n \tce->ce_flags = create_ce_flags(stage);\n \tce->ce_namelen = len;\n\n\n### Patches\n\nJameson Miller (8):\n read-cache: teach refresh_cache_entry to take istate\n read-cache: teach make_cache_entry to take object_id\n block alloc: add lifecycle APIs for cache_entry structs\n mem-pool: only search head block for available space\n mem-pool: add life cycle management functions\n mem-pool: fill out functionality\n block alloc: allocate cache entries from mem_pool\n block alloc: add validations around cache_entry lifecyle\n\n apply.c | 24 ++--\n blame.c | 5 +-\n builtin/checkout.c | 8 +-\n builtin/difftool.c | 6 +-\n builtin/reset.c | 2 +-\n builtin/update-index.c | 26 ++--\n cache.h | 64 +++++++++-\n git.c | 3 +\n mem-pool.c | 114 ++++++++++++++++--\n mem-pool.h | 23 ++++\n merge-recursive.c | 4 +-\n read-cache.c | 264 ++++++++++++++++++++++++++++++++++-------\n resolve-undo.c | 4 +-\n split-index.c | 58 +++++++--\n tree.c | 4 +-\n unpack-trees.c | 40 ++++---\n 16 files changed, 515 insertions(+), 134 deletions(-)\n\n\nbase-commit: ed843436dd4924c10669820cc73daf50f0b4dabd\n-- \n2.17.1\n\n\n"},{"id":"351567","messageId":"CAGZ79kYDyGg8z5FiYNXWAr+wT3r6Qm2Y_hh=o7xEKRkonO8Jaw@mail.gmail.com","threadId":"48310","inReplyTo":"20180702193745.8940-3-jamill@microsoft.com","subject":"Re: [PATCH v6 2/8] read-cache: teach make_cache_entry to take object_id","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-07-02T21:23:02Z","receivedAt":"2018-07-02T21:23:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jul 2, 2018 at 12:49 PM Jameson Miller wrote:\n>\n> Teach make_cache_entry function to take object_id instead of a SHA-1.\n\nThis repeats the subject line?\n\nSign off missing.\n"},{"id":"351707","messageId":"DM5PR21MB0780A4D3EFECC8FAE2A031DACE400@DM5PR21MB0780.namprd21.prod.outlook.com","threadId":"48310","inReplyTo":"CAGZ79kYDyGg8z5FiYNXWAr+wT3r6Qm2Y_hh=o7xEKRkonO8Jaw@mail.gmail.com","subject":"RE: [PATCH v6 2/8] read-cache: teach make_cache_entry to take object_id","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-07-05T15:20:58Z","receivedAt":"2018-07-05T15:22:57Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"> >\n> > Teach make_cache_entry function to take object_id instead of a SHA-1.\n> \n> This repeats the subject line?\n> \n> Sign off missing.\n\nThank you Stefan for pointing this out. This does not add much information to the subject line. I could clean it up if I re-roll to fix up the missing sign off.\n\nJunio - would you like me to re-roll this patch series and include the correct sign off, or would you be able to correct this? I can do whichever is easiest for you.\n"},{"id":"353305","messageId":"CACsJy8DfK=QaNsYRNBH8r0JJERAK0ppn4v6jZK03T22VfZA9Aw@mail.gmail.com","threadId":"48310","inReplyTo":"20180702193745.8940-4-jamill@microsoft.com","subject":"Re: [PATCH v6 3/8] block alloc: add lifecycle APIs for cache_entry structs","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-07-22T09:23:23Z","receivedAt":"2018-07-22T09:23:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 2, 2018 at 9:49 PM Jameson Miller wrote:\n> +struct cache_entry *make_transient_cache_entry(unsigned int mode, const struct object_id *oid,\n> + const char *path, int stage)\n> +{\n> + struct cache_entry *ce;\n> + int len;\n> +\n> + if (!verify_path(path, mode)) {\n> + error(\"Invalid path '%s'\", path);\n\nPlease wrap all new user-visible strings in _().\n\n> + return NULL;\n> + }\n-- \nDuy\n"}]}