{"thread":{"id":"48394","subject":"[PATCH 00/13] object store: alloc","startedAt":"2018-05-01T21:34:12Z","lastAt":"2018-05-16T02:27:11Z","messageCount":95,"participants":["Stefan Beller","Eric Sunshine","Duy Nguyen","Jameson Miller","Jonathan Tan","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"346330","messageId":"20180501213403.14643-1-sbeller@google.com","threadId":"48394","inReplyTo":null,"subject":"[PATCH 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:50Z","receivedAt":"2018-05-01T21:34:12Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This applies on top of sb/oid-object-info and is the logical continuum of\nthe series that it builds on; this brings the object store into more of\nGits code, removing global state, such that reasoning about the state of\nthe in-memory representation of the repository is easier.\n\nMy original plan was to convert lookup_commit_graft as the next series,\nwhich would be similar to lookup_replace_object, as in sb/object-store-replace.\nThe grafts and shallow mechanism are very close to each other, such that\nthey need to be converted at the same time, both depending on the\n\"parsed object store\" that is introduced in this commit.\n\nThe next series will then convert code in {object/blob/tree/commit/tag}.c\nhopefully finishing the lookup_* functions.\n\nI also debated if it is worth converting alloc.c via this patch series\nor if it might make more sense to use the new mem-pool by Jameson[1].\n\nI vaguely wonder about the performance impact, as the object allocation\ncode seemed to be relevant in the past.\n\n[1] https://public-inbox.org/git/20180430153122.243976-1-jamill@microsoft.com/\n\nAny comments welcome,\nThanks,\nStefan\n\nJonathan Nieder (1):\n object: add repository argument to grow_object_hash\n\nStefan Beller (12):\n repository: introduce object parser field\n object: add repository argument to create_object\n alloc: add repository argument to alloc_blob_node\n alloc: add repository argument to alloc_tree_node\n alloc: add repository argument to alloc_commit_node\n alloc: add repository argument to alloc_tag_node\n alloc: add repository argument to alloc_object_node\n alloc: add repository argument to alloc_report\n alloc: add repository argument to alloc_commit_index\n object: allow grow_object_hash to handle arbitrary repositories\n object: allow create_object to handle arbitrary repositories\n alloc: allow arbitrary repositories for alloc functions\n\n alloc.c | 69 +++++++++++++++++++++++------------\n alloc.h | 21 +++++++++++\n blame.c | 3 +-\n blob.c | 5 ++-\n cache.h | 9 -----\n commit.c | 4 +-\n merge-recursive.c | 3 +-\n object.c | 93 +++++++++++++++++++++++++++++++++--------------\n object.h | 18 ++++++++-\n repository.c | 7 ++++\n repository.h | 11 +++++-\n tag.c | 4 +-\n tree.c | 4 +-\n 13 files changed, 182 insertions(+), 69 deletions(-)\n create mode 100644 alloc.h\n\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346331","messageId":"20180501213403.14643-2-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 01/13] repository: introduce object parser field","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:51Z","receivedAt":"2018-05-01T21:34:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Git's object access code can be thought of as containing two layers:\nthe raw object store provides access to raw object content, while the\nhigher level obj_hash code parses raw objects and keeps track of\nparenthood and other object relationships using 'struct object'.\nKeeping these layers separate should make it easier to find relevant\nfunctions and to change the implementation of one without having to\ntouch the other.\n\nAdd an object_parser field to 'struct repository' to prepare obj_hash\nto be handled per repository. Callers still only use the_repository\nfor now --- later patches will adapt them to handle arbitrary\nrepositories.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 63 +++++++++++++++++++++++++++++++++-------------------\n object.h | 8 +++++++\n repository.c | 7 ++++++\n repository.h | 11 ++++++++-\n 4 files changed, 65 insertions(+), 24 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 5044d08e96c..503c73e2d1f 100644\n--- a/object.c\n+++ b/object.c\n@@ -8,17 +8,14 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n-static struct object **obj_hash;\n-static int nr_objs, obj_hash_size;\n-\n unsigned int get_max_object_index(void)\n {\n-\treturn obj_hash_size;\n+\treturn the_repository->parsed_objects->obj_hash_size;\n }\n \n struct object *get_indexed_object(unsigned int idx)\n {\n-\treturn obj_hash[idx];\n+\treturn the_repository->parsed_objects->obj_hash[idx];\n }\n \n static const char *object_type_strings[] = {\n@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)\n \tunsigned int i, first;\n \tstruct object *obj;\n \n-\tif (!obj_hash)\n+\tif (!the_repository->parsed_objects->obj_hash)\n \t\treturn NULL;\n \n-\tfirst = i = hash_obj(sha1, obj_hash_size);\n-\twhile ((obj = obj_hash[i]) != NULL) {\n+\tfirst = i = hash_obj(sha1,\n+\t\t\t the_repository->parsed_objects->obj_hash_size);\n+\twhile ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {\n \t\tif (!hashcmp(sha1, obj->oid.hash))\n \t\t\tbreak;\n \t\ti++;\n-\t\tif (i == obj_hash_size)\n+\t\tif (i == the_repository->parsed_objects->obj_hash_size)\n \t\t\ti = 0;\n \t}\n \tif (obj && i != first) {\n@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)\n \t\t * that we do not need to walk the hash table the next\n \t\t * time we look for it.\n \t\t */\n-\t\tSWAP(obj_hash[i], obj_hash[first]);\n+\t\tSWAP(the_repository->parsed_objects->obj_hash[i],\n+\t\t the_repository->parsed_objects->obj_hash[first]);\n \t}\n \treturn obj;\n }\n@@ -124,19 +123,19 @@ static void grow_object_hash(void)\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;\n+\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(obj_hash);\n-\tobj_hash = new_hash;\n-\tobj_hash_size = new_hash_size;\n+\tfree(the_repository->parsed_objects->obj_hash);\n+\tthe_repository->parsed_objects->obj_hash = new_hash;\n+\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object(const unsigned char *sha1, void *o)\n@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (obj_hash_size - 1 <= nr_objs * 2)\n+\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n \t\tgrow_object_hash();\n \n-\tinsert_obj_hash(obj, obj_hash, obj_hash_size);\n-\tnr_objs++;\n+\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n+\t\t\tthe_repository->parsed_objects->obj_hash_size);\n+\tthe_repository->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \n@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)\n {\n \tint i;\n \n-\tfor (i=0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj)\n \t\t\tobj->flags &= ~flags;\n \t}\n@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)\n {\n \tint i;\n \n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj && obj->type == OBJ_COMMIT)\n \t\t\tobj->flags &= ~flags;\n \t}\n }\n \n+struct object_parser *object_parser_new(void)\n+{\n+\tstruct object_parser *o = xmalloc(sizeof(*o));\n+\tmemset(o, 0, sizeof(*o));\n+\treturn o;\n+}\n+\n struct raw_object_store *raw_object_store_new(void)\n {\n \tstruct raw_object_store *o = xmalloc(sizeof(*o));\n@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)\n \tclose_all_packs(o);\n \to->packed_git = NULL;\n }\n+\n+void object_parser_clear(struct object_parser *o)\n+{\n+\t/*\n+\t * TOOD free objects in o->obj_hash.\n+\t *\n+\t * As objects are allocated in slabs (see alloc.c), we do\n+\t * not need to free each object, but each slab instead.\n+\t */\n+}\ndiff --git a/object.h b/object.h\nindex f13f85b2a94..84380b2b4d5 100644\n--- a/object.h\n+++ b/object.h\n@@ -1,6 +1,14 @@\n #ifndef OBJECT_H\n #define OBJECT_H\n \n+struct object_parser {\n+\tstruct object **obj_hash;\n+\tint nr_objs, obj_hash_size;\n+};\n+\n+struct object_parser *object_parser_new(void);\n+void object_parser_clear(struct object_parser *o);\n+\n struct object_list {\n \tstruct object *item;\n \tstruct object_list *next;\ndiff --git a/repository.c b/repository.c\nindex a4848c1bd05..208ee10071c 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"config.h\"\n+#include \"object.h\"\n #include \"submodule-config.h\"\n \n /* The main repository */\n@@ -14,6 +15,8 @@ void initialize_the_repository(void)\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n+\tthe_repo.parsed_objects = object_parser_new();\n+\n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n \n@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n+\trepo->parsed_objects = object_parser_new();\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\n@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)\n \traw_object_store_clear(repo->objects);\n \tFREE_AND_NULL(repo->objects);\n \n+\tobject_parser_clear(repo->parsed_objects);\n+\tFREE_AND_NULL(repo->parsed_objects);\n+\n \tif (repo->config) {\n \t\tgit_configset_clear(repo->config);\n \t\tFREE_AND_NULL(repo->config);\ndiff --git a/repository.h b/repository.h\nindex e6e00f541bd..8d042e0fa11 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -22,10 +22,19 @@ struct repository {\n \tchar *commondir;\n \n \t/*\n-\t * Holds any information related to accessing the raw object content.\n+\t * Holds any information needed to retrieve the raw content\n+\t * of objects. The object_parser uses this to get object\n+\t * content which it then parses.\n \t */\n \tstruct raw_object_store *objects;\n \n+\t/*\n+\t * State for the object parser. This owns all parsed objects\n+\t * (struct object) so callers do not have to manage their\n+\t * lifetime.\n+\t */\n+\tstruct object_parser *parsed_objects;\n+\n \t/* The store in which the refs are held. */\n \tstruct ref_store *refs;\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346332","messageId":"20180501213403.14643-3-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 02/13] object: add repository argument to create_object","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:52Z","receivedAt":"2018-05-01T21:34:18Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Add a repository argument to allow the callers of create_object\nto be more specific about which repository to act on. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nAs with the previous commits, use a macro to catch callers passing a\nrepository other than the_repository at compile time.\n\nAdd the cocci patch that converted the callers.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n blob.c | 4 +++-\n commit.c | 3 ++-\n object.c | 5 +++--\n object.h | 3 ++-\n tag.c | 3 ++-\n tree.c | 3 ++-\n 6 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/blob.c b/blob.c\nindex fa2ab4f7a74..85c2143f299 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"blob.h\"\n+#include \"repository.h\"\n \n const char *blob_type = \"blob\";\n \n@@ -7,7 +8,8 @@ struct blob *lookup_blob(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_blob_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_blob_node());\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/commit.c b/commit.c\nindex ca474a7c112..9106acf0aad 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -50,7 +50,8 @@ struct commit *lookup_commit(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_commit_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_commit_node());\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/object.c b/object.c\nindex 503c73e2d1f..933921e35c9 100644\n--- a/object.c\n+++ b/object.c\n@@ -138,7 +138,7 @@ static void grow_object_hash(void)\n \tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object(const unsigned char *sha1, void *o)\n+void *create_object_the_repository(const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -178,7 +178,8 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n-\t\tobj = create_object(sha1, alloc_object_node());\n+\t\tobj = create_object(the_repository, sha1,\n+\t\t\t\t alloc_object_node());\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex 84380b2b4d5..d1869dbc502 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,7 +93,8 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-extern void *create_object(const unsigned char *sha1, void *obj);\n+#define create_object(r, s, o) create_object_##r(s, o)\n+extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \ndiff --git a/tag.c b/tag.c\nindex 3d37c1bd251..7150b759d66 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -93,7 +93,8 @@ struct tag *lookup_tag(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tag_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tag_node());\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \ndiff --git a/tree.c b/tree.c\nindex 1c68ea586bd..63730e3fb46 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -196,7 +196,8 @@ struct tree *lookup_tree(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tree_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tree_node());\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346333","messageId":"20180501213403.14643-6-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 05/13] alloc: add repository argument to alloc_tree_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:55Z","receivedAt":"2018-05-01T21:34:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tree.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 6c5c376a25a..2c8d1430758 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -57,7 +57,7 @@ void *alloc_blob_node_the_repository(void)\n }\n \n static struct alloc_state tree_state;\n-void *alloc_tree_node(void)\n+void *alloc_tree_node_the_repository(void)\n {\n \tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\ndiff --git a/cache.h b/cache.h\nindex 2258e611275..1717d07a2c5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1766,7 +1766,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n /* alloc.c */\n #define alloc_blob_node(r) alloc_blob_node_##r()\n extern void *alloc_blob_node_the_repository(void);\n-extern void *alloc_tree_node(void);\n+#define alloc_tree_node(r) alloc_tree_node_##r()\n+extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\ndiff --git a/tree.c b/tree.c\nindex 63730e3fb46..58cf19b4fa8 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -197,7 +197,7 @@ struct tree *lookup_tree(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tree_node());\n+\t\t\t\t alloc_tree_node(the_repository));\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346334","messageId":"20180501213403.14643-8-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 07/13] alloc: add repository argument to alloc_tag_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:57Z","receivedAt":"2018-05-01T21:34:22Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tag.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 9e2b897ec1d..290250e3595 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -65,7 +65,7 @@ void *alloc_tree_node_the_repository(void)\n }\n \n static struct alloc_state tag_state;\n-void *alloc_tag_node(void)\n+void *alloc_tag_node_the_repository(void)\n {\n \tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\ndiff --git a/cache.h b/cache.h\nindex bf6e8c87d83..32f340cde59 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1770,7 +1770,8 @@ extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node_the_repository(void);\n #define alloc_commit_node(r) alloc_commit_node_##r()\n extern void *alloc_commit_node_the_repository(void);\n-extern void *alloc_tag_node(void);\n+#define alloc_tag_node(r) alloc_tag_node_##r()\n+extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\ndiff --git a/tag.c b/tag.c\nindex 7150b759d66..02ef4eaafc0 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -94,7 +94,7 @@ struct tag *lookup_tag(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tag_node());\n+\t\t\t\t alloc_tag_node(the_repository));\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346335","messageId":"20180501213403.14643-10-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 09/13] alloc: add repository argument to alloc_report","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:59Z","receivedAt":"2018-05-01T21:34:26Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n 2 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex f031ce422d9..28b85b22144 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -105,7 +105,7 @@ static void report(const char *name, unsigned int count, size_t size)\n #define REPORT(name, type)\t\\\n report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n \n-void alloc_report(void)\n+void alloc_report_the_repository(void)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/cache.h b/cache.h\nindex 2d60359a964..01cc207d218 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1774,7 +1774,8 @@ extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node_the_repository(void);\n #define alloc_object_node(r) alloc_object_node_##r()\n extern void *alloc_object_node_the_repository(void);\n-extern void alloc_report(void);\n+#define alloc_report(r) alloc_report_##r()\n+extern void alloc_report_the_repository(void);\n extern unsigned int alloc_commit_index(void);\n \n /* pkt-line.c */\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346336","messageId":"20180501213403.14643-11-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 10/13] alloc: add repository argument to alloc_commit_index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:34:00Z","receivedAt":"2018-05-01T21:34:28Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 4 ++--\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 28b85b22144..277dadd221b 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -82,7 +82,7 @@ void *alloc_object_node_the_repository(void)\n \n static struct alloc_state commit_state;\n \n-unsigned int alloc_commit_index(void)\n+unsigned int alloc_commit_index_the_repository(void)\n {\n \tstatic unsigned int count;\n \treturn count++;\n@@ -92,7 +92,7 @@ void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index();\n+\tc->index = alloc_commit_index(the_repository);\n \treturn c;\n }\n \ndiff --git a/cache.h b/cache.h\nindex 01cc207d218..0e6c5dd5639 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1776,7 +1776,8 @@ extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node_the_repository(void);\n #define alloc_report(r) alloc_report_##r()\n extern void alloc_report_the_repository(void);\n-extern unsigned int alloc_commit_index(void);\n+#define alloc_commit_index(r) alloc_commit_index_##r()\n+extern unsigned int alloc_commit_index_the_repository(void);\n \n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\ndiff --git a/object.c b/object.c\nindex 7d36323445b..ddf4b7b196e 100644\n--- a/object.c\n+++ b/object.c\n@@ -162,7 +162,7 @@ void *object_as_type(struct object *obj, enum object_type type, int quiet)\n \t\treturn obj;\n \telse if (obj->type == OBJ_NONE) {\n \t\tif (type == OBJ_COMMIT)\n-\t\t\t((struct commit *)obj)->index = alloc_commit_index();\n+\t\t\t((struct commit *)obj)->index = alloc_commit_index(the_repository);\n \t\tobj->type = type;\n \t\treturn obj;\n \t}\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346337","messageId":"20180501213403.14643-13-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 12/13] object: allow create_object to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:34:02Z","receivedAt":"2018-05-01T21:34:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 12 ++++++------\n object.h | 3 +--\n 2 files changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 43954fadf93..fd27cf54faa 100644\n--- a/object.c\n+++ b/object.c\n@@ -139,7 +139,7 @@ static void grow_object_hash(struct repository *r)\n \tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object_the_repository(const unsigned char *sha1, void *o)\n+void *create_object(struct repository *r, const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -147,12 +147,12 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash(the_repository);\n+\tif (r->parsed_objects->obj_hash_size - 1 <= r->parsed_objects->nr_objs * 2)\n+\t\tgrow_object_hash(r);\n \n-\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n-\t\t\tthe_repository->parsed_objects->obj_hash_size);\n-\tthe_repository->parsed_objects->nr_objs++;\n+\tinsert_obj_hash(obj, r->parsed_objects->obj_hash,\n+\t\t\tr->parsed_objects->obj_hash_size);\n+\tr->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex d1869dbc502..5ef6ce1ea96 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,8 +93,7 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-#define create_object(r, s, o) create_object_##r(s, o)\n-extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n+extern void *create_object(struct repository *r, const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346338","messageId":"20180501213403.14643-14-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:34:03Z","receivedAt":"2018-05-01T21:34:31Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 69 ++++++++++++++++++++++++++++++-----------------\n alloc.h | 21 +++++++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 -----------\n commit.c | 1 +\n merge-recursive.c | 1 +\n object.c | 24 ++++++++++++++---\n object.h | 10 ++++++-\n repository.c | 4 +--\n tag.c | 1 +\n tree.c | 1 +\n 12 files changed, 104 insertions(+), 46 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..66a3d07ba2d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,10 +4,11 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n+ *\n+ * TODO: Combine this with mem-pool?\n */\n #include \"cache.h\"\n #include \"object.h\"\n@@ -30,8 +31,25 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -45,54 +63,56 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n+\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\ts->slabs[s->slab_nr++] = ret;\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+struct alloc_state the_repository_blob_state;\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+struct alloc_state the_repository_tree_state;\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+struct alloc_state the_repository_tag_state;\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+struct alloc_state the_repository_object_state;\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+struct alloc_state the_repository_commit_state;\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n@@ -103,9 +123,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..a62d7a06307\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,21 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+extern struct alloc_state the_repository_blob_state;\n+extern struct alloc_state the_repository_tree_state;\n+extern struct alloc_state the_repository_commit_state;\n+extern struct alloc_state the_repository_tag_state;\n+extern struct alloc_state the_repository_object_state;\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..c3b400d5930 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..aa086a85089 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -14,6 +14,7 @@\n #include \"tree-walk.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"alloc.h\"\n #include \"tag.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\ndiff --git a/object.c b/object.c\nindex fd27cf54faa..003ec55a291 100644\n--- a/object.c\n+++ b/object.c\n@@ -4,6 +4,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"commit.h\"\n+#include \"alloc.h\"\n #include \"tag.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n@@ -451,10 +452,24 @@ void clear_commit_marks_all(unsigned int flags)\n \t}\n }\n \n-struct object_parser *object_parser_new(void)\n+struct object_parser *object_parser_new(int is_the_repo)\n {\n \tstruct object_parser *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\tif (is_the_repo) {\n+\t\to->blob_state = &the_repository_blob_state;\n+\t\to->tree_state = &the_repository_tree_state;\n+\t\to->commit_state = &the_repository_commit_state;\n+\t\to->tag_state = &the_repository_tag_state;\n+\t\to->object_state = &the_repository_object_state;\n+\t} else {\n+\t\to->blob_state = allocate_alloc_state();\n+\t\to->tree_state = allocate_alloc_state();\n+\t\to->commit_state = allocate_alloc_state();\n+\t\to->tag_state = allocate_alloc_state();\n+\t\to->object_state = allocate_alloc_state();\n+\t}\n \treturn o;\n }\n \n@@ -501,9 +516,12 @@ void raw_object_store_clear(struct raw_object_store *o)\n void object_parser_clear(struct object_parser *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n \t */\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex 5ef6ce1ea96..cf97dc0d472 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,9 +4,17 @@\n struct object_parser {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n-struct object_parser *object_parser_new(void);\n+struct object_parser *object_parser_new(int is_the_repo);\n void object_parser_clear(struct object_parser *o);\n \n struct object_list {\ndiff --git a/repository.c b/repository.c\nindex 208ee10071c..51eb876617d 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -15,7 +15,7 @@ void initialize_the_repository(void)\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n-\tthe_repo.parsed_objects = object_parser_new();\n+\tthe_repo.parsed_objects = object_parser_new(1);\n \n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n@@ -146,7 +146,7 @@ static int repo_init(struct repository *repo,\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n-\trepo->parsed_objects = object_parser_new();\n+\trepo->parsed_objects = object_parser_new(0);\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..af6a0725b6a 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346339","messageId":"20180501213403.14643-12-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 11/13] object: allow grow_object_hash to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:34:01Z","receivedAt":"2018-05-01T21:34:38Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex ddf4b7b196e..43954fadf93 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,27 +116,27 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-#define grow_object_hash(r) grow_object_hash_##r()\n-static void grow_object_hash_the_repository(void)\n+static void grow_object_hash(struct repository *r)\n {\n \tint i;\n \t/*\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n+\tint new_hash_size = r->parsed_objects->obj_hash_size < 32 ? 32 : 2 * r->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n-\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n+\tfor (i = 0; i < r->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = r->parsed_objects->obj_hash[i];\n+\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(the_repository->parsed_objects->obj_hash);\n-\tthe_repository->parsed_objects->obj_hash = new_hash;\n-\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n+\tfree(r->parsed_objects->obj_hash);\n+\tr->parsed_objects->obj_hash = new_hash;\n+\tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object_the_repository(const unsigned char *sha1, void *o)\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346340","messageId":"20180501213403.14643-7-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 06/13] alloc: add repository argument to alloc_commit_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:56Z","receivedAt":"2018-05-01T21:34:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n blame.c | 2 +-\n cache.h | 3 ++-\n commit.c | 2 +-\n merge-recursive.c | 2 +-\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 2c8d1430758..9e2b897ec1d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -88,7 +88,7 @@ unsigned int alloc_commit_index(void)\n \treturn count++;\n }\n \n-void *alloc_commit_node(void)\n+void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\ndiff --git a/blame.c b/blame.c\nindex dfa24473dc6..ba9b18e7542 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -161,7 +161,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \n \tread_cache();\n \ttime(&now);\n-\tcommit = alloc_commit_node();\n+\tcommit = alloc_commit_node(the_repository);\n \tcommit->object.parsed = 1;\n \tcommit->date = now;\n \tparent_tail = &commit->parents;\ndiff --git a/cache.h b/cache.h\nindex 1717d07a2c5..bf6e8c87d83 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1768,7 +1768,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n extern void *alloc_blob_node_the_repository(void);\n #define alloc_tree_node(r) alloc_tree_node_##r()\n extern void *alloc_tree_node_the_repository(void);\n-extern void *alloc_commit_node(void);\n+#define alloc_commit_node(r) alloc_commit_node_##r()\n+extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\ndiff --git a/commit.c b/commit.c\nindex 9106acf0aad..a9a43e79bae 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -51,7 +51,7 @@ struct commit *lookup_commit(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_commit_node());\n+\t\t\t\t alloc_commit_node(the_repository));\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624da..6dac8908648 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -98,7 +98,7 @@ static struct tree *shift_tree_object(struct tree *one, struct tree *two,\n \n static struct commit *make_virtual_commit(struct tree *tree, const char *comment)\n {\n-\tstruct commit *commit = alloc_commit_node();\n+\tstruct commit *commit = alloc_commit_node(the_repository);\n \n \tset_merge_remote_desc(commit, comment, (struct object *)commit);\n \tcommit->tree = tree;\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346341","messageId":"20180501213403.14643-9-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 08/13] alloc: add repository argument to alloc_object_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:58Z","receivedAt":"2018-05-01T21:34:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 290250e3595..f031ce422d9 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -73,7 +73,7 @@ void *alloc_tag_node_the_repository(void)\n }\n \n static struct alloc_state object_state;\n-void *alloc_object_node(void)\n+void *alloc_object_node_the_repository(void)\n {\n \tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\ndiff --git a/cache.h b/cache.h\nindex 32f340cde59..2d60359a964 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1772,7 +1772,8 @@ extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node_the_repository(void);\n #define alloc_tag_node(r) alloc_tag_node_##r()\n extern void *alloc_tag_node_the_repository(void);\n-extern void *alloc_object_node(void);\n+#define alloc_object_node(r) alloc_object_node_##r()\n+extern void *alloc_object_node_the_repository(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\n \ndiff --git a/object.c b/object.c\nindex a6202d11292..7d36323445b 100644\n--- a/object.c\n+++ b/object.c\n@@ -180,7 +180,7 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n \t\tobj = create_object(the_repository, sha1,\n-\t\t\t\t alloc_object_node());\n+\t\t\t\t alloc_object_node(the_repository));\n \treturn obj;\n }\n \n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346342","messageId":"20180501213403.14643-4-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 03/13] object: add repository argument to grow_object_hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:53Z","receivedAt":"2018-05-01T21:34:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"From: Jonathan Nieder \n\nAdd a repository argument to allow the caller of grow_object_hash to\nbe more specific about which repository to handle. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nAs with the previous commits, use a macro to catch callers passing a\nrepository other than the_repository at compile time.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 933921e35c9..a6202d11292 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,7 +116,8 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-static void grow_object_hash(void)\n+#define grow_object_hash(r) grow_object_hash_##r()\n+static void grow_object_hash_the_repository(void)\n {\n \tint i;\n \t/*\n@@ -147,7 +148,7 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \thashcpy(obj->oid.hash, sha1);\n \n \tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash();\n+\t\tgrow_object_hash(the_repository);\n \n \tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n \t\t\tthe_repository->parsed_objects->obj_hash_size);\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346343","messageId":"20180501213403.14643-5-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH 04/13] alloc: add repository argument to alloc_blob_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-01T21:33:54Z","receivedAt":"2018-05-01T21:34:49Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n blob.c | 2 +-\n cache.h | 3 ++-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacdd..6c5c376a25a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -49,7 +49,7 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n }\n \n static struct alloc_state blob_state;\n-void *alloc_blob_node(void)\n+void *alloc_blob_node_the_repository(void)\n {\n \tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\ndiff --git a/blob.c b/blob.c\nindex 85c2143f299..9e64f301895 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -9,7 +9,7 @@ struct blob *lookup_blob(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_blob_node());\n+\t\t\t\t alloc_blob_node(the_repository));\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 3a4d80e92bf..2258e611275 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1764,7 +1764,8 @@ int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n /* alloc.c */\n-extern void *alloc_blob_node(void);\n+#define alloc_blob_node(r) alloc_blob_node_##r()\n+extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n-- \n2.17.0.441.gb46fe60e1d-goog\n\n"},{"id":"346344","messageId":"CAPig+cSYxib5NEGM+J9RZ=v6GbaMf_0R2nWpfQWybrcn7bPOaw@mail.gmail.com","threadId":"48394","inReplyTo":"20180501213403.14643-3-sbeller@google.com","subject":"Re: [PATCH 02/13] object: add repository argument to create_object","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-05-01T21:43:00Z","receivedAt":"2018-05-01T21:43:05Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, May 1, 2018 at 5:33 PM, Stefan Beller wrote:\n> Add a repository argument to allow the callers of create_object\n> to be more specific about which repository to act on. This is a small\n> mechanical change; it doesn't change the implementation to handle\n> repositories other than the_repository yet.\n>\n> As with the previous commits, use a macro to catch callers passing a\n> repository other than the_repository at compile time.\n\nThis is only patch 2, and patch 1 did not use a macro trick to catch\ncallers passing the wrong argument.\n\n> Add the cocci patch that converted the callers.\n\nNo cocci patch anywhere in sight.\n\n> Signed-off-by: Jonathan Nieder \n> Signed-off-by: Stefan Beller \n"},{"id":"346466","messageId":"CACsJy8DKnBRDZC=oBd8nTOAVzQf3UtoLCeoGKjcMHgUmXf-KqQ@mail.gmail.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"Re: [PATCH 00/13] object store: alloc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-02T17:01:42Z","receivedAt":"2018-05-02T17:02:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 1, 2018 at 11:33 PM, Stefan Beller wrote:\n> I also debated if it is worth converting alloc.c via this patch series\n> or if it might make more sense to use the new mem-pool by Jameson[1].\n>\n> I vaguely wonder about the performance impact, as the object allocation\n> code seemed to be relevant in the past.\n\nIf I remember correctly, alloc.c was added because malloc() has too\nhigh overhead per allocation (and we create like millions of them). As\nlong as you keep allocation overhead low, it should be ok. Note that\nwe allocate a lot more than the mem-pool's main target (cache entries\nif I remember correctly). We may have a couple thousands cache\nentries. We already deal with a couple million of struct object.\n-- \nDuy\n"},{"id":"346467","messageId":"CACsJy8CgX6BME=EhiDBfMRzBOYDBNHE6Ouxv4fZC-GW7Rsi81Q@mail.gmail.com","threadId":"48394","inReplyTo":"20180501213403.14643-2-sbeller@google.com","subject":"Re: [PATCH 01/13] repository: introduce object parser field","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-02T17:17:40Z","receivedAt":"2018-05-02T17:18:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 1, 2018 at 11:33 PM, Stefan Beller wrote:\n> /*\n> - * Holds any information related to accessing the raw object content.\n> + * Holds any information needed to retrieve the raw content\n> + * of objects. The object_parser uses this to get object\n> + * content which it then parses.\n> */\n> struct raw_object_store *objects;\n>\n> + /*\n> + * State for the object parser. This owns all parsed objects\n> + * (struct object) so callers do not have to manage their\n> + * lifetime.\n> + */\n> + struct object_parser *parsed_objects;\n\nI like this name 'parsed_objects'. Should we rename the struct after\nit (e.g. parsed_object_store as opposed to raw_object_store above)?\n\nAnother suggestion is object_pool, if we keep 'struct object' instead\nof 'struct parsed_object' and also want to keep current allocation\nbehavior: no individual deallocation. If you free, you free the whole\npool (e.g. you could run rev-list --all --objects in a separate pool,\nonce you're done, you delete the whole pool).\n-- \nDuy\n"},{"id":"346469","messageId":"CAGZ79ka6p33W8msji9kFMaWGOuDzSyx=y0AFPaQSX_08AZSgag@mail.gmail.com","threadId":"48394","inReplyTo":"CACsJy8CgX6BME=EhiDBfMRzBOYDBNHE6Ouxv4fZC-GW7Rsi81Q@mail.gmail.com","subject":"Re: [PATCH 01/13] repository: introduce object parser field","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-02T17:26:48Z","receivedAt":"2018-05-02T17:26:53Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 2, 2018 at 10:17 AM, Duy Nguyen wrote:\n> On Tue, May 1, 2018 at 11:33 PM, Stefan Beller wrote:\n>> /*\n>> - * Holds any information related to accessing the raw object content.\n>> + * Holds any information needed to retrieve the raw content\n>> + * of objects. The object_parser uses this to get object\n>> + * content which it then parses.\n>> */\n>> struct raw_object_store *objects;\n>>\n>> + /*\n>> + * State for the object parser. This owns all parsed objects\n>> + * (struct object) so callers do not have to manage their\n>> + * lifetime.\n>> + */\n>> + struct object_parser *parsed_objects;\n>\n> I like this name 'parsed_objects'. Should we rename the struct after\n> it (e.g. parsed_object_store as opposed to raw_object_store above)?\n\nI can rename it to parsed_object_store for consistency.\n\n> Another suggestion is object_pool, if we keep 'struct object' instead\n> of 'struct parsed_object' and also want to keep current allocation\n> behavior: no individual deallocation. If you free, you free the whole\n> pool (e.g. you could run rev-list --all --objects in a separate pool,\n> once you're done, you delete the whole pool).\n\nThat is what the following patches will be about, you can\nonly free the whole set of parsed objects.\n\nSo if you want to do a separate rev walk, you may need to\ninstantiate a new repository for it (ideally you'd only need a\nseparate parsed object store).\n\nI'd want to have the ability to have separate pools for submodules,\nsuch that they can be free'd on a per-repo basis.\n\n> --\n> Duy\n"},{"id":"346470","messageId":"CACsJy8B=i_0EKGRLxS5N5tfz8666hm_aTns949eb9DAW1--XWg@mail.gmail.com","threadId":"48394","inReplyTo":"20180501213403.14643-14-sbeller@google.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-02T17:44:15Z","receivedAt":"2018-05-02T17:44:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 1, 2018 at 11:34 PM, Stefan Beller wrote:\n> #include \"cache.h\"\n> #include \"object.h\"\n> @@ -30,8 +31,25 @@ struct alloc_state {\n> int count; /* total number of nodes allocated */\n> int nr; /* number of nodes left in current allocation */\n> void *p; /* first free node in current allocation */\n> +\n> + /* bookkeeping of allocations */\n> + void **slabs;\n\nAnother way to manage this is linked list: you could reserve one\n\"object\" in each slab to store the \"next\" (or \"prev\") pointer to\nanother slab, then you can just walk through all slabs and free. It's\na bit cheaper than reallocating slabs[], but I guess we reallocate so\nfew times that readability matters more (whichever way is chosen).\n\n> + int slab_nr, slab_alloc;\n> };\n>\n> +void *allocate_alloc_state(void)\n> +{\n> + return xcalloc(1, sizeof(struct alloc_state));\n> +}\n> +\n> +void clear_alloc_state(struct alloc_state *s)\n> +{\n> + while (s->slab_nr > 0) {\n> + s->slab_nr--;\n> + free(s->slabs[s->slab_nr]);\n\nI think you're leaking memory here. Commit and tree objects may have\nmore allocations in them (especially trees, but I think we have\ncommit_list in struct commit too). Those need to be freed as well.\n\n> + }\n> +}\n> +\n> static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n> {\n> void *ret;\n-- \nDuy\n"},{"id":"346471","messageId":"CACsJy8CouvcppoHDEG-PvpuKh9b-r5R1j4Q6HaJ8XPb1G8FSNw@mail.gmail.com","threadId":"48394","inReplyTo":"CAGZ79ka6p33W8msji9kFMaWGOuDzSyx=y0AFPaQSX_08AZSgag@mail.gmail.com","subject":"Re: [PATCH 01/13] repository: introduce object parser field","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-02T17:58:06Z","receivedAt":"2018-05-02T17:58:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 2, 2018 at 7:26 PM, Stefan Beller wrote:\n>> Another suggestion is object_pool, if we keep 'struct object' instead\n>> of 'struct parsed_object' and also want to keep current allocation\n>> behavior: no individual deallocation. If you free, you free the whole\n>> pool (e.g. you could run rev-list --all --objects in a separate pool,\n>> once you're done, you delete the whole pool).\n>\n> That is what the following patches will be about, you can\n> only free the whole set of parsed objects.\n>\n> So if you want to do a separate rev walk, you may need to\n> instantiate a new repository for it (ideally you'd only need a\n> separate parsed object store).\n\nI'm not sure if it's a good idea to create a separate struct\nrepository just because you want to free this parsed object store.\nWhat if updates are made in both repositories? All the cache (well,\nmostly the delta base cache in sha1_file.c) will double memory usage\nas well. Yeah not ideal. But I guess making rev-list related code use\na separate parsed object store is no small task (and kinda risky as\nwell since we migrate from global lookup_* functions to local ones and\nneed to choose the correct parsed object store to look up from)\n\nFor your information, there is already a good use case for this\nwholesale memory free: if we can free the rev-list related memory\nearly in pack-objects (e.g. part of repack operation) then it could\nlower memory pressure significantly when running on large repos. This\nhas been discussed a bit lately.\n-- \nDuy\n"},{"id":"346473","messageId":"BL0PR2101MB1106C07DEE184F8DA2C7CA62CE800@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48394","inReplyTo":"CACsJy8DKnBRDZC=oBd8nTOAVzQf3UtoLCeoGKjcMHgUmXf-KqQ@mail.gmail.com","subject":"RE: [PATCH 00/13] object store: alloc","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-02T18:07:42Z","receivedAt":"2018-05-02T18:07:47Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"\n\n> -----Original Message-----\n> From: Duy Nguyen \n> Sent: Wednesday, May 2, 2018 1:02 PM\n> To: Stefan Beller \n> Cc: Git Mailing List ; Jameson Miller\n> \n> Subject: Re: [PATCH 00/13] object store: alloc\n> \n> On Tue, May 1, 2018 at 11:33 PM, Stefan Beller wrote:\n> > I also debated if it is worth converting alloc.c via this patch series\n> > or if it might make more sense to use the new mem-pool by Jameson[1].\n> >\n> > I vaguely wonder about the performance impact, as the object\n> > allocation code seemed to be relevant in the past.\n> \n> If I remember correctly, alloc.c was added because malloc() has too high\n> overhead per allocation (and we create like millions of them). As long as you\n> keep allocation overhead low, it should be ok. Note that we allocate a lot more\n> than the mem-pool's main target (cache entries if I remember correctly). We\n> may have a couple thousands cache entries. We already deal with a couple\n> million of struct object.\n\nThe work to move cache entry allocation onto a memory pool was motivated by\nthe fact that we are looking at indexes with millions of entries. If there is scaling\nconcern with the current version of mem-pool, we would like to address it there\nas well. Or if there is improvements that can be shared, that would be nice as well.\n\n> --\n> Duy\n"},{"id":"346475","messageId":"CACsJy8CR1RYYdsc5=8hz2Sx9_bOY9eb_-vwEqDo62Hex1FxBmw@mail.gmail.com","threadId":"48394","inReplyTo":"BL0PR2101MB1106C07DEE184F8DA2C7CA62CE800@BL0PR2101MB1106.namprd21.prod.outlook.com","subject":"Re: [PATCH 00/13] object store: alloc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-02T18:22:44Z","receivedAt":"2018-05-02T18:23:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 2, 2018 at 8:07 PM, Jameson Miller wrote:\n>\n>\n>> -----Original Message-----\n>> From: Duy Nguyen \n>> Sent: Wednesday, May 2, 2018 1:02 PM\n>> To: Stefan Beller \n>> Cc: Git Mailing List ; Jameson Miller\n>> \n>> Subject: Re: [PATCH 00/13] object store: alloc\n>>\n>> On Tue, May 1, 2018 at 11:33 PM, Stefan Beller wrote:\n>> > I also debated if it is worth converting alloc.c via this patch series\n>> > or if it might make more sense to use the new mem-pool by Jameson[1].\n>> >\n>> > I vaguely wonder about the performance impact, as the object\n>> > allocation code seemed to be relevant in the past.\n>>\n>> If I remember correctly, alloc.c was added because malloc() has too high\n>> overhead per allocation (and we create like millions of them). As long as you\n>> keep allocation overhead low, it should be ok. Note that we allocate a lot more\n>> than the mem-pool's main target (cache entries if I remember correctly). We\n>> may have a couple thousands cache entries. We already deal with a couple\n>> million of struct object.\n>\n> The work to move cache entry allocation onto a memory pool was motivated by\n> the fact that we are looking at indexes with millions of entries. If there is scaling\n> concern with the current version of mem-pool, we would like to address it there\n> as well. Or if there is improvements that can be shared, that would be nice as well.\n\nI think the two have quite different characteristics. alloc.c code is\ndriven by overhead. struct blob is only 24 bytes each and about 1/3\nthe repo is blobs, and each malloc has 16 bytes overhead or so if I\nremember correctly. struct cache_entry at minimum in 88 bytes so\nrelative overhead is not that a big deal (but sure reducing it is\nstill very nice).\n\nmem-pool is about allocation speed, but I think that's not a concern\nfor alloc.c because when we do full rev walk, I think I/O is always\nthe bottleneck (maybe object lookup as well). I don't see a good way\nto have the one memory allocator that satisfyies both to be honest. If\nyou could allocate fixed-size cache entries most of the time (e.g.\nlarger entries will be allocated using malloc or something, and should\nbe a small number), then perhaps we can unify. Or if mem-pool can have\nan option to allocated fixed size pieces with no overhead, that would\nbe great (sorry I still have not read that mem-pool series yet)\n-- \nDuy\n"},{"id":"346477","messageId":"BL0PR2101MB1106A9F695338C4E045F2AA1CE800@BL0PR2101MB1106.namprd21.prod.outlook.com","threadId":"48394","inReplyTo":"CACsJy8CR1RYYdsc5=8hz2Sx9_bOY9eb_-vwEqDo62Hex1FxBmw@mail.gmail.com","subject":"RE: [PATCH 00/13] object store: alloc","fromName":"Jameson Miller","fromEmail":"jamill@microsoft.com","sentAt":"2018-05-02T18:44:53Z","receivedAt":"2018-05-02T18:44:58Z","isPatch":true,"sender":{"key":"jamill@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/2045976?v=4"},"body":"\n\n> -----Original Message-----\n> From: Duy Nguyen \n> Sent: Wednesday, May 2, 2018 2:23 PM\n> To: Jameson Miller \n> Cc: Stefan Beller ; Git Mailing List \n> Subject: Re: [PATCH 00/13] object store: alloc\n> \n> On Wed, May 2, 2018 at 8:07 PM, Jameson Miller \n> wrote:\n> >\n> >\n> >> -----Original Message-----\n> >> From: Duy Nguyen \n> >> Sent: Wednesday, May 2, 2018 1:02 PM\n> >> To: Stefan Beller \n> >> Cc: Git Mailing List ; Jameson Miller\n> >> \n> >> Subject: Re: [PATCH 00/13] object store: alloc\n> >>\n> >> On Tue, May 1, 2018 at 11:33 PM, Stefan Beller \n> wrote:\n> >> > I also debated if it is worth converting alloc.c via this patch\n> >> > series or if it might make more sense to use the new mem-pool by\n> Jameson[1].\n> >> >\n> >> > I vaguely wonder about the performance impact, as the object\n> >> > allocation code seemed to be relevant in the past.\n> >>\n> >> If I remember correctly, alloc.c was added because malloc() has too\n> >> high overhead per allocation (and we create like millions of them).\n> >> As long as you keep allocation overhead low, it should be ok. Note\n> >> that we allocate a lot more than the mem-pool's main target (cache\n> >> entries if I remember correctly). We may have a couple thousands\n> >> cache entries. We already deal with a couple million of struct object.\n> >\n> > The work to move cache entry allocation onto a memory pool was\n> > motivated by the fact that we are looking at indexes with millions of\n> > entries. If there is scaling concern with the current version of\n> > mem-pool, we would like to address it there as well. Or if there is\n> improvements that can be shared, that would be nice as well.\n> \n> I think the two have quite different characteristics. alloc.c code is driven by\n> overhead. struct blob is only 24 bytes each and about 1/3 the repo is blobs, and\n> each malloc has 16 bytes overhead or so if I remember correctly. struct\n> cache_entry at minimum in 88 bytes so relative overhead is not that a big deal\n> (but sure reducing it is still very nice).\n> \n> mem-pool is about allocation speed, but I think that's not a concern for alloc.c\n> because when we do full rev walk, I think I/O is always the bottleneck (maybe\n> object lookup as well). I don't see a good way to have the one memory allocator\n> that satisfyies both to be honest. If you could allocate fixed-size cache entries\n> most of the time (e.g.\n> larger entries will be allocated using malloc or something, and should be a small\n> number), then perhaps we can unify. Or if mem-pool can have an option to\n> allocated fixed size pieces with no overhead, that would be great (sorry I still\n> have not read that mem-pool series yet)\n> --\n> Duy\n\nThank you for the extra details - the extra context was helpful -\nespecially the motivations for each of the areas. I agree with\nyour general analysis, but with the extra point that the memory\npool does allocate memory (variable sized) without any overhead,\nexcept for possible alignment considerations and differences in\nbookkeeping the larger \"blocks\" of memory from which small\nallocations are made from - but I don't think this would be\nenough to have a meaningful overall impact.\n\nThe mem-pool only tracks the pointer to the next available bit of\nmemory, and the end of the available memory range. It has a\nsimilar constraint in that individual allocations cannot be freed\n- you have to free the whole block.\n\nIt may be that the requirements are different enough (or the\ngains worth it) to have another dedicated pooling allocator, but\nI think the current design of the memory pool would satisfy both\nconsumers, even if the memory considerations are a bigger\nmotivation for blob structs. I would be interested in your\nthoughts if you get the opportunity to read the mem-pool series.\n"},{"id":"346479","messageId":"20180502133005.668e9eefda445f37ba808b96@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-2-sbeller@google.com","subject":"Re: [PATCH 01/13] repository: introduce object parser field","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-02T20:30:05Z","receivedAt":"2018-05-02T20:30:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 1 May 2018 14:33:51 -0700\nStefan Beller wrote:\n\n> Git's object access code can be thought of as containing two layers:\n> the raw object store provides access to raw object content, while the\n> higher level obj_hash code parses raw objects and keeps track of\n> parenthood and other object relationships using 'struct object'.\n> Keeping these layers separate should make it easier to find relevant\n> functions and to change the implementation of one without having to\n> touch the other.\n\nI only understood this after reading the code below. I think this\nparagraph can be removed; I don't see its relevance - of course we need\nto store metadata about how to load objects somewhere, and caching\nobjects we have already loaded is a good idea: and the metadata and\ncache are two separate things before and after this patch anyway.\n\n> Add an object_parser field to 'struct repository' to prepare obj_hash\n> to be handled per repository. Callers still only use the_repository\n> for now --- later patches will adapt them to handle arbitrary\n> repositories.\n\nI think this is better reworded as:\n\n Convert the existing global cache for parsed objects (obj_hash) into\n repository-specific parsed object caches. Existing code that uses\n obj_hash are modified to use the parsed object cache of\n the_repository; future patches will use the parsed object caches of\n other repositories.\n\n> +struct object_parser *object_parser_new(void)\n> +{\n> +\tstruct object_parser *o = xmalloc(sizeof(*o));\n> +\tmemset(o, 0, sizeof(*o));\n> +\treturn o;\n> +}\n\nI'm not sure that I like this style of code (I prefer the strbuf style\nwith _INIT and release(), which I think is more flexible) but I don't\nfeel too strongly about it.\n\n> +struct object_parser {\n> +\tstruct object **obj_hash;\n> +\tint nr_objs, obj_hash_size;\n> +};\n\nobject_parser is probably a bad name. I think Duy had some good\nsuggestions in [1].\n\n[1] https://public-inbox.org/git/CACsJy8CgX6BME=EhiDBfMRzBOYDBNHE6Ouxv4fZC-GW7Rsi81Q@mail.gmail.com/\n\n> \t/*\n> -\t * Holds any information related to accessing the raw object content.\n> +\t * Holds any information needed to retrieve the raw content\n> +\t * of objects. The object_parser uses this to get object\n> +\t * content which it then parses.\n> \t */\n> \tstruct raw_object_store *objects;\n\nI think the additional sentence (\"The object_parser uses this...\") is\nunnecessary and confusing, especially if its identity is going to be one\nof storage (like \"parsed_objects\" implies).\n\n> +\t/*\n> +\t * State for the object parser. This owns all parsed objects\n> +\t * (struct object) so callers do not have to manage their\n> +\t * lifetime.\n> +\t */\n> +\tstruct object_parser *parsed_objects;\n\nEven after all the commits in this patch set, this does not store any\nstate for parsing. Maybe just document as:\n\n All objects in this repository that have been parsed. This structure\n owns all objects it references, so users of \"struct object *\"\n generally do not need to free them; instead, when a repository is no\n longer used, call object_parser_clear() on this structure.\n\n(And maybe say that the freeing method on struct repository will\nautomatically call object_parser_clear().)\n"},{"id":"346480","messageId":"20180502133404.d039e81a6279213ae6943cc7@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-5-sbeller@google.com","subject":"Re: [PATCH 04/13] alloc: add repository argument to alloc_blob_node","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-02T20:34:04Z","receivedAt":"2018-05-02T20:34:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 1 May 2018 14:33:54 -0700\nStefan Beller wrote:\n\n> Signed-off-by: Stefan Beller \n\nAdd the same boilerplate explanation to this and subsequent commits. If\nediting it for every new function name is cumbersome, maybe use this\nshortened version:\n\n This is a small mechanical change; it doesn't change the\n implementation to handle repositories other than the_repository yet.\n Use a macro to catch callers passing a repository other than\n the_repository at compile time.\n"},{"id":"346481","messageId":"20180502133604.bac43123a7082ff6e6c5a1ed@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-13-sbeller@google.com","subject":"Re: [PATCH 12/13] object: allow create_object to handle arbitrary repositories","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-02T20:36:04Z","receivedAt":"2018-05-02T20:36:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 1 May 2018 14:34:02 -0700\nStefan Beller wrote:\n\n> Signed-off-by: Jonathan Nieder \n> Signed-off-by: Stefan Beller \n\nReviewed-by: Jonathan Tan \n\nDownloading this patch set and viewing the whole function modified in\nthis patch shows that globals are no longer referenced, so this is good.\nSame comment for patch 11.\n"},{"id":"346482","messageId":"20180502135029.051dcaed75b18fb95ae6b28f@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-14-sbeller@google.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-02T20:50:29Z","receivedAt":"2018-05-02T20:50:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 1 May 2018 14:34:03 -0700\nStefan Beller wrote:\n\n> +void *allocate_alloc_state(void)\n> +{\n> +\treturn xcalloc(1, sizeof(struct alloc_state));\n> +}\n> +\n> +void clear_alloc_state(struct alloc_state *s)\n> +{\n> +\twhile (s->slab_nr > 0) {\n> +\t\ts->slab_nr--;\n> +\t\tfree(s->slabs[s->slab_nr]);\n> +\t}\n> +}\n\nThese functions are asymmetrical. I understand why it is this way\n(because we use pointers, and we want to use FREE_AND_NULL), but would\nstill prefer _INIT and _release().\n\n> static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n> {\n> \tvoid *ret;\n> @@ -45,54 +63,56 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n> \tret = s->p;\n> \ts->p = (char *)s->p + node_size;\n> \tmemset(ret, 0, node_size);\n> +\n> +\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n> +\ts->slabs[s->slab_nr++] = ret;\n> +\n> \treturn ret;\n> }\n\nThis unconditionally grows the slabs for each node allocation. Shouldn't\nmore than one node fit in each slab?\n\n> +extern struct alloc_state the_repository_blob_state;\n> +extern struct alloc_state the_repository_tree_state;\n> +extern struct alloc_state the_repository_commit_state;\n> +extern struct alloc_state the_repository_tag_state;\n> +extern struct alloc_state the_repository_object_state;\n\n(Context: these were in alloc.h)\n\nThese seem to be used only in object.c, and only in one function - could\nwe declare them \"static\" inside that function instead?\n\n> -struct object_parser *object_parser_new(void)\n> +struct object_parser *object_parser_new(int is_the_repo)\n> {\n> \tstruct object_parser *o = xmalloc(sizeof(*o));\n> \tmemset(o, 0, sizeof(*o));\n> +\n> +\tif (is_the_repo) {\n> +\t\to->blob_state = &the_repository_blob_state;\n> +\t\to->tree_state = &the_repository_tree_state;\n> +\t\to->commit_state = &the_repository_commit_state;\n> +\t\to->tag_state = &the_repository_tag_state;\n> +\t\to->object_state = &the_repository_object_state;\n> +\t} else {\n> +\t\to->blob_state = allocate_alloc_state();\n> +\t\to->tree_state = allocate_alloc_state();\n> +\t\to->commit_state = allocate_alloc_state();\n> +\t\to->tag_state = allocate_alloc_state();\n> +\t\to->object_state = allocate_alloc_state();\n> +\t}\n> \treturn o;\n> }\n\nI don't think saving 5 allocations is worth the code complexity (of the\nadditional parameter).\n"},{"id":"346515","messageId":"CACsJy8BwAL=fmHG1kDJ+hK4gLOaYMjfVgDqYtEOnos09XhYKFw@mail.gmail.com","threadId":"48394","inReplyTo":"20180501213403.14643-14-sbeller@google.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T14:58:18Z","receivedAt":"2018-05-03T14:58:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 1, 2018 at 11:34 PM, Stefan Beller wrote:\n> @@ -501,9 +516,12 @@ void raw_object_store_clear(struct raw_object_store *o)\n> void object_parser_clear(struct object_parser *o)\n> {\n> /*\n> - * TOOD free objects in o->obj_hash.\n> - *\n\nYou need to free(o->obj_hash) too. If you just want to reuse existing\nobj_hash[] then at least clear it, leave no dangling pointers behind.\n\n> * As objects are allocated in slabs (see alloc.c), we do\n> * not need to free each object, but each slab instead.\n> */\n> + clear_alloc_state(o->blob_state);\n> + clear_alloc_state(o->tree_state);\n> + clear_alloc_state(o->commit_state);\n> + clear_alloc_state(o->tag_state);\n> + clear_alloc_state(o->object_state);\n> }\n-- \nDuy\n"},{"id":"346551","messageId":"CAGZ79kaNNFYyTAH_Kq=bO_Q9tLGMgnnxQKpDZSftuh7deDoFxA@mail.gmail.com","threadId":"48394","inReplyTo":"CACsJy8B=i_0EKGRLxS5N5tfz8666hm_aTns949eb9DAW1--XWg@mail.gmail.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T17:24:07Z","receivedAt":"2018-05-03T17:24:11Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 2, 2018 at 10:44 AM, Duy Nguyen wrote:\n> On Tue, May 1, 2018 at 11:34 PM, Stefan Beller wrote:\n>> #include \"cache.h\"\n>> #include \"object.h\"\n>> @@ -30,8 +31,25 @@ struct alloc_state {\n>> int count; /* total number of nodes allocated */\n>> int nr; /* number of nodes left in current allocation */\n>> void *p; /* first free node in current allocation */\n>> +\n>> + /* bookkeeping of allocations */\n>> + void **slabs;\n>\n> Another way to manage this is linked list: you could reserve one\n> \"object\" in each slab to store the \"next\" (or \"prev\") pointer to\n> another slab, then you can just walk through all slabs and free. It's\n> a bit cheaper than reallocating slabs[], but I guess we reallocate so\n> few times that readability matters more (whichever way is chosen).\n\nThis is a good idea. I'll do so in a resend.\n\n>> +void clear_alloc_state(struct alloc_state *s)\n>> +{\n>> + while (s->slab_nr > 0) {\n>> + s->slab_nr--;\n>> + free(s->slabs[s->slab_nr]);\n>\n> I think you're leaking memory here. Commit and tree objects may have\n> more allocations in them (especially trees, but I think we have\n> commit_list in struct commit too). Those need to be freed as well.\n\nI would think that tree and commit memory will be free'd in the\nparsed_objects release function? (TODO: I need to add it over there)\n\nThanks,\nStefan\n"},{"id":"346552","messageId":"CAGZ79ka6rTsRL7e-Dr5TZZeQRDCq6m4C-_DsRpDuuxAh48=-VQ@mail.gmail.com","threadId":"48394","inReplyTo":"20180502135029.051dcaed75b18fb95ae6b28f@google.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T17:25:27Z","receivedAt":"2018-05-03T17:25:31Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 2, 2018 at 1:50 PM, Jonathan Tan wrote:\n> On Tue, 1 May 2018 14:34:03 -0700\n> Stefan Beller wrote:\n>\n>> +void *allocate_alloc_state(void)\n>> +{\n>> + return xcalloc(1, sizeof(struct alloc_state));\n>> +}\n>> +\n>> +void clear_alloc_state(struct alloc_state *s)\n>> +{\n>> + while (s->slab_nr > 0) {\n>> + s->slab_nr--;\n>> + free(s->slabs[s->slab_nr]);\n>> + }\n>> +}\n>\n> These functions are asymmetrical. I understand why it is this way\n> (because we use pointers, and we want to use FREE_AND_NULL), but would\n> still prefer _INIT and _release().\n>\n>> static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n>> {\n>> void *ret;\n>> @@ -45,54 +63,56 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n>> ret = s->p;\n>> s->p = (char *)s->p + node_size;\n>> memset(ret, 0, node_size);\n>> +\n>> + ALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n>> + s->slabs[s->slab_nr++] = ret;\n>> +\n>> return ret;\n>> }\n>\n> This unconditionally grows the slabs for each node allocation. Shouldn't\n> more than one node fit in each slab?\n\nYes. I'll go with Duy's idea and make it a linked list by using the first\nobject as a pointer to the next slab.\n\n>\n>> +extern struct alloc_state the_repository_blob_state;\n>> +extern struct alloc_state the_repository_tree_state;\n>> +extern struct alloc_state the_repository_commit_state;\n>> +extern struct alloc_state the_repository_tag_state;\n>> +extern struct alloc_state the_repository_object_state;\n>\n> (Context: these were in alloc.h)\n>\n> These seem to be used only in object.c, and only in one function - could\n> we declare them \"static\" inside that function instead?\n\nok\n\n>\n>> -struct object_parser *object_parser_new(void)\n>> +struct object_parser *object_parser_new(int is_the_repo)\n>> {\n>> struct object_parser *o = xmalloc(sizeof(*o));\n>> memset(o, 0, sizeof(*o));\n>> +\n>> + if (is_the_repo) {\n>> + o->blob_state = &the_repository_blob_state;\n>> + o->tree_state = &the_repository_tree_state;\n>> + o->commit_state = &the_repository_commit_state;\n>> + o->tag_state = &the_repository_tag_state;\n>> + o->object_state = &the_repository_object_state;\n>> + } else {\n>> + o->blob_state = allocate_alloc_state();\n>> + o->tree_state = allocate_alloc_state();\n>> + o->commit_state = allocate_alloc_state();\n>> + o->tag_state = allocate_alloc_state();\n>> + o->object_state = allocate_alloc_state();\n>> + }\n>> return o;\n>> }\n>\n> I don't think saving 5 allocations is worth the code complexity (of the\n> additional parameter).\n\nOk, I'll remove this overhead.\n\nThanks,\nStefan\n"},{"id":"346553","messageId":"CACsJy8BoiCjz889q0cAjSZ4DvhLbRArqG9+CqQJZxss4vd9sFQ@mail.gmail.com","threadId":"48394","inReplyTo":"CAGZ79kaNNFYyTAH_Kq=bO_Q9tLGMgnnxQKpDZSftuh7deDoFxA@mail.gmail.com","subject":"Re: [PATCH 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-03T17:35:50Z","receivedAt":"2018-05-03T17:36:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, May 3, 2018 at 7:24 PM, Stefan Beller wrote:\n>>> +void clear_alloc_state(struct alloc_state *s)\n>>> +{\n>>> + while (s->slab_nr > 0) {\n>>> + s->slab_nr--;\n>>> + free(s->slabs[s->slab_nr]);\n>>\n>> I think you're leaking memory here. Commit and tree objects may have\n>> more allocations in them (especially trees, but I think we have\n>> commit_list in struct commit too). Those need to be freed as well.\n>\n> I would think that tree and commit memory will be free'd in the\n> parsed_objects release function? (TODO: I need to add it over there)\n\nWhat release function? I know tree->buffer is often released since you\ndon't want to keep much in memory. But the user should not be required\nto release all objects before calling object_parser_clear(). For\nstruct commit, I'm pretty we make the commit_list (for commit parents)\nand never free. Now we need to do it, somewhere.\n\nOh you mean object_parser_clear() as release function? Yes. I've been\nwondering if it makes more sense to go through obj_hash[] and release\nobjects before free(obj_hash) than doing it here. It's probably better\nto do it there since obj_hash[] would contain the very last pointers\nto these objects and look like the right place to release resources.\n\n> Thanks,\n> Stefan\n-- \nDuy\n"},{"id":"346593","messageId":"CAGZ79kYUdQ8G8jZYi=iYuzSwfsZua2TbCFX_1B_MBU-sENNmKw@mail.gmail.com","threadId":"48394","inReplyTo":"CACsJy8CR1RYYdsc5=8hz2Sx9_bOY9eb_-vwEqDo62Hex1FxBmw@mail.gmail.com","subject":"Re: [PATCH 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-03T22:45:57Z","receivedAt":"2018-05-03T22:46:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 2, 2018 at 11:22 AM, Duy Nguyen wrote:\n\n> I think the two have quite different characteristics. alloc.c code is\n> driven by overhead. struct blob is only 24 bytes each and about 1/3\n> the repo is blobs, and each malloc has 16 bytes overhead or so if I\n> remember correctly. struct cache_entry at minimum in 88 bytes so\n> relative overhead is not that a big deal (but sure reducing it is\n> still very nice).\n>\n> mem-pool is about allocation speed,\n\nI don't think so, given that we do a linear search in each block allocation.\n\n> but I think that's not a concern\n> for alloc.c because when we do full rev walk, I think I/O is always\n> the bottleneck (maybe object lookup as well). I don't see a good way\n> to have the one memory allocator that satisfyies both to be honest.\n\nBy changing the allocation size of a block to be larger than 1024 entries\nin alloc. we should lessen the impact of management overhead, and then\nthe mem pool can be more than feasible.\n"},{"id":"346866","messageId":"xmqqo9hrpnn6.fsf@gitster-ct.c.googlers.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"Re: [PATCH 00/13] object store: alloc","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-07T14:05:01Z","receivedAt":"2018-05-07T14:05:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> This applies on top of sb/oid-object-info and is the logical continuum of\n> the series that it builds on; this brings the object store into more of\n> Gits code, removing global state, such that reasoning about the state of\n> the in-memory representation of the repository is easier.\n\nI am not sure how well this topic is done, but I've queued the\nfollowing patch at the tip of the topic to make it compile after\ngetting merged to integration branches (curiously, the topic by\nitself compiled file for whatever reason). I think I haven't send\nthat fixup patch out, so here it is.\n\n-- >8 --\nFrom: Junio C Hamano \nDate: Wed, 2 May 2018 19:09:50 +0900\nSubject: [PATCH] alloc.c: include alloc.h\n\nOtherwise the definition in alloc.c would not see the matching decl\nin alloc.h, triggering warning from compiler.\n\nSigned-off-by: Junio C Hamano \n---\n alloc.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/alloc.c b/alloc.c\nindex 66a3d07ba2..f47a67153b 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -16,6 +16,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n-- \n2.17.0-391-g1f1cddd558\n\n"},{"id":"346908","messageId":"CAGZ79kbb8KOTjAVSrxN2GhRmem_A9BdUmaHjM64UoDP3MfHqSQ@mail.gmail.com","threadId":"48394","inReplyTo":"xmqqo9hrpnn6.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T20:53:12Z","receivedAt":"2018-05-07T20:53:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Hi Junio,\n\nOn Mon, May 7, 2018 at 7:05 AM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> This applies on top of sb/oid-object-info and is the logical continuum of\n>> the series that it builds on; this brings the object store into more of\n>> Gits code, removing global state, such that reasoning about the state of\n>> the in-memory representation of the repository is easier.\n>\n> I am not sure how well this topic is done, but I've queued the\n> following patch at the tip of the topic to make it compile after\n> getting merged to integration branches (curiously, the topic by\n> itself compiled file for whatever reason).\n\nThanks for the fixup; I will include it with other fixes in a reroll.\n\nThe investigation why it would not compile is not found in alloc.c\nbut in 1da1580e4c2 (Makefile: detect compiler and enable more\nwarnings in DEVELOPER=1, 2018-04-14), which enabled\n-Werror=missing-prototypes, that requires a prototype which\nis found in alloc.h\n\nThanks,\nStefan\n"},{"id":"346913","messageId":"20180507225916.155236-1-sbeller@google.com","threadId":"48394","inReplyTo":"20180501213403.14643-1-sbeller@google.com","subject":"[PATCH v2 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:03Z","receivedAt":"2018-05-07T22:59:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v2:\n* I decided to stick with alloc.c and not migrate it to the mem-pool for now.\n The reasoning for that is that mem-pool.h would introduce some alignment\n waste, which I really did not want to.\n* renamed to struct parsed_object_pool;\n* free'd the additional memory of trees and commits.\n* do not special case the_repository for allocation purposes\n* corrected&polished commit messages\n* I used the (soon to be renamed?) branch-diff tool to attach a diff below.\n (I still need to get used to that format, I find an interdiff of the\n branches easier to read, but that would not yield the commit messages)\n \n \n\nv1:\nThis applies on top of sb/oid-object-info and is the logical continuum of\nthe series that it builds on; this brings the object store into more of\nGits code, removing global state, such that reasoning about the state of\nthe in-memory representation of the repository is easier.\n\nMy original plan was to convert lookup_commit_graft as the next series,\nwhich would be similar to lookup_replace_object, as in sb/object-store-replace.\nThe grafts and shallow mechanism are very close to each other, such that\nthey need to be converted at the same time, both depending on the\n\"parsed object store\" that is introduced in this commit.\n\nThe next series will then convert code in {object/blob/tree/commit/tag}.c\nhopefully finishing the lookup_* functions.\n\nI also debated if it is worth converting alloc.c via this patch series\nor if it might make more sense to use the new mem-pool by Jameson[1].\n\nI vaguely wonder about the performance impact, as the object allocation\ncode seemed to be relevant in the past.\n\n[1] https://public-inbox.org/git/20180430153122.243976-1-jamill@microsoft.com/\n\nAny comments welcome,\nThanks,\nStefan\n\nJonathan Nieder (1):\n object: add repository argument to grow_object_hash\n\nStefan Beller (12):\n repository: introduce parsed objects field\n object: add repository argument to create_object\n alloc: add repository argument to alloc_blob_node\n alloc: add repository argument to alloc_tree_node\n alloc: add repository argument to alloc_commit_node\n alloc: add repository argument to alloc_tag_node\n alloc: add repository argument to alloc_object_node\n alloc: add repository argument to alloc_report\n alloc: add repository argument to alloc_commit_index\n object: allow grow_object_hash to handle arbitrary repositories\n object: allow create_object to handle arbitrary repositories\n alloc: allow arbitrary repositories for alloc functions\n\n alloc.c | 63 +++++++++++++++++-----------\n alloc.h | 15 +++++++\n blame.c | 3 +-\n blob.c | 5 ++-\n cache.h | 9 ----\n commit.c | 4 +-\n merge-recursive.c | 3 +-\n object.c | 105 +++++++++++++++++++++++++++++++++-------------\n object.h | 18 +++++++-\n repository.c | 7 ++++\n repository.h | 13 +++++-\n tag.c | 4 +-\n tree.c | 4 +-\n 13 files changed, 184 insertions(+), 69 deletions(-)\n create mode 100644 alloc.h\n\n\n1: 94a4aa2a825 ! 1: c40aae31a1e repository: introduce object parser field\n @@ -1,23 +1,20 @@\n Author: Stefan Beller \n \n - repository: introduce object parser field\n + repository: introduce parsed objects field\n \n - Git's object access code can be thought of as containing two layers:\n - the raw object store provides access to raw object content, while the\n - higher level obj_hash code parses raw objects and keeps track of\n - parenthood and other object relationships using 'struct object'.\n - Keeping these layers separate should make it easier to find relevant\n - functions and to change the implementation of one without having to\n - touch the other.\n + Convert the existing global cache for parsed objects (obj_hash) into\n + repository-specific parsed object caches. Existing code that uses\n + obj_hash are modified to use the parsed object cache of\n + the_repository; future patches will use the parsed object caches of\n + other repositories.\n \n - Add an object_parser field to 'struct repository' to prepare obj_hash\n - to be handled per repository. Callers still only use the_repository\n - for now --- later patches will adapt them to handle arbitrary\n - repositories.\n + Another future use case for a pool of objects is ease of memory management\n + in revision walking: If we can free the rev-list related memory early in\n + pack-objects (e.g. part of repack operation) then it could lower memory\n + pressure significantly when running on large repos. While this has been\n + discussed on the mailing list lately, this series doesn't implement this.\n \n - Signed-off-by: Jonathan Nieder \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/object.c b/object.c\n --- a/object.c\n @@ -139,9 +136,9 @@\n \t}\n }\n \n -+struct object_parser *object_parser_new(void)\n ++struct parsed_object_pool *parsed_object_pool_new(void)\n +{\n -+\tstruct object_parser *o = xmalloc(sizeof(*o));\n ++\tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n +\tmemset(o, 0, sizeof(*o));\n +\treturn o;\n +}\n @@ -154,7 +151,7 @@\n \to->packed_git = NULL;\n }\n +\n -+void object_parser_clear(struct object_parser *o)\n ++void parsed_object_pool_clear(struct parsed_object_pool *o)\n +{\n +\t/*\n +\t * TOOD free objects in o->obj_hash.\n @@ -171,13 +168,13 @@\n #ifndef OBJECT_H\n #define OBJECT_H\n \n -+struct object_parser {\n ++struct parsed_object_pool {\n +\tstruct object **obj_hash;\n +\tint nr_objs, obj_hash_size;\n +};\n +\n -+struct object_parser *object_parser_new(void);\n -+void object_parser_clear(struct object_parser *o);\n ++struct parsed_object_pool *parsed_object_pool_new(void);\n ++void parsed_object_pool_clear(struct parsed_object_pool *o);\n +\n struct object_list {\n \tstruct object *item;\n @@ -198,7 +195,7 @@\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n -+\tthe_repo.parsed_objects = object_parser_new();\n ++\tthe_repo.parsed_objects = parsed_object_pool_new();\n +\n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n @@ -207,7 +204,7 @@\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n -+\trepo->parsed_objects = object_parser_new();\n ++\trepo->parsed_objects = parsed_object_pool_new();\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\n @@ -215,7 +212,7 @@\n \traw_object_store_clear(repo->objects);\n \tFREE_AND_NULL(repo->objects);\n \n -+\tobject_parser_clear(repo->parsed_objects);\n ++\tparsed_object_pool_clear(repo->parsed_objects);\n +\tFREE_AND_NULL(repo->parsed_objects);\n +\n \tif (repo->config) {\n @@ -237,11 +234,13 @@\n \tstruct raw_object_store *objects;\n \n +\t/*\n -+\t * State for the object parser. This owns all parsed objects\n -+\t * (struct object) so callers do not have to manage their\n -+\t * lifetime.\n ++\t * All objects in this repository that have been parsed. This structure\n ++\t * owns all objects it references, so users of \"struct object *\"\n ++\t * generally do not need to free them; instead, when a repository is no\n ++\t * longer used, call parsed_object_pool_clear() on this structure, which\n ++\t * is called by the repositories repo_clear on its desconstruction.\n +\t */\n -+\tstruct object_parser *parsed_objects;\n ++\tstruct parsed_object_pool *parsed_objects;\n +\n \t/* The store in which the refs are held. */\n \tstruct ref_store *refs;\n2: 9ddead5bb7e ! 2: 4ce05bb8b04 object: add repository argument to create_object\n @@ -7,14 +7,8 @@\n mechanical change; it doesn't change the implementation to handle\n repositories other than the_repository yet.\n \n - As with the previous commits, use a macro to catch callers passing a\n - repository other than the_repository at compile time.\n -\n - Add the cocci patch that converted the callers.\n -\n Signed-off-by: Jonathan Nieder \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/blob.c b/blob.c\n --- a/blob.c\n3: 5ec44fe74de ! 3: 27e1d2621c4 object: add repository argument to grow_object_hash\n @@ -7,12 +7,8 @@\n mechanical change; it doesn't change the implementation to handle\n repositories other than the_repository yet.\n \n - As with the previous commits, use a macro to catch callers passing a\n - repository other than the_repository at compile time.\n -\n Signed-off-by: Jonathan Nieder \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/object.c b/object.c\n --- a/object.c\n4: 2a1d0220eb9 ! 4: f8c7fc9b26f alloc: add repository argument to alloc_blob_node\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_blob_node\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n5: 286cde18ff2 ! 5: b682e50bd80 alloc: add repository argument to alloc_tree_node\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_tree_node\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n6: b05f0588f37 ! 6: c89a3f0ca8b alloc: add repository argument to alloc_commit_node\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_commit_node\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n7: e8e25cf61fc ! 7: 59e74c6ef43 alloc: add repository argument to alloc_tag_node\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_tag_node\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n8: dbd4e184121 ! 8: 6c11cf164b1 alloc: add repository argument to alloc_object_node\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_object_node\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n9: ca4a23489b7 ! 9: eb824871f17 alloc: add repository argument to alloc_report\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_report\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n10: f0f84627ce8 ! 10: 76e214a2254 alloc: add repository argument to alloc_commit_index\n @@ -2,8 +2,12 @@\n \n alloc: add repository argument to alloc_commit_index\n \n + This is a small mechanical change; it doesn't change the\n + implementation to handle repositories other than the_repository yet.\n + Use a macro to catch callers passing a repository other than\n + the_repository at compile time.\n +\n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n11: 60f4e269bb8 ! 11: 3a13500d480 object: allow grow_object_hash to handle arbitrary repositories\n @@ -2,9 +2,9 @@\n \n object: allow grow_object_hash to handle arbitrary repositories\n \n + Reviewed-by: Jonathan Tan \n Signed-off-by: Jonathan Nieder \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/object.c b/object.c\n --- a/object.c\n12: 5c0b35cfabf ! 12: 8e06fb06d60 object: allow create_object to handle arbitrary repositories\n @@ -2,9 +2,9 @@\n \n object: allow create_object to handle arbitrary repositories\n \n + Reviewed-by: Jonathan Tan \n Signed-off-by: Jonathan Nieder \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/object.c b/object.c\n --- a/object.c\n13: 10de7382d15 ! 13: d15807445ad alloc: allow arbitrary repositories for alloc functions\n @@ -12,7 +12,6 @@\n the_repository.\n \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/alloc.c b/alloc.c\n --- a/alloc.c\n @@ -26,11 +25,15 @@\n + * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n -+ *\n -+ * TODO: Combine this with mem-pool?\n */\n - #include \"cache.h\"\n - #include \"object.h\"\n +@@\n + #include \"tree.h\"\n + #include \"commit.h\"\n + #include \"tag.h\"\n ++#include \"alloc.h\"\n + \n + #define BLOCKING 1024\n + \n @@\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n @@ -58,19 +61,24 @@\n {\n \tvoid *ret;\n @@\n + \tif (!s->nr) {\n + \t\ts->nr = BLOCKING;\n + \t\ts->p = xmalloc(BLOCKING * node_size);\n ++\n ++\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n ++\t\ts->slabs[s->slab_nr++] = s->p;\n + \t}\n + \ts->nr--;\n + \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n -+\n -+\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n -+\ts->slabs[s->slab_nr++] = ret;\n +\n \treturn ret;\n }\n \n -static struct alloc_state blob_state;\n -void *alloc_blob_node_the_repository(void)\n -+struct alloc_state the_repository_blob_state;\n +void *alloc_blob_node(struct repository *r)\n {\n -\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n @@ -81,7 +89,6 @@\n \n -static struct alloc_state tree_state;\n -void *alloc_tree_node_the_repository(void)\n -+struct alloc_state the_repository_tree_state;\n +void *alloc_tree_node(struct repository *r)\n {\n -\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n @@ -92,7 +99,6 @@\n \n -static struct alloc_state tag_state;\n -void *alloc_tag_node_the_repository(void)\n -+struct alloc_state the_repository_tag_state;\n +void *alloc_tag_node(struct repository *r)\n {\n -\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n @@ -103,7 +109,6 @@\n \n -static struct alloc_state object_state;\n -void *alloc_object_node_the_repository(void)\n -+struct alloc_state the_repository_object_state;\n +void *alloc_object_node(struct repository *r)\n {\n -\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n @@ -123,7 +128,6 @@\n }\n \n -void *alloc_commit_node_the_repository(void)\n -+struct alloc_state the_repository_commit_state;\n +void *alloc_commit_node(struct repository *r)\n {\n -\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n @@ -167,12 +171,6 @@\n +void *allocate_alloc_state(void);\n +void clear_alloc_state(struct alloc_state *s);\n +\n -+extern struct alloc_state the_repository_blob_state;\n -+extern struct alloc_state the_repository_tree_state;\n -+extern struct alloc_state the_repository_commit_state;\n -+extern struct alloc_state the_repository_tag_state;\n -+extern struct alloc_state the_repository_object_state;\n -+\n +#endif\n \n diff --git a/blame.c b/blame.c\n @@ -262,40 +260,47 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n @@\n - \t}\n - }\n - \n --struct object_parser *object_parser_new(void)\n -+struct object_parser *object_parser_new(int is_the_repo)\n {\n - \tstruct object_parser *o = xmalloc(sizeof(*o));\n + \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n +\n -+\tif (is_the_repo) {\n -+\t\to->blob_state = &the_repository_blob_state;\n -+\t\to->tree_state = &the_repository_tree_state;\n -+\t\to->commit_state = &the_repository_commit_state;\n -+\t\to->tag_state = &the_repository_tag_state;\n -+\t\to->object_state = &the_repository_object_state;\n -+\t} else {\n -+\t\to->blob_state = allocate_alloc_state();\n -+\t\to->tree_state = allocate_alloc_state();\n -+\t\to->commit_state = allocate_alloc_state();\n -+\t\to->tag_state = allocate_alloc_state();\n -+\t\to->object_state = allocate_alloc_state();\n -+\t}\n ++\to->blob_state = allocate_alloc_state();\n ++\to->tree_state = allocate_alloc_state();\n ++\to->commit_state = allocate_alloc_state();\n ++\to->tag_state = allocate_alloc_state();\n ++\to->object_state = allocate_alloc_state();\n ++\n \treturn o;\n }\n \n @@\n - void object_parser_clear(struct object_parser *o)\n + void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n -\t * TOOD free objects in o->obj_hash.\n -\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n ++\t *\n ++\t * Before doing so, we need to free any additional memory\n ++\t * the objects may hold.\n \t */\n ++\tunsigned i;\n ++\n ++\tfor (i = 0; i < o->obj_hash_size; i++) {\n ++\t\tstruct object *obj = o->obj_hash[i];\n ++\n ++\t\tif (!obj)\n ++\t\t\tcontinue;\n ++\n ++\t\tif (obj->type == OBJ_TREE) {\n ++\t\t\tfree(((struct tree*)obj)->buffer);\n ++\t\t} else if (obj->type == OBJ_COMMIT) {\n ++\t\t\tfree_commit_list(((struct commit*)obj)->parents);\n ++\t\t\tfree(&((struct commit*)obj)->util);\n ++\t\t}\n ++\t}\n ++\n +\tclear_alloc_state(o->blob_state);\n +\tclear_alloc_state(o->tree_state);\n +\tclear_alloc_state(o->commit_state);\n @@ -307,7 +312,7 @@\n --- a/object.h\n +++ b/object.h\n @@\n - struct object_parser {\n + struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n +\n @@ -320,33 +325,7 @@\n +\tunsigned commit_count;\n };\n \n --struct object_parser *object_parser_new(void);\n -+struct object_parser *object_parser_new(int is_the_repo);\n - void object_parser_clear(struct object_parser *o);\n - \n - struct object_list {\n -\n -diff --git a/repository.c b/repository.c\n ---- a/repository.c\n -+++ b/repository.c\n -@@\n - \n - \tthe_repo.index = &the_index;\n - \tthe_repo.objects = raw_object_store_new();\n --\tthe_repo.parsed_objects = object_parser_new();\n -+\tthe_repo.parsed_objects = object_parser_new(1);\n - \n - \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n - }\n -@@\n - \tmemset(repo, 0, sizeof(*repo));\n - \n - \trepo->objects = raw_object_store_new();\n --\trepo->parsed_objects = object_parser_new();\n -+\trepo->parsed_objects = object_parser_new(0);\n - \n - \tif (repo_init_gitdir(repo, gitdir))\n - \t\tgoto error;\n + struct parsed_object_pool *parsed_object_pool_new(void);\n \n diff --git a/tag.c b/tag.c\n --- a/tag.c\n14: 12e8de9e65c < -: ----------- alloc.c: include alloc.h\n\n\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346914","messageId":"20180507225916.155236-3-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 02/13] object: add repository argument to create_object","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:05Z","receivedAt":"2018-05-07T22:59:34Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Add a repository argument to allow the callers of create_object\nto be more specific about which repository to act on. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n blob.c | 4 +++-\n commit.c | 3 ++-\n object.c | 5 +++--\n object.h | 3 ++-\n tag.c | 3 ++-\n tree.c | 3 ++-\n 6 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/blob.c b/blob.c\nindex fa2ab4f7a74..85c2143f299 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"blob.h\"\n+#include \"repository.h\"\n \n const char *blob_type = \"blob\";\n \n@@ -7,7 +8,8 @@ struct blob *lookup_blob(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_blob_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_blob_node());\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/commit.c b/commit.c\nindex ca474a7c112..9106acf0aad 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -50,7 +50,8 @@ struct commit *lookup_commit(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_commit_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_commit_node());\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/object.c b/object.c\nindex f7c624a7ba6..2de029275bc 100644\n--- a/object.c\n+++ b/object.c\n@@ -138,7 +138,7 @@ static void grow_object_hash(void)\n \tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object(const unsigned char *sha1, void *o)\n+void *create_object_the_repository(const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -178,7 +178,8 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n-\t\tobj = create_object(sha1, alloc_object_node());\n+\t\tobj = create_object(the_repository, sha1,\n+\t\t\t\t alloc_object_node());\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex cecda7da370..2cb0b241083 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,7 +93,8 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-extern void *create_object(const unsigned char *sha1, void *obj);\n+#define create_object(r, s, o) create_object_##r(s, o)\n+extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \ndiff --git a/tag.c b/tag.c\nindex 3d37c1bd251..7150b759d66 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -93,7 +93,8 @@ struct tag *lookup_tag(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tag_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tag_node());\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \ndiff --git a/tree.c b/tree.c\nindex 1c68ea586bd..63730e3fb46 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -196,7 +196,8 @@ struct tree *lookup_tree(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tree_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tree_node());\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346915","messageId":"20180507225916.155236-4-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 03/13] object: add repository argument to grow_object_hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:06Z","receivedAt":"2018-05-07T22:59:37Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"From: Jonathan Nieder \n\nAdd a repository argument to allow the caller of grow_object_hash to\nbe more specific about which repository to handle. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 2de029275bc..91edc30770c 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,7 +116,8 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-static void grow_object_hash(void)\n+#define grow_object_hash(r) grow_object_hash_##r()\n+static void grow_object_hash_the_repository(void)\n {\n \tint i;\n \t/*\n@@ -147,7 +148,7 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \thashcpy(obj->oid.hash, sha1);\n \n \tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash();\n+\t\tgrow_object_hash(the_repository);\n \n \tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n \t\t\tthe_repository->parsed_objects->obj_hash_size);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346916","messageId":"20180507225916.155236-5-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 04/13] alloc: add repository argument to alloc_blob_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:07Z","receivedAt":"2018-05-07T22:59:41Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n blob.c | 2 +-\n cache.h | 3 ++-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacdd..6c5c376a25a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -49,7 +49,7 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n }\n \n static struct alloc_state blob_state;\n-void *alloc_blob_node(void)\n+void *alloc_blob_node_the_repository(void)\n {\n \tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\ndiff --git a/blob.c b/blob.c\nindex 85c2143f299..9e64f301895 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -9,7 +9,7 @@ struct blob *lookup_blob(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_blob_node());\n+\t\t\t\t alloc_blob_node(the_repository));\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 3a4d80e92bf..2258e611275 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1764,7 +1764,8 @@ int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n /* alloc.c */\n-extern void *alloc_blob_node(void);\n+#define alloc_blob_node(r) alloc_blob_node_##r()\n+extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346917","messageId":"20180507225916.155236-9-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 08/13] alloc: add repository argument to alloc_object_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:11Z","receivedAt":"2018-05-07T22:59:43Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 290250e3595..f031ce422d9 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -73,7 +73,7 @@ void *alloc_tag_node_the_repository(void)\n }\n \n static struct alloc_state object_state;\n-void *alloc_object_node(void)\n+void *alloc_object_node_the_repository(void)\n {\n \tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\ndiff --git a/cache.h b/cache.h\nindex 32f340cde59..2d60359a964 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1772,7 +1772,8 @@ extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node_the_repository(void);\n #define alloc_tag_node(r) alloc_tag_node_##r()\n extern void *alloc_tag_node_the_repository(void);\n-extern void *alloc_object_node(void);\n+#define alloc_object_node(r) alloc_object_node_##r()\n+extern void *alloc_object_node_the_repository(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\n \ndiff --git a/object.c b/object.c\nindex 91edc30770c..b8c3f923c51 100644\n--- a/object.c\n+++ b/object.c\n@@ -180,7 +180,7 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n \t\tobj = create_object(the_repository, sha1,\n-\t\t\t\t alloc_object_node());\n+\t\t\t\t alloc_object_node(the_repository));\n \treturn obj;\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346918","messageId":"20180507225916.155236-11-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 10/13] alloc: add repository argument to alloc_commit_index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:13Z","receivedAt":"2018-05-07T22:59:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 4 ++--\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 28b85b22144..277dadd221b 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -82,7 +82,7 @@ void *alloc_object_node_the_repository(void)\n \n static struct alloc_state commit_state;\n \n-unsigned int alloc_commit_index(void)\n+unsigned int alloc_commit_index_the_repository(void)\n {\n \tstatic unsigned int count;\n \treturn count++;\n@@ -92,7 +92,7 @@ void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index();\n+\tc->index = alloc_commit_index(the_repository);\n \treturn c;\n }\n \ndiff --git a/cache.h b/cache.h\nindex 01cc207d218..0e6c5dd5639 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1776,7 +1776,8 @@ extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node_the_repository(void);\n #define alloc_report(r) alloc_report_##r()\n extern void alloc_report_the_repository(void);\n-extern unsigned int alloc_commit_index(void);\n+#define alloc_commit_index(r) alloc_commit_index_##r()\n+extern unsigned int alloc_commit_index_the_repository(void);\n \n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\ndiff --git a/object.c b/object.c\nindex b8c3f923c51..a365a910859 100644\n--- a/object.c\n+++ b/object.c\n@@ -162,7 +162,7 @@ void *object_as_type(struct object *obj, enum object_type type, int quiet)\n \t\treturn obj;\n \telse if (obj->type == OBJ_NONE) {\n \t\tif (type == OBJ_COMMIT)\n-\t\t\t((struct commit *)obj)->index = alloc_commit_index();\n+\t\t\t((struct commit *)obj)->index = alloc_commit_index(the_repository);\n \t\tobj->type = type;\n \t\treturn obj;\n \t}\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346919","messageId":"20180507225916.155236-12-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 11/13] object: allow grow_object_hash to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:14Z","receivedAt":"2018-05-07T22:59:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex a365a910859..0fcd6f6df42 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,27 +116,27 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-#define grow_object_hash(r) grow_object_hash_##r()\n-static void grow_object_hash_the_repository(void)\n+static void grow_object_hash(struct repository *r)\n {\n \tint i;\n \t/*\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n+\tint new_hash_size = r->parsed_objects->obj_hash_size < 32 ? 32 : 2 * r->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n-\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n+\tfor (i = 0; i < r->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = r->parsed_objects->obj_hash[i];\n+\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(the_repository->parsed_objects->obj_hash);\n-\tthe_repository->parsed_objects->obj_hash = new_hash;\n-\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n+\tfree(r->parsed_objects->obj_hash);\n+\tr->parsed_objects->obj_hash = new_hash;\n+\tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object_the_repository(const unsigned char *sha1, void *o)\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346920","messageId":"20180507225916.155236-13-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 12/13] object: allow create_object to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:15Z","receivedAt":"2018-05-07T22:59:49Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \n---\n object.c | 12 ++++++------\n object.h | 3 +--\n 2 files changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 0fcd6f6df42..49b952e9299 100644\n--- a/object.c\n+++ b/object.c\n@@ -139,7 +139,7 @@ static void grow_object_hash(struct repository *r)\n \tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object_the_repository(const unsigned char *sha1, void *o)\n+void *create_object(struct repository *r, const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -147,12 +147,12 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash(the_repository);\n+\tif (r->parsed_objects->obj_hash_size - 1 <= r->parsed_objects->nr_objs * 2)\n+\t\tgrow_object_hash(r);\n \n-\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n-\t\t\tthe_repository->parsed_objects->obj_hash_size);\n-\tthe_repository->parsed_objects->nr_objs++;\n+\tinsert_obj_hash(obj, r->parsed_objects->obj_hash,\n+\t\t\tr->parsed_objects->obj_hash_size);\n+\tr->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex 2cb0b241083..b41d7a3accb 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,8 +93,7 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-#define create_object(r, s, o) create_object_##r(s, o)\n-extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n+extern void *create_object(struct repository *r, const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346921","messageId":"20180507225916.155236-14-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:16Z","receivedAt":"2018-05-07T22:59:52Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 63 +++++++++++++++++++++++++++++------------------\n alloc.h | 15 +++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 ------------\n commit.c | 1 +\n merge-recursive.c | 1 +\n object.c | 34 +++++++++++++++++++++++--\n object.h | 8 ++++++\n tag.c | 1 +\n tree.c | 1 +\n 11 files changed, 100 insertions(+), 42 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..cbdbbc1dd2d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,8 +4,7 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n */\n@@ -15,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n@@ -30,8 +30,25 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -39,60 +56,57 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tif (!s->nr) {\n \t\ts->nr = BLOCKING;\n \t\ts->p = xmalloc(BLOCKING * node_size);\n+\n+\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\t\ts->slabs[s->slab_nr++] = s->p;\n \t}\n \ts->nr--;\n \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n@@ -103,9 +117,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..26dcf96020e\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,15 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..c3b400d5930 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..aa086a85089 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -14,6 +14,7 @@\n #include \"tree-walk.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n+#include \"alloc.h\"\n #include \"tag.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\ndiff --git a/object.c b/object.c\nindex 49b952e9299..2ea2dc7a1cd 100644\n--- a/object.c\n+++ b/object.c\n@@ -4,6 +4,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"commit.h\"\n+#include \"alloc.h\"\n #include \"tag.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n@@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\to->blob_state = allocate_alloc_state();\n+\to->tree_state = allocate_alloc_state();\n+\to->commit_state = allocate_alloc_state();\n+\to->tag_state = allocate_alloc_state();\n+\to->object_state = allocate_alloc_state();\n+\n \treturn o;\n }\n \n@@ -501,9 +509,31 @@ void raw_object_store_clear(struct raw_object_store *o)\n void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n+\t *\n+\t * Before doing so, we need to free any additional memory\n+\t * the objects may hold.\n \t */\n+\tunsigned i;\n+\n+\tfor (i = 0; i < o->obj_hash_size; i++) {\n+\t\tstruct object *obj = o->obj_hash[i];\n+\n+\t\tif (!obj)\n+\t\t\tcontinue;\n+\n+\t\tif (obj->type == OBJ_TREE) {\n+\t\t\tfree(((struct tree*)obj)->buffer);\n+\t\t} else if (obj->type == OBJ_COMMIT) {\n+\t\t\tfree_commit_list(((struct commit*)obj)->parents);\n+\t\t\tfree(&((struct commit*)obj)->util);\n+\t\t}\n+\t}\n+\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex b41d7a3accb..7916edb4edf 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,6 +4,14 @@\n struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n struct parsed_object_pool *parsed_object_pool_new(void);\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..af6a0725b6a 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346922","messageId":"20180507225916.155236-6-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 05/13] alloc: add repository argument to alloc_tree_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:08Z","receivedAt":"2018-05-07T22:59:55Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tree.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 6c5c376a25a..2c8d1430758 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -57,7 +57,7 @@ void *alloc_blob_node_the_repository(void)\n }\n \n static struct alloc_state tree_state;\n-void *alloc_tree_node(void)\n+void *alloc_tree_node_the_repository(void)\n {\n \tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\ndiff --git a/cache.h b/cache.h\nindex 2258e611275..1717d07a2c5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1766,7 +1766,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n /* alloc.c */\n #define alloc_blob_node(r) alloc_blob_node_##r()\n extern void *alloc_blob_node_the_repository(void);\n-extern void *alloc_tree_node(void);\n+#define alloc_tree_node(r) alloc_tree_node_##r()\n+extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\ndiff --git a/tree.c b/tree.c\nindex 63730e3fb46..58cf19b4fa8 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -197,7 +197,7 @@ struct tree *lookup_tree(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tree_node());\n+\t\t\t\t alloc_tree_node(the_repository));\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346923","messageId":"20180507225916.155236-7-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 06/13] alloc: add repository argument to alloc_commit_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:09Z","receivedAt":"2018-05-07T22:59:58Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n blame.c | 2 +-\n cache.h | 3 ++-\n commit.c | 2 +-\n merge-recursive.c | 2 +-\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 2c8d1430758..9e2b897ec1d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -88,7 +88,7 @@ unsigned int alloc_commit_index(void)\n \treturn count++;\n }\n \n-void *alloc_commit_node(void)\n+void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\ndiff --git a/blame.c b/blame.c\nindex dfa24473dc6..ba9b18e7542 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -161,7 +161,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \n \tread_cache();\n \ttime(&now);\n-\tcommit = alloc_commit_node();\n+\tcommit = alloc_commit_node(the_repository);\n \tcommit->object.parsed = 1;\n \tcommit->date = now;\n \tparent_tail = &commit->parents;\ndiff --git a/cache.h b/cache.h\nindex 1717d07a2c5..bf6e8c87d83 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1768,7 +1768,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n extern void *alloc_blob_node_the_repository(void);\n #define alloc_tree_node(r) alloc_tree_node_##r()\n extern void *alloc_tree_node_the_repository(void);\n-extern void *alloc_commit_node(void);\n+#define alloc_commit_node(r) alloc_commit_node_##r()\n+extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\ndiff --git a/commit.c b/commit.c\nindex 9106acf0aad..a9a43e79bae 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -51,7 +51,7 @@ struct commit *lookup_commit(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_commit_node());\n+\t\t\t\t alloc_commit_node(the_repository));\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624da..6dac8908648 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -98,7 +98,7 @@ static struct tree *shift_tree_object(struct tree *one, struct tree *two,\n \n static struct commit *make_virtual_commit(struct tree *tree, const char *comment)\n {\n-\tstruct commit *commit = alloc_commit_node();\n+\tstruct commit *commit = alloc_commit_node(the_repository);\n \n \tset_merge_remote_desc(commit, comment, (struct object *)commit);\n \tcommit->tree = tree;\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346924","messageId":"20180507225916.155236-8-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 07/13] alloc: add repository argument to alloc_tag_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:10Z","receivedAt":"2018-05-07T23:00:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tag.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 9e2b897ec1d..290250e3595 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -65,7 +65,7 @@ void *alloc_tree_node_the_repository(void)\n }\n \n static struct alloc_state tag_state;\n-void *alloc_tag_node(void)\n+void *alloc_tag_node_the_repository(void)\n {\n \tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\ndiff --git a/cache.h b/cache.h\nindex bf6e8c87d83..32f340cde59 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1770,7 +1770,8 @@ extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node_the_repository(void);\n #define alloc_commit_node(r) alloc_commit_node_##r()\n extern void *alloc_commit_node_the_repository(void);\n-extern void *alloc_tag_node(void);\n+#define alloc_tag_node(r) alloc_tag_node_##r()\n+extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\ndiff --git a/tag.c b/tag.c\nindex 7150b759d66..02ef4eaafc0 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -94,7 +94,7 @@ struct tag *lookup_tag(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tag_node());\n+\t\t\t\t alloc_tag_node(the_repository));\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346925","messageId":"20180507225916.155236-10-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 09/13] alloc: add repository argument to alloc_report","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:12Z","receivedAt":"2018-05-07T23:00:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n 2 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex f031ce422d9..28b85b22144 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -105,7 +105,7 @@ static void report(const char *name, unsigned int count, size_t size)\n #define REPORT(name, type)\t\\\n report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n \n-void alloc_report(void)\n+void alloc_report_the_repository(void)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/cache.h b/cache.h\nindex 2d60359a964..01cc207d218 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1774,7 +1774,8 @@ extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node_the_repository(void);\n #define alloc_object_node(r) alloc_object_node_##r()\n extern void *alloc_object_node_the_repository(void);\n-extern void alloc_report(void);\n+#define alloc_report(r) alloc_report_##r()\n+extern void alloc_report_the_repository(void);\n extern unsigned int alloc_commit_index(void);\n \n /* pkt-line.c */\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346926","messageId":"20180507225916.155236-2-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v2 01/13] repository: introduce parsed objects field","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-07T22:59:04Z","receivedAt":"2018-05-07T23:00:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Convert the existing global cache for parsed objects (obj_hash) into\nrepository-specific parsed object caches. Existing code that uses\nobj_hash are modified to use the parsed object cache of\nthe_repository; future patches will use the parsed object caches of\nother repositories.\n\nAnother future use case for a pool of objects is ease of memory management\nin revision walking: If we can free the rev-list related memory early in\npack-objects (e.g. part of repack operation) then it could lower memory\npressure significantly when running on large repos. While this has been\ndiscussed on the mailing list lately, this series doesn't implement this.\n\nSigned-off-by: Stefan Beller \n---\n object.c | 63 +++++++++++++++++++++++++++++++++-------------------\n object.h | 8 +++++++\n repository.c | 7 ++++++\n repository.h | 13 ++++++++++-\n 4 files changed, 67 insertions(+), 24 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 5044d08e96c..f7c624a7ba6 100644\n--- a/object.c\n+++ b/object.c\n@@ -8,17 +8,14 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n-static struct object **obj_hash;\n-static int nr_objs, obj_hash_size;\n-\n unsigned int get_max_object_index(void)\n {\n-\treturn obj_hash_size;\n+\treturn the_repository->parsed_objects->obj_hash_size;\n }\n \n struct object *get_indexed_object(unsigned int idx)\n {\n-\treturn obj_hash[idx];\n+\treturn the_repository->parsed_objects->obj_hash[idx];\n }\n \n static const char *object_type_strings[] = {\n@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)\n \tunsigned int i, first;\n \tstruct object *obj;\n \n-\tif (!obj_hash)\n+\tif (!the_repository->parsed_objects->obj_hash)\n \t\treturn NULL;\n \n-\tfirst = i = hash_obj(sha1, obj_hash_size);\n-\twhile ((obj = obj_hash[i]) != NULL) {\n+\tfirst = i = hash_obj(sha1,\n+\t\t\t the_repository->parsed_objects->obj_hash_size);\n+\twhile ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {\n \t\tif (!hashcmp(sha1, obj->oid.hash))\n \t\t\tbreak;\n \t\ti++;\n-\t\tif (i == obj_hash_size)\n+\t\tif (i == the_repository->parsed_objects->obj_hash_size)\n \t\t\ti = 0;\n \t}\n \tif (obj && i != first) {\n@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)\n \t\t * that we do not need to walk the hash table the next\n \t\t * time we look for it.\n \t\t */\n-\t\tSWAP(obj_hash[i], obj_hash[first]);\n+\t\tSWAP(the_repository->parsed_objects->obj_hash[i],\n+\t\t the_repository->parsed_objects->obj_hash[first]);\n \t}\n \treturn obj;\n }\n@@ -124,19 +123,19 @@ static void grow_object_hash(void)\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;\n+\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(obj_hash);\n-\tobj_hash = new_hash;\n-\tobj_hash_size = new_hash_size;\n+\tfree(the_repository->parsed_objects->obj_hash);\n+\tthe_repository->parsed_objects->obj_hash = new_hash;\n+\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object(const unsigned char *sha1, void *o)\n@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (obj_hash_size - 1 <= nr_objs * 2)\n+\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n \t\tgrow_object_hash();\n \n-\tinsert_obj_hash(obj, obj_hash, obj_hash_size);\n-\tnr_objs++;\n+\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n+\t\t\tthe_repository->parsed_objects->obj_hash_size);\n+\tthe_repository->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \n@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)\n {\n \tint i;\n \n-\tfor (i=0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj)\n \t\t\tobj->flags &= ~flags;\n \t}\n@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)\n {\n \tint i;\n \n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj && obj->type == OBJ_COMMIT)\n \t\t\tobj->flags &= ~flags;\n \t}\n }\n \n+struct parsed_object_pool *parsed_object_pool_new(void)\n+{\n+\tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n+\tmemset(o, 0, sizeof(*o));\n+\treturn o;\n+}\n+\n struct raw_object_store *raw_object_store_new(void)\n {\n \tstruct raw_object_store *o = xmalloc(sizeof(*o));\n@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)\n \tclose_all_packs(o);\n \to->packed_git = NULL;\n }\n+\n+void parsed_object_pool_clear(struct parsed_object_pool *o)\n+{\n+\t/*\n+\t * TOOD free objects in o->obj_hash.\n+\t *\n+\t * As objects are allocated in slabs (see alloc.c), we do\n+\t * not need to free each object, but each slab instead.\n+\t */\n+}\ndiff --git a/object.h b/object.h\nindex f13f85b2a94..cecda7da370 100644\n--- a/object.h\n+++ b/object.h\n@@ -1,6 +1,14 @@\n #ifndef OBJECT_H\n #define OBJECT_H\n \n+struct parsed_object_pool {\n+\tstruct object **obj_hash;\n+\tint nr_objs, obj_hash_size;\n+};\n+\n+struct parsed_object_pool *parsed_object_pool_new(void);\n+void parsed_object_pool_clear(struct parsed_object_pool *o);\n+\n struct object_list {\n \tstruct object *item;\n \tstruct object_list *next;\ndiff --git a/repository.c b/repository.c\nindex a4848c1bd05..c23404677eb 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"config.h\"\n+#include \"object.h\"\n #include \"submodule-config.h\"\n \n /* The main repository */\n@@ -14,6 +15,8 @@ void initialize_the_repository(void)\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n+\tthe_repo.parsed_objects = parsed_object_pool_new();\n+\n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n \n@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n+\trepo->parsed_objects = parsed_object_pool_new();\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\n@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)\n \traw_object_store_clear(repo->objects);\n \tFREE_AND_NULL(repo->objects);\n \n+\tparsed_object_pool_clear(repo->parsed_objects);\n+\tFREE_AND_NULL(repo->parsed_objects);\n+\n \tif (repo->config) {\n \t\tgit_configset_clear(repo->config);\n \t\tFREE_AND_NULL(repo->config);\ndiff --git a/repository.h b/repository.h\nindex e6e00f541bd..73389e81afd 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -22,10 +22,21 @@ struct repository {\n \tchar *commondir;\n \n \t/*\n-\t * Holds any information related to accessing the raw object content.\n+\t * Holds any information needed to retrieve the raw content\n+\t * of objects. The object_parser uses this to get object\n+\t * content which it then parses.\n \t */\n \tstruct raw_object_store *objects;\n \n+\t/*\n+\t * All objects in this repository that have been parsed. This structure\n+\t * owns all objects it references, so users of \"struct object *\"\n+\t * generally do not need to free them; instead, when a repository is no\n+\t * longer used, call parsed_object_pool_clear() on this structure, which\n+\t * is called by the repositories repo_clear on its desconstruction.\n+\t */\n+\tstruct parsed_object_pool *parsed_objects;\n+\n \t/* The store in which the refs are held. */\n \tstruct ref_store *refs;\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"346973","messageId":"20180508101022.GA17501@sigill.intra.peff.net","threadId":"48394","inReplyTo":"20180507225916.155236-14-sbeller@google.com","subject":"Re: [PATCH v2 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-05-08T10:10:22Z","receivedAt":"2018-05-08T10:10:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, May 07, 2018 at 03:59:16PM -0700, Stefan Beller wrote:\n\n> @@ -501,9 +509,31 @@ void raw_object_store_clear(struct raw_object_store *o)\n> void parsed_object_pool_clear(struct parsed_object_pool *o)\n> [...]\n> +\tfor (i = 0; i < o->obj_hash_size; i++) {\n> +\t\tstruct object *obj = o->obj_hash[i];\n> +\n> +\t\tif (!obj)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (obj->type == OBJ_TREE) {\n> +\t\t\tfree(((struct tree*)obj)->buffer);\n> +\t\t} else if (obj->type == OBJ_COMMIT) {\n> +\t\t\tfree_commit_list(((struct commit*)obj)->parents);\n> +\t\t\tfree(&((struct commit*)obj)->util);\n> +\t\t}\n> +\t}\n\nCoverity complains about this final free(). I think the \"&\" is doing an\nincorrect extra level of indirection?\n\nThat said, I'm not sure if it is safe to blindly free the util field. We\ndon't necessarily know what downstream code has pointed it to. It may\nnot be allocated memory[1], or it may even be a more complicated data\nstructure that has sub-components that need freeing[2].\n\nIn the long run, it may be worth trying to get rid of this util field\ncompletely, in favor of having callers use a commit_slab. That has\nbetter memory-ownership semantics, and it would save 8 bytes in struct\ncommit.\n\n[1] Grepping for \"commit->util =\", sequencer.c seems to assign pointers\n into other arrays, as well as the \"(void *)1\".\n\n[2] Most assignments seem to be flex-structs, but blame.c assigns a\n linked list.\n\n-Peff\n"},{"id":"346989","messageId":"CACsJy8D-e-bff3S+LQAMfwB-w8OpkjrfFrV9O5S3ku+M5aAjQA@mail.gmail.com","threadId":"48394","inReplyTo":"20180507225916.155236-14-sbeller@google.com","subject":"Re: [PATCH v2 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-08T15:00:33Z","receivedAt":"2018-05-08T15:01:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 8, 2018 at 12:59 AM, Stefan Beller wrote:\n> @@ -501,9 +509,31 @@ void raw_object_store_clear(struct raw_object_store *o)\n> void parsed_object_pool_clear(struct parsed_object_pool *o)\n> {\n> /*\n> - * TOOD free objects in o->obj_hash.\n> - *\n> * As objects are allocated in slabs (see alloc.c), we do\n> * not need to free each object, but each slab instead.\n> + *\n> + * Before doing so, we need to free any additional memory\n> + * the objects may hold.\n> */\n> + unsigned i;\n> +\n> + for (i = 0; i < o->obj_hash_size; i++) {\n> + struct object *obj = o->obj_hash[i];\n> +\n> + if (!obj)\n> + continue;\n> +\n> + if (obj->type == OBJ_TREE) {\n> + free(((struct tree*)obj)->buffer);\n\nIt would be nicer to keep this in separate functions, e.g.\nrelease_tree_node() and release_commit_node() to go with\nalloc_xxx_node().\n\n> + } else if (obj->type == OBJ_COMMIT) {\n> + free_commit_list(((struct commit*)obj)->parents);\n> + free(&((struct commit*)obj)->util);\n> + }\n> + }\n\nI still don't see who frees obj_hash[] (or at least clears it if not\nfreed). If I'm going to use this to free memory in pack-objects then\nI'd really prefer obj_hash[] freed because it's a big _big_ array.\n\nJust to be clear, what I mean is\n\nFREE_AND_NULL(o->obj_hash);\no->obj_hash_size = 0;\n\n> +\n> + clear_alloc_state(o->blob_state);\n> + clear_alloc_state(o->tree_state);\n> + clear_alloc_state(o->commit_state);\n> + clear_alloc_state(o->tag_state);\n> + clear_alloc_state(o->object_state);\n> }\n-- \nDuy\n"},{"id":"346996","messageId":"20180508102352.2d6d77fb07c60fc788debc33@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-2-sbeller@google.com","subject":"Re: [PATCH v2 01/13] repository: introduce parsed objects field","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-08T17:23:52Z","receivedAt":"2018-05-08T17:23:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 7 May 2018 15:59:04 -0700\nStefan Beller wrote:\n\n> \t/*\n> -\t * Holds any information related to accessing the raw object content.\n> +\t * Holds any information needed to retrieve the raw content\n> +\t * of objects. The object_parser uses this to get object\n> +\t * content which it then parses.\n\nUpdate this comment - there is no more object_parser. (Maybe just delete\nthe last sentence, or specifically name some of the functions that\naccess this field.)\n\n> \t */\n> \tstruct raw_object_store *objects;\n\nOther than that, this patch looks good.\n"},{"id":"346999","messageId":"20180508104551.8af4e13823a26680f07611be@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-14-sbeller@google.com","subject":"Re: [PATCH v2 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-08T17:45:51Z","receivedAt":"2018-05-08T17:45:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 7 May 2018 15:59:16 -0700\nStefan Beller wrote:\n\n> +\tfor (i = 0; i < o->obj_hash_size; i++) {\n> +\t\tstruct object *obj = o->obj_hash[i];\n> +\n> +\t\tif (!obj)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (obj->type == OBJ_TREE) {\n> +\t\t\tfree(((struct tree*)obj)->buffer);\n> +\t\t} else if (obj->type == OBJ_COMMIT) {\n> +\t\t\tfree_commit_list(((struct commit*)obj)->parents);\n> +\t\t\tfree(&((struct commit*)obj)->util);\n> +\t\t}\n\nBesides the other comments by Peff and Duy, should the \"tag\" field of a\ntag object be freed too? It is allocated by xmemdupz in tag.c, and is\nnot assigned to by any other code (verified by renaming it and then\nfixing the compile errors one by one).\n\nOther than that, and other than my small comment on patch 1, this patch\nset looks good to me.\n"},{"id":"347010","messageId":"CAGZ79kZhAjArSksrWMzJc49BKu1HF-R6kBHEzkus-in4fbTZvQ@mail.gmail.com","threadId":"48394","inReplyTo":"CACsJy8D-e-bff3S+LQAMfwB-w8OpkjrfFrV9O5S3ku+M5aAjQA@mail.gmail.com","subject":"Re: [PATCH v2 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T18:38:01Z","receivedAt":"2018-05-08T18:38:08Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, May 8, 2018 at 8:00 AM, Duy Nguyen wrote:\n> On Tue, May 8, 2018 at 12:59 AM, Stefan Beller wrote:\n>> @@ -501,9 +509,31 @@ void raw_object_store_clear(struct raw_object_store *o)\n>> void parsed_object_pool_clear(struct parsed_object_pool *o)\n>> {\n>> /*\n>> - * TOOD free objects in o->obj_hash.\n>> - *\n>> * As objects are allocated in slabs (see alloc.c), we do\n>> * not need to free each object, but each slab instead.\n>> + *\n>> + * Before doing so, we need to free any additional memory\n>> + * the objects may hold.\n>> */\n>> + unsigned i;\n>> +\n>> + for (i = 0; i < o->obj_hash_size; i++) {\n>> + struct object *obj = o->obj_hash[i];\n>> +\n>> + if (!obj)\n>> + continue;\n>> +\n>> + if (obj->type == OBJ_TREE) {\n>> + free(((struct tree*)obj)->buffer);\n>\n> It would be nicer to keep this in separate functions, e.g.\n> release_tree_node() and release_commit_node() to go with\n> alloc_xxx_node().\n\nok, I can introduce that, although it seems unnecessary complicated\nfor now.\n\nOn top of this series I started an experiment (which rewrites alloc\nand object.c a whole lot more; for performance reasons), which gets\nrid of the multiple alloc_states. There will be only one allocation for\none repository, it can allocate across multiple types without alignment\noverhead. It will reduce memory footprint of obj_hash by half, via\nstoring indexes instead of pointers in there.\nThat said, the experiment shall not influence the\ndirection of this series. Will fix.\n\n>> + } else if (obj->type == OBJ_COMMIT) {\n>> + free_commit_list(((struct commit*)obj)->parents);\n>> + free(&((struct commit*)obj)->util);\n>> + }\n>> + }\n>\n> I still don't see who frees obj_hash[] (or at least clears it if not\n> freed). If I'm going to use this to free memory in pack-objects then\n> I'd really prefer obj_hash[] freed because it's a big _big_ array.\n\ngah!\n\n> Just to be clear, what I mean is\n>\n> FREE_AND_NULL(o->obj_hash);\n> o->obj_hash_size = 0;\n\nok, I just put it here, just before the calls\nto clear_alloc_state()s.\n"},{"id":"347013","messageId":"20180508193736.14883-1-sbeller@google.com","threadId":"48394","inReplyTo":"20180507225916.155236-1-sbeller@google.com","subject":"[PATCH v3 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:23Z","receivedAt":"2018-05-08T19:37:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v3:\n\n* I used the (soon to be renamed?) branch-diff tool to attach a diff below\n between v2 and v3 \n \n* fixed comment in patch 1\n* correctly free objects and its hashmap in the last patch.\n* drop free'ing the commit->util pointer as we do not know where\n it points to.\n\nv2:\n* I decided to stick with alloc.c and not migrate it to the mem-pool for now.\n The reasoning for that is that mem-pool.h would introduce some alignment\n waste, which I really did not want to.\n* renamed to struct parsed_object_pool;\n* free'd the additional memory of trees and commits.\n* do not special case the_repository for allocation purposes\n* corrected&polished commit messages\n* I used the (soon to be renamed?) branch-diff tool to attach a diff below.\n (I still need to get used to that format, I find an interdiff of the\n branches easier to read, but that would not yield the commit messages)\n\n\n\nv1:\nThis applies on top of sb/oid-object-info and is the logical continuum of\nthe series that it builds on; this brings the object store into more of\nGits code, removing global state, such that reasoning about the state of\nthe in-memory representation of the repository is easier.\n\nMy original plan was to convert lookup_commit_graft as the next series,\nwhich would be similar to lookup_replace_object, as in sb/object-store-replace.\nThe grafts and shallow mechanism are very close to each other, such that\nthey need to be converted at the same time, both depending on the\n\"parsed object store\" that is introduced in this commit.\n\nThe next series will then convert code in {object/blob/tree/commit/tag}.c\nhopefully finishing the lookup_* functions.\n\nI also debated if it is worth converting alloc.c via this patch series\nor if it might make more sense to use the new mem-pool by Jameson[1].\n\nI vaguely wonder about the performance impact, as the object allocation\ncode seemed to be relevant in the past.\n\n[1] https://public-inbox.org/git/20180430153122.243976-1-jamill@microsoft.com/\n\nAny comments welcome,\nThanks,\nStefan\n\nJonathan Nieder (1):\n object: add repository argument to grow_object_hash\n\nStefan Beller (12):\n repository: introduce parsed objects field\n object: add repository argument to create_object\n alloc: add repository argument to alloc_blob_node\n alloc: add repository argument to alloc_tree_node\n alloc: add repository argument to alloc_commit_node\n alloc: add repository argument to alloc_tag_node\n alloc: add repository argument to alloc_object_node\n alloc: add repository argument to alloc_report\n alloc: add repository argument to alloc_commit_index\n object: allow grow_object_hash to handle arbitrary repositories\n object: allow create_object to handle arbitrary repositories\n alloc: allow arbitrary repositories for alloc functions\n\n alloc.c | 79 ++++++++++++++++++++++-----------\n alloc.h | 23 ++++++++++\n blame.c | 3 +-\n blob.c | 5 ++-\n cache.h | 9 ----\n commit.c | 4 +-\n merge-recursive.c | 3 +-\n object.c | 108 ++++++++++++++++++++++++++++++++++------------\n object.h | 18 +++++++-\n repository.c | 7 +++\n repository.h | 9 ++++\n tag.c | 4 +-\n tree.c | 4 +-\n 13 files changed, 208 insertions(+), 68 deletions(-)\n create mode 100644 alloc.h\n\n-- \n2.17.0.255.g8bfb7c0704\n\n1: 9efc685875b ! 1: f8e521c7c11 repository: introduce parsed objects field\n @@ -15,7 +15,6 @@\n discussed on the mailing list lately, this series doesn't implement this.\n \n Signed-off-by: Stefan Beller \n - Signed-off-by: Junio C Hamano \n \n diff --git a/object.c b/object.c\n --- a/object.c\n @@ -224,13 +223,6 @@\n --- a/repository.h\n +++ b/repository.h\n @@\n - \tchar *commondir;\n - \n - \t/*\n --\t * Holds any information related to accessing the raw object content.\n -+\t * Holds any information needed to retrieve the raw content\n -+\t * of objects. The object_parser uses this to get object\n -+\t * content which it then parses.\n \t */\n \tstruct raw_object_store *objects;\n \n2: 0d41290a9e6 = 2: 55c555b32eb object: add repository argument to create_object\n3: 0242ec870f5 = 3: f1661c9e46a object: add repository argument to grow_object_hash\n4: 9a6aeee10db = 4: f72b25db946 alloc: add repository argument to alloc_blob_node\n5: f7ed8da3909 = 5: 87b7ddda195 alloc: add repository argument to alloc_tree_node\n6: 253f1bf5c2a = 6: 4480e916bdf alloc: add repository argument to alloc_commit_node\n7: 4f8d3dfd460 = 7: c3aa2a7c252 alloc: add repository argument to alloc_tag_node\n8: 6ce5d5b0f0e = 8: 59d33cfaff2 alloc: add repository argument to alloc_object_node\n9: 104f158fc37 = 9: 2ba78c289c1 alloc: add repository argument to alloc_report\n10: 38d90052c29 = 10: 10ce6c44d4b alloc: add repository argument to alloc_commit_index\n11: eae3dea5763 = 11: eae95e75b0b object: allow grow_object_hash to handle arbitrary repositories\n12: d08b382662f = 12: c6d86c8b5db object: allow create_object to handle arbitrary repositories\n13: f87e600c439 ! 13: 2a56520e7af alloc: allow arbitrary repositories for alloc functions\n @@ -139,6 +139,25 @@\n \treturn c;\n }\n \n ++void release_tree_node(struct tree *t)\n ++{\n ++\tfree(t->buffer);\n ++}\n ++\n ++void release_commit_node(struct commit *c)\n ++{\n ++\tfree_commit_list(c->parents);\n ++\t/* TODO: what about commit->util? */\n ++}\n ++\n ++void release_tag_node(struct tag *t)\n ++{\n ++\tfree(t->tag);\n ++}\n ++\n + static void report(const char *name, unsigned int count, size_t size)\n + {\n + \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\n @@\n }\n \n @@ -161,6 +180,10 @@\n +#ifndef ALLOC_H\n +#define ALLOC_H\n +\n ++struct tree;\n ++struct commit;\n ++struct tag;\n ++\n +void *alloc_blob_node(struct repository *r);\n +void *alloc_tree_node(struct repository *r);\n +void *alloc_commit_node(struct repository *r);\n @@ -172,6 +195,10 @@\n +void *allocate_alloc_state(void);\n +void clear_alloc_state(struct alloc_state *s);\n +\n ++void release_tree_node(struct tree *t);\n ++void release_commit_node(struct commit *c);\n ++void release_tag_node(struct tag *t);\n ++\n +#endif\n \n diff --git a/blame.c b/blame.c\n @@ -241,25 +268,25 @@\n --- a/merge-recursive.c\n +++ b/merge-recursive.c\n @@\n - #include \"tree-walk.h\"\n #include \"diff.h\"\n #include \"diffcore.h\"\n -+#include \"alloc.h\"\n #include \"tag.h\"\n ++#include \"alloc.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\n + #include \"xdiff-interface.h\"\n \n diff --git a/object.c b/object.c\n --- a/object.c\n +++ b/object.c\n @@\n - #include \"blob.h\"\n #include \"tree.h\"\n #include \"commit.h\"\n -+#include \"alloc.h\"\n #include \"tag.h\"\n ++#include \"alloc.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n + \n @@\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n @@ -294,14 +321,17 @@\n +\t\tif (!obj)\n +\t\t\tcontinue;\n +\n -+\t\tif (obj->type == OBJ_TREE) {\n -+\t\t\tfree(((struct tree*)obj)->buffer);\n -+\t\t} else if (obj->type == OBJ_COMMIT) {\n -+\t\t\tfree_commit_list(((struct commit*)obj)->parents);\n -+\t\t\tfree(&((struct commit*)obj)->util);\n -+\t\t}\n ++\t\tif (obj->type == OBJ_TREE)\n ++\t\t\trelease_tree_node((struct tree*)obj);\n ++\t\telse if (obj->type == OBJ_COMMIT)\n ++\t\t\trelease_commit_node((struct commit*)obj);\n ++\t\telse if (obj->type == OBJ_TAG)\n ++\t\t\trelease_tag_node((struct tag*)obj);\n +\t}\n +\n ++\tFREE_AND_NULL(o->obj_hash);\n ++\to->obj_hash_size = 0;\n ++\n +\tclear_alloc_state(o->blob_state);\n +\tclear_alloc_state(o->tree_state);\n +\tclear_alloc_state(o->commit_state);\n"},{"id":"347014","messageId":"20180508193736.14883-2-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 01/13] repository: introduce parsed objects field","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:24Z","receivedAt":"2018-05-08T19:37:49Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Convert the existing global cache for parsed objects (obj_hash) into\nrepository-specific parsed object caches. Existing code that uses\nobj_hash are modified to use the parsed object cache of\nthe_repository; future patches will use the parsed object caches of\nother repositories.\n\nAnother future use case for a pool of objects is ease of memory management\nin revision walking: If we can free the rev-list related memory early in\npack-objects (e.g. part of repack operation) then it could lower memory\npressure significantly when running on large repos. While this has been\ndiscussed on the mailing list lately, this series doesn't implement this.\n\nSigned-off-by: Stefan Beller \n---\n object.c | 63 +++++++++++++++++++++++++++++++++-------------------\n object.h | 8 +++++++\n repository.c | 7 ++++++\n repository.h | 9 ++++++++\n 4 files changed, 64 insertions(+), 23 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 5044d08e96c..f7c624a7ba6 100644\n--- a/object.c\n+++ b/object.c\n@@ -8,17 +8,14 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n-static struct object **obj_hash;\n-static int nr_objs, obj_hash_size;\n-\n unsigned int get_max_object_index(void)\n {\n-\treturn obj_hash_size;\n+\treturn the_repository->parsed_objects->obj_hash_size;\n }\n \n struct object *get_indexed_object(unsigned int idx)\n {\n-\treturn obj_hash[idx];\n+\treturn the_repository->parsed_objects->obj_hash[idx];\n }\n \n static const char *object_type_strings[] = {\n@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)\n \tunsigned int i, first;\n \tstruct object *obj;\n \n-\tif (!obj_hash)\n+\tif (!the_repository->parsed_objects->obj_hash)\n \t\treturn NULL;\n \n-\tfirst = i = hash_obj(sha1, obj_hash_size);\n-\twhile ((obj = obj_hash[i]) != NULL) {\n+\tfirst = i = hash_obj(sha1,\n+\t\t\t the_repository->parsed_objects->obj_hash_size);\n+\twhile ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {\n \t\tif (!hashcmp(sha1, obj->oid.hash))\n \t\t\tbreak;\n \t\ti++;\n-\t\tif (i == obj_hash_size)\n+\t\tif (i == the_repository->parsed_objects->obj_hash_size)\n \t\t\ti = 0;\n \t}\n \tif (obj && i != first) {\n@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)\n \t\t * that we do not need to walk the hash table the next\n \t\t * time we look for it.\n \t\t */\n-\t\tSWAP(obj_hash[i], obj_hash[first]);\n+\t\tSWAP(the_repository->parsed_objects->obj_hash[i],\n+\t\t the_repository->parsed_objects->obj_hash[first]);\n \t}\n \treturn obj;\n }\n@@ -124,19 +123,19 @@ static void grow_object_hash(void)\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;\n+\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(obj_hash);\n-\tobj_hash = new_hash;\n-\tobj_hash_size = new_hash_size;\n+\tfree(the_repository->parsed_objects->obj_hash);\n+\tthe_repository->parsed_objects->obj_hash = new_hash;\n+\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object(const unsigned char *sha1, void *o)\n@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (obj_hash_size - 1 <= nr_objs * 2)\n+\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n \t\tgrow_object_hash();\n \n-\tinsert_obj_hash(obj, obj_hash, obj_hash_size);\n-\tnr_objs++;\n+\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n+\t\t\tthe_repository->parsed_objects->obj_hash_size);\n+\tthe_repository->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \n@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)\n {\n \tint i;\n \n-\tfor (i=0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj)\n \t\t\tobj->flags &= ~flags;\n \t}\n@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)\n {\n \tint i;\n \n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj && obj->type == OBJ_COMMIT)\n \t\t\tobj->flags &= ~flags;\n \t}\n }\n \n+struct parsed_object_pool *parsed_object_pool_new(void)\n+{\n+\tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n+\tmemset(o, 0, sizeof(*o));\n+\treturn o;\n+}\n+\n struct raw_object_store *raw_object_store_new(void)\n {\n \tstruct raw_object_store *o = xmalloc(sizeof(*o));\n@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)\n \tclose_all_packs(o);\n \to->packed_git = NULL;\n }\n+\n+void parsed_object_pool_clear(struct parsed_object_pool *o)\n+{\n+\t/*\n+\t * TOOD free objects in o->obj_hash.\n+\t *\n+\t * As objects are allocated in slabs (see alloc.c), we do\n+\t * not need to free each object, but each slab instead.\n+\t */\n+}\ndiff --git a/object.h b/object.h\nindex f13f85b2a94..cecda7da370 100644\n--- a/object.h\n+++ b/object.h\n@@ -1,6 +1,14 @@\n #ifndef OBJECT_H\n #define OBJECT_H\n \n+struct parsed_object_pool {\n+\tstruct object **obj_hash;\n+\tint nr_objs, obj_hash_size;\n+};\n+\n+struct parsed_object_pool *parsed_object_pool_new(void);\n+void parsed_object_pool_clear(struct parsed_object_pool *o);\n+\n struct object_list {\n \tstruct object *item;\n \tstruct object_list *next;\ndiff --git a/repository.c b/repository.c\nindex a4848c1bd05..c23404677eb 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"config.h\"\n+#include \"object.h\"\n #include \"submodule-config.h\"\n \n /* The main repository */\n@@ -14,6 +15,8 @@ void initialize_the_repository(void)\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n+\tthe_repo.parsed_objects = parsed_object_pool_new();\n+\n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n \n@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n+\trepo->parsed_objects = parsed_object_pool_new();\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\n@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)\n \traw_object_store_clear(repo->objects);\n \tFREE_AND_NULL(repo->objects);\n \n+\tparsed_object_pool_clear(repo->parsed_objects);\n+\tFREE_AND_NULL(repo->parsed_objects);\n+\n \tif (repo->config) {\n \t\tgit_configset_clear(repo->config);\n \t\tFREE_AND_NULL(repo->config);\ndiff --git a/repository.h b/repository.h\nindex e6e00f541bd..6d199819905 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -26,6 +26,15 @@ struct repository {\n \t */\n \tstruct raw_object_store *objects;\n \n+\t/*\n+\t * All objects in this repository that have been parsed. This structure\n+\t * owns all objects it references, so users of \"struct object *\"\n+\t * generally do not need to free them; instead, when a repository is no\n+\t * longer used, call parsed_object_pool_clear() on this structure, which\n+\t * is called by the repositories repo_clear on its desconstruction.\n+\t */\n+\tstruct parsed_object_pool *parsed_objects;\n+\n \t/* The store in which the refs are held. */\n \tstruct ref_store *refs;\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347015","messageId":"20180508193736.14883-3-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 02/13] object: add repository argument to create_object","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:25Z","receivedAt":"2018-05-08T19:37:53Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Add a repository argument to allow the callers of create_object\nto be more specific about which repository to act on. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n blob.c | 4 +++-\n commit.c | 3 ++-\n object.c | 5 +++--\n object.h | 3 ++-\n tag.c | 3 ++-\n tree.c | 3 ++-\n 6 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/blob.c b/blob.c\nindex fa2ab4f7a74..85c2143f299 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"blob.h\"\n+#include \"repository.h\"\n \n const char *blob_type = \"blob\";\n \n@@ -7,7 +8,8 @@ struct blob *lookup_blob(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_blob_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_blob_node());\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/commit.c b/commit.c\nindex ca474a7c112..9106acf0aad 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -50,7 +50,8 @@ struct commit *lookup_commit(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_commit_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_commit_node());\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/object.c b/object.c\nindex f7c624a7ba6..2de029275bc 100644\n--- a/object.c\n+++ b/object.c\n@@ -138,7 +138,7 @@ static void grow_object_hash(void)\n \tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object(const unsigned char *sha1, void *o)\n+void *create_object_the_repository(const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -178,7 +178,8 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n-\t\tobj = create_object(sha1, alloc_object_node());\n+\t\tobj = create_object(the_repository, sha1,\n+\t\t\t\t alloc_object_node());\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex cecda7da370..2cb0b241083 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,7 +93,8 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-extern void *create_object(const unsigned char *sha1, void *obj);\n+#define create_object(r, s, o) create_object_##r(s, o)\n+extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \ndiff --git a/tag.c b/tag.c\nindex 3d37c1bd251..7150b759d66 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -93,7 +93,8 @@ struct tag *lookup_tag(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tag_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tag_node());\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \ndiff --git a/tree.c b/tree.c\nindex 1c68ea586bd..63730e3fb46 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -196,7 +196,8 @@ struct tree *lookup_tree(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tree_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tree_node());\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347016","messageId":"20180508193736.14883-6-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 05/13] alloc: add repository argument to alloc_tree_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:28Z","receivedAt":"2018-05-08T19:37:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tree.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 6c5c376a25a..2c8d1430758 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -57,7 +57,7 @@ void *alloc_blob_node_the_repository(void)\n }\n \n static struct alloc_state tree_state;\n-void *alloc_tree_node(void)\n+void *alloc_tree_node_the_repository(void)\n {\n \tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\ndiff --git a/cache.h b/cache.h\nindex 2258e611275..1717d07a2c5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1766,7 +1766,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n /* alloc.c */\n #define alloc_blob_node(r) alloc_blob_node_##r()\n extern void *alloc_blob_node_the_repository(void);\n-extern void *alloc_tree_node(void);\n+#define alloc_tree_node(r) alloc_tree_node_##r()\n+extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\ndiff --git a/tree.c b/tree.c\nindex 63730e3fb46..58cf19b4fa8 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -197,7 +197,7 @@ struct tree *lookup_tree(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tree_node());\n+\t\t\t\t alloc_tree_node(the_repository));\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347017","messageId":"20180508193736.14883-8-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 07/13] alloc: add repository argument to alloc_tag_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:30Z","receivedAt":"2018-05-08T19:37:59Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tag.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 9e2b897ec1d..290250e3595 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -65,7 +65,7 @@ void *alloc_tree_node_the_repository(void)\n }\n \n static struct alloc_state tag_state;\n-void *alloc_tag_node(void)\n+void *alloc_tag_node_the_repository(void)\n {\n \tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\ndiff --git a/cache.h b/cache.h\nindex bf6e8c87d83..32f340cde59 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1770,7 +1770,8 @@ extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node_the_repository(void);\n #define alloc_commit_node(r) alloc_commit_node_##r()\n extern void *alloc_commit_node_the_repository(void);\n-extern void *alloc_tag_node(void);\n+#define alloc_tag_node(r) alloc_tag_node_##r()\n+extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\ndiff --git a/tag.c b/tag.c\nindex 7150b759d66..02ef4eaafc0 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -94,7 +94,7 @@ struct tag *lookup_tag(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tag_node());\n+\t\t\t\t alloc_tag_node(the_repository));\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347018","messageId":"20180508193736.14883-10-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 09/13] alloc: add repository argument to alloc_report","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:32Z","receivedAt":"2018-05-08T19:38:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n 2 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex f031ce422d9..28b85b22144 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -105,7 +105,7 @@ static void report(const char *name, unsigned int count, size_t size)\n #define REPORT(name, type)\t\\\n report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n \n-void alloc_report(void)\n+void alloc_report_the_repository(void)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/cache.h b/cache.h\nindex 2d60359a964..01cc207d218 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1774,7 +1774,8 @@ extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node_the_repository(void);\n #define alloc_object_node(r) alloc_object_node_##r()\n extern void *alloc_object_node_the_repository(void);\n-extern void alloc_report(void);\n+#define alloc_report(r) alloc_report_##r()\n+extern void alloc_report_the_repository(void);\n extern unsigned int alloc_commit_index(void);\n \n /* pkt-line.c */\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347019","messageId":"20180508193736.14883-11-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 10/13] alloc: add repository argument to alloc_commit_index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:33Z","receivedAt":"2018-05-08T19:38:03Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 4 ++--\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 28b85b22144..277dadd221b 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -82,7 +82,7 @@ void *alloc_object_node_the_repository(void)\n \n static struct alloc_state commit_state;\n \n-unsigned int alloc_commit_index(void)\n+unsigned int alloc_commit_index_the_repository(void)\n {\n \tstatic unsigned int count;\n \treturn count++;\n@@ -92,7 +92,7 @@ void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index();\n+\tc->index = alloc_commit_index(the_repository);\n \treturn c;\n }\n \ndiff --git a/cache.h b/cache.h\nindex 01cc207d218..0e6c5dd5639 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1776,7 +1776,8 @@ extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node_the_repository(void);\n #define alloc_report(r) alloc_report_##r()\n extern void alloc_report_the_repository(void);\n-extern unsigned int alloc_commit_index(void);\n+#define alloc_commit_index(r) alloc_commit_index_##r()\n+extern unsigned int alloc_commit_index_the_repository(void);\n \n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\ndiff --git a/object.c b/object.c\nindex b8c3f923c51..a365a910859 100644\n--- a/object.c\n+++ b/object.c\n@@ -162,7 +162,7 @@ void *object_as_type(struct object *obj, enum object_type type, int quiet)\n \t\treturn obj;\n \telse if (obj->type == OBJ_NONE) {\n \t\tif (type == OBJ_COMMIT)\n-\t\t\t((struct commit *)obj)->index = alloc_commit_index();\n+\t\t\t((struct commit *)obj)->index = alloc_commit_index(the_repository);\n \t\tobj->type = type;\n \t\treturn obj;\n \t}\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347020","messageId":"20180508193736.14883-12-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 11/13] object: allow grow_object_hash to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:34Z","receivedAt":"2018-05-08T19:38:07Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex a365a910859..0fcd6f6df42 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,27 +116,27 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-#define grow_object_hash(r) grow_object_hash_##r()\n-static void grow_object_hash_the_repository(void)\n+static void grow_object_hash(struct repository *r)\n {\n \tint i;\n \t/*\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n+\tint new_hash_size = r->parsed_objects->obj_hash_size < 32 ? 32 : 2 * r->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n-\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n+\tfor (i = 0; i < r->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = r->parsed_objects->obj_hash[i];\n+\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(the_repository->parsed_objects->obj_hash);\n-\tthe_repository->parsed_objects->obj_hash = new_hash;\n-\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n+\tfree(r->parsed_objects->obj_hash);\n+\tr->parsed_objects->obj_hash = new_hash;\n+\tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object_the_repository(const unsigned char *sha1, void *o)\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347021","messageId":"20180508193736.14883-13-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 12/13] object: allow create_object to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:35Z","receivedAt":"2018-05-08T19:38:10Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 12 ++++++------\n object.h | 3 +--\n 2 files changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 0fcd6f6df42..49b952e9299 100644\n--- a/object.c\n+++ b/object.c\n@@ -139,7 +139,7 @@ static void grow_object_hash(struct repository *r)\n \tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object_the_repository(const unsigned char *sha1, void *o)\n+void *create_object(struct repository *r, const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -147,12 +147,12 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash(the_repository);\n+\tif (r->parsed_objects->obj_hash_size - 1 <= r->parsed_objects->nr_objs * 2)\n+\t\tgrow_object_hash(r);\n \n-\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n-\t\t\tthe_repository->parsed_objects->obj_hash_size);\n-\tthe_repository->parsed_objects->nr_objs++;\n+\tinsert_obj_hash(obj, r->parsed_objects->obj_hash,\n+\t\t\tr->parsed_objects->obj_hash_size);\n+\tr->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex 2cb0b241083..b41d7a3accb 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,8 +93,7 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-#define create_object(r, s, o) create_object_##r(s, o)\n-extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n+extern void *create_object(struct repository *r, const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347022","messageId":"20180508193736.14883-14-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:36Z","receivedAt":"2018-05-08T19:38:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 79 +++++++++++++++++++++++++++++++++--------------\n alloc.h | 23 ++++++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 ----------\n commit.c | 1 +\n merge-recursive.c | 1 +\n object.c | 37 ++++++++++++++++++++--\n object.h | 8 +++++\n tag.c | 1 +\n tree.c | 1 +\n 11 files changed, 127 insertions(+), 42 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..4ecf0f160f4 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,8 +4,7 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n */\n@@ -15,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n@@ -30,8 +30,25 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -39,63 +56,76 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tif (!s->nr) {\n \t\ts->nr = BLOCKING;\n \t\ts->p = xmalloc(BLOCKING * node_size);\n+\n+\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\t\ts->slabs[s->slab_nr++] = s->p;\n \t}\n \ts->nr--;\n \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n+void release_tree_node(struct tree *t)\n+{\n+\tfree(t->buffer);\n+}\n+\n+void release_commit_node(struct commit *c)\n+{\n+\tfree_commit_list(c->parents);\n+\t/* TODO: what about commit->util? */\n+}\n+\n+void release_tag_node(struct tag *t)\n+{\n+\tfree(t->tag);\n+}\n+\n static void report(const char *name, unsigned int count, size_t size)\n {\n \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\n@@ -103,9 +133,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..941d71960fb\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,23 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+struct tree;\n+struct commit;\n+struct tag;\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+void release_tree_node(struct tree *t);\n+void release_commit_node(struct commit *c);\n+void release_tag_node(struct tag *t);\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..c3b400d5930 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..cbded673c28 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -15,6 +15,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\n #include \"xdiff-interface.h\"\ndiff --git a/object.c b/object.c\nindex 49b952e9299..803d34ae189 100644\n--- a/object.c\n+++ b/object.c\n@@ -5,6 +5,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n@@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\to->blob_state = allocate_alloc_state();\n+\to->tree_state = allocate_alloc_state();\n+\to->commit_state = allocate_alloc_state();\n+\to->tag_state = allocate_alloc_state();\n+\to->object_state = allocate_alloc_state();\n+\n \treturn o;\n }\n \n@@ -501,9 +509,34 @@ void raw_object_store_clear(struct raw_object_store *o)\n void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n+\t *\n+\t * Before doing so, we need to free any additional memory\n+\t * the objects may hold.\n \t */\n+\tunsigned i;\n+\n+\tfor (i = 0; i < o->obj_hash_size; i++) {\n+\t\tstruct object *obj = o->obj_hash[i];\n+\n+\t\tif (!obj)\n+\t\t\tcontinue;\n+\n+\t\tif (obj->type == OBJ_TREE)\n+\t\t\trelease_tree_node((struct tree*)obj);\n+\t\telse if (obj->type == OBJ_COMMIT)\n+\t\t\trelease_commit_node((struct commit*)obj);\n+\t\telse if (obj->type == OBJ_TAG)\n+\t\t\trelease_tag_node((struct tag*)obj);\n+\t}\n+\n+\tFREE_AND_NULL(o->obj_hash);\n+\to->obj_hash_size = 0;\n+\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex b41d7a3accb..7916edb4edf 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,6 +4,14 @@\n struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n struct parsed_object_pool *parsed_object_pool_new(void);\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..af6a0725b6a 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347023","messageId":"20180508193736.14883-9-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 08/13] alloc: add repository argument to alloc_object_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:31Z","receivedAt":"2018-05-08T19:38:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 290250e3595..f031ce422d9 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -73,7 +73,7 @@ void *alloc_tag_node_the_repository(void)\n }\n \n static struct alloc_state object_state;\n-void *alloc_object_node(void)\n+void *alloc_object_node_the_repository(void)\n {\n \tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\ndiff --git a/cache.h b/cache.h\nindex 32f340cde59..2d60359a964 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1772,7 +1772,8 @@ extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node_the_repository(void);\n #define alloc_tag_node(r) alloc_tag_node_##r()\n extern void *alloc_tag_node_the_repository(void);\n-extern void *alloc_object_node(void);\n+#define alloc_object_node(r) alloc_object_node_##r()\n+extern void *alloc_object_node_the_repository(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\n \ndiff --git a/object.c b/object.c\nindex 91edc30770c..b8c3f923c51 100644\n--- a/object.c\n+++ b/object.c\n@@ -180,7 +180,7 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n \t\tobj = create_object(the_repository, sha1,\n-\t\t\t\t alloc_object_node());\n+\t\t\t\t alloc_object_node(the_repository));\n \treturn obj;\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347024","messageId":"20180508193736.14883-4-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 03/13] object: add repository argument to grow_object_hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:26Z","receivedAt":"2018-05-08T19:38:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"From: Jonathan Nieder \n\nAdd a repository argument to allow the caller of grow_object_hash to\nbe more specific about which repository to handle. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 2de029275bc..91edc30770c 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,7 +116,8 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-static void grow_object_hash(void)\n+#define grow_object_hash(r) grow_object_hash_##r()\n+static void grow_object_hash_the_repository(void)\n {\n \tint i;\n \t/*\n@@ -147,7 +148,7 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \thashcpy(obj->oid.hash, sha1);\n \n \tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash();\n+\t\tgrow_object_hash(the_repository);\n \n \tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n \t\t\tthe_repository->parsed_objects->obj_hash_size);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347025","messageId":"20180508193736.14883-5-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 04/13] alloc: add repository argument to alloc_blob_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:27Z","receivedAt":"2018-05-08T19:38:25Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n blob.c | 2 +-\n cache.h | 3 ++-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacdd..6c5c376a25a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -49,7 +49,7 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n }\n \n static struct alloc_state blob_state;\n-void *alloc_blob_node(void)\n+void *alloc_blob_node_the_repository(void)\n {\n \tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\ndiff --git a/blob.c b/blob.c\nindex 85c2143f299..9e64f301895 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -9,7 +9,7 @@ struct blob *lookup_blob(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_blob_node());\n+\t\t\t\t alloc_blob_node(the_repository));\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 3a4d80e92bf..2258e611275 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1764,7 +1764,8 @@ int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n /* alloc.c */\n-extern void *alloc_blob_node(void);\n+#define alloc_blob_node(r) alloc_blob_node_##r()\n+extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347026","messageId":"20180508193736.14883-7-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v3 06/13] alloc: add repository argument to alloc_commit_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T19:37:29Z","receivedAt":"2018-05-08T19:38:27Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n blame.c | 2 +-\n cache.h | 3 ++-\n commit.c | 2 +-\n merge-recursive.c | 2 +-\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 2c8d1430758..9e2b897ec1d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -88,7 +88,7 @@ unsigned int alloc_commit_index(void)\n \treturn count++;\n }\n \n-void *alloc_commit_node(void)\n+void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\ndiff --git a/blame.c b/blame.c\nindex dfa24473dc6..ba9b18e7542 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -161,7 +161,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \n \tread_cache();\n \ttime(&now);\n-\tcommit = alloc_commit_node();\n+\tcommit = alloc_commit_node(the_repository);\n \tcommit->object.parsed = 1;\n \tcommit->date = now;\n \tparent_tail = &commit->parents;\ndiff --git a/cache.h b/cache.h\nindex 1717d07a2c5..bf6e8c87d83 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1768,7 +1768,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n extern void *alloc_blob_node_the_repository(void);\n #define alloc_tree_node(r) alloc_tree_node_##r()\n extern void *alloc_tree_node_the_repository(void);\n-extern void *alloc_commit_node(void);\n+#define alloc_commit_node(r) alloc_commit_node_##r()\n+extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\ndiff --git a/commit.c b/commit.c\nindex 9106acf0aad..a9a43e79bae 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -51,7 +51,7 @@ struct commit *lookup_commit(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_commit_node());\n+\t\t\t\t alloc_commit_node(the_repository));\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624da..6dac8908648 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -98,7 +98,7 @@ static struct tree *shift_tree_object(struct tree *one, struct tree *two,\n \n static struct commit *make_virtual_commit(struct tree *tree, const char *comment)\n {\n-\tstruct commit *commit = alloc_commit_node();\n+\tstruct commit *commit = alloc_commit_node(the_repository);\n \n \tset_merge_remote_desc(commit, comment, (struct object *)commit);\n \tcommit->tree = tree;\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347027","messageId":"20180508130431.287a9f273a847c375b3b1e2b@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-14-sbeller@google.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-08T20:04:31Z","receivedAt":"2018-05-08T20:04:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 8 May 2018 12:37:36 -0700\nStefan Beller wrote:\n\n> +void clear_alloc_state(struct alloc_state *s)\n> +{\n> +\twhile (s->slab_nr > 0) {\n> +\t\ts->slab_nr--;\n> +\t\tfree(s->slabs[s->slab_nr]);\n> +\t}\n\nI should have caught this earlier, but you need to free s->slabs itself\ntoo.\n\n> +void release_tree_node(struct tree *t);\n> +void release_commit_node(struct commit *c);\n> +void release_tag_node(struct tag *t);\n\nDo these really need to be defined in alloc.c? I would think that it\nwould be sufficient to define them as static in object.c.\n\nHaving said that, opinions differ (e.g. Duy said he thinks that release_\ngoes with alloc_ [1]) so I'm OK either way.\n\n[1] https://public-inbox.org/git/CACsJy8D-e-bff3S+LQAMfwB-w8OpkjrfFrV9O5S3ku+M5aAjQA@mail.gmail.com/\n"},{"id":"347028","messageId":"CAGZ79kbgxFJ-+t=50r9gVMTm4xomh+pgm81r2QLnDvTqxpxV6g@mail.gmail.com","threadId":"48394","inReplyTo":"20180508130431.287a9f273a847c375b3b1e2b@google.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-08T20:37:33Z","receivedAt":"2018-05-08T20:37:37Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, May 8, 2018 at 1:04 PM, Jonathan Tan wrote:\n> On Tue, 8 May 2018 12:37:36 -0700\n> Stefan Beller wrote:\n>\n>> +void clear_alloc_state(struct alloc_state *s)\n>> +{\n>> + while (s->slab_nr > 0) {\n>> + s->slab_nr--;\n>> + free(s->slabs[s->slab_nr]);\n>> + }\n>\n> I should have caught this earlier, but you need to free s->slabs itself\n> too.\n\nok.\n\n>\n>> +void release_tree_node(struct tree *t);\n>> +void release_commit_node(struct commit *c);\n>> +void release_tag_node(struct tag *t);\n>\n> Do these really need to be defined in alloc.c? I would think that it\n> would be sufficient to define them as static in object.c.\n>\n> Having said that, opinions differ (e.g. Duy said he thinks that release_\n> goes with alloc_ [1]) so I'm OK either way.\n\nI would have preferred static as well, but went with Duys suggestion of\nhaving it in alloc.c.\n\nI can change that.\n\nThanks,\nStefan\n"},{"id":"347088","messageId":"CACsJy8A3xBz8_MEN-dUcSh317AQn03EP7=TOQ99vzo5zyyd_jg@mail.gmail.com","threadId":"48394","inReplyTo":"CAGZ79kbgxFJ-+t=50r9gVMTm4xomh+pgm81r2QLnDvTqxpxV6g@mail.gmail.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-09T15:54:54Z","receivedAt":"2018-05-09T15:55:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 8, 2018 at 10:37 PM, Stefan Beller wrote:\n>>> +void release_tree_node(struct tree *t);\n>>> +void release_commit_node(struct commit *c);\n>>> +void release_tag_node(struct tag *t);\n>>\n>> Do these really need to be defined in alloc.c? I would think that it\n>> would be sufficient to define them as static in object.c.\n>>\n>> Having said that, opinions differ (e.g. Duy said he thinks that release_\n>> goes with alloc_ [1]) so I'm OK either way.\n>\n> I would have preferred static as well, but went with Duys suggestion of\n> having it in alloc.c.\n>\n> I can change that.\n\nHeh I thought you would make them static ;-) I just wanted to keep\nrelease logic outside that object pool, which is clearer and also\nmakes it easier to replace it with mem-pool.c later. I'm ok with\nmaking it static. Or if you do export these, please move them close to\nthe parse_* functions where memory is actually allocated. E.g.\nrelease_commit_node() is moved to commit.c, close to\nparse_commit_gently(), release_tree_node() close to\nparse_tree_gently().\n-- \nDuy\n"},{"id":"347107","messageId":"CACsJy8C7N2W821H8YR8VaKdCSOSCDtQi_YT7z8hHNDO-VxJmEA@mail.gmail.com","threadId":"48394","inReplyTo":"20180508130431.287a9f273a847c375b3b1e2b@google.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-09T17:18:06Z","receivedAt":"2018-05-09T17:18:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 8, 2018 at 10:04 PM, Jonathan Tan wrote:\n> On Tue, 8 May 2018 12:37:36 -0700\n> Stefan Beller wrote:\n>\n>> +void clear_alloc_state(struct alloc_state *s)\n>> +{\n>> + while (s->slab_nr > 0) {\n>> + s->slab_nr--;\n>> + free(s->slabs[s->slab_nr]);\n>> + }\n>\n> I should have caught this earlier, but you need to free s->slabs itself\n> too.\n\nAnd nobody frees 's' either. I'm not saying cler_alloc_state() should,\nbut somebody should. When I tried repo_clear(the_repository) with\ngitster/sb/object-store-alloc I got this\n\n==13250== 944 (32 direct, 912 indirect) bytes in 1 blocks are\ndefinitely lost in loss record 62 of 88\n==13250== at 0x4C2CF25: calloc (vg_replace_malloc.c:718)\n==13250== by 0x1AB7A8: xcalloc (wrapper.c:160)\n==13250== by 0x1BF666: allocate_alloc_state (alloc.c:41)\n==13250== by 0x140090: parsed_object_pool_new (object.c:462)\n==13250== by 0x16CCF4: initialize_the_repository (repository.c:18)\n==13250== by 0x110BD0: main (common-main.c:37)\n\nIf you want to reproduce, this is what I used to test this with.\n\nhttps://gist.github.com/pclouds/86a2df6c28043f1b6fa3d4e72e7a1276\n-- \nDuy\n"},{"id":"347125","messageId":"CAGZ79kbYuodP12Y2WSwmF+EmStv0pHONwfVbqjDS7Kyhvx103Q@mail.gmail.com","threadId":"48394","inReplyTo":"CACsJy8C7N2W821H8YR8VaKdCSOSCDtQi_YT7z8hHNDO-VxJmEA@mail.gmail.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-09T19:20:48Z","receivedAt":"2018-05-09T19:20:53Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, May 9, 2018 at 10:18 AM, Duy Nguyen wrote:\n>\n> If you want to reproduce, this is what I used to test this with.\n>\n> https://gist.github.com/pclouds/86a2df6c28043f1b6fa3d4e72e7a1276\n\nThis only applied cleanly after I created an empty file at\nt/helper/test-abc.c, using git-apply. I'll use it to have no leaks here.\n\nThanks!\nStefan\n"},{"id":"347153","messageId":"20180510004024.93974-1-sbeller@google.com","threadId":"48394","inReplyTo":"20180508193736.14883-1-sbeller@google.com","subject":"[PATCH v4 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:11Z","receivedAt":"2018-05-10T00:40:33Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"v4:\n* address the memory issues, an interdiff is below.\n\nv3:\n\n* I used the (soon to be renamed?) branch-diff tool to attach a diff below\n between v2 and v3 \n \n* fixed comment in patch 1\n* correctly free objects and its hashmap in the last patch.\n* drop free'ing the commit->util pointer as we do not know where\n it points to.\n\nv2:\n* I decided to stick with alloc.c and not migrate it to the mem-pool for now.\n The reasoning for that is that mem-pool.h would introduce some alignment\n waste, which I really did not want to.\n* renamed to struct parsed_object_pool;\n* free'd the additional memory of trees and commits.\n* do not special case the_repository for allocation purposes\n* corrected&polished commit messages\n* I used the (soon to be renamed?) branch-diff tool to attach a diff below.\n (I still need to get used to that format, I find an interdiff of the\n branches easier to read, but that would not yield the commit messages)\n\n\n\nv1:\nThis applies on top of sb/oid-object-info and is the logical continuum of\nthe series that it builds on; this brings the object store into more of\nGits code, removing global state, such that reasoning about the state of\nthe in-memory representation of the repository is easier.\n\nMy original plan was to convert lookup_commit_graft as the next series,\nwhich would be similar to lookup_replace_object, as in sb/object-store-replace.\nThe grafts and shallow mechanism are very close to each other, such that\nthey need to be converted at the same time, both depending on the\n\"parsed object store\" that is introduced in this commit.\n\nThe next series will then convert code in {object/blob/tree/commit/tag}.c\nhopefully finishing the lookup_* functions.\n\nI also debated if it is worth converting alloc.c via this patch series\nor if it might make more sense to use the new mem-pool by Jameson[1].\n\nI vaguely wonder about the performance impact, as the object allocation\ncode seemed to be relevant in the past.\n\n[1] https://public-inbox.org/git/20180430153122.243976-1-jamill@microsoft.com/\n\nAny comments welcome,\nThanks,\nStefan\n\nJonathan Nieder (1):\n object: add repository argument to grow_object_hash\n\nStefan Beller (12):\n repository: introduce parsed objects field\n object: add repository argument to create_object\n alloc: add repository argument to alloc_blob_node\n alloc: add repository argument to alloc_tree_node\n alloc: add repository argument to alloc_commit_node\n alloc: add repository argument to alloc_tag_node\n alloc: add repository argument to alloc_object_node\n alloc: add repository argument to alloc_report\n alloc: add repository argument to alloc_commit_index\n object: allow grow_object_hash to handle arbitrary repositories\n object: allow create_object to handle arbitrary repositories\n alloc: allow arbitrary repositories for alloc functions\n\n alloc.c | 65 ++++++++++++++++----------\n alloc.h | 19 ++++++++\n blame.c | 3 +-\n blob.c | 5 +-\n cache.h | 9 ----\n commit.c | 11 ++++-\n commit.h | 6 +++\n merge-recursive.c | 3 +-\n object.c | 113 ++++++++++++++++++++++++++++++++++------------\n object.h | 18 +++++++-\n repository.c | 7 +++\n repository.h | 9 ++++\n tag.c | 9 +++-\n tag.h | 1 +\n tree.c | 4 +-\n 15 files changed, 214 insertions(+), 68 deletions(-)\n create mode 100644 alloc.h\n\n-- \n2.17.0.255.g8bfb7c0704\n\ndiff --git c/alloc.c w/alloc.c\nindex 4ecf0f160f4..714df633169 100644\n--- c/alloc.c\n+++ w/alloc.c\n@@ -47,6 +47,8 @@ void clear_alloc_state(struct alloc_state *s)\n \t\ts->slab_nr--;\n \t\tfree(s->slabs[s->slab_nr]);\n \t}\n+\n+\tFREE_AND_NULL(s->slabs);\n }\n \n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n@@ -110,22 +112,6 @@ void *alloc_commit_node(struct repository *r)\n \treturn c;\n }\n \n-void release_tree_node(struct tree *t)\n-{\n-\tfree(t->buffer);\n-}\n-\n-void release_commit_node(struct commit *c)\n-{\n-\tfree_commit_list(c->parents);\n-\t/* TODO: what about commit->util? */\n-}\n-\n-void release_tag_node(struct tag *t)\n-{\n-\tfree(t->tag);\n-}\n-\n static void report(const char *name, unsigned int count, size_t size)\n {\n \tfprintf(stderr, \"%10s: %8u (%\"PRIuMAX\" kB)\\n\",\ndiff --git c/alloc.h w/alloc.h\nindex 941d71960fb..3e4e828db48 100644\n--- c/alloc.h\n+++ w/alloc.h\n@@ -16,8 +16,4 @@ unsigned int alloc_commit_index(struct repository *r);\n void *allocate_alloc_state(void);\n void clear_alloc_state(struct alloc_state *s);\n \n-void release_tree_node(struct tree *t);\n-void release_commit_node(struct commit *c);\n-void release_tag_node(struct tag *t);\n-\n #endif\ndiff --git c/commit.c w/commit.c\nindex c3b400d5930..612ccf7b053 100644\n--- c/commit.c\n+++ w/commit.c\n@@ -297,6 +297,13 @@ void free_commit_buffer(struct commit *commit)\n \t}\n }\n \n+void release_commit_memory(struct commit *c)\n+{\n+\tfree_commit_buffer(c);\n+\tfree_commit_list(c->parents);\n+\t/* TODO: what about commit->util? */\n+}\n+\n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n {\n \tstruct commit_buffer *v = buffer_slab_peek(&buffer_slab, commit);\ndiff --git c/commit.h w/commit.h\nindex 0fb8271665c..2d764ab7d8e 100644\n--- c/commit.h\n+++ w/commit.h\n@@ -99,6 +99,12 @@ void unuse_commit_buffer(const struct commit *, const void *buffer);\n */\n void free_commit_buffer(struct commit *);\n \n+/*\n+ * Release memory related to a commit, including the parent list and\n+ * any cached object buffer.\n+ */\n+void release_commit_memory(struct commit *c);\n+\n /*\n * Disassociate any cached object buffer from the commit, but do not free it.\n * The buffer (or NULL, if none) is returned.\ndiff --git c/object.c w/object.c\nindex 803d34ae189..9d5b10d5a20 100644\n--- c/object.c\n+++ w/object.c\n@@ -524,11 +524,11 @@ void parsed_object_pool_clear(struct parsed_object_pool *o)\n \t\t\tcontinue;\n \n \t\tif (obj->type == OBJ_TREE)\n-\t\t\trelease_tree_node((struct tree*)obj);\n+\t\t\tfree_tree_buffer((struct tree*)obj);\n \t\telse if (obj->type == OBJ_COMMIT)\n-\t\t\trelease_commit_node((struct commit*)obj);\n+\t\t\trelease_commit_memory((struct commit*)obj);\n \t\telse if (obj->type == OBJ_TAG)\n-\t\t\trelease_tag_node((struct tag*)obj);\n+\t\t\tfree_tag_buffer((struct tag*)obj);\n \t}\n \n \tFREE_AND_NULL(o->obj_hash);\n@@ -539,4 +539,9 @@ void parsed_object_pool_clear(struct parsed_object_pool *o)\n \tclear_alloc_state(o->commit_state);\n \tclear_alloc_state(o->tag_state);\n \tclear_alloc_state(o->object_state);\n+\tFREE_AND_NULL(o->blob_state);\n+\tFREE_AND_NULL(o->tree_state);\n+\tFREE_AND_NULL(o->commit_state);\n+\tFREE_AND_NULL(o->tag_state);\n+\tFREE_AND_NULL(o->object_state);\n }\ndiff --git c/tag.c w/tag.c\nindex af6a0725b6a..254352c30c6 100644\n--- c/tag.c\n+++ w/tag.c\n@@ -116,6 +116,11 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n+void free_tag_buffer(struct tag *t)\n+{\n+\tfree(t->tag);\n+}\n+\n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n {\n \tstruct object_id oid;\ndiff --git c/tag.h w/tag.h\nindex d469534e82a..b241fe67bc5 100644\n--- c/tag.h\n+++ w/tag.h\n@@ -15,6 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n+extern void free_tag_buffer(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\n"},{"id":"347154","messageId":"20180510004024.93974-2-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 01/13] repository: introduce parsed objects field","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:12Z","receivedAt":"2018-05-10T00:40:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Convert the existing global cache for parsed objects (obj_hash) into\nrepository-specific parsed object caches. Existing code that uses\nobj_hash are modified to use the parsed object cache of\nthe_repository; future patches will use the parsed object caches of\nother repositories.\n\nAnother future use case for a pool of objects is ease of memory management\nin revision walking: If we can free the rev-list related memory early in\npack-objects (e.g. part of repack operation) then it could lower memory\npressure significantly when running on large repos. While this has been\ndiscussed on the mailing list lately, this series doesn't implement this.\n\nSigned-off-by: Stefan Beller \n---\n object.c | 63 +++++++++++++++++++++++++++++++++-------------------\n object.h | 8 +++++++\n repository.c | 7 ++++++\n repository.h | 9 ++++++++\n 4 files changed, 64 insertions(+), 23 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 5044d08e96c..f7c624a7ba6 100644\n--- a/object.c\n+++ b/object.c\n@@ -8,17 +8,14 @@\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n-static struct object **obj_hash;\n-static int nr_objs, obj_hash_size;\n-\n unsigned int get_max_object_index(void)\n {\n-\treturn obj_hash_size;\n+\treturn the_repository->parsed_objects->obj_hash_size;\n }\n \n struct object *get_indexed_object(unsigned int idx)\n {\n-\treturn obj_hash[idx];\n+\treturn the_repository->parsed_objects->obj_hash[idx];\n }\n \n static const char *object_type_strings[] = {\n@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)\n \tunsigned int i, first;\n \tstruct object *obj;\n \n-\tif (!obj_hash)\n+\tif (!the_repository->parsed_objects->obj_hash)\n \t\treturn NULL;\n \n-\tfirst = i = hash_obj(sha1, obj_hash_size);\n-\twhile ((obj = obj_hash[i]) != NULL) {\n+\tfirst = i = hash_obj(sha1,\n+\t\t\t the_repository->parsed_objects->obj_hash_size);\n+\twhile ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {\n \t\tif (!hashcmp(sha1, obj->oid.hash))\n \t\t\tbreak;\n \t\ti++;\n-\t\tif (i == obj_hash_size)\n+\t\tif (i == the_repository->parsed_objects->obj_hash_size)\n \t\t\ti = 0;\n \t}\n \tif (obj && i != first) {\n@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)\n \t\t * that we do not need to walk the hash table the next\n \t\t * time we look for it.\n \t\t */\n-\t\tSWAP(obj_hash[i], obj_hash[first]);\n+\t\tSWAP(the_repository->parsed_objects->obj_hash[i],\n+\t\t the_repository->parsed_objects->obj_hash[first]);\n \t}\n \treturn obj;\n }\n@@ -124,19 +123,19 @@ static void grow_object_hash(void)\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;\n+\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(obj_hash);\n-\tobj_hash = new_hash;\n-\tobj_hash_size = new_hash_size;\n+\tfree(the_repository->parsed_objects->obj_hash);\n+\tthe_repository->parsed_objects->obj_hash = new_hash;\n+\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object(const unsigned char *sha1, void *o)\n@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (obj_hash_size - 1 <= nr_objs * 2)\n+\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n \t\tgrow_object_hash();\n \n-\tinsert_obj_hash(obj, obj_hash, obj_hash_size);\n-\tnr_objs++;\n+\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n+\t\t\tthe_repository->parsed_objects->obj_hash_size);\n+\tthe_repository->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \n@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)\n {\n \tint i;\n \n-\tfor (i=0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj)\n \t\t\tobj->flags &= ~flags;\n \t}\n@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)\n {\n \tint i;\n \n-\tfor (i = 0; i < obj_hash_size; i++) {\n-\t\tstruct object *obj = obj_hash[i];\n+\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n \t\tif (obj && obj->type == OBJ_COMMIT)\n \t\t\tobj->flags &= ~flags;\n \t}\n }\n \n+struct parsed_object_pool *parsed_object_pool_new(void)\n+{\n+\tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n+\tmemset(o, 0, sizeof(*o));\n+\treturn o;\n+}\n+\n struct raw_object_store *raw_object_store_new(void)\n {\n \tstruct raw_object_store *o = xmalloc(sizeof(*o));\n@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)\n \tclose_all_packs(o);\n \to->packed_git = NULL;\n }\n+\n+void parsed_object_pool_clear(struct parsed_object_pool *o)\n+{\n+\t/*\n+\t * TOOD free objects in o->obj_hash.\n+\t *\n+\t * As objects are allocated in slabs (see alloc.c), we do\n+\t * not need to free each object, but each slab instead.\n+\t */\n+}\ndiff --git a/object.h b/object.h\nindex f13f85b2a94..cecda7da370 100644\n--- a/object.h\n+++ b/object.h\n@@ -1,6 +1,14 @@\n #ifndef OBJECT_H\n #define OBJECT_H\n \n+struct parsed_object_pool {\n+\tstruct object **obj_hash;\n+\tint nr_objs, obj_hash_size;\n+};\n+\n+struct parsed_object_pool *parsed_object_pool_new(void);\n+void parsed_object_pool_clear(struct parsed_object_pool *o);\n+\n struct object_list {\n \tstruct object *item;\n \tstruct object_list *next;\ndiff --git a/repository.c b/repository.c\nindex a4848c1bd05..c23404677eb 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"object-store.h\"\n #include \"config.h\"\n+#include \"object.h\"\n #include \"submodule-config.h\"\n \n /* The main repository */\n@@ -14,6 +15,8 @@ void initialize_the_repository(void)\n \n \tthe_repo.index = &the_index;\n \tthe_repo.objects = raw_object_store_new();\n+\tthe_repo.parsed_objects = parsed_object_pool_new();\n+\n \trepo_set_hash_algo(&the_repo, GIT_HASH_SHA1);\n }\n \n@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,\n \tmemset(repo, 0, sizeof(*repo));\n \n \trepo->objects = raw_object_store_new();\n+\trepo->parsed_objects = parsed_object_pool_new();\n \n \tif (repo_init_gitdir(repo, gitdir))\n \t\tgoto error;\n@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)\n \traw_object_store_clear(repo->objects);\n \tFREE_AND_NULL(repo->objects);\n \n+\tparsed_object_pool_clear(repo->parsed_objects);\n+\tFREE_AND_NULL(repo->parsed_objects);\n+\n \tif (repo->config) {\n \t\tgit_configset_clear(repo->config);\n \t\tFREE_AND_NULL(repo->config);\ndiff --git a/repository.h b/repository.h\nindex e6e00f541bd..6d199819905 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -26,6 +26,15 @@ struct repository {\n \t */\n \tstruct raw_object_store *objects;\n \n+\t/*\n+\t * All objects in this repository that have been parsed. This structure\n+\t * owns all objects it references, so users of \"struct object *\"\n+\t * generally do not need to free them; instead, when a repository is no\n+\t * longer used, call parsed_object_pool_clear() on this structure, which\n+\t * is called by the repositories repo_clear on its desconstruction.\n+\t */\n+\tstruct parsed_object_pool *parsed_objects;\n+\n \t/* The store in which the refs are held. */\n \tstruct ref_store *refs;\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347155","messageId":"20180510004024.93974-4-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 03/13] object: add repository argument to grow_object_hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:14Z","receivedAt":"2018-05-10T00:40:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"From: Jonathan Nieder \n\nAdd a repository argument to allow the caller of grow_object_hash to\nbe more specific about which repository to handle. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 2de029275bc..91edc30770c 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,7 +116,8 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-static void grow_object_hash(void)\n+#define grow_object_hash(r) grow_object_hash_##r()\n+static void grow_object_hash_the_repository(void)\n {\n \tint i;\n \t/*\n@@ -147,7 +148,7 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \thashcpy(obj->oid.hash, sha1);\n \n \tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash();\n+\t\tgrow_object_hash(the_repository);\n \n \tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n \t\t\tthe_repository->parsed_objects->obj_hash_size);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347156","messageId":"20180510004024.93974-7-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 06/13] alloc: add repository argument to alloc_commit_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:17Z","receivedAt":"2018-05-10T00:40:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n blame.c | 2 +-\n cache.h | 3 ++-\n commit.c | 2 +-\n merge-recursive.c | 2 +-\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 2c8d1430758..9e2b897ec1d 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -88,7 +88,7 @@ unsigned int alloc_commit_index(void)\n \treturn count++;\n }\n \n-void *alloc_commit_node(void)\n+void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\ndiff --git a/blame.c b/blame.c\nindex dfa24473dc6..ba9b18e7542 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -161,7 +161,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \n \tread_cache();\n \ttime(&now);\n-\tcommit = alloc_commit_node();\n+\tcommit = alloc_commit_node(the_repository);\n \tcommit->object.parsed = 1;\n \tcommit->date = now;\n \tparent_tail = &commit->parents;\ndiff --git a/cache.h b/cache.h\nindex 1717d07a2c5..bf6e8c87d83 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1768,7 +1768,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n extern void *alloc_blob_node_the_repository(void);\n #define alloc_tree_node(r) alloc_tree_node_##r()\n extern void *alloc_tree_node_the_repository(void);\n-extern void *alloc_commit_node(void);\n+#define alloc_commit_node(r) alloc_commit_node_##r()\n+extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\ndiff --git a/commit.c b/commit.c\nindex 9106acf0aad..a9a43e79bae 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -51,7 +51,7 @@ struct commit *lookup_commit(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_commit_node());\n+\t\t\t\t alloc_commit_node(the_repository));\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 0c0d48624da..6dac8908648 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -98,7 +98,7 @@ static struct tree *shift_tree_object(struct tree *one, struct tree *two,\n \n static struct commit *make_virtual_commit(struct tree *tree, const char *comment)\n {\n-\tstruct commit *commit = alloc_commit_node();\n+\tstruct commit *commit = alloc_commit_node(the_repository);\n \n \tset_merge_remote_desc(commit, comment, (struct object *)commit);\n \tcommit->tree = tree;\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347157","messageId":"20180510004024.93974-9-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 08/13] alloc: add repository argument to alloc_object_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:19Z","receivedAt":"2018-05-10T00:40:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 290250e3595..f031ce422d9 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -73,7 +73,7 @@ void *alloc_tag_node_the_repository(void)\n }\n \n static struct alloc_state object_state;\n-void *alloc_object_node(void)\n+void *alloc_object_node_the_repository(void)\n {\n \tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\ndiff --git a/cache.h b/cache.h\nindex 32f340cde59..2d60359a964 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1772,7 +1772,8 @@ extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node_the_repository(void);\n #define alloc_tag_node(r) alloc_tag_node_##r()\n extern void *alloc_tag_node_the_repository(void);\n-extern void *alloc_object_node(void);\n+#define alloc_object_node(r) alloc_object_node_##r()\n+extern void *alloc_object_node_the_repository(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\n \ndiff --git a/object.c b/object.c\nindex 91edc30770c..b8c3f923c51 100644\n--- a/object.c\n+++ b/object.c\n@@ -180,7 +180,7 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n \t\tobj = create_object(the_repository, sha1,\n-\t\t\t\t alloc_object_node());\n+\t\t\t\t alloc_object_node(the_repository));\n \treturn obj;\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347158","messageId":"20180510004024.93974-11-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 10/13] alloc: add repository argument to alloc_commit_index","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:21Z","receivedAt":"2018-05-10T00:40:50Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 4 ++--\n cache.h | 3 ++-\n object.c | 2 +-\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 28b85b22144..277dadd221b 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -82,7 +82,7 @@ void *alloc_object_node_the_repository(void)\n \n static struct alloc_state commit_state;\n \n-unsigned int alloc_commit_index(void)\n+unsigned int alloc_commit_index_the_repository(void)\n {\n \tstatic unsigned int count;\n \treturn count++;\n@@ -92,7 +92,7 @@ void *alloc_commit_node_the_repository(void)\n {\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index();\n+\tc->index = alloc_commit_index(the_repository);\n \treturn c;\n }\n \ndiff --git a/cache.h b/cache.h\nindex 01cc207d218..0e6c5dd5639 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1776,7 +1776,8 @@ extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node_the_repository(void);\n #define alloc_report(r) alloc_report_##r()\n extern void alloc_report_the_repository(void);\n-extern unsigned int alloc_commit_index(void);\n+#define alloc_commit_index(r) alloc_commit_index_##r()\n+extern unsigned int alloc_commit_index_the_repository(void);\n \n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\ndiff --git a/object.c b/object.c\nindex b8c3f923c51..a365a910859 100644\n--- a/object.c\n+++ b/object.c\n@@ -162,7 +162,7 @@ void *object_as_type(struct object *obj, enum object_type type, int quiet)\n \t\treturn obj;\n \telse if (obj->type == OBJ_NONE) {\n \t\tif (type == OBJ_COMMIT)\n-\t\t\t((struct commit *)obj)->index = alloc_commit_index();\n+\t\t\t((struct commit *)obj)->index = alloc_commit_index(the_repository);\n \t\tobj->type = type;\n \t\treturn obj;\n \t}\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347159","messageId":"20180510004024.93974-13-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 12/13] object: allow create_object to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:23Z","receivedAt":"2018-05-10T00:40:54Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 12 ++++++------\n object.h | 3 +--\n 2 files changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex 0fcd6f6df42..49b952e9299 100644\n--- a/object.c\n+++ b/object.c\n@@ -139,7 +139,7 @@ static void grow_object_hash(struct repository *r)\n \tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object_the_repository(const unsigned char *sha1, void *o)\n+void *create_object(struct repository *r, const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -147,12 +147,12 @@ void *create_object_the_repository(const unsigned char *sha1, void *o)\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n-\tif (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)\n-\t\tgrow_object_hash(the_repository);\n+\tif (r->parsed_objects->obj_hash_size - 1 <= r->parsed_objects->nr_objs * 2)\n+\t\tgrow_object_hash(r);\n \n-\tinsert_obj_hash(obj, the_repository->parsed_objects->obj_hash,\n-\t\t\tthe_repository->parsed_objects->obj_hash_size);\n-\tthe_repository->parsed_objects->nr_objs++;\n+\tinsert_obj_hash(obj, r->parsed_objects->obj_hash,\n+\t\t\tr->parsed_objects->obj_hash_size);\n+\tr->parsed_objects->nr_objs++;\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex 2cb0b241083..b41d7a3accb 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,8 +93,7 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-#define create_object(r, s, o) create_object_##r(s, o)\n-extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n+extern void *create_object(struct repository *r, const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347160","messageId":"20180510004024.93974-12-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 11/13] object: allow grow_object_hash to handle arbitrary repositories","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:22Z","receivedAt":"2018-05-10T00:40:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Reviewed-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n object.c | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/object.c b/object.c\nindex a365a910859..0fcd6f6df42 100644\n--- a/object.c\n+++ b/object.c\n@@ -116,27 +116,27 @@ struct object *lookup_object(const unsigned char *sha1)\n * power of 2 (but at least 32). Copy the existing values to the new\n * hash map.\n */\n-#define grow_object_hash(r) grow_object_hash_##r()\n-static void grow_object_hash_the_repository(void)\n+static void grow_object_hash(struct repository *r)\n {\n \tint i;\n \t/*\n \t * Note that this size must always be power-of-2 to match hash_obj\n \t * above.\n \t */\n-\tint new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;\n+\tint new_hash_size = r->parsed_objects->obj_hash_size < 32 ? 32 : 2 * r->parsed_objects->obj_hash_size;\n \tstruct object **new_hash;\n \n \tnew_hash = xcalloc(new_hash_size, sizeof(struct object *));\n-\tfor (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {\n-\t\tstruct object *obj = the_repository->parsed_objects->obj_hash[i];\n+\tfor (i = 0; i < r->parsed_objects->obj_hash_size; i++) {\n+\t\tstruct object *obj = r->parsed_objects->obj_hash[i];\n+\n \t\tif (!obj)\n \t\t\tcontinue;\n \t\tinsert_obj_hash(obj, new_hash, new_hash_size);\n \t}\n-\tfree(the_repository->parsed_objects->obj_hash);\n-\tthe_repository->parsed_objects->obj_hash = new_hash;\n-\tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n+\tfree(r->parsed_objects->obj_hash);\n+\tr->parsed_objects->obj_hash = new_hash;\n+\tr->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n void *create_object_the_repository(const unsigned char *sha1, void *o)\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347161","messageId":"20180510004024.93974-14-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:24Z","receivedAt":"2018-05-10T00:40:57Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 65 ++++++++++++++++++++++++++++++-----------------\n alloc.h | 19 ++++++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 ------------\n commit.c | 8 ++++++\n commit.h | 6 +++++\n merge-recursive.c | 1 +\n object.c | 42 ++++++++++++++++++++++++++++--\n object.h | 8 ++++++\n tag.c | 6 +++++\n tag.h | 1 +\n tree.c | 1 +\n 13 files changed, 133 insertions(+), 42 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..714df633169 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,8 +4,7 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n */\n@@ -15,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n@@ -30,8 +30,27 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+\n+\tFREE_AND_NULL(s->slabs);\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -39,60 +58,57 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tif (!s->nr) {\n \t\ts->nr = BLOCKING;\n \t\ts->p = xmalloc(BLOCKING * node_size);\n+\n+\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\t\ts->slabs[s->slab_nr++] = s->p;\n \t}\n \ts->nr--;\n \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n@@ -103,9 +119,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..3e4e828db48\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,19 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+struct tree;\n+struct commit;\n+struct tag;\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..612ccf7b053 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\n@@ -296,6 +297,13 @@ void free_commit_buffer(struct commit *commit)\n \t}\n }\n \n+void release_commit_memory(struct commit *c)\n+{\n+\tfree_commit_buffer(c);\n+\tfree_commit_list(c->parents);\n+\t/* TODO: what about commit->util? */\n+}\n+\n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n {\n \tstruct commit_buffer *v = buffer_slab_peek(&buffer_slab, commit);\ndiff --git a/commit.h b/commit.h\nindex 0fb8271665c..2d764ab7d8e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -99,6 +99,12 @@ void unuse_commit_buffer(const struct commit *, const void *buffer);\n */\n void free_commit_buffer(struct commit *);\n \n+/*\n+ * Release memory related to a commit, including the parent list and\n+ * any cached object buffer.\n+ */\n+void release_commit_memory(struct commit *c);\n+\n /*\n * Disassociate any cached object buffer from the commit, but do not free it.\n * The buffer (or NULL, if none) is returned.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..cbded673c28 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -15,6 +15,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\n #include \"xdiff-interface.h\"\ndiff --git a/object.c b/object.c\nindex 49b952e9299..9d5b10d5a20 100644\n--- a/object.c\n+++ b/object.c\n@@ -5,6 +5,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n@@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\to->blob_state = allocate_alloc_state();\n+\to->tree_state = allocate_alloc_state();\n+\to->commit_state = allocate_alloc_state();\n+\to->tag_state = allocate_alloc_state();\n+\to->object_state = allocate_alloc_state();\n+\n \treturn o;\n }\n \n@@ -501,9 +509,39 @@ void raw_object_store_clear(struct raw_object_store *o)\n void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n+\t *\n+\t * Before doing so, we need to free any additional memory\n+\t * the objects may hold.\n \t */\n+\tunsigned i;\n+\n+\tfor (i = 0; i < o->obj_hash_size; i++) {\n+\t\tstruct object *obj = o->obj_hash[i];\n+\n+\t\tif (!obj)\n+\t\t\tcontinue;\n+\n+\t\tif (obj->type == OBJ_TREE)\n+\t\t\tfree_tree_buffer((struct tree*)obj);\n+\t\telse if (obj->type == OBJ_COMMIT)\n+\t\t\trelease_commit_memory((struct commit*)obj);\n+\t\telse if (obj->type == OBJ_TAG)\n+\t\t\tfree_tag_buffer((struct tag*)obj);\n+\t}\n+\n+\tFREE_AND_NULL(o->obj_hash);\n+\to->obj_hash_size = 0;\n+\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n+\tFREE_AND_NULL(o->blob_state);\n+\tFREE_AND_NULL(o->tree_state);\n+\tFREE_AND_NULL(o->commit_state);\n+\tFREE_AND_NULL(o->tag_state);\n+\tFREE_AND_NULL(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex b41d7a3accb..7916edb4edf 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,6 +4,14 @@\n struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n struct parsed_object_pool *parsed_object_pool_new(void);\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..254352c30c6 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\n@@ -115,6 +116,11 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n+void free_tag_buffer(struct tag *t)\n+{\n+\tfree(t->tag);\n+}\n+\n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n {\n \tstruct object_id oid;\ndiff --git a/tag.h b/tag.h\nindex d469534e82a..b241fe67bc5 100644\n--- a/tag.h\n+++ b/tag.h\n@@ -15,6 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n+extern void free_tag_buffer(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347162","messageId":"20180510004024.93974-6-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 05/13] alloc: add repository argument to alloc_tree_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:16Z","receivedAt":"2018-05-10T00:41:04Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tree.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 6c5c376a25a..2c8d1430758 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -57,7 +57,7 @@ void *alloc_blob_node_the_repository(void)\n }\n \n static struct alloc_state tree_state;\n-void *alloc_tree_node(void)\n+void *alloc_tree_node_the_repository(void)\n {\n \tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\ndiff --git a/cache.h b/cache.h\nindex 2258e611275..1717d07a2c5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1766,7 +1766,8 @@ void encode_85(char *buf, const unsigned char *data, int bytes);\n /* alloc.c */\n #define alloc_blob_node(r) alloc_blob_node_##r()\n extern void *alloc_blob_node_the_repository(void);\n-extern void *alloc_tree_node(void);\n+#define alloc_tree_node(r) alloc_tree_node_##r()\n+extern void *alloc_tree_node_the_repository(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n extern void *alloc_object_node(void);\ndiff --git a/tree.c b/tree.c\nindex 63730e3fb46..58cf19b4fa8 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -197,7 +197,7 @@ struct tree *lookup_tree(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tree_node());\n+\t\t\t\t alloc_tree_node(the_repository));\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347163","messageId":"20180510004024.93974-10-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 09/13] alloc: add repository argument to alloc_report","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:20Z","receivedAt":"2018-05-10T00:41:06Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n 2 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex f031ce422d9..28b85b22144 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -105,7 +105,7 @@ static void report(const char *name, unsigned int count, size_t size)\n #define REPORT(name, type)\t\\\n report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n \n-void alloc_report(void)\n+void alloc_report_the_repository(void)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/cache.h b/cache.h\nindex 2d60359a964..01cc207d218 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1774,7 +1774,8 @@ extern void *alloc_commit_node_the_repository(void);\n extern void *alloc_tag_node_the_repository(void);\n #define alloc_object_node(r) alloc_object_node_##r()\n extern void *alloc_object_node_the_repository(void);\n-extern void alloc_report(void);\n+#define alloc_report(r) alloc_report_##r()\n+extern void alloc_report_the_repository(void);\n extern unsigned int alloc_commit_index(void);\n \n /* pkt-line.c */\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347164","messageId":"20180510004024.93974-8-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 07/13] alloc: add repository argument to alloc_tag_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:18Z","receivedAt":"2018-05-10T00:41:07Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n cache.h | 3 ++-\n tag.c | 2 +-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 9e2b897ec1d..290250e3595 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -65,7 +65,7 @@ void *alloc_tree_node_the_repository(void)\n }\n \n static struct alloc_state tag_state;\n-void *alloc_tag_node(void)\n+void *alloc_tag_node_the_repository(void)\n {\n \tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\ndiff --git a/cache.h b/cache.h\nindex bf6e8c87d83..32f340cde59 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1770,7 +1770,8 @@ extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node_the_repository(void);\n #define alloc_commit_node(r) alloc_commit_node_##r()\n extern void *alloc_commit_node_the_repository(void);\n-extern void *alloc_tag_node(void);\n+#define alloc_tag_node(r) alloc_tag_node_##r()\n+extern void *alloc_tag_node_the_repository(void);\n extern void *alloc_object_node(void);\n extern void alloc_report(void);\n extern unsigned int alloc_commit_index(void);\ndiff --git a/tag.c b/tag.c\nindex 7150b759d66..02ef4eaafc0 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -94,7 +94,7 @@ struct tag *lookup_tag(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_tag_node());\n+\t\t\t\t alloc_tag_node(the_repository));\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347165","messageId":"20180510004024.93974-3-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 02/13] object: add repository argument to create_object","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:13Z","receivedAt":"2018-05-10T00:41:11Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Add a repository argument to allow the callers of create_object\nto be more specific about which repository to act on. This is a small\nmechanical change; it doesn't change the implementation to handle\nrepositories other than the_repository yet.\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n blob.c | 4 +++-\n commit.c | 3 ++-\n object.c | 5 +++--\n object.h | 3 ++-\n tag.c | 3 ++-\n tree.c | 3 ++-\n 6 files changed, 14 insertions(+), 7 deletions(-)\n\ndiff --git a/blob.c b/blob.c\nindex fa2ab4f7a74..85c2143f299 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"blob.h\"\n+#include \"repository.h\"\n \n const char *blob_type = \"blob\";\n \n@@ -7,7 +8,8 @@ struct blob *lookup_blob(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_blob_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_blob_node());\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/commit.c b/commit.c\nindex ca474a7c112..9106acf0aad 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -50,7 +50,8 @@ struct commit *lookup_commit(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_commit_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_commit_node());\n \treturn object_as_type(obj, OBJ_COMMIT, 0);\n }\n \ndiff --git a/object.c b/object.c\nindex f7c624a7ba6..2de029275bc 100644\n--- a/object.c\n+++ b/object.c\n@@ -138,7 +138,7 @@ static void grow_object_hash(void)\n \tthe_repository->parsed_objects->obj_hash_size = new_hash_size;\n }\n \n-void *create_object(const unsigned char *sha1, void *o)\n+void *create_object_the_repository(const unsigned char *sha1, void *o)\n {\n \tstruct object *obj = o;\n \n@@ -178,7 +178,8 @@ struct object *lookup_unknown_object(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n \tif (!obj)\n-\t\tobj = create_object(sha1, alloc_object_node());\n+\t\tobj = create_object(the_repository, sha1,\n+\t\t\t\t alloc_object_node());\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex cecda7da370..2cb0b241083 100644\n--- a/object.h\n+++ b/object.h\n@@ -93,7 +93,8 @@ extern struct object *get_indexed_object(unsigned int);\n */\n struct object *lookup_object(const unsigned char *sha1);\n \n-extern void *create_object(const unsigned char *sha1, void *obj);\n+#define create_object(r, s, o) create_object_##r(s, o)\n+extern void *create_object_the_repository(const unsigned char *sha1, void *obj);\n \n void *object_as_type(struct object *obj, enum object_type type, int quiet);\n \ndiff --git a/tag.c b/tag.c\nindex 3d37c1bd251..7150b759d66 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -93,7 +93,8 @@ struct tag *lookup_tag(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tag_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tag_node());\n \treturn object_as_type(obj, OBJ_TAG, 0);\n }\n \ndiff --git a/tree.c b/tree.c\nindex 1c68ea586bd..63730e3fb46 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -196,7 +196,8 @@ struct tree *lookup_tree(const struct object_id *oid)\n {\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n-\t\treturn create_object(oid->hash, alloc_tree_node());\n+\t\treturn create_object(the_repository, oid->hash,\n+\t\t\t\t alloc_tree_node());\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347166","messageId":"20180510004024.93974-5-sbeller@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"[PATCH v4 04/13] alloc: add repository argument to alloc_blob_node","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T00:40:15Z","receivedAt":"2018-05-10T00:41:13Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"This is a small mechanical change; it doesn't change the\nimplementation to handle repositories other than the_repository yet.\nUse a macro to catch callers passing a repository other than\nthe_repository at compile time.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n alloc.c | 2 +-\n blob.c | 2 +-\n cache.h | 3 ++-\n 3 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacdd..6c5c376a25a 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -49,7 +49,7 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n }\n \n static struct alloc_state blob_state;\n-void *alloc_blob_node(void)\n+void *alloc_blob_node_the_repository(void)\n {\n \tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\ndiff --git a/blob.c b/blob.c\nindex 85c2143f299..9e64f301895 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -9,7 +9,7 @@ struct blob *lookup_blob(const struct object_id *oid)\n \tstruct object *obj = lookup_object(oid->hash);\n \tif (!obj)\n \t\treturn create_object(the_repository, oid->hash,\n-\t\t\t\t alloc_blob_node());\n+\t\t\t\t alloc_blob_node(the_repository));\n \treturn object_as_type(obj, OBJ_BLOB, 0);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 3a4d80e92bf..2258e611275 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1764,7 +1764,8 @@ int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n /* alloc.c */\n-extern void *alloc_blob_node(void);\n+#define alloc_blob_node(r) alloc_blob_node_##r()\n+extern void *alloc_blob_node_the_repository(void);\n extern void *alloc_tree_node(void);\n extern void *alloc_commit_node(void);\n extern void *alloc_tag_node(void);\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347251","messageId":"CACsJy8CJu-Og1Mn4Sn+eCyv+KkbJ__21oTYRaMyTdOsjAVxpqw@mail.gmail.com","threadId":"48394","inReplyTo":"CAGZ79kbYuodP12Y2WSwmF+EmStv0pHONwfVbqjDS7Kyhvx103Q@mail.gmail.com","subject":"Re: [PATCH v3 13/13] alloc: allow arbitrary repositories for alloc functions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-05-10T15:43:39Z","receivedAt":"2018-05-10T15:44:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, May 9, 2018 at 9:20 PM, Stefan Beller wrote:\n> On Wed, May 9, 2018 at 10:18 AM, Duy Nguyen wrote:\n>>\n>> If you want to reproduce, this is what I used to test this with.\n>>\n>> https://gist.github.com/pclouds/86a2df6c28043f1b6fa3d4e72e7a1276\n>\n> This only applied cleanly after I created an empty file at\n> t/helper/test-abc.c, using git-apply.\n\nRight. I created the patch with \"git add -N\". I know exactly what the\nbug is but I'll need to be careful with renaming a bit before trying\nto fix this. Thanks for reminding me.\n-- \nDuy\n"},{"id":"347259","messageId":"20180510101615.931eb0d219eac4c84cf8140d@google.com","threadId":"48394","inReplyTo":"20180510004024.93974-1-sbeller@google.com","subject":"Re: [PATCH v4 00/13] object store: alloc","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-10T17:16:15Z","receivedAt":"2018-05-10T17:16:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 9 May 2018 17:40:11 -0700\nStefan Beller wrote:\n\n> \t\tif (obj->type == OBJ_TREE)\n> -\t\t\trelease_tree_node((struct tree*)obj);\n> +\t\t\tfree_tree_buffer((struct tree*)obj);\n> \t\telse if (obj->type == OBJ_COMMIT)\n> -\t\t\trelease_commit_node((struct commit*)obj);\n> +\t\t\trelease_commit_memory((struct commit*)obj);\n> \t\telse if (obj->type == OBJ_TAG)\n> -\t\t\trelease_tag_node((struct tag*)obj);\n> +\t\t\tfree_tag_buffer((struct tag*)obj);\n\nThis might seem a bit bikesheddy, but I wouldn't call it\nfree_tag_buffer(), since what's being freed is not the buffer of the\nobject itself, but just a string. If you want such a function, I would\njust call it release_tag_memory() to match release_commit_memory().\n\nOther than that, all the patches look fine to me.\n\nSome optional comments (this is almost certainly bikeshedding):\n\n - I would call them release_commit() and release_tag(), to match\n strbuf_release().\n - It might be better to just inline the handling of releasing commit\n and tag memory. This code already knows that, for a tree, it needs to\n free its buffer and only its buffer, so it is not much of a stretch\n to think that it similarly knows the details of commit and tag\n objects too.\n"},{"id":"347262","messageId":"CAGZ79kab0RDE1D81-uxrJPRQF7LoWkTfHj=nLntd8v2cz8THHg@mail.gmail.com","threadId":"48394","inReplyTo":"20180510101615.931eb0d219eac4c84cf8140d@google.com","subject":"Re: [PATCH v4 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T17:32:09Z","receivedAt":"2018-05-10T17:32:14Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 10, 2018 at 10:16 AM, Jonathan Tan wrote:\n> On Wed, 9 May 2018 17:40:11 -0700\n> Stefan Beller wrote:\n>\n>> if (obj->type == OBJ_TREE)\n>> - release_tree_node((struct tree*)obj);\n>> + free_tree_buffer((struct tree*)obj);\n>> else if (obj->type == OBJ_COMMIT)\n>> - release_commit_node((struct commit*)obj);\n>> + release_commit_memory((struct commit*)obj);\n>> else if (obj->type == OBJ_TAG)\n>> - release_tag_node((struct tag*)obj);\n>> + free_tag_buffer((struct tag*)obj);\n>\n> This might seem a bit bikesheddy, but I wouldn't call it\n> free_tag_buffer(), since what's being freed is not the buffer of the\n> object itself, but just a string. If you want such a function, I would\n> just call it release_tag_memory() to match release_commit_memory().\n>\n> Other than that, all the patches look fine to me.\n>\n> Some optional comments (this is almost certainly bikeshedding):\n\nWho doesn't love some bikeshedding in late spring?\n\n>\n> - I would call them release_commit() and release_tag(), to match\n> strbuf_release().\n\nWhy not commit_release and tag_release to also have the same order\nof words as in strbuf_release ?\n\n> - It might be better to just inline the handling of releasing commit\n> and tag memory. This code already knows that, for a tree, it needs to\n> free its buffer and only its buffer, so it is not much of a stretch\n> to think that it similarly knows the details of commit and tag\n> objects too.\n\nWe still call out to free_tree_buffer? Not sure I understand.\n"},{"id":"347311","messageId":"20180510135615.aee82c0d9720fd6c1be80585@google.com","threadId":"48394","inReplyTo":"CAGZ79kab0RDE1D81-uxrJPRQF7LoWkTfHj=nLntd8v2cz8THHg@mail.gmail.com","subject":"Re: [PATCH v4 00/13] object store: alloc","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-05-10T20:56:15Z","receivedAt":"2018-05-10T20:56:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 10 May 2018 10:32:09 -0700\nStefan Beller wrote:\n\n> > - I would call them release_commit() and release_tag(), to match\n> > strbuf_release().\n> \n> Why not commit_release and tag_release to also have the same order\n> of words as in strbuf_release ?\n\nAt this point in the discussion, either is fine.\n\n> > - It might be better to just inline the handling of releasing commit\n> > and tag memory. This code already knows that, for a tree, it needs to\n> > free its buffer and only its buffer, so it is not much of a stretch\n> > to think that it similarly knows the details of commit and tag\n> > objects too.\n> \n> We still call out to free_tree_buffer? Not sure I understand.\n\nI meant that since we call out to free_tree_buffer (as you said), this\nshows that the code knows the internal details of a tree object (in that\nit has a buffer, and that needs to be freed, and that is the only thing\nthat needs to be freed), so maybe the code should operate on the\ninternal details of commits and tags as well. But again, this is a minor\npoint.\n"},{"id":"347317","messageId":"CAGZ79kYWzcGaAOUXTxMV2VmkXgcu8oS9RKkeag3+BnBy3krj5g@mail.gmail.com","threadId":"48394","inReplyTo":"20180510135615.aee82c0d9720fd6c1be80585@google.com","subject":"Re: [PATCH v4 00/13] object store: alloc","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T22:36:15Z","receivedAt":"2018-05-10T22:36:19Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 10, 2018 at 1:56 PM, Jonathan Tan wrote:\n> On Thu, 10 May 2018 10:32:09 -0700\n> Stefan Beller wrote:\n>\n>> > - I would call them release_commit() and release_tag(), to match\n>> > strbuf_release().\n>>\n>> Why not commit_release and tag_release to also have the same order\n>> of words as in strbuf_release ?\n>\n> At this point in the discussion, either is fine.\n\nok, please express your opinion in form of a patch. ;)\n\nI do not plan on resending this series unless someone comments\nsomething that they themselves do not consider bikesheddy.\n\nStefan\n"},{"id":"347378","messageId":"20180511191725.54664-1-sbeller@google.com","threadId":"48394","inReplyTo":"20180510101615.931eb0d219eac4c84cf8140d@google.com","subject":"[PATCH] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-11T19:17:25Z","receivedAt":"2018-05-11T19:17:32Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n\n> This might seem a bit bikesheddy, but I wouldn't call it\n> free_tag_buffer(), since what's being freed is not the buffer of the\n> object itself, but just a string. If you want such a function, I would\n> just call it release_tag_memory() to match release_commit_memory().\n\nSo you would replace the last commit with a patch like this?\n\nThanks,\nStefan\n\nNotes:\n diff to what is currently queued:\n diff --git c/commit.c w/commit.c\n index 612ccf7b053..f3a5872c393 100644\n --- c/commit.c\n +++ w/commit.c\n @@ -297,11 +297,15 @@ void free_commit_buffer(struct commit *commit)\n \t}\n }\n \n -void release_commit_memory(struct commit *c)\n +void relase_commit_memory(struct commit *c)\n {\n +\tc->tree = NULL;\n +\tc->index = 0;\n \tfree_commit_buffer(c);\n \tfree_commit_list(c->parents);\n \t/* TODO: what about commit->util? */\n +\n +\tc->object.parsed = 0;\n }\n \n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n diff --git c/commit.h w/commit.h\n index 2d764ab7d8e..366c151e0cb 100644\n --- c/commit.h\n +++ w/commit.h\n @@ -103,7 +103,7 @@ void free_commit_buffer(struct commit *);\n * Release memory related to a commit, including the parent list and\n * any cached object buffer.\n */\n -void release_commit_memory(struct commit *c);\n +void relase_commit_memory(struct commit *c);\n \n /*\n * Disassociate any cached object buffer from the commit, but do not free it.\n diff --git c/object.c w/object.c\n index 9d5b10d5a20..a7d1fd4a20b 100644\n --- c/object.c\n +++ w/object.c\n @@ -526,9 +526,9 @@ void parsed_object_pool_clear(struct parsed_object_pool *o)\n \t\tif (obj->type == OBJ_TREE)\n \t\t\tfree_tree_buffer((struct tree*)obj);\n \t\telse if (obj->type == OBJ_COMMIT)\n -\t\t\trelease_commit_memory((struct commit*)obj);\n +\t\t\trelase_commit_memory((struct commit*)obj);\n \t\telse if (obj->type == OBJ_TAG)\n -\t\t\tfree_tag_buffer((struct tag*)obj);\n +\t\t\trelease_tag_memory((struct tag*)obj);\n \t}\n \n \tFREE_AND_NULL(o->obj_hash);\n diff --git c/tag.c w/tag.c\n index 254352c30c6..7c12426b4ea 100644\n --- c/tag.c\n +++ w/tag.c\n @@ -116,9 +116,12 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n -void free_tag_buffer(struct tag *t)\n +void release_tag_memory(struct tag *t)\n {\n \tfree(t->tag);\n +\tt->tagged = NULL;\n +\tt->object.parsed = 0;\n +\tt->date = 0;\n }\n \n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n diff --git c/tag.h w/tag.h\n index b241fe67bc5..9057d76a506 100644\n --- c/tag.h\n +++ w/tag.h\n @@ -15,7 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n -extern void free_tag_buffer(struct tag *t);\n +extern void release_tag_memory(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\n\n alloc.c | 65 ++++++++++++++++++++++++++++++-----------------\n alloc.h | 19 ++++++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 ------------\n commit.c | 12 +++++++++\n commit.h | 6 +++++\n merge-recursive.c | 1 +\n object.c | 42 ++++++++++++++++++++++++++++--\n object.h | 8 ++++++\n tag.c | 9 +++++++\n tag.h | 1 +\n tree.c | 1 +\n 13 files changed, 140 insertions(+), 42 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..714df633169 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,8 +4,7 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n */\n@@ -15,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n@@ -30,8 +30,27 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+\n+\tFREE_AND_NULL(s->slabs);\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -39,60 +58,57 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tif (!s->nr) {\n \t\ts->nr = BLOCKING;\n \t\ts->p = xmalloc(BLOCKING * node_size);\n+\n+\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\t\ts->slabs[s->slab_nr++] = s->p;\n \t}\n \ts->nr--;\n \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n@@ -103,9 +119,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..3e4e828db48\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,19 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+struct tree;\n+struct commit;\n+struct tag;\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..f3a5872c393 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\n@@ -296,6 +297,17 @@ void free_commit_buffer(struct commit *commit)\n \t}\n }\n \n+void relase_commit_memory(struct commit *c)\n+{\n+\tc->tree = NULL;\n+\tc->index = 0;\n+\tfree_commit_buffer(c);\n+\tfree_commit_list(c->parents);\n+\t/* TODO: what about commit->util? */\n+\n+\tc->object.parsed = 0;\n+}\n+\n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n {\n \tstruct commit_buffer *v = buffer_slab_peek(&buffer_slab, commit);\ndiff --git a/commit.h b/commit.h\nindex 0fb8271665c..366c151e0cb 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -99,6 +99,12 @@ void unuse_commit_buffer(const struct commit *, const void *buffer);\n */\n void free_commit_buffer(struct commit *);\n \n+/*\n+ * Release memory related to a commit, including the parent list and\n+ * any cached object buffer.\n+ */\n+void relase_commit_memory(struct commit *c);\n+\n /*\n * Disassociate any cached object buffer from the commit, but do not free it.\n * The buffer (or NULL, if none) is returned.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..cbded673c28 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -15,6 +15,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\n #include \"xdiff-interface.h\"\ndiff --git a/object.c b/object.c\nindex 49b952e9299..a7d1fd4a20b 100644\n--- a/object.c\n+++ b/object.c\n@@ -5,6 +5,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n@@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\to->blob_state = allocate_alloc_state();\n+\to->tree_state = allocate_alloc_state();\n+\to->commit_state = allocate_alloc_state();\n+\to->tag_state = allocate_alloc_state();\n+\to->object_state = allocate_alloc_state();\n+\n \treturn o;\n }\n \n@@ -501,9 +509,39 @@ void raw_object_store_clear(struct raw_object_store *o)\n void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n+\t *\n+\t * Before doing so, we need to free any additional memory\n+\t * the objects may hold.\n \t */\n+\tunsigned i;\n+\n+\tfor (i = 0; i < o->obj_hash_size; i++) {\n+\t\tstruct object *obj = o->obj_hash[i];\n+\n+\t\tif (!obj)\n+\t\t\tcontinue;\n+\n+\t\tif (obj->type == OBJ_TREE)\n+\t\t\tfree_tree_buffer((struct tree*)obj);\n+\t\telse if (obj->type == OBJ_COMMIT)\n+\t\t\trelase_commit_memory((struct commit*)obj);\n+\t\telse if (obj->type == OBJ_TAG)\n+\t\t\trelease_tag_memory((struct tag*)obj);\n+\t}\n+\n+\tFREE_AND_NULL(o->obj_hash);\n+\to->obj_hash_size = 0;\n+\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n+\tFREE_AND_NULL(o->blob_state);\n+\tFREE_AND_NULL(o->tree_state);\n+\tFREE_AND_NULL(o->commit_state);\n+\tFREE_AND_NULL(o->tag_state);\n+\tFREE_AND_NULL(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex b41d7a3accb..7916edb4edf 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,6 +4,14 @@\n struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n struct parsed_object_pool *parsed_object_pool_new(void);\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..7c12426b4ea 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\n@@ -115,6 +116,14 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n+void release_tag_memory(struct tag *t)\n+{\n+\tfree(t->tag);\n+\tt->tagged = NULL;\n+\tt->object.parsed = 0;\n+\tt->date = 0;\n+}\n+\n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n {\n \tstruct object_id oid;\ndiff --git a/tag.h b/tag.h\nindex d469534e82a..9057d76a506 100644\n--- a/tag.h\n+++ b/tag.h\n@@ -15,6 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n+extern void release_tag_memory(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.255.g8bfb7c0704\n\n"},{"id":"347380","messageId":"CAPig+cTMeYFsn1iTto_wsG0Myik_Euh+UZh3pZwks=AEt8D13Q@mail.gmail.com","threadId":"48394","inReplyTo":"20180511191725.54664-1-sbeller@google.com","subject":"Re: [PATCH] alloc: allow arbitrary repositories for alloc functions","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-05-11T19:38:49Z","receivedAt":"2018-05-11T19:39:05Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 11, 2018 at 3:17 PM Stefan Beller wrote:\n> diff --git a/commit.c b/commit.c\n> @@ -296,6 +297,17 @@ void free_commit_buffer(struct commit *commit)\n> +void relase_commit_memory(struct commit *c)\n\ns/relase/release/\n"},{"id":"347722","messageId":"20180515214842.108713-1-sbeller@google.com","threadId":"48394","inReplyTo":"CAPig+cTMeYFsn1iTto_wsG0Myik_Euh+UZh3pZwks=AEt8D13Q@mail.gmail.com","subject":"[PATCH] alloc: allow arbitrary repositories for alloc functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-15T21:48:42Z","receivedAt":"2018-05-15T21:48:51Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"We have to convert all of the alloc functions at once, because alloc_report\nuses a funky macro for reporting. It is better for the sake of mechanical\nconversion to convert multiple functions at once rather than changing the\nstructure of the reporting function.\n\nWe record all memory allocation in alloc.c, and free them in\nclear_alloc_state, which is called for all repositories except\nthe_repository.\n\nSigned-off-by: Stefan Beller \nSigned-off-by: Junio C Hamano \n---\n\nNotes:\n\n Eric, I have fixed s/relase/release/\n\n\nJonathan,\n\n> This might seem a bit bikesheddy, but I wouldn't call it\n> free_tag_buffer(), since what's being freed is not the buffer of the\n> object itself, but just a string. If you want such a function, I would\n> just call it release_tag_memory() to match release_commit_memory().\n\nSo you would replace the last commit with a patch like this?\n\nThanks,\nStefan\n\n diff to what is currently queued:\n\n diff --git c/commit.c w/commit.c\n index 612ccf7b053..5eb4d2f08f8 100644\n --- c/commit.c\n +++ w/commit.c\n @@ -299,9 +299,13 @@ void free_commit_buffer(struct commit *commit)\n \n void release_commit_memory(struct commit *c)\n {\n +\tc->tree = NULL;\n +\tc->index = 0;\n \tfree_commit_buffer(c);\n \tfree_commit_list(c->parents);\n \t/* TODO: what about commit->util? */\n +\n +\tc->object.parsed = 0;\n }\n \n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n diff --git c/object.c w/object.c\n index 9d5b10d5a20..8e29f63bf23 100644\n --- c/object.c\n +++ w/object.c\n @@ -528,7 +528,7 @@ void parsed_object_pool_clear(struct parsed_object_pool *o)\n \t\telse if (obj->type == OBJ_COMMIT)\n \t\t\trelease_commit_memory((struct commit*)obj);\n \t\telse if (obj->type == OBJ_TAG)\n -\t\t\tfree_tag_buffer((struct tag*)obj);\n +\t\t\trelease_tag_memory((struct tag*)obj);\n \t}\n \n \tFREE_AND_NULL(o->obj_hash);\n diff --git c/tag.c w/tag.c\n index 254352c30c6..7c12426b4ea 100644\n --- c/tag.c\n +++ w/tag.c\n @@ -116,9 +116,12 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n -void free_tag_buffer(struct tag *t)\n +void release_tag_memory(struct tag *t)\n {\n \tfree(t->tag);\n +\tt->tagged = NULL;\n +\tt->object.parsed = 0;\n +\tt->date = 0;\n }\n \n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n diff --git c/tag.h w/tag.h\n index b241fe67bc5..9057d76a506 100644\n --- c/tag.h\n +++ w/tag.h\n @@ -15,7 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n -extern void free_tag_buffer(struct tag *t);\n +extern void release_tag_memory(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\n\n alloc.c | 65 ++++++++++++++++++++++++++++++-----------------\n alloc.h | 19 ++++++++++++++\n blame.c | 1 +\n blob.c | 1 +\n cache.h | 16 ------------\n commit.c | 12 +++++++++\n commit.h | 6 +++++\n merge-recursive.c | 1 +\n object.c | 42 ++++++++++++++++++++++++++++--\n object.h | 8 ++++++\n tag.c | 9 +++++++\n tag.h | 1 +\n tree.c | 1 +\n 13 files changed, 140 insertions(+), 42 deletions(-)\n create mode 100644 alloc.h\n\ndiff --git a/alloc.c b/alloc.c\nindex 277dadd221b..714df633169 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -4,8 +4,7 @@\n * Copyright (C) 2006 Linus Torvalds\n *\n * The standard malloc/free wastes too much space for objects, partly because\n- * it maintains all the allocation infrastructure (which isn't needed, since\n- * we never free an object descriptor anyway), but even more because it ends\n+ * it maintains all the allocation infrastructure, but even more because it ends\n * up with maximal alignment because it doesn't know what the object alignment\n * for the new allocation is.\n */\n@@ -15,6 +14,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n \n #define BLOCKING 1024\n \n@@ -30,8 +30,27 @@ struct alloc_state {\n \tint count; /* total number of nodes allocated */\n \tint nr; /* number of nodes left in current allocation */\n \tvoid *p; /* first free node in current allocation */\n+\n+\t/* bookkeeping of allocations */\n+\tvoid **slabs;\n+\tint slab_nr, slab_alloc;\n };\n \n+void *allocate_alloc_state(void)\n+{\n+\treturn xcalloc(1, sizeof(struct alloc_state));\n+}\n+\n+void clear_alloc_state(struct alloc_state *s)\n+{\n+\twhile (s->slab_nr > 0) {\n+\t\ts->slab_nr--;\n+\t\tfree(s->slabs[s->slab_nr]);\n+\t}\n+\n+\tFREE_AND_NULL(s->slabs);\n+}\n+\n static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n {\n \tvoid *ret;\n@@ -39,60 +58,57 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n \tif (!s->nr) {\n \t\ts->nr = BLOCKING;\n \t\ts->p = xmalloc(BLOCKING * node_size);\n+\n+\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n+\t\ts->slabs[s->slab_nr++] = s->p;\n \t}\n \ts->nr--;\n \ts->count++;\n \tret = s->p;\n \ts->p = (char *)s->p + node_size;\n \tmemset(ret, 0, node_size);\n+\n \treturn ret;\n }\n \n-static struct alloc_state blob_state;\n-void *alloc_blob_node_the_repository(void)\n+void *alloc_blob_node(struct repository *r)\n {\n-\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n+\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n \tb->object.type = OBJ_BLOB;\n \treturn b;\n }\n \n-static struct alloc_state tree_state;\n-void *alloc_tree_node_the_repository(void)\n+void *alloc_tree_node(struct repository *r)\n {\n-\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n+\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n \tt->object.type = OBJ_TREE;\n \treturn t;\n }\n \n-static struct alloc_state tag_state;\n-void *alloc_tag_node_the_repository(void)\n+void *alloc_tag_node(struct repository *r)\n {\n-\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n+\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n \tt->object.type = OBJ_TAG;\n \treturn t;\n }\n \n-static struct alloc_state object_state;\n-void *alloc_object_node_the_repository(void)\n+void *alloc_object_node(struct repository *r)\n {\n-\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n+\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n \tobj->type = OBJ_NONE;\n \treturn obj;\n }\n \n-static struct alloc_state commit_state;\n-\n-unsigned int alloc_commit_index_the_repository(void)\n+unsigned int alloc_commit_index(struct repository *r)\n {\n-\tstatic unsigned int count;\n-\treturn count++;\n+\treturn r->parsed_objects->commit_count++;\n }\n \n-void *alloc_commit_node_the_repository(void)\n+void *alloc_commit_node(struct repository *r)\n {\n-\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n+\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n-\tc->index = alloc_commit_index(the_repository);\n+\tc->index = alloc_commit_index(r);\n \treturn c;\n }\n \n@@ -103,9 +119,10 @@ static void report(const char *name, unsigned int count, size_t size)\n }\n \n #define REPORT(name, type)\t\\\n- report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n+ report(#name, r->parsed_objects->name##_state->count, \\\n+\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n \n-void alloc_report_the_repository(void)\n+void alloc_report(struct repository *r)\n {\n \tREPORT(blob, struct blob);\n \tREPORT(tree, struct tree);\ndiff --git a/alloc.h b/alloc.h\nnew file mode 100644\nindex 00000000000..3e4e828db48\n--- /dev/null\n+++ b/alloc.h\n@@ -0,0 +1,19 @@\n+#ifndef ALLOC_H\n+#define ALLOC_H\n+\n+struct tree;\n+struct commit;\n+struct tag;\n+\n+void *alloc_blob_node(struct repository *r);\n+void *alloc_tree_node(struct repository *r);\n+void *alloc_commit_node(struct repository *r);\n+void *alloc_tag_node(struct repository *r);\n+void *alloc_object_node(struct repository *r);\n+void alloc_report(struct repository *r);\n+unsigned int alloc_commit_index(struct repository *r);\n+\n+void *allocate_alloc_state(void);\n+void clear_alloc_state(struct alloc_state *s);\n+\n+#endif\ndiff --git a/blame.c b/blame.c\nindex ba9b18e7542..3a11f1ce52b 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -6,6 +6,7 @@\n #include \"diffcore.h\"\n #include \"tag.h\"\n #include \"blame.h\"\n+#include \"alloc.h\"\n \n void blame_origin_decref(struct blame_origin *o)\n {\ndiff --git a/blob.c b/blob.c\nindex 9e64f301895..458dafa811e 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"repository.h\"\n+#include \"alloc.h\"\n \n const char *blob_type = \"blob\";\n \ndiff --git a/cache.h b/cache.h\nindex 0e6c5dd5639..c75559b7d38 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n int decode_85(char *dst, const char *line, int linelen);\n void encode_85(char *buf, const unsigned char *data, int bytes);\n \n-/* alloc.c */\n-#define alloc_blob_node(r) alloc_blob_node_##r()\n-extern void *alloc_blob_node_the_repository(void);\n-#define alloc_tree_node(r) alloc_tree_node_##r()\n-extern void *alloc_tree_node_the_repository(void);\n-#define alloc_commit_node(r) alloc_commit_node_##r()\n-extern void *alloc_commit_node_the_repository(void);\n-#define alloc_tag_node(r) alloc_tag_node_##r()\n-extern void *alloc_tag_node_the_repository(void);\n-#define alloc_object_node(r) alloc_object_node_##r()\n-extern void *alloc_object_node_the_repository(void);\n-#define alloc_report(r) alloc_report_##r()\n-extern void alloc_report_the_repository(void);\n-#define alloc_commit_index(r) alloc_commit_index_##r()\n-extern unsigned int alloc_commit_index_the_repository(void);\n-\n /* pkt-line.c */\n void packet_trace_identity(const char *prog);\n \ndiff --git a/commit.c b/commit.c\nindex a9a43e79bae..5eb4d2f08f8 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -6,6 +6,7 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"notes.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n #include \"mergesort.h\"\n #include \"commit-slab.h\"\n@@ -296,6 +297,17 @@ void free_commit_buffer(struct commit *commit)\n \t}\n }\n \n+void release_commit_memory(struct commit *c)\n+{\n+\tc->tree = NULL;\n+\tc->index = 0;\n+\tfree_commit_buffer(c);\n+\tfree_commit_list(c->parents);\n+\t/* TODO: what about commit->util? */\n+\n+\tc->object.parsed = 0;\n+}\n+\n const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n {\n \tstruct commit_buffer *v = buffer_slab_peek(&buffer_slab, commit);\ndiff --git a/commit.h b/commit.h\nindex 0fb8271665c..2d764ab7d8e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -99,6 +99,12 @@ void unuse_commit_buffer(const struct commit *, const void *buffer);\n */\n void free_commit_buffer(struct commit *);\n \n+/*\n+ * Release memory related to a commit, including the parent list and\n+ * any cached object buffer.\n+ */\n+void release_commit_memory(struct commit *c);\n+\n /*\n * Disassociate any cached object buffer from the commit, but do not free it.\n * The buffer (or NULL, if none) is returned.\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 6dac8908648..cbded673c28 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -15,6 +15,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"unpack-trees.h\"\n #include \"string-list.h\"\n #include \"xdiff-interface.h\"\ndiff --git a/object.c b/object.c\nindex 49b952e9299..8e29f63bf23 100644\n--- a/object.c\n+++ b/object.c\n@@ -5,6 +5,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"object-store.h\"\n #include \"packfile.h\"\n \n@@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n {\n \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n \tmemset(o, 0, sizeof(*o));\n+\n+\to->blob_state = allocate_alloc_state();\n+\to->tree_state = allocate_alloc_state();\n+\to->commit_state = allocate_alloc_state();\n+\to->tag_state = allocate_alloc_state();\n+\to->object_state = allocate_alloc_state();\n+\n \treturn o;\n }\n \n@@ -501,9 +509,39 @@ void raw_object_store_clear(struct raw_object_store *o)\n void parsed_object_pool_clear(struct parsed_object_pool *o)\n {\n \t/*\n-\t * TOOD free objects in o->obj_hash.\n-\t *\n \t * As objects are allocated in slabs (see alloc.c), we do\n \t * not need to free each object, but each slab instead.\n+\t *\n+\t * Before doing so, we need to free any additional memory\n+\t * the objects may hold.\n \t */\n+\tunsigned i;\n+\n+\tfor (i = 0; i < o->obj_hash_size; i++) {\n+\t\tstruct object *obj = o->obj_hash[i];\n+\n+\t\tif (!obj)\n+\t\t\tcontinue;\n+\n+\t\tif (obj->type == OBJ_TREE)\n+\t\t\tfree_tree_buffer((struct tree*)obj);\n+\t\telse if (obj->type == OBJ_COMMIT)\n+\t\t\trelease_commit_memory((struct commit*)obj);\n+\t\telse if (obj->type == OBJ_TAG)\n+\t\t\trelease_tag_memory((struct tag*)obj);\n+\t}\n+\n+\tFREE_AND_NULL(o->obj_hash);\n+\to->obj_hash_size = 0;\n+\n+\tclear_alloc_state(o->blob_state);\n+\tclear_alloc_state(o->tree_state);\n+\tclear_alloc_state(o->commit_state);\n+\tclear_alloc_state(o->tag_state);\n+\tclear_alloc_state(o->object_state);\n+\tFREE_AND_NULL(o->blob_state);\n+\tFREE_AND_NULL(o->tree_state);\n+\tFREE_AND_NULL(o->commit_state);\n+\tFREE_AND_NULL(o->tag_state);\n+\tFREE_AND_NULL(o->object_state);\n }\ndiff --git a/object.h b/object.h\nindex b41d7a3accb..7916edb4edf 100644\n--- a/object.h\n+++ b/object.h\n@@ -4,6 +4,14 @@\n struct parsed_object_pool {\n \tstruct object **obj_hash;\n \tint nr_objs, obj_hash_size;\n+\n+\t/* TODO: migrate alloc_states to mem-pool? */\n+\tstruct alloc_state *blob_state;\n+\tstruct alloc_state *tree_state;\n+\tstruct alloc_state *commit_state;\n+\tstruct alloc_state *tag_state;\n+\tstruct alloc_state *object_state;\n+\tunsigned commit_count;\n };\n \n struct parsed_object_pool *parsed_object_pool_new(void);\ndiff --git a/tag.c b/tag.c\nindex 02ef4eaafc0..7c12426b4ea 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"tree.h\"\n #include \"blob.h\"\n+#include \"alloc.h\"\n #include \"gpg-interface.h\"\n \n const char *tag_type = \"tag\";\n@@ -115,6 +116,14 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n \treturn parse_timestamp(dateptr, NULL, 10);\n }\n \n+void release_tag_memory(struct tag *t)\n+{\n+\tfree(t->tag);\n+\tt->tagged = NULL;\n+\tt->object.parsed = 0;\n+\tt->date = 0;\n+}\n+\n int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n {\n \tstruct object_id oid;\ndiff --git a/tag.h b/tag.h\nindex d469534e82a..9057d76a506 100644\n--- a/tag.h\n+++ b/tag.h\n@@ -15,6 +15,7 @@ struct tag {\n extern struct tag *lookup_tag(const struct object_id *oid);\n extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n extern int parse_tag(struct tag *item);\n+extern void release_tag_memory(struct tag *t);\n extern struct object *deref_tag(struct object *, const char *, int);\n extern struct object *deref_tag_noverify(struct object *);\n extern int gpg_verify_tag(const struct object_id *oid,\ndiff --git a/tree.c b/tree.c\nindex 58cf19b4fa8..8f8ef3189af 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -5,6 +5,7 @@\n #include \"blob.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"alloc.h\"\n #include \"tree-walk.h\"\n \n const char *tree_type = \"tree\";\n-- \n2.17.0.582.gccdcbd54c44.dirty\n\n"},{"id":"347778","messageId":"xmqqfu2s72tk.fsf@gitster-ct.c.googlers.com","threadId":"48394","inReplyTo":"20180515214842.108713-1-sbeller@google.com","subject":"Re: [PATCH] alloc: allow arbitrary repositories for alloc functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-16T02:27:03Z","receivedAt":"2018-05-16T02:27:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> We have to convert all of the alloc functions at once, because alloc_report\n> uses a funky macro for reporting. It is better for the sake of mechanical\n> conversion to convert multiple functions at once rather than changing the\n> structure of the reporting function.\n>\n> We record all memory allocation in alloc.c, and free them in\n> clear_alloc_state, which is called for all repositories except\n> the_repository.\n>\n> Signed-off-by: Stefan Beller \n> Signed-off-by: Junio C Hamano \n> ---\n>\n> Notes:\n>\n> Eric, I have fixed s/relase/release/\n>\n>\n> Jonathan,\n>\n>> This might seem a bit bikesheddy, but I wouldn't call it\n>> free_tag_buffer(), since what's being freed is not the buffer of the\n>> object itself, but just a string. If you want such a function, I would\n>> just call it release_tag_memory() to match release_commit_memory().\n>\n> So you would replace the last commit with a patch like this?\n\nQuite honestly, I do not see much difference either way between free\nand release. None of the resetting the .parsed bit, .index to 0,\nand tagged to NULL is about releasing or freeing resources. These\noperations are about de-initializing the object, which may well\ninvolve releasing resources directly associated with the object\nitself, but even more. The release_commit_memory() function may\nwant to do something to various commit slabs that refer back to the\ncommit, for example.\n\nBut as I said, I do not deeply care either way ;-)\n\nThanks, the patch looks sensible.\n\n>\n> Thanks,\n> Stefan\n>\n> diff to what is currently queued:\n>\n> diff --git c/commit.c w/commit.c\n> index 612ccf7b053..5eb4d2f08f8 100644\n> --- c/commit.c\n> +++ w/commit.c\n> @@ -299,9 +299,13 @@ void free_commit_buffer(struct commit *commit)\n> \n> void release_commit_memory(struct commit *c)\n> {\n> +\tc->tree = NULL;\n> +\tc->index = 0;\n> \tfree_commit_buffer(c);\n> \tfree_commit_list(c->parents);\n> \t/* TODO: what about commit->util? */\n> +\n> +\tc->object.parsed = 0;\n> }\n> \n> const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n> diff --git c/object.c w/object.c\n> index 9d5b10d5a20..8e29f63bf23 100644\n> --- c/object.c\n> +++ w/object.c\n> @@ -528,7 +528,7 @@ void parsed_object_pool_clear(struct parsed_object_pool *o)\n> \t\telse if (obj->type == OBJ_COMMIT)\n> \t\t\trelease_commit_memory((struct commit*)obj);\n> \t\telse if (obj->type == OBJ_TAG)\n> -\t\t\tfree_tag_buffer((struct tag*)obj);\n> +\t\t\trelease_tag_memory((struct tag*)obj);\n> \t}\n> \n> \tFREE_AND_NULL(o->obj_hash);\n> diff --git c/tag.c w/tag.c\n> index 254352c30c6..7c12426b4ea 100644\n> --- c/tag.c\n> +++ w/tag.c\n> @@ -116,9 +116,12 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n> \treturn parse_timestamp(dateptr, NULL, 10);\n> }\n> \n> -void free_tag_buffer(struct tag *t)\n> +void release_tag_memory(struct tag *t)\n> {\n> \tfree(t->tag);\n> +\tt->tagged = NULL;\n> +\tt->object.parsed = 0;\n> +\tt->date = 0;\n> }\n> \n> int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n> diff --git c/tag.h w/tag.h\n> index b241fe67bc5..9057d76a506 100644\n> --- c/tag.h\n> +++ w/tag.h\n> @@ -15,7 +15,7 @@ struct tag {\n> extern struct tag *lookup_tag(const struct object_id *oid);\n> extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n> extern int parse_tag(struct tag *item);\n> -extern void free_tag_buffer(struct tag *t);\n> +extern void release_tag_memory(struct tag *t);\n> extern struct object *deref_tag(struct object *, const char *, int);\n> extern struct object *deref_tag_noverify(struct object *);\n> extern int gpg_verify_tag(const struct object_id *oid,\n>\n> alloc.c | 65 ++++++++++++++++++++++++++++++-----------------\n> alloc.h | 19 ++++++++++++++\n> blame.c | 1 +\n> blob.c | 1 +\n> cache.h | 16 ------------\n> commit.c | 12 +++++++++\n> commit.h | 6 +++++\n> merge-recursive.c | 1 +\n> object.c | 42 ++++++++++++++++++++++++++++--\n> object.h | 8 ++++++\n> tag.c | 9 +++++++\n> tag.h | 1 +\n> tree.c | 1 +\n> 13 files changed, 140 insertions(+), 42 deletions(-)\n> create mode 100644 alloc.h\n>\n> diff --git a/alloc.c b/alloc.c\n> index 277dadd221b..714df633169 100644\n> --- a/alloc.c\n> +++ b/alloc.c\n> @@ -4,8 +4,7 @@\n> * Copyright (C) 2006 Linus Torvalds\n> *\n> * The standard malloc/free wastes too much space for objects, partly because\n> - * it maintains all the allocation infrastructure (which isn't needed, since\n> - * we never free an object descriptor anyway), but even more because it ends\n> + * it maintains all the allocation infrastructure, but even more because it ends\n> * up with maximal alignment because it doesn't know what the object alignment\n> * for the new allocation is.\n> */\n> @@ -15,6 +14,7 @@\n> #include \"tree.h\"\n> #include \"commit.h\"\n> #include \"tag.h\"\n> +#include \"alloc.h\"\n> \n> #define BLOCKING 1024\n> \n> @@ -30,8 +30,27 @@ struct alloc_state {\n> \tint count; /* total number of nodes allocated */\n> \tint nr; /* number of nodes left in current allocation */\n> \tvoid *p; /* first free node in current allocation */\n> +\n> +\t/* bookkeeping of allocations */\n> +\tvoid **slabs;\n> +\tint slab_nr, slab_alloc;\n> };\n> \n> +void *allocate_alloc_state(void)\n> +{\n> +\treturn xcalloc(1, sizeof(struct alloc_state));\n> +}\n> +\n> +void clear_alloc_state(struct alloc_state *s)\n> +{\n> +\twhile (s->slab_nr > 0) {\n> +\t\ts->slab_nr--;\n> +\t\tfree(s->slabs[s->slab_nr]);\n> +\t}\n> +\n> +\tFREE_AND_NULL(s->slabs);\n> +}\n> +\n> static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n> {\n> \tvoid *ret;\n> @@ -39,60 +58,57 @@ static inline void *alloc_node(struct alloc_state *s, size_t node_size)\n> \tif (!s->nr) {\n> \t\ts->nr = BLOCKING;\n> \t\ts->p = xmalloc(BLOCKING * node_size);\n> +\n> +\t\tALLOC_GROW(s->slabs, s->slab_nr + 1, s->slab_alloc);\n> +\t\ts->slabs[s->slab_nr++] = s->p;\n> \t}\n> \ts->nr--;\n> \ts->count++;\n> \tret = s->p;\n> \ts->p = (char *)s->p + node_size;\n> \tmemset(ret, 0, node_size);\n> +\n> \treturn ret;\n> }\n> \n> -static struct alloc_state blob_state;\n> -void *alloc_blob_node_the_repository(void)\n> +void *alloc_blob_node(struct repository *r)\n> {\n> -\tstruct blob *b = alloc_node(&blob_state, sizeof(struct blob));\n> +\tstruct blob *b = alloc_node(r->parsed_objects->blob_state, sizeof(struct blob));\n> \tb->object.type = OBJ_BLOB;\n> \treturn b;\n> }\n> \n> -static struct alloc_state tree_state;\n> -void *alloc_tree_node_the_repository(void)\n> +void *alloc_tree_node(struct repository *r)\n> {\n> -\tstruct tree *t = alloc_node(&tree_state, sizeof(struct tree));\n> +\tstruct tree *t = alloc_node(r->parsed_objects->tree_state, sizeof(struct tree));\n> \tt->object.type = OBJ_TREE;\n> \treturn t;\n> }\n> \n> -static struct alloc_state tag_state;\n> -void *alloc_tag_node_the_repository(void)\n> +void *alloc_tag_node(struct repository *r)\n> {\n> -\tstruct tag *t = alloc_node(&tag_state, sizeof(struct tag));\n> +\tstruct tag *t = alloc_node(r->parsed_objects->tag_state, sizeof(struct tag));\n> \tt->object.type = OBJ_TAG;\n> \treturn t;\n> }\n> \n> -static struct alloc_state object_state;\n> -void *alloc_object_node_the_repository(void)\n> +void *alloc_object_node(struct repository *r)\n> {\n> -\tstruct object *obj = alloc_node(&object_state, sizeof(union any_object));\n> +\tstruct object *obj = alloc_node(r->parsed_objects->object_state, sizeof(union any_object));\n> \tobj->type = OBJ_NONE;\n> \treturn obj;\n> }\n> \n> -static struct alloc_state commit_state;\n> -\n> -unsigned int alloc_commit_index_the_repository(void)\n> +unsigned int alloc_commit_index(struct repository *r)\n> {\n> -\tstatic unsigned int count;\n> -\treturn count++;\n> +\treturn r->parsed_objects->commit_count++;\n> }\n> \n> -void *alloc_commit_node_the_repository(void)\n> +void *alloc_commit_node(struct repository *r)\n> {\n> -\tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n> +\tstruct commit *c = alloc_node(r->parsed_objects->commit_state, sizeof(struct commit));\n> \tc->object.type = OBJ_COMMIT;\n> -\tc->index = alloc_commit_index(the_repository);\n> +\tc->index = alloc_commit_index(r);\n> \treturn c;\n> }\n> \n> @@ -103,9 +119,10 @@ static void report(const char *name, unsigned int count, size_t size)\n> }\n> \n> #define REPORT(name, type)\t\\\n> - report(#name, name##_state.count, name##_state.count * sizeof(type) >> 10)\n> + report(#name, r->parsed_objects->name##_state->count, \\\n> +\t\t r->parsed_objects->name##_state->count * sizeof(type) >> 10)\n> \n> -void alloc_report_the_repository(void)\n> +void alloc_report(struct repository *r)\n> {\n> \tREPORT(blob, struct blob);\n> \tREPORT(tree, struct tree);\n> diff --git a/alloc.h b/alloc.h\n> new file mode 100644\n> index 00000000000..3e4e828db48\n> --- /dev/null\n> +++ b/alloc.h\n> @@ -0,0 +1,19 @@\n> +#ifndef ALLOC_H\n> +#define ALLOC_H\n> +\n> +struct tree;\n> +struct commit;\n> +struct tag;\n> +\n> +void *alloc_blob_node(struct repository *r);\n> +void *alloc_tree_node(struct repository *r);\n> +void *alloc_commit_node(struct repository *r);\n> +void *alloc_tag_node(struct repository *r);\n> +void *alloc_object_node(struct repository *r);\n> +void alloc_report(struct repository *r);\n> +unsigned int alloc_commit_index(struct repository *r);\n> +\n> +void *allocate_alloc_state(void);\n> +void clear_alloc_state(struct alloc_state *s);\n> +\n> +#endif\n> diff --git a/blame.c b/blame.c\n> index ba9b18e7542..3a11f1ce52b 100644\n> --- a/blame.c\n> +++ b/blame.c\n> @@ -6,6 +6,7 @@\n> #include \"diffcore.h\"\n> #include \"tag.h\"\n> #include \"blame.h\"\n> +#include \"alloc.h\"\n> \n> void blame_origin_decref(struct blame_origin *o)\n> {\n> diff --git a/blob.c b/blob.c\n> index 9e64f301895..458dafa811e 100644\n> --- a/blob.c\n> +++ b/blob.c\n> @@ -1,6 +1,7 @@\n> #include \"cache.h\"\n> #include \"blob.h\"\n> #include \"repository.h\"\n> +#include \"alloc.h\"\n> \n> const char *blob_type = \"blob\";\n> \n> diff --git a/cache.h b/cache.h\n> index 0e6c5dd5639..c75559b7d38 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1763,22 +1763,6 @@ extern const char *excludes_file;\n> int decode_85(char *dst, const char *line, int linelen);\n> void encode_85(char *buf, const unsigned char *data, int bytes);\n> \n> -/* alloc.c */\n> -#define alloc_blob_node(r) alloc_blob_node_##r()\n> -extern void *alloc_blob_node_the_repository(void);\n> -#define alloc_tree_node(r) alloc_tree_node_##r()\n> -extern void *alloc_tree_node_the_repository(void);\n> -#define alloc_commit_node(r) alloc_commit_node_##r()\n> -extern void *alloc_commit_node_the_repository(void);\n> -#define alloc_tag_node(r) alloc_tag_node_##r()\n> -extern void *alloc_tag_node_the_repository(void);\n> -#define alloc_object_node(r) alloc_object_node_##r()\n> -extern void *alloc_object_node_the_repository(void);\n> -#define alloc_report(r) alloc_report_##r()\n> -extern void alloc_report_the_repository(void);\n> -#define alloc_commit_index(r) alloc_commit_index_##r()\n> -extern unsigned int alloc_commit_index_the_repository(void);\n> -\n> /* pkt-line.c */\n> void packet_trace_identity(const char *prog);\n> \n> diff --git a/commit.c b/commit.c\n> index a9a43e79bae..5eb4d2f08f8 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -6,6 +6,7 @@\n> #include \"diff.h\"\n> #include \"revision.h\"\n> #include \"notes.h\"\n> +#include \"alloc.h\"\n> #include \"gpg-interface.h\"\n> #include \"mergesort.h\"\n> #include \"commit-slab.h\"\n> @@ -296,6 +297,17 @@ void free_commit_buffer(struct commit *commit)\n> \t}\n> }\n> \n> +void release_commit_memory(struct commit *c)\n> +{\n> +\tc->tree = NULL;\n> +\tc->index = 0;\n> +\tfree_commit_buffer(c);\n> +\tfree_commit_list(c->parents);\n> +\t/* TODO: what about commit->util? */\n> +\n> +\tc->object.parsed = 0;\n> +}\n> +\n> const void *detach_commit_buffer(struct commit *commit, unsigned long *sizep)\n> {\n> \tstruct commit_buffer *v = buffer_slab_peek(&buffer_slab, commit);\n> diff --git a/commit.h b/commit.h\n> index 0fb8271665c..2d764ab7d8e 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -99,6 +99,12 @@ void unuse_commit_buffer(const struct commit *, const void *buffer);\n> */\n> void free_commit_buffer(struct commit *);\n> \n> +/*\n> + * Release memory related to a commit, including the parent list and\n> + * any cached object buffer.\n> + */\n> +void release_commit_memory(struct commit *c);\n> +\n> /*\n> * Disassociate any cached object buffer from the commit, but do not free it.\n> * The buffer (or NULL, if none) is returned.\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index 6dac8908648..cbded673c28 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -15,6 +15,7 @@\n> #include \"diff.h\"\n> #include \"diffcore.h\"\n> #include \"tag.h\"\n> +#include \"alloc.h\"\n> #include \"unpack-trees.h\"\n> #include \"string-list.h\"\n> #include \"xdiff-interface.h\"\n> diff --git a/object.c b/object.c\n> index 49b952e9299..8e29f63bf23 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -5,6 +5,7 @@\n> #include \"tree.h\"\n> #include \"commit.h\"\n> #include \"tag.h\"\n> +#include \"alloc.h\"\n> #include \"object-store.h\"\n> #include \"packfile.h\"\n> \n> @@ -455,6 +456,13 @@ struct parsed_object_pool *parsed_object_pool_new(void)\n> {\n> \tstruct parsed_object_pool *o = xmalloc(sizeof(*o));\n> \tmemset(o, 0, sizeof(*o));\n> +\n> +\to->blob_state = allocate_alloc_state();\n> +\to->tree_state = allocate_alloc_state();\n> +\to->commit_state = allocate_alloc_state();\n> +\to->tag_state = allocate_alloc_state();\n> +\to->object_state = allocate_alloc_state();\n> +\n> \treturn o;\n> }\n> \n> @@ -501,9 +509,39 @@ void raw_object_store_clear(struct raw_object_store *o)\n> void parsed_object_pool_clear(struct parsed_object_pool *o)\n> {\n> \t/*\n> -\t * TOOD free objects in o->obj_hash.\n> -\t *\n> \t * As objects are allocated in slabs (see alloc.c), we do\n> \t * not need to free each object, but each slab instead.\n> +\t *\n> +\t * Before doing so, we need to free any additional memory\n> +\t * the objects may hold.\n> \t */\n> +\tunsigned i;\n> +\n> +\tfor (i = 0; i < o->obj_hash_size; i++) {\n> +\t\tstruct object *obj = o->obj_hash[i];\n> +\n> +\t\tif (!obj)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (obj->type == OBJ_TREE)\n> +\t\t\tfree_tree_buffer((struct tree*)obj);\n> +\t\telse if (obj->type == OBJ_COMMIT)\n> +\t\t\trelease_commit_memory((struct commit*)obj);\n> +\t\telse if (obj->type == OBJ_TAG)\n> +\t\t\trelease_tag_memory((struct tag*)obj);\n> +\t}\n> +\n> +\tFREE_AND_NULL(o->obj_hash);\n> +\to->obj_hash_size = 0;\n> +\n> +\tclear_alloc_state(o->blob_state);\n> +\tclear_alloc_state(o->tree_state);\n> +\tclear_alloc_state(o->commit_state);\n> +\tclear_alloc_state(o->tag_state);\n> +\tclear_alloc_state(o->object_state);\n> +\tFREE_AND_NULL(o->blob_state);\n> +\tFREE_AND_NULL(o->tree_state);\n> +\tFREE_AND_NULL(o->commit_state);\n> +\tFREE_AND_NULL(o->tag_state);\n> +\tFREE_AND_NULL(o->object_state);\n> }\n> diff --git a/object.h b/object.h\n> index b41d7a3accb..7916edb4edf 100644\n> --- a/object.h\n> +++ b/object.h\n> @@ -4,6 +4,14 @@\n> struct parsed_object_pool {\n> \tstruct object **obj_hash;\n> \tint nr_objs, obj_hash_size;\n> +\n> +\t/* TODO: migrate alloc_states to mem-pool? */\n> +\tstruct alloc_state *blob_state;\n> +\tstruct alloc_state *tree_state;\n> +\tstruct alloc_state *commit_state;\n> +\tstruct alloc_state *tag_state;\n> +\tstruct alloc_state *object_state;\n> +\tunsigned commit_count;\n> };\n> \n> struct parsed_object_pool *parsed_object_pool_new(void);\n> diff --git a/tag.c b/tag.c\n> index 02ef4eaafc0..7c12426b4ea 100644\n> --- a/tag.c\n> +++ b/tag.c\n> @@ -3,6 +3,7 @@\n> #include \"commit.h\"\n> #include \"tree.h\"\n> #include \"blob.h\"\n> +#include \"alloc.h\"\n> #include \"gpg-interface.h\"\n> \n> const char *tag_type = \"tag\";\n> @@ -115,6 +116,14 @@ static timestamp_t parse_tag_date(const char *buf, const char *tail)\n> \treturn parse_timestamp(dateptr, NULL, 10);\n> }\n> \n> +void release_tag_memory(struct tag *t)\n> +{\n> +\tfree(t->tag);\n> +\tt->tagged = NULL;\n> +\tt->object.parsed = 0;\n> +\tt->date = 0;\n> +}\n> +\n> int parse_tag_buffer(struct tag *item, const void *data, unsigned long size)\n> {\n> \tstruct object_id oid;\n> diff --git a/tag.h b/tag.h\n> index d469534e82a..9057d76a506 100644\n> --- a/tag.h\n> +++ b/tag.h\n> @@ -15,6 +15,7 @@ struct tag {\n> extern struct tag *lookup_tag(const struct object_id *oid);\n> extern int parse_tag_buffer(struct tag *item, const void *data, unsigned long size);\n> extern int parse_tag(struct tag *item);\n> +extern void release_tag_memory(struct tag *t);\n> extern struct object *deref_tag(struct object *, const char *, int);\n> extern struct object *deref_tag_noverify(struct object *);\n> extern int gpg_verify_tag(const struct object_id *oid,\n> diff --git a/tree.c b/tree.c\n> index 58cf19b4fa8..8f8ef3189af 100644\n> --- a/tree.c\n> +++ b/tree.c\n> @@ -5,6 +5,7 @@\n> #include \"blob.h\"\n> #include \"commit.h\"\n> #include \"tag.h\"\n> +#include \"alloc.h\"\n> #include \"tree-walk.h\"\n> \n> const char *tree_type = \"tree\";\n"}]}