{"thread":{"id":"30195","subject":"[PATCH 0/5] multithread traverse_commit_list (aka rev-list)","startedAt":"2012-04-10T14:39:26Z","lastAt":"2012-04-10T16:51:55Z","messageCount":7,"participants":["Nguyễn Thái Ngọc Duy","Martin Fick"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"188864","messageId":"1334068771-32725-1-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":null,"subject":"[PATCH 0/5] multithread traverse_commit_list (aka rev-list)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:26Z","receivedAt":"2012-04-10T14:39:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"rev-list is quite cpu extensive. While there's better approach to\ntackle this (e.g. Shawn's bitmap cache idea), I was curious if\nmaking it parellel could give us anything. So this is a naive (and\nfailed) attempt at that. I post it anyway, so people, especially gsoc\nstudents if the related proposal is chosen, might learn from it, or\nlearn to avoid it, or just laugh at my attempt to relieve stress.\n\nWhile this series does not break rev-list (but breaks all external\ncommands, I don't care anyway), and it proves lock-free pack access\nworks, it does not improve rev-list performance. I suspect extensive\nlocking around \"struct object *\" may be the culprit.  Or maybe it's\njust my dumb partition method. A better partition method that avoids\nlocking might improve it. We may split the object hash table into 4,\none for each object type and assign one thread for tree, one for blob.\nOr something else..\n\nAnyway I'll stop here to finish my ongoing topics for 1.7.11 and maybe\nimplement Shawn's idea if I still have more time. Effectively generating\nthe cache from index-pack could be challenging.\n\nNguyễn Thái Ngọc Duy (5):\n  Remove global pointer \"packed_git\" in favor or set/get function pair\n  sha1_file: stuff various pack reading variables into a struct\n  Make lookup_*() functions thread-safe\n  Teach traverse_commit_list callsites about new parameter, nr_threads\n  Support multithread in traverse_commit_list and rev-list\n\n blob.c                   |   11 ++-\n builtin/count-objects.c  |    4 +-\n builtin/fsck.c           |    4 +-\n builtin/gc.c             |    2 +-\n builtin/pack-objects.c   |   12 +-\n builtin/pack-redundant.c |    4 +-\n builtin/rev-list.c       |    5 +-\n cache.h                  |    8 +-\n commit.c                 |   12 ++-\n git.c                    |    1 +\n list-objects.c           |  157 +++++++++++++++++++++++++-\n list-objects.h           |    2 +-\n object.c                 |    9 ++-\n object.h                 |    6 +\n pack-revindex.c          |    4 +-\n revision.c               |   16 +++\n revision.h               |    2 +\n server-info.c            |    4 +-\n sha1_file.c              |  276 ++++++++++++++++++++++++++++------------------\n sha1_name.c              |    2 +-\n tag.c                    |    9 +-\n tree.c                   |   18 +++-\n upload-pack.c            |    2 +-\n 23 files changed, 419 insertions(+), 151 deletions(-)\n\n-- \n1.7.8.36.g69ee2\n"},{"id":"188865","messageId":"1334068771-32725-2-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/5] Remove global pointer \"packed_git\" in favor or set/get function pair","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:27Z","receivedAt":"2012-04-10T14:39:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/count-objects.c  |    4 ++--\n builtin/fsck.c           |    4 ++--\n builtin/gc.c             |    2 +-\n builtin/pack-objects.c   |   10 +++++-----\n builtin/pack-redundant.c |    4 ++--\n cache.h                  |    7 +++++--\n pack-revindex.c          |    4 ++--\n server-info.c            |    4 ++--\n sha1_file.c              |   33 ++++++++++++++++++++++-----------\n sha1_name.c              |    2 +-\n 10 files changed, 44 insertions(+), 30 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex c37cb98..dc217db 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -104,9 +104,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)\n \t\tstruct packed_git *p;\n \t\tunsigned long num_pack = 0;\n \t\toff_t size_pack = 0;\n-\t\tif (!packed_git)\n+\t\tif (!get_packed_git())\n \t\t\tprepare_packed_git();\n-\t\tfor (p = packed_git; p; p = p->next) {\n+\t\tfor (p = get_packed_git(); p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n \t\t\tif (open_pack_index(p))\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 67eb553..9dbe6d8 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -669,7 +669,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tprepare_packed_git();\n \n \t\tif (show_progress) {\n-\t\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\tfor (p = get_packed_git(); p; p = p->next) {\n \t\t\t\tif (open_pack_index(p))\n \t\t\t\t\tcontinue;\n \t\t\t\ttotal += p->num_objects;\n@@ -677,7 +677,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \t\t\tprogress = start_progress(\"Checking objects\", total);\n \t\t}\n-\t\tfor (p = packed_git; p; p = p->next) {\n+\t\tfor (p = get_packed_git(); p; p = p->next) {\n \t\t\t/* verify gives error messages itself */\n \t\t\tif (verify_pack(p, fsck_obj_buffer,\n \t\t\t\t\tprogress, count))\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 271376d..3007650 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -130,7 +130,7 @@ static int too_many_packs(void)\n \t\treturn 0;\n \n \tprepare_packed_git();\n-\tfor (cnt = 0, p = packed_git; p; p = p->next) {\n+\tfor (cnt = 0, p = get_packed_git(); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7b07c09..2f242c4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -840,7 +840,7 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \tif (!exclude && local && has_loose_object_nonlocal(sha1))\n \t\treturn 0;\n \n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\toff_t offset = find_pack_entry_one(sha1, p);\n \t\tif (offset) {\n \t\t\tif (!found_pack) {\n@@ -2183,7 +2183,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)\n \n \tmemset(&in_pack, 0, sizeof(in_pack));\n \n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\tconst unsigned char *sha1;\n \t\tstruct object *o;\n \n@@ -2221,7 +2221,7 @@ static int has_sha1_pack_kept_or_nonlocal(const unsigned char *sha1)\n \tstatic struct packed_git *last_found = (void *)1;\n \tstruct packed_git *p;\n \n-\tp = (last_found != (void *)1) ? last_found : packed_git;\n+\tp = (last_found != (void *)1) ? last_found : get_packed_git();\n \n \twhile (p) {\n \t\tif ((!p->pack_local || p->pack_keep) &&\n@@ -2230,7 +2230,7 @@ static int has_sha1_pack_kept_or_nonlocal(const unsigned char *sha1)\n \t\t\treturn 1;\n \t\t}\n \t\tif (p == last_found)\n-\t\t\tp = packed_git;\n+\t\t\tp = get_packed_git();\n \t\telse\n \t\t\tp = p->next;\n \t\tif (p == last_found)\n@@ -2245,7 +2245,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)\n \tuint32_t i;\n \tconst unsigned char *sha1;\n \n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\tif (!p->pack_local || p->pack_keep)\n \t\t\tcontinue;\n \ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex f5c6afc..8fece6a 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -569,7 +569,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n static struct pack_list * add_pack_file(const char *filename)\n {\n-\tstruct packed_git *p = packed_git;\n+\tstruct packed_git *p = get_packed_git();\n \n \tif (strlen(filename) < 40)\n \t\tdie(\"Bad pack filename: %s\", filename);\n@@ -584,7 +584,7 @@ static struct pack_list * add_pack_file(const char *filename)\n \n static void load_all(void)\n {\n-\tstruct packed_git *p = packed_git;\n+\tstruct packed_git *p = get_packed_git();\n \n \twhile (p) {\n \t\tadd_pack(p);\ndiff --git a/cache.h b/cache.h\nindex e5e1aa4..26d14b4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -975,7 +975,7 @@ struct pack_window {\n \tunsigned int inuse_cnt;\n };\n \n-extern struct packed_git {\n+struct packed_git {\n \tstruct packed_git *next;\n \tstruct pack_window *windows;\n \toff_t pack_size;\n@@ -993,7 +993,10 @@ extern struct packed_git {\n \tunsigned char sha1[20];\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n-} *packed_git;\n+};\n+\n+extern struct packed_git *get_packed_git(void);\n+extern void set_packed_git(struct packed_git *);\n \n struct pack_entry {\n \toff_t offset;\ndiff --git a/pack-revindex.c b/pack-revindex.c\nindex 77a0465..636d35d 100644\n--- a/pack-revindex.c\n+++ b/pack-revindex.c\n@@ -45,13 +45,13 @@ static void init_pack_revindex(void)\n \tint num;\n \tstruct packed_git *p;\n \n-\tfor (num = 0, p = packed_git; p; p = p->next)\n+\tfor (num = 0, p = get_packed_git(); p; p = p->next)\n \t\tnum++;\n \tif (!num)\n \t\treturn;\n \tpack_revindex_hashsz = num * 11;\n \tpack_revindex = xcalloc(sizeof(*pack_revindex), pack_revindex_hashsz);\n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\tnum = pack_revindex_ix(p);\n \t\tnum = - 1 - num;\n \t\tpack_revindex[num].p = p;\ndiff --git a/server-info.c b/server-info.c\nindex 9ec744e..b7951ca 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -161,7 +161,7 @@ static void init_pack_info(const char *infofile, int force)\n \tobjdirlen = strlen(objdir);\n \n \tprepare_packed_git();\n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\t/* we ignore things on alternate path since they are\n \t\t * not available to the pullers in general.\n \t\t */\n@@ -171,7 +171,7 @@ static void init_pack_info(const char *infofile, int force)\n \t}\n \tnum_pack = i;\n \tinfo = xcalloc(num_pack, sizeof(struct pack_info *));\n-\tfor (i = 0, p = packed_git; p; p = p->next) {\n+\tfor (i = 0, p = get_packed_git(); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tinfo[i] = xcalloc(1, sizeof(struct pack_info));\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4f06a0e..4fd4e2c 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -33,6 +33,17 @@ static inline uintmax_t sz_fmt(size_t s) { return s; }\n \n const unsigned char null_sha1[20];\n \n+static struct packed_git *packed_git_;\n+struct packed_git *get_packed_git(void)\n+{\n+\treturn packed_git_;\n+}\n+void set_packed_git(struct packed_git *p)\n+{\n+\tpacked_git_ = p;\n+}\n+\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want read_sha1_file() to be able to return, but yet you do not want\n@@ -618,7 +629,7 @@ static int unuse_one_window(struct packed_git *current, int keep_fd)\n \n \tif (current)\n \t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = packed_git; p; p = p->next)\n+\tfor (p = get_packed_git(); p; p = p->next)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n@@ -708,7 +719,7 @@ void close_pack_index(struct packed_git *p)\n  */\n void free_pack_by_name(const char *pack_name)\n {\n-\tstruct packed_git *p, **pp = &packed_git;\n+\tstruct packed_git *p, **pp = &packed_git_;\n \n \twhile (*pp) {\n \t\tp = *pp;\n@@ -990,8 +1001,8 @@ void install_packed_git(struct packed_git *pack)\n \tif (pack->pack_fd != -1)\n \t\tpack_open_fds++;\n \n-\tpack->next = packed_git;\n-\tpacked_git = pack;\n+\tpack->next = get_packed_git();\n+\tset_packed_git(pack);\n }\n \n static void prepare_packed_git_one(char *objdir, int local)\n@@ -1026,7 +1037,7 @@ static void prepare_packed_git_one(char *objdir, int local)\n \n \t\t/* Don't reopen a pack we already have. */\n \t\tstrcpy(path + len, de->d_name);\n-\t\tfor (p = packed_git; p; p = p->next) {\n+\t\tfor (p = get_packed_git(); p; p = p->next) {\n \t\t\tif (!memcmp(path, p->pack_name, len + namelen - 4))\n \t\t\t\tbreak;\n \t\t}\n@@ -1076,14 +1087,14 @@ static void rearrange_packed_git(void)\n \tstruct packed_git **ary, *p;\n \tint i, n;\n \n-\tfor (n = 0, p = packed_git; p; p = p->next)\n+\tfor (n = 0, p = get_packed_git(); p; p = p->next)\n \t\tn++;\n \tif (n < 2)\n \t\treturn;\n \n \t/* prepare an array of packed_git for easier sorting */\n \tary = xcalloc(n, sizeof(struct packed_git *));\n-\tfor (n = 0, p = packed_git; p; p = p->next)\n+\tfor (n = 0, p = get_packed_git(); p; p = p->next)\n \t\tary[n++] = p;\n \n \tqsort(ary, n, sizeof(struct packed_git *), sort_pack);\n@@ -1092,7 +1103,7 @@ static void rearrange_packed_git(void)\n \tfor (i = 0; i < n - 1; i++)\n \t\tary[i]->next = ary[i + 1];\n \tary[n - 1]->next = NULL;\n-\tpacked_git = ary[0];\n+\tset_packed_git(ary[0]);\n \n \tfree(ary);\n }\n@@ -1139,7 +1150,7 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n \tstruct packed_git *p;\n \tunsigned i;\n \n-\tfor (p = packed_git; p; p = p->next)\n+\tfor (p = get_packed_git(); p; p = p->next)\n \t\tfor (i = 0; i < p->num_bad_objects; i++)\n \t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n \t\t\t\treturn p;\n@@ -2058,13 +2069,13 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \tstruct packed_git *p;\n \n \tprepare_packed_git();\n-\tif (!packed_git)\n+\tif (!get_packed_git())\n \t\treturn 0;\n \n \tif (last_found_pack && fill_pack_entry(sha1, e, last_found_pack))\n \t\treturn 1;\n \n-\tfor (p = packed_git; p; p = p->next) {\n+\tfor (p = get_packed_git(); p; p = p->next) {\n \t\tif (p == last_found_pack || !fill_pack_entry(sha1, e, p))\n \t\t\tcontinue;\n \ndiff --git a/sha1_name.c b/sha1_name.c\nindex 03ffc2c..756b226 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -78,7 +78,7 @@ static int find_short_packed_object(int len, const unsigned char *match, unsigne\n \tint found = 0;\n \n \tprepare_packed_git();\n-\tfor (p = packed_git; p && found < 2; p = p->next) {\n+\tfor (p = get_packed_git(); p && found < 2; p = p->next) {\n \t\tuint32_t num, last;\n \t\tuint32_t first = 0;\n \t\topen_pack_index(p);\n-- \n1.7.8.36.g69ee2\n"},{"id":"188866","messageId":"1334068771-32725-3-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/5] sha1_file: stuff various pack reading variables into a struct","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:28Z","receivedAt":"2012-04-10T14:39:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This requires any threads (including the main one) to call\ninit_pack_context() first. All pack data is per thread.\n\nOriginal patch is written by Thomas Rast <trast@student.ethz.ch>\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h     |    1 +\n git.c       |    1 +\n sha1_file.c |  251 +++++++++++++++++++++++++++++++++++------------------------\n 3 files changed, 152 insertions(+), 101 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 26d14b4..868ef48 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -997,6 +997,7 @@ struct packed_git {\n \n extern struct packed_git *get_packed_git(void);\n extern void set_packed_git(struct packed_git *);\n+extern void init_pack_context(void);\n \n struct pack_entry {\n \toff_t offset;\ndiff --git a/git.c b/git.c\nindex 3805616..7eee270 100644\n--- a/git.c\n+++ b/git.c\n@@ -533,6 +533,7 @@ int main(int argc, const char **argv)\n \tconst char *cmd;\n \n \tstartup_info = &git_startup_info;\n+\tinit_pack_context();\n \n \tcmd = git_extract_argv0_path(argv[0]);\n \tif (!cmd)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4fd4e2c..170c0b1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -33,16 +33,79 @@ static inline uintmax_t sz_fmt(size_t s) { return s; }\n \n const unsigned char null_sha1[20];\n \n-static struct packed_git *packed_git_;\n+#define MAX_DELTA_CACHE (256)\n+\n+struct delta_base_cache_lru_list {\n+\tstruct delta_base_cache_lru_list *prev;\n+\tstruct delta_base_cache_lru_list *next;\n+};\n+\n+struct delta_base_cache_entry {\n+\tstruct delta_base_cache_lru_list lru;\n+\tvoid *data;\n+\tstruct packed_git *p;\n+\toff_t base_offset;\n+\tunsigned long size;\n+\tenum object_type type;\n+};\n+\n+struct pack_context {\n+\tsize_t delta_base_cached;\n+\tstruct delta_base_cache_entry *delta_base_cache;\n+\tstruct delta_base_cache_lru_list *delta_base_cache_lru;\n+\tstruct packed_git *last_found_pack;\n+\n+\tunsigned int pack_used_ctr;\n+\tunsigned int pack_mmap_calls;\n+\tunsigned int peak_pack_open_windows;\n+\tunsigned int pack_open_windows;\n+\tunsigned int pack_open_fds;\n+\tunsigned int pack_max_fds;\n+\tsize_t peak_pack_mapped;\n+\tsize_t pack_mapped;\n+\tstruct packed_git *packed_git;\n+\tint prepare_packed_git_run_once;\n+\n+\tchar sha1_file_name_buf[PATH_MAX];\n+\tchar *sha1_pack_name, *sha1_pack_base;\n+\tchar *sha1_pack_index_name, *sha1_pack_index_base;\n+};\n+\n+static pthread_key_t pack_key;\n+struct pack_context *get_thread_pack_context(void)\n+{\n+\treturn pthread_getspecific(pack_key);\n+}\n+\n struct packed_git *get_packed_git(void)\n {\n-\treturn packed_git_;\n+\treturn get_thread_pack_context()->packed_git;\n }\n void set_packed_git(struct packed_git *p)\n {\n-\tpacked_git_ = p;\n+\tget_thread_pack_context()->packed_git = p;\n }\n \n+void init_pack_context(void)\n+{\n+\tstatic int initialized = 0;\n+\tstruct pack_context *ctx = xmalloc(sizeof(struct pack_context));\n+\n+\tif (!initialized) {\n+\t\tpthread_key_create(&pack_key, NULL);\n+\t\tinitialized = 1;\n+\t}\n+\n+\tmemset(ctx, 0, sizeof(*ctx));\n+\tctx->delta_base_cached = 0;\n+\tctx->delta_base_cache_lru = xmalloc(sizeof(struct delta_base_cache_lru_list));\n+\tctx->delta_base_cache_lru->prev = ctx->delta_base_cache_lru;\n+\tctx->delta_base_cache_lru->next = ctx->delta_base_cache_lru;\n+\tctx->delta_base_cache = xcalloc(MAX_DELTA_CACHE, sizeof(struct delta_base_cache_entry));\n+\tctx->last_found_pack = NULL;\n+\tctx->packed_git = NULL;\n+\tpthread_setspecific(pack_key, ctx);\n+}\n \n /*\n  * This is meant to hold a *small* number of objects that you would\n@@ -65,8 +128,6 @@ static struct cached_object empty_tree = {\n \t0\n };\n \n-static struct packed_git *last_found_pack;\n-\n static struct cached_object *find_cached_object(const unsigned char *sha1)\n {\n \tint i;\n@@ -177,7 +238,7 @@ static void fill_sha1_path(char *pathbuf, const unsigned char *sha1)\n  */\n char *sha1_file_name(const unsigned char *sha1)\n {\n-\tstatic char buf[PATH_MAX];\n+\tchar *buf = get_thread_pack_context()->sha1_file_name_buf;\n \tconst char *objdir;\n \tint len;\n \n@@ -224,16 +285,14 @@ static char *sha1_get_pack_name(const unsigned char *sha1,\n \n char *sha1_pack_name(const unsigned char *sha1)\n {\n-\tstatic char *name, *base;\n-\n-\treturn sha1_get_pack_name(sha1, &name, &base, \"pack\");\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\treturn sha1_get_pack_name(sha1, &ctx->sha1_pack_name, &ctx->sha1_pack_base, \"pack\");\n }\n \n char *sha1_pack_index_name(const unsigned char *sha1)\n {\n-\tstatic char *name, *base;\n-\n-\treturn sha1_get_pack_name(sha1, &name, &base, \"idx\");\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\treturn sha1_get_pack_name(sha1, &ctx->sha1_pack_index_name, &ctx->sha1_pack_index_base, \"idx\");\n }\n \n struct alternate_object_database *alt_odb_list;\n@@ -455,18 +514,10 @@ static int has_loose_object(const unsigned char *sha1)\n \t       has_loose_object_nonlocal(sha1);\n }\n \n-static unsigned int pack_used_ctr;\n-static unsigned int pack_mmap_calls;\n-static unsigned int peak_pack_open_windows;\n-static unsigned int pack_open_windows;\n-static unsigned int pack_open_fds;\n-static unsigned int pack_max_fds;\n-static size_t peak_pack_mapped;\n-static size_t pack_mapped;\n-struct packed_git *packed_git;\n-\n-void pack_report(void)\n+void pack_report()\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\n \tfprintf(stderr,\n \t\t\"pack_report: getpagesize()            = %10\" SZ_FMT \"\\n\"\n \t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n@@ -480,10 +531,10 @@ void pack_report(void)\n \t\t\"pack_report: pack_open_windows        = %10u / %10u\\n\"\n \t\t\"pack_report: pack_mapped              = \"\n \t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n-\t\tpack_used_ctr,\n-\t\tpack_mmap_calls,\n-\t\tpack_open_windows, peak_pack_open_windows,\n-\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n+\t\tctx->pack_used_ctr,\n+\t\tctx->pack_mmap_calls,\n+\t\tctx->pack_open_windows, ctx->peak_pack_open_windows,\n+\t\tsz_fmt(ctx->pack_mapped), sz_fmt(ctx->peak_pack_mapped));\n }\n \n static int check_packed_git_idx(const char *path,  struct packed_git *p)\n@@ -624,6 +675,7 @@ static void scan_windows(struct packed_git *p,\n \n static int unuse_one_window(struct packed_git *current, int keep_fd)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tstruct packed_git *p, *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n \n@@ -633,7 +685,7 @@ static int unuse_one_window(struct packed_git *current, int keep_fd)\n \t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n-\t\tpack_mapped -= lru_w->len;\n+\t\tctx->pack_mapped -= lru_w->len;\n \t\tif (lru_l)\n \t\t\tlru_l->next = lru_w->next;\n \t\telse {\n@@ -641,12 +693,12 @@ static int unuse_one_window(struct packed_git *current, int keep_fd)\n \t\t\tif (!lru_p->windows && lru_p->pack_fd != -1\n \t\t\t\t&& lru_p->pack_fd != keep_fd) {\n \t\t\t\tclose(lru_p->pack_fd);\n-\t\t\t\tpack_open_fds--;\n+\t\t\t\tctx->pack_open_fds--;\n \t\t\t\tlru_p->pack_fd = -1;\n \t\t\t}\n \t\t}\n \t\tfree(lru_w);\n-\t\tpack_open_windows--;\n+\t\tctx->pack_open_windows--;\n \t\treturn 1;\n \t}\n \treturn 0;\n@@ -654,8 +706,9 @@ static int unuse_one_window(struct packed_git *current, int keep_fd)\n \n void release_pack_memory(size_t need, int fd)\n {\n-\tsize_t cur = pack_mapped;\n-\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL, fd))\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\tsize_t cur = ctx->pack_mapped;\n+\twhile (need >= (cur - ctx->pack_mapped) && unuse_one_window(NULL, fd))\n \t\t; /* nothing */\n }\n \n@@ -676,6 +729,7 @@ void *xmmap(void *start, size_t length,\n \n void close_pack_windows(struct packed_git *p)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \twhile (p->windows) {\n \t\tstruct pack_window *w = p->windows;\n \n@@ -683,8 +737,8 @@ void close_pack_windows(struct packed_git *p)\n \t\t\tdie(\"pack '%s' still has open windows to it\",\n \t\t\t    p->pack_name);\n \t\tmunmap(w->base, w->len);\n-\t\tpack_mapped -= w->len;\n-\t\tpack_open_windows--;\n+\t\tctx->pack_mapped -= w->len;\n+\t\tctx->pack_open_windows--;\n \t\tp->windows = w->next;\n \t\tfree(w);\n \t}\n@@ -719,7 +773,8 @@ void close_pack_index(struct packed_git *p)\n  */\n void free_pack_by_name(const char *pack_name)\n {\n-\tstruct packed_git *p, **pp = &packed_git_;\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\tstruct packed_git *p, **pp = &ctx->packed_git;\n \n \twhile (*pp) {\n \t\tp = *pp;\n@@ -728,13 +783,13 @@ void free_pack_by_name(const char *pack_name)\n \t\t\tclose_pack_windows(p);\n \t\t\tif (p->pack_fd != -1) {\n \t\t\t\tclose(p->pack_fd);\n-\t\t\t\tpack_open_fds--;\n+\t\t\t\tctx->pack_open_fds--;\n \t\t\t}\n \t\t\tclose_pack_index(p);\n \t\t\tfree(p->bad_object_sha1);\n \t\t\t*pp = p->next;\n-\t\t\tif (last_found_pack == p)\n-\t\t\t\tlast_found_pack = NULL;\n+\t\t\tif (ctx->last_found_pack == p)\n+\t\t\t\tctx->last_found_pack = NULL;\n \t\t\tfree(p);\n \t\t\treturn;\n \t\t}\n@@ -748,6 +803,7 @@ void free_pack_by_name(const char *pack_name)\n  */\n static int open_packed_git_1(struct packed_git *p)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tstruct stat st;\n \tstruct pack_header hdr;\n \tunsigned char sha1[20];\n@@ -757,7 +813,7 @@ static int open_packed_git_1(struct packed_git *p)\n \tif (!p->index_data && open_pack_index(p))\n \t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n \n-\tif (!pack_max_fds) {\n+\tif (!ctx->pack_max_fds) {\n \t\tstruct rlimit lim;\n \t\tunsigned int max_fds;\n \n@@ -768,18 +824,18 @@ static int open_packed_git_1(struct packed_git *p)\n \n \t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n \t\tif (25 < max_fds)\n-\t\t\tpack_max_fds = max_fds - 25;\n+\t\t\tctx->pack_max_fds = max_fds - 25;\n \t\telse\n-\t\t\tpack_max_fds = 1;\n+\t\t\tctx->pack_max_fds = 1;\n \t}\n \n-\twhile (pack_max_fds <= pack_open_fds && unuse_one_window(NULL, -1))\n+\twhile (ctx->pack_max_fds <= ctx->pack_open_fds && unuse_one_window(NULL, -1))\n \t\t; /* nothing */\n \n \tp->pack_fd = git_open_noatime(p->pack_name);\n \tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n \t\treturn -1;\n-\tpack_open_fds++;\n+\tctx->pack_open_fds++;\n \n \t/* If we created the struct before we had the pack we lack size. */\n \tif (!p->pack_size) {\n@@ -827,11 +883,12 @@ static int open_packed_git_1(struct packed_git *p)\n \n static int open_packed_git(struct packed_git *p)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tif (!open_packed_git_1(p))\n \t\treturn 0;\n \tif (p->pack_fd != -1) {\n \t\tclose(p->pack_fd);\n-\t\tpack_open_fds--;\n+\t\tctx->pack_open_fds--;\n \t\tp->pack_fd = -1;\n \t}\n \treturn -1;\n@@ -855,6 +912,7 @@ unsigned char *use_pack(struct packed_git *p,\n \t\toff_t offset,\n \t\tunsigned long *left)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tstruct pack_window *win = *w_cursor;\n \n \t/* Since packfiles end in a hash of their content and it's\n@@ -887,8 +945,8 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t\tif (len > packed_git_window_size)\n \t\t\t\tlen = packed_git_window_size;\n \t\t\twin->len = (size_t)len;\n-\t\t\tpack_mapped += win->len;\n-\t\t\twhile (packed_git_limit < pack_mapped\n+\t\t\tctx->pack_mapped += win->len;\n+\t\t\twhile (packed_git_limit < ctx->pack_mapped\n \t\t\t\t&& unuse_one_window(p, p->pack_fd))\n \t\t\t\t; /* nothing */\n \t\t\twin->base = xmmap(NULL, win->len,\n@@ -901,21 +959,21 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t\tif (!win->offset && win->len == p->pack_size\n \t\t\t\t&& !p->do_not_close) {\n \t\t\t\tclose(p->pack_fd);\n-\t\t\t\tpack_open_fds--;\n+\t\t\t\tctx->pack_open_fds--;\n \t\t\t\tp->pack_fd = -1;\n \t\t\t}\n-\t\t\tpack_mmap_calls++;\n-\t\t\tpack_open_windows++;\n-\t\t\tif (pack_mapped > peak_pack_mapped)\n-\t\t\t\tpeak_pack_mapped = pack_mapped;\n-\t\t\tif (pack_open_windows > peak_pack_open_windows)\n-\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n+\t\t\tctx->pack_mmap_calls++;\n+\t\t\tctx->pack_open_windows++;\n+\t\t\tif (ctx->pack_mapped > ctx->peak_pack_mapped)\n+\t\t\t\tctx->peak_pack_mapped = ctx->pack_mapped;\n+\t\t\tif (ctx->pack_open_windows > ctx->peak_pack_open_windows)\n+\t\t\t\tctx->peak_pack_open_windows = ctx->pack_open_windows;\n \t\t\twin->next = p->windows;\n \t\t\tp->windows = win;\n \t\t}\n \t}\n \tif (win != *w_cursor) {\n-\t\twin->last_used = pack_used_ctr++;\n+\t\twin->last_used = ctx->pack_used_ctr++;\n \t\twin->inuse_cnt++;\n \t\t*w_cursor = win;\n \t}\n@@ -998,8 +1056,9 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n \n void install_packed_git(struct packed_git *pack)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tif (pack->pack_fd != -1)\n-\t\tpack_open_fds++;\n+\t\tctx->pack_open_fds++;\n \n \tpack->next = get_packed_git();\n \tset_packed_git(pack);\n@@ -1108,12 +1167,12 @@ static void rearrange_packed_git(void)\n \tfree(ary);\n }\n \n-static int prepare_packed_git_run_once = 0;\n void prepare_packed_git(void)\n {\n \tstruct alternate_object_database *alt;\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \n-\tif (prepare_packed_git_run_once)\n+\tif (ctx->prepare_packed_git_run_once)\n \t\treturn;\n \tprepare_packed_git_one(get_object_directory(), 1);\n \tprepare_alt_odb();\n@@ -1123,13 +1182,14 @@ void prepare_packed_git(void)\n \t\talt->name[-1] = '/';\n \t}\n \trearrange_packed_git();\n-\tprepare_packed_git_run_once = 1;\n+\tctx->prepare_packed_git_run_once = 1;\n }\n \n void reprepare_packed_git(void)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tdiscard_revindex();\n-\tprepare_packed_git_run_once = 0;\n+\tctx->prepare_packed_git_run_once = 0;\n \tprepare_packed_git();\n }\n \n@@ -1664,24 +1724,6 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \treturn buffer;\n }\n \n-#define MAX_DELTA_CACHE (256)\n-\n-static size_t delta_base_cached;\n-\n-static struct delta_base_cache_lru_list {\n-\tstruct delta_base_cache_lru_list *prev;\n-\tstruct delta_base_cache_lru_list *next;\n-} delta_base_cache_lru = { &delta_base_cache_lru, &delta_base_cache_lru };\n-\n-static struct delta_base_cache_entry {\n-\tstruct delta_base_cache_lru_list lru;\n-\tvoid *data;\n-\tstruct packed_git *p;\n-\toff_t base_offset;\n-\tunsigned long size;\n-\tenum object_type type;\n-} delta_base_cache[MAX_DELTA_CACHE];\n-\n static unsigned long pack_entry_hash(struct packed_git *p, off_t base_offset)\n {\n \tunsigned long hash;\n@@ -1694,7 +1736,8 @@ static unsigned long pack_entry_hash(struct packed_git *p, off_t base_offset)\n static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n {\n \tunsigned long hash = pack_entry_hash(p, base_offset);\n-\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\tstruct delta_base_cache_entry *ent\n+\t\t= get_thread_pack_context()->delta_base_cache + hash;\n \treturn (ent->data && ent->p == p && ent->base_offset == base_offset);\n }\n \n@@ -1703,7 +1746,8 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n {\n \tvoid *ret;\n \tunsigned long hash = pack_entry_hash(p, base_offset);\n-\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\tstruct delta_base_cache_entry *ent = ctx->delta_base_cache + hash;\n \n \tret = ent->data;\n \tif (!ret || ent->p != p || ent->base_offset != base_offset)\n@@ -1713,7 +1757,7 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \t\tent->data = NULL;\n \t\tent->lru.next->prev = ent->lru.prev;\n \t\tent->lru.prev->next = ent->lru.next;\n-\t\tdelta_base_cached -= ent->size;\n+\t\tctx->delta_base_cached -= ent->size;\n \t} else {\n \t\tret = xmemdupz(ent->data, ent->size);\n \t}\n@@ -1722,48 +1766,52 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \treturn ret;\n }\n \n-static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)\n+static inline void release_delta_base_cache(struct pack_context *ctx,\n+\t\t\t\t\t    struct delta_base_cache_entry *ent)\n {\n \tif (ent->data) {\n \t\tfree(ent->data);\n \t\tent->data = NULL;\n \t\tent->lru.next->prev = ent->lru.prev;\n \t\tent->lru.prev->next = ent->lru.next;\n-\t\tdelta_base_cached -= ent->size;\n+\t\tctx->delta_base_cached -= ent->size;\n \t}\n }\n \n void clear_delta_base_cache(void)\n {\n \tunsigned long p;\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\tstruct delta_base_cache_entry *delta_base_cache = ctx->delta_base_cache;\n \tfor (p = 0; p < MAX_DELTA_CACHE; p++)\n-\t\trelease_delta_base_cache(&delta_base_cache[p]);\n+\t\trelease_delta_base_cache(ctx, &delta_base_cache[p]);\n }\n \n static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n \tvoid *base, unsigned long base_size, enum object_type type)\n {\n \tunsigned long hash = pack_entry_hash(p, base_offset);\n-\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\tstruct pack_context *ctx = get_thread_pack_context();\n+\tstruct delta_base_cache_entry *ent = ctx->delta_base_cache + hash;\n \tstruct delta_base_cache_lru_list *lru;\n \n-\trelease_delta_base_cache(ent);\n-\tdelta_base_cached += base_size;\n+\trelease_delta_base_cache(ctx, ent);\n+\tctx->delta_base_cached += base_size;\n \n-\tfor (lru = delta_base_cache_lru.next;\n-\t     delta_base_cached > delta_base_cache_limit\n-\t     && lru != &delta_base_cache_lru;\n+\tfor (lru = ctx->delta_base_cache_lru->next;\n+\t     ctx->delta_base_cached > delta_base_cache_limit\n+\t     && lru != ctx->delta_base_cache_lru;\n \t     lru = lru->next) {\n \t\tstruct delta_base_cache_entry *f = (void *)lru;\n \t\tif (f->type == OBJ_BLOB)\n-\t\t\trelease_delta_base_cache(f);\n+\t\t\trelease_delta_base_cache(ctx, f);\n \t}\n-\tfor (lru = delta_base_cache_lru.next;\n-\t     delta_base_cached > delta_base_cache_limit\n-\t     && lru != &delta_base_cache_lru;\n+\tfor (lru = ctx->delta_base_cache_lru->next;\n+\t     ctx->delta_base_cached > delta_base_cache_limit\n+\t     && lru != ctx->delta_base_cache_lru;\n \t     lru = lru->next) {\n \t\tstruct delta_base_cache_entry *f = (void *)lru;\n-\t\trelease_delta_base_cache(f);\n+\t\trelease_delta_base_cache(ctx, f);\n \t}\n \n \tent->p = p;\n@@ -1771,10 +1819,10 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n \tent->type = type;\n \tent->data = base;\n \tent->size = base_size;\n-\tent->lru.next = &delta_base_cache_lru;\n-\tent->lru.prev = delta_base_cache_lru.prev;\n-\tdelta_base_cache_lru.prev->next = &ent->lru;\n-\tdelta_base_cache_lru.prev = &ent->lru;\n+\tent->lru.next = ctx->delta_base_cache_lru;\n+\tent->lru.prev = ctx->delta_base_cache_lru->prev;\n+\tctx->delta_base_cache_lru->prev->next = &ent->lru;\n+\tctx->delta_base_cache_lru->prev = &ent->lru;\n }\n \n static void *read_object(const unsigned char *sha1, enum object_type *type,\n@@ -2066,20 +2114,21 @@ static int fill_pack_entry(const unsigned char *sha1,\n \n static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n {\n+\tstruct pack_context *ctx = get_thread_pack_context();\n \tstruct packed_git *p;\n \n \tprepare_packed_git();\n \tif (!get_packed_git())\n \t\treturn 0;\n \n-\tif (last_found_pack && fill_pack_entry(sha1, e, last_found_pack))\n+\tif (ctx->last_found_pack && fill_pack_entry(sha1, e, ctx->last_found_pack))\n \t\treturn 1;\n \n \tfor (p = get_packed_git(); p; p = p->next) {\n-\t\tif (p == last_found_pack || !fill_pack_entry(sha1, e, p))\n+\t\tif (p == ctx->last_found_pack || !fill_pack_entry(sha1, e, p))\n \t\t\tcontinue;\n \n-\t\tlast_found_pack = p;\n+\t\tctx->last_found_pack = p;\n \t\treturn 1;\n \t}\n \treturn 0;\n-- \n1.7.8.36.g69ee2\n"},{"id":"188868","messageId":"1334068771-32725-4-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] Make lookup_*() functions thread-safe","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:29Z","receivedAt":"2012-04-10T14:39:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n blob.c         |   11 +++++++++--\n commit.c       |   12 +++++++++---\n list-objects.c |    2 ++\n object.c       |    9 ++++++++-\n object.h       |    6 ++++++\n tag.c          |    9 +++++++--\n tree.c         |   18 ++++++++++++++----\n 7 files changed, 55 insertions(+), 12 deletions(-)\n\ndiff --git a/blob.c b/blob.c\nindex ae320bd..606e2be 100644\n--- a/blob.c\n+++ b/blob.c\n@@ -6,8 +6,13 @@ const char *blob_type = \"blob\";\n struct blob *lookup_blob(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n-\tif (!obj)\n-\t\treturn create_object(sha1, OBJ_BLOB, alloc_blob_node());\n+\tLOCK();\n+\tif (!obj) {\n+\t\tobj = create_object(sha1, OBJ_BLOB, alloc_blob_node());\n+\t\tUNLOCK();\n+\t\treturn obj;\n+\t}\n+\tUNLOCK();\n \tif (!obj->type)\n \t\tobj->type = OBJ_BLOB;\n \tif (obj->type != OBJ_BLOB) {\n@@ -20,6 +25,8 @@ struct blob *lookup_blob(const unsigned char *sha1)\n \n int parse_blob_buffer(struct blob *item, void *buffer, unsigned long size)\n {\n+\tLOCK();\n \titem->object.parsed = 1;\n+\tUNLOCK();\n \treturn 0;\n }\ndiff --git a/commit.c b/commit.c\nindex 4b39c19..f84152c 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -54,9 +54,15 @@ struct commit *lookup_commit_or_die(const unsigned char *sha1, const char *ref_n\n \n struct commit *lookup_commit(const unsigned char *sha1)\n {\n-\tstruct object *obj = lookup_object(sha1);\n-\tif (!obj)\n-\t\treturn create_object(sha1, OBJ_COMMIT, alloc_commit_node());\n+\tstruct object *obj;\n+\tLOCK();\n+\tobj = lookup_object(sha1);\n+\tif (!obj) {\n+\t\tobj = create_object(sha1, OBJ_COMMIT, alloc_commit_node());\n+\t\tUNLOCK();\n+\t\treturn (struct commit*)obj;\n+\t}\n+\tUNLOCK();\n \tif (!obj->type)\n \t\tobj->type = OBJ_COMMIT;\n \treturn check_commit(obj, sha1, 0);\ndiff --git a/list-objects.c b/list-objects.c\nindex 3dd4a96..2605b63 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -123,8 +123,10 @@ static void process_tree(struct rev_info *revs,\n \t\t\t\t     cb_data);\n \t}\n \tstrbuf_setlen(base, baselen);\n+\tLOCK();\n \tfree(tree->buffer);\n \ttree->buffer = NULL;\n+\tUNLOCK();\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\ndiff --git a/object.c b/object.c\nindex 6b06297..05fb91f 100644\n--- a/object.c\n+++ b/object.c\n@@ -8,6 +8,9 @@\n static struct object **obj_hash;\n static int nr_objs, obj_hash_size;\n \n+int multithread_object_access;\n+pthread_mutex_t mtx;\n+\n unsigned int get_max_object_index(void)\n {\n \treturn obj_hash_size;\n@@ -126,9 +129,13 @@ void *create_object(const unsigned char *sha1, int type, void *o)\n \n struct object *lookup_unknown_object(const unsigned char *sha1)\n {\n-\tstruct object *obj = lookup_object(sha1);\n+\tstruct object *obj;\n+\n+\tLOCK();\n+\tobj = lookup_object(sha1);\n \tif (!obj)\n \t\tobj = create_object(sha1, OBJ_NONE, alloc_object_node());\n+\tUNLOCK();\n \treturn obj;\n }\n \ndiff --git a/object.h b/object.h\nindex b6618d9..17c7c65 100644\n--- a/object.h\n+++ b/object.h\n@@ -32,6 +32,12 @@ struct object {\n \tunsigned char sha1[20];\n };\n \n+extern int multithread_object_access;\n+extern pthread_mutex_t mtx;\n+\n+#define LOCK() { if (multithread_object_access) pthread_mutex_lock(&mtx); }\n+#define UNLOCK() { if (multithread_object_access) pthread_mutex_unlock(&mtx); }\n+\n extern const char *typename(unsigned int type);\n extern int type_from_string(const char *str);\n \ndiff --git a/tag.c b/tag.c\nindex 78d272b..4f16c5b 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -39,8 +39,13 @@ struct object *deref_tag_noverify(struct object *o)\n struct tag *lookup_tag(const unsigned char *sha1)\n {\n \tstruct object *obj = lookup_object(sha1);\n-\tif (!obj)\n-\t\treturn create_object(sha1, OBJ_TAG, alloc_tag_node());\n+\tLOCK();\n+\tif (!obj) {\n+\t\tobj = create_object(sha1, OBJ_TAG, alloc_tag_node());\n+\t\tUNLOCK();\n+\t\treturn obj;\n+\t}\n+\tUNLOCK();\n \tif (!obj->type)\n \t\tobj->type = OBJ_TAG;\n \tif (obj->type != OBJ_TAG) {\ndiff --git a/tree.c b/tree.c\nindex 676e9f7..6d3c6a2 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -180,9 +180,15 @@ int read_tree(struct tree *tree, int stage, struct pathspec *match)\n \n struct tree *lookup_tree(const unsigned char *sha1)\n {\n-\tstruct object *obj = lookup_object(sha1);\n-\tif (!obj)\n-\t\treturn create_object(sha1, OBJ_TREE, alloc_tree_node());\n+\tstruct object *obj;\n+\tLOCK();\n+\tobj = lookup_object(sha1);\n+\tif (!obj) {\n+\t\tobj = create_object(sha1, OBJ_TREE, alloc_tree_node());\n+\t\tUNLOCK();\n+\t\treturn obj;\n+\t}\n+\tUNLOCK();\n \tif (!obj->type)\n \t\tobj->type = OBJ_TREE;\n \tif (obj->type != OBJ_TREE) {\n@@ -195,11 +201,15 @@ struct tree *lookup_tree(const unsigned char *sha1)\n \n int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size)\n {\n-\tif (item->object.parsed)\n+\tLOCK();\n+\tif (item->object.parsed) {\n+\t\tUNLOCK();\n \t\treturn 0;\n+\t}\n \titem->object.parsed = 1;\n \titem->buffer = buffer;\n \titem->size = size;\n+\tUNLOCK();\n \n \treturn 0;\n }\n-- \n1.7.8.36.g69ee2\n"},{"id":"188867","messageId":"1334068771-32725-5-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/5] Teach traverse_commit_list callsites about new parameter, nr_threads","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:30Z","receivedAt":"2012-04-10T14:39:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/pack-objects.c |    2 +-\n builtin/rev-list.c     |    4 ++--\n list-objects.c         |    2 +-\n list-objects.h         |    2 +-\n upload-pack.c          |    2 +-\n 5 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2f242c4..0702cd1 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2292,7 +2292,7 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\ttraverse_commit_list(&revs, show_commit, show_object, NULL, 0);\n \n \tif (keep_unreachable)\n \t\tadd_objects_in_unpacked_packs(&revs);\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 4c4d404..e720561 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -290,7 +290,7 @@ static int show_bisect_vars(struct rev_list_info *info, int reaches, int all)\n \t\tstrcpy(hex, sha1_to_hex(revs->commits->item->object.sha1));\n \n \tif (flags & BISECT_SHOW_ALL) {\n-\t\ttraverse_commit_list(revs, show_commit, show_object, info);\n+\t\ttraverse_commit_list(revs, show_commit, show_object, info, 0);\n \t\tprintf(\"------\\n\");\n \t}\n \n@@ -395,7 +395,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n-\ttraverse_commit_list(&revs, show_commit, show_object, &info);\n+\ttraverse_commit_list(&revs, show_commit, show_object, &info, 0);\n \n \tif (revs.count) {\n \t\tif (revs.left_right && revs.cherry_mark)\ndiff --git a/list-objects.c b/list-objects.c\nindex 2605b63..4f365e8 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -170,7 +170,7 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t  show_commit_fn show_commit,\n \t\t\t  show_object_fn show_object,\n-\t\t\t  void *data)\n+\t\t\t  void *data, int nr_threads_)\n {\n \tint i;\n \tstruct commit *commit;\ndiff --git a/list-objects.h b/list-objects.h\nindex 3db7bb6..aa604ba 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -3,7 +3,7 @@\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const struct name_path *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n+void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *, int);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct commit_list *, struct rev_info *, show_edge_fn);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex bb08e2e..698f966 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -128,7 +128,7 @@ static int do_rev_list(int in, int out, void *user_data)\n \t\tfor (i = 0; i < extra_edge_obj.nr; i++)\n \t\t\tfprintf(pack_pipe, \"-%s\\n\", sha1_to_hex(\n \t\t\t\t\textra_edge_obj.objects[i].item->sha1));\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\ttraverse_commit_list(&revs, show_commit, show_object, NULL, 0);\n \tfflush(pack_pipe);\n \tfclose(pack_pipe);\n \treturn 0;\n-- \n1.7.8.36.g69ee2\n"},{"id":"188869","messageId":"1334068771-32725-6-git-send-email-pclouds@gmail.com","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/5] Support multithread in traverse_commit_list and rev-list","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-04-10T14:39:31Z","receivedAt":"2012-04-10T14:39:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/rev-list.c |    3 +-\n list-objects.c     |  153 +++++++++++++++++++++++++++++++++++++++++++++++++--\n revision.c         |   16 ++++++\n revision.h         |    2 +\n 4 files changed, 167 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex e720561..e07ba40 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -395,7 +395,8 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\treturn show_bisect_vars(&info, reaches, all);\n \t}\n \n-\ttraverse_commit_list(&revs, show_commit, show_object, &info, 0);\n+\ttraverse_commit_list(&revs, show_commit, show_object, &info,\n+\t\t\t     getenv(\"REV_LIST\") ? 2 : 0);\n \n \tif (revs.count) {\n \t\tif (revs.left_right && revs.cherry_mark)\ndiff --git a/list-objects.c b/list-objects.c\nindex 4f365e8..59a7c33 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -7,6 +7,29 @@\n #include \"tree-walk.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"semaphore.h\"\n+\n+struct thread_data {\n+\tpthread_t thread;\n+\tpthread_mutex_t mutex;\n+\tsem_t sem;\n+\n+\tstruct rev_info *revs;\n+\tstruct tree *tree;\n+\tshow_object_fn show;\n+\tstruct name_path *path;\n+\tstruct strbuf base;\n+\tconst char *name;\n+\tvoid *cb_data;\n+};\n+\n+static struct thread_data *threads;\n+static int nr_threads;\n+static pthread_mutex_t obj_lock;\n+static int done;\n+\n+#define LOCK_OBJ() { if (nr_threads) pthread_mutex_lock(&obj_lock); }\n+#define UNLOCK_OBJ() { if (nr_threads) pthread_mutex_unlock(&obj_lock); }\n \n static void process_blob(struct rev_info *revs,\n \t\t\t struct blob *blob,\n@@ -21,10 +44,14 @@ static void process_blob(struct rev_info *revs,\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad blob object\");\n-\tif (obj->flags & (UNINTERESTING | SEEN))\n+\tLOCK_OBJ();\n+\tif (obj->flags & (UNINTERESTING | SEEN)) {\n+\t\tUNLOCK_OBJ();\n \t\treturn;\n+\t}\n \tobj->flags |= SEEN;\n \tshow(obj, path, name, cb_data);\n+\tUNLOCK_OBJ();\n }\n \n /*\n@@ -79,12 +106,20 @@ static void process_tree(struct rev_info *revs,\n \t\treturn;\n \tif (!obj)\n \t\tdie(\"bad tree object\");\n-\tif (obj->flags & (UNINTERESTING | SEEN))\n+\n+\tLOCK_OBJ();\n+\tif (obj->flags & (UNINTERESTING | SEEN)) {\n+\t\tUNLOCK_OBJ();\n \t\treturn;\n+\t}\n+\tobj->flags |= SEEN;\n+\tUNLOCK_OBJ();\n \tif (parse_tree(tree) < 0)\n \t\tdie(\"bad tree object %s\", sha1_to_hex(obj->sha1));\n-\tobj->flags |= SEEN;\n+\tLOCK_OBJ();\n \tshow(obj, path, name, cb_data);\n+\tUNLOCK_OBJ();\n+\n \tme.up = path;\n \tme.elem = name;\n \tme.elem_len = strlen(name);\n@@ -167,6 +202,80 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n+static void *woodchipper(void *arg)\n+{\n+\tstruct thread_data *data = arg;\n+\tstruct tree *tree;\n+\n+\tinit_pack_context();\n+\n+\twhile (!done) {\n+\t\tpthread_mutex_lock(&data->mutex);\n+\t\ttree = data->tree;\n+\t\tpthread_mutex_unlock(&data->mutex);\n+\t\tif (!tree) {\n+\t\t\tsem_wait(&data->sem);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tprocess_tree(data->revs, tree, data->show, data->path,\n+\t\t\t     &data->base, data->name, data->cb_data);\n+\t\tfree(data->path);\n+\t\tstrbuf_reset(&data->base);\n+\n+\t\tpthread_mutex_lock(&data->mutex);\n+\t\tdata->tree = NULL;\n+\t\tpthread_mutex_unlock(&data->mutex);\n+\n+\t\tsem_wait(&data->sem);\n+\t}\n+\treturn NULL;\n+}\n+\n+static void distribute_tree(struct rev_info *revs,\n+\t\t\t    struct tree *tree,\n+\t\t\t    show_object_fn show,\n+\t\t\t    struct name_path *path,\n+\t\t\t    struct strbuf *base,\n+\t\t\t    const char *name,\n+\t\t\t    void *cb_data)\n+{\n+\tint i = 0;\n+\n+\twhile (nr_threads && i < nr_threads) {\n+\t\tstruct tree *old_tree;\n+\t\tstruct thread_data *thr = threads + i;\n+\t\tpthread_mutex_lock(&thr->mutex);\n+\t\told_tree = thr->tree;\n+\t\tpthread_mutex_unlock(&thr->mutex);\n+#if 0\n+\t\tif (old_tree) {\n+\t\t\ti = (i + 1) % nr_threads;\n+\t\t\tsched_yield();\n+\t\t\tcontinue;\n+\t\t}\n+#else\n+\t\tif (old_tree) {\n+\t\t\ti++;\n+\t\t\tcontinue;\n+\t\t}\n+#endif\n+\n+\t\t/* the thread must be asleep, or going to sleep by now */\n+\t\tthr->revs = revs;\n+\t\tthr->tree = tree;\n+\t\tthr->show = show;\n+\t\tthr->path = dup_name_path(path);\n+\t\tstrbuf_addbuf(&thr->base, base);\n+\t\tthr->name = name;\n+\t\tthr->cb_data = cb_data;\n+\t\tsem_post(&thr->sem);\n+\t\treturn;\n+\t}\n+\n+\tprocess_tree(revs, tree, show, path, base, name, cb_data);\n+}\n+\n void traverse_commit_list(struct rev_info *revs,\n \t\t\t  show_commit_fn show_commit,\n \t\t\t  show_object_fn show_object,\n@@ -186,20 +295,45 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\tadd_pending_tree(revs, commit->tree);\n \t\tshow_commit(commit, data);\n \t}\n+\n+\tnr_threads = nr_threads_;\n+\tif (nr_threads) {\n+\t\tthreads = xmalloc((sizeof*threads) * nr_threads);\n+\t\tmemset(threads, 0, sizeof(*threads) * nr_threads);\n+\t\tpthread_mutex_init(&obj_lock, NULL);\n+\t\tdone = 0;\n+\t\tfor (i = 0; i < nr_threads; i++) {\n+\t\t\tthreads[i].tree = NULL;\n+\t\t\tstrbuf_init(&threads[i].base, PATH_MAX);\n+\t\t\tpthread_mutex_init(&threads[i].mutex, NULL);\n+\t\t\tsem_init(&threads[i].sem, 0, 0);\n+\t\t\tpthread_create(&threads[i].thread, NULL, woodchipper, threads + i);\n+\t\t}\n+\n+\t\tpthread_mutex_init(&mtx, NULL);\n+\t\tmultithread_object_access = 1;\n+\t}\n+\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n-\t\tif (obj->flags & (UNINTERESTING | SEEN))\n+\t\tLOCK_OBJ();\n+\t\tif (obj->flags & (UNINTERESTING | SEEN)) {\n+\t\t\tUNLOCK_OBJ();\n \t\t\tcontinue;\n+\t\t}\n+\t\tUNLOCK_OBJ();\n \t\tif (obj->type == OBJ_TAG) {\n+\t\t\tLOCK_OBJ();\n \t\t\tobj->flags |= SEEN;\n \t\t\tshow_object(obj, NULL, name, data);\n+\t\t\tUNLOCK_OBJ();\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_TREE) {\n-\t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n-\t\t\t\t     NULL, &base, name, data);\n+\t\t\tdistribute_tree(revs, (struct tree *)obj, show_object,\n+\t\t\t\t\tNULL, &base, name, data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n@@ -210,6 +344,13 @@ void traverse_commit_list(struct rev_info *revs,\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t    sha1_to_hex(obj->sha1), name);\n \t}\n+\tif (nr_threads) {\n+\t\tdone = 1;\n+\t\tfor (i = 0; i < nr_threads; i++) {\n+\t\t\tsem_post(&threads[i].sem);\n+\t\t\tpthread_join(threads[i].thread, NULL);\n+\t\t}\n+\t}\n \tif (revs->pending.nr) {\n \t\tfree(revs->pending.objects);\n \t\trevs->pending.nr = 0;\ndiff --git a/revision.c b/revision.c\nindex b3554ed..40b6e15 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -40,6 +40,22 @@ char *path_name(const struct name_path *path, const char *name)\n \treturn n;\n }\n \n+struct name_path *dup_name_path(const struct name_path *path)\n+{\n+\tconst struct name_path *src;\n+\tstruct name_path *dup, *dst;\n+\tint len;\n+\tfor (len = 0, src = path; src; src = src->up)\n+\t\tlen++;\n+\tdup = xmalloc(sizeof(*dup) * len);\n+\tfor (src = path, dst = dup; src; src = src->up, dst++) {\n+\t\tmemcpy(dst, src, sizeof(*src));\n+\t\tif (dst > dup)\n+\t\t\tdst[-1].up = dst;\n+\t}\n+\treturn dup;\n+}\n+\n static int show_path_component_truncated(FILE *out, const char *name, int len)\n {\n \tint cnt;\ndiff --git a/revision.h b/revision.h\nindex b8e9223..0da4636 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -208,6 +208,8 @@ struct name_path {\n \n char *path_name(const struct name_path *path, const char *name);\n \n+struct name_path *dup_name_path(const struct name_path *);\n+\n extern void show_object_with_name(FILE *, struct object *, const struct name_path *, const char *);\n \n extern void add_object(struct object *obj,\n-- \n1.7.8.36.g69ee2\n"},{"id":"188880","messageId":"201204101051.56288.mfick@codeaurora.org","threadId":"30195","inReplyTo":"1334068771-32725-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 0/5] multithread traverse_commit_list (aka rev-list)","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2012-04-10T16:51:55Z","receivedAt":"2012-04-10T16:51:55Z","isPatch":true,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"On Tuesday, April 10, 2012 08:39:26 am Nguyễn Thái Ngọc Duy \nwrote:\n> While this series does not break rev-list (but breaks all\n> external commands, I don't care anyway), and it proves\n> lock-free pack access works, it does not improve\n> rev-list performance. I suspect extensive locking around\n> \"struct object *\" may be the culprit.  \n\nOr if it is memory contention and your threads share the \nsame cache, it won't help much either,\n\n-Martin\n\n-- \nEmployee of Qualcomm Innovation Center, Inc. which is a \nmember of Code Aurora Forum\n"}]}