{"thread":{"id":"36517","subject":"[PATCH 00/32] Split index mode for very large indexes","startedAt":"2014-04-28T10:55:21Z","lastAt":"2014-05-22T16:38:10Z","messageCount":75,"participants":["Nguyễn Thái Ngọc Duy","Shawn Pearce","Junio C Hamano","Duy Nguyen","Eric Sunshine","Richard Hansen","Erik Faye-Lund","Stefan Beller","David Turner"],"isPatch":true,"patchVersion":1,"patchTotal":32},"messages":[{"id":"239864","messageId":"1398682553-11634-1-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":null,"subject":"[PATCH 00/32] Split index mode for very large indexes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:21Z","receivedAt":"2014-04-28T10:55:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"I hinted about it earlier [1]. It now passes the test suite and with a\ndesign that I'm happy with (thanks to Junio for a suggestion about the\nrename problem).\n\nFrom the user point of view, this reduces the writable size of index\ndown to the number of updated files. For example my webkit index v4 is\n14MB. With a fresh split, I only have to update an index of 200KB.\nEvery file I touch will add about 80 bytes to that. As long as I don't\ntouch every single tracked file in my worktree, I should not pay\npenalty for writing 14MB index file on every operation.\n\nThe read penalty is not addressed here, so I still pay 14MB hashing\ncost. But that's an easy problem. We could cache the validated index\nin a daemon. Whenever git needs to load an index, it pokes the daemon.\nThe daemon verifies that the on-disk index still has the same\nsignature, then sends the in-mem index to git. When git updates the\nindex, it pokes the daemon again to update in-mem index. Next time git\nreads the index, it does not have to pay I/O cost any more (actually\nit does but the cost is hidden away when you do not have to read it\nyet).\n\nThe forth patch is not really necessary. I started out with a\ndifferent approach that needed that abstraction. But I think it's\nstill a nice thing to keep. The real meat starts from 0017 to 0025. In\nessence, the new index is more like a journal, where the real index is\nput away unchanged.\n\nDoing this in other implementations should be easy (at least the\nreading part) and with small code change. The whole index format is\nretained. All you need is to read a new extension that contains two\newah-bitmaps and apply the changes to create the final index.\n\nThis is a preparation step for my untracked file cache. With writing\n(and later on reading) index becoming cheap, I can start to put more\nthings in there.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/246471/focus=247031\n\nNguyễn Thái Ngọc Duy (32):\n ewah: fix constness of ewah_read_mmap\n ewah: delete unused ewah_read_mmap_native declaration\n sequencer: do not update/refresh index if the lock cannot be held\n read-cache: new API write_locked_index instead of write_index/write_cache\n read-cache: relocate and unexport commit_locked_index()\n read-cache: store in-memory flags in the first 12 bits of ce_flags\n read-cache: be strict about \"changed\" in remove_marked_cache_entries()\n read-cache: be specific what part of the index has changed\n update-index: be specific what part of the index has changed\n resolve-undo: be specific what part of the index has changed\n unpack-trees: be specific what part of the index has changed\n cache-tree: mark istate->cache_changed on cache tree invalidation\n cache-tree: mark istate->cache_changed on cache tree update\n cache-tree: mark istate->cache_changed on prime_cache_tree()\n entry.c: update cache_changed if refresh_cache is set in checkout_entry()\n read-cache: save index SHA-1 after reading\n read-cache: split-index mode\n read-cache: mark new entries for split index\n read-cache: save deleted entries in split index\n read-cache: mark updated entries for split index\n split-index: the writing part\n split-index: the reading part\n split-index: do not invalidate cache-tree at read time\n split-index: strip pathname of on-disk replaced entries\n update-index: new options to enable/disable split index mode\n update-index --split-index: do not split if $GIT_DIR is read only\n rev-parse: add --shared-index-path to get shared index path\n read-tree: force split-index mode off on --index-output\n read-tree: note about dropping split-index mode or index version\n read-cache: force split index mode with GIT_TEST_SPLIT_INDEX\n t2104: make sure split index mode is off for the version test\n t1700: new tests for split-index mode\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239865","messageId":"1398682553-11634-2-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 01/32] ewah: fix constness of ewah_read_mmap","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:22Z","receivedAt":"2014-04-28T10:55:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n ewah/ewah_io.c | 4 ++--\n ewah/ewok.h | 2 +-\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/ewah/ewah_io.c b/ewah/ewah_io.c\nindex f7f700e..1c2d7af 100644\n--- a/ewah/ewah_io.c\n+++ b/ewah/ewah_io.c\n@@ -110,9 +110,9 @@ int ewah_serialize(struct ewah_bitmap *self, int fd)\n \treturn ewah_serialize_to(self, write_helper, (void *)(intptr_t)fd);\n }\n \n-int ewah_read_mmap(struct ewah_bitmap *self, void *map, size_t len)\n+int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len)\n {\n-\tuint8_t *ptr = map;\n+\tconst uint8_t *ptr = map;\n \tsize_t i;\n \n \tself->bit_size = get_be32(ptr);\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 43adeb5..0556ca5 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -99,7 +99,7 @@ int ewah_serialize(struct ewah_bitmap *self, int fd);\n int ewah_serialize_native(struct ewah_bitmap *self, int fd);\n \n int ewah_deserialize(struct ewah_bitmap *self, int fd);\n-int ewah_read_mmap(struct ewah_bitmap *self, void *map, size_t len);\n+int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len);\n int ewah_read_mmap_native(struct ewah_bitmap *self, void *map, size_t len);\n \n uint32_t ewah_checksum(struct ewah_bitmap *self);\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239896","messageId":"1398682553-11634-3-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 02/32] ewah: delete unused ewah_read_mmap_native declaration","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:23Z","receivedAt":"2014-04-28T10:55:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n ewah/ewok.h | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex 0556ca5..f6ad190 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -100,7 +100,6 @@ int ewah_serialize_native(struct ewah_bitmap *self, int fd);\n \n int ewah_deserialize(struct ewah_bitmap *self, int fd);\n int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len);\n-int ewah_read_mmap_native(struct ewah_bitmap *self, void *map, size_t len);\n \n uint32_t ewah_checksum(struct ewah_bitmap *self);\n \n-- \n1.9.1.346.ga2b5940\n"},{"id":"239866","messageId":"1398682553-11634-4-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 03/32] sequencer: do not update/refresh index if the lock cannot be held","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:24Z","receivedAt":"2014-04-28T10:55:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n sequencer.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sequencer.c b/sequencer.c\nindex bde5f04..7b886a6 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -679,7 +679,7 @@ static void read_and_refresh_cache(struct replay_opts *opts)\n \tif (read_index_preload(&the_index, NULL) < 0)\n \t\tdie(_(\"git %s: failed to read the index\"), action_name(opts));\n \trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, NULL, NULL, NULL);\n-\tif (the_index.cache_changed) {\n+\tif (the_index.cache_changed && index_fd >= 0) {\n \t\tif (write_index(&the_index, index_fd) ||\n \t\t commit_locked_index(&index_lock))\n \t\t\tdie(_(\"git %s: failed to refresh the index\"), action_name(opts));\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239867","messageId":"1398682553-11634-5-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 04/32] read-cache: new API write_locked_index instead of write_index/write_cache","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:25Z","receivedAt":"2014-04-28T10:55:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/add.c | 6 ++----\n builtin/apply.c | 9 ++++-----\n builtin/checkout-index.c | 3 +--\n builtin/checkout.c | 11 ++++-------\n builtin/clone.c | 7 +++----\n builtin/commit.c | 33 ++++++++++++++-------------------\n builtin/merge.c | 12 ++++--------\n builtin/mv.c | 7 +++----\n builtin/read-tree.c | 7 +++----\n builtin/reset.c | 5 ++---\n builtin/rm.c | 7 +++----\n builtin/update-index.c | 3 +--\n cache-tree.c | 3 +--\n cache.h | 6 ++++--\n merge-recursive.c | 7 +++----\n merge.c | 7 +++----\n read-cache.c | 28 ++++++++++++++++++++++++----\n rerere.c | 3 +--\n sequencer.c | 12 +++++-------\n test-scrap-cache-tree.c | 5 ++---\n 20 files changed, 87 insertions(+), 94 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 459208a..4baf3a5 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -299,7 +299,6 @@ static int add_files(struct dir_struct *dir, int flags)\n int cmd_add(int argc, const char **argv, const char *prefix)\n {\n \tint exit_status = 0;\n-\tint newfd;\n \tstruct pathspec pathspec;\n \tstruct dir_struct dir;\n \tint flags;\n@@ -345,7 +344,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \tadd_new_files = !take_worktree_changes && !refresh_only;\n \trequire_pathspec = !take_worktree_changes;\n \n-\tnewfd = hold_locked_index(&lock_file, 1);\n+\thold_locked_index(&lock_file, 1);\n \n \tflags = ((verbose ? ADD_CACHE_VERBOSE : 0) |\n \t\t (show_only ? ADD_CACHE_PRETEND : 0) |\n@@ -443,8 +442,7 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \n finish:\n \tif (active_cache_changed) {\n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(&lock_file))\n+\t\tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\t\tdie(_(\"Unable to write new index file\"));\n \t}\n \ndiff --git a/builtin/apply.c b/builtin/apply.c\nindex 87439fa..5e13444 100644\n--- a/builtin/apply.c\n+++ b/builtin/apply.c\n@@ -3644,7 +3644,7 @@ static void build_fake_ancestor(struct patch *list, const char *filename)\n {\n \tstruct patch *patch;\n \tstruct index_state result = { NULL };\n-\tint fd;\n+\tstatic struct lock_file lock;\n \n \t/* Once we start supporting the reverse patch, it may be\n \t * worth showing the new sha1 prefix, but until then...\n@@ -3682,8 +3682,8 @@ static void build_fake_ancestor(struct patch *list, const char *filename)\n \t\t\tdie (\"Could not add %s to temporary index\", name);\n \t}\n \n-\tfd = open(filename, O_WRONLY | O_CREAT, 0666);\n-\tif (fd < 0 || write_index(&result, fd) || close(fd))\n+\thold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);\n+\tif (write_locked_index(&result, &lock, COMMIT_LOCK))\n \t\tdie (\"Could not write temporary index to %s\", filename);\n \n \tdiscard_index(&result);\n@@ -4501,8 +4501,7 @@ int cmd_apply(int argc, const char **argv, const char *prefix_)\n \t}\n \n \tif (update_index) {\n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(&lock_file))\n+\t\tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\t\tdie(_(\"Unable to write new index file\"));\n \t}\n \ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 61e75eb..9e49bf2 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -279,8 +279,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t\tcheckout_all(prefix, prefix_length);\n \n \tif (0 <= newfd &&\n-\t (write_cache(newfd, active_cache, active_nr) ||\n-\t commit_locked_index(&lock_file)))\n+\t write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(\"Unable to write new index file\");\n \treturn 0;\n }\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 07cf555..944a634 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -225,7 +225,6 @@ static int checkout_paths(const struct checkout_opts *opts,\n \tint flag;\n \tstruct commit *head;\n \tint errs = 0;\n-\tint newfd;\n \tstruct lock_file *lock_file;\n \n \tif (opts->track != BRANCH_TRACK_UNSPECIFIED)\n@@ -256,7 +255,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \n \tlock_file = xcalloc(1, sizeof(struct lock_file));\n \n-\tnewfd = hold_locked_index(lock_file, 1);\n+\thold_locked_index(lock_file, 1);\n \tif (read_cache_preload(&opts->pathspec) < 0)\n \t\treturn error(_(\"corrupt index file\"));\n \n@@ -352,8 +351,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t\t}\n \t}\n \n-\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t commit_locked_index(lock_file))\n+\tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\n \n \tread_ref_full(\"HEAD\", rev, 0, &flag);\n@@ -444,8 +442,8 @@ static int merge_working_tree(const struct checkout_opts *opts,\n {\n \tint ret;\n \tstruct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file));\n-\tint newfd = hold_locked_index(lock_file, 1);\n \n+\thold_locked_index(lock_file, 1);\n \tif (read_cache_preload(NULL) < 0)\n \t\treturn error(_(\"corrupt index file\"));\n \n@@ -553,8 +551,7 @@ static int merge_working_tree(const struct checkout_opts *opts,\n \t\t}\n \t}\n \n-\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t commit_locked_index(lock_file))\n+\tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\n \n \tif (!opts->force && !opts->quiet)\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9b3c04d..48f91f5 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -616,7 +616,7 @@ static int checkout(void)\n \tstruct unpack_trees_options opts;\n \tstruct tree *tree;\n \tstruct tree_desc t;\n-\tint err = 0, fd;\n+\tint err = 0;\n \n \tif (option_no_checkout)\n \t\treturn 0;\n@@ -640,7 +640,7 @@ static int checkout(void)\n \tsetup_work_tree();\n \n \tlock_file = xcalloc(1, sizeof(struct lock_file));\n-\tfd = hold_locked_index(lock_file, 1);\n+\thold_locked_index(lock_file, 1);\n \n \tmemset(&opts, 0, sizeof opts);\n \topts.update = 1;\n@@ -656,8 +656,7 @@ static int checkout(void)\n \tif (unpack_trees(1, &t, &opts) < 0)\n \t\tdie(_(\"unable to checkout working tree\"));\n \n-\tif (write_cache(fd, active_cache, active_nr) ||\n-\t commit_locked_index(lock_file))\n+\tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\n \n \terr |= run_hook_le(NULL, \"post-checkout\", sha1_to_hex(null_sha1),\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 9cfef6c..243b0c3 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -305,7 +305,6 @@ static void refresh_cache_or_die(int refresh_flags)\n static char *prepare_index(int argc, const char **argv, const char *prefix,\n \t\t\t const struct commit *current_head, int is_status)\n {\n-\tint fd;\n \tstruct string_list partial;\n \tstruct pathspec pathspec;\n \tint refresh_flags = REFRESH_QUIET;\n@@ -321,12 +320,11 @@ static char *prepare_index(int argc, const char **argv, const char *prefix,\n \n \tif (interactive) {\n \t\tchar *old_index_env = NULL;\n-\t\tfd = hold_locked_index(&index_lock, 1);\n+\t\thold_locked_index(&index_lock, 1);\n \n \t\trefresh_cache_or_die(refresh_flags);\n \n-\t\tif (write_cache(fd, active_cache, active_nr) ||\n-\t\t close_lock_file(&index_lock))\n+\t\tif (write_locked_index(&the_index, &index_lock, CLOSE_LOCK))\n \t\t\tdie(_(\"unable to create temporary index\"));\n \n \t\told_index_env = getenv(INDEX_ENVIRONMENT);\n@@ -360,12 +358,11 @@ static char *prepare_index(int argc, const char **argv, const char *prefix,\n \t * (B) on failure, rollback the real index.\n \t */\n \tif (all || (also && pathspec.nr)) {\n-\t\tfd = hold_locked_index(&index_lock, 1);\n+\t\thold_locked_index(&index_lock, 1);\n \t\tadd_files_to_cache(also ? prefix : NULL, &pathspec, 0);\n \t\trefresh_cache_or_die(refresh_flags);\n \t\tupdate_main_cache_tree(WRITE_TREE_SILENT);\n-\t\tif (write_cache(fd, active_cache, active_nr) ||\n-\t\t close_lock_file(&index_lock))\n+\t\tif (write_locked_index(&the_index, &index_lock, CLOSE_LOCK))\n \t\t\tdie(_(\"unable to write new_index file\"));\n \t\tcommit_style = COMMIT_NORMAL;\n \t\treturn index_lock.filename;\n@@ -381,12 +378,12 @@ static char *prepare_index(int argc, const char **argv, const char *prefix,\n \t * We still need to refresh the index here.\n \t */\n \tif (!only && !pathspec.nr) {\n-\t\tfd = hold_locked_index(&index_lock, 1);\n+\t\thold_locked_index(&index_lock, 1);\n \t\trefresh_cache_or_die(refresh_flags);\n \t\tif (active_cache_changed) {\n \t\t\tupdate_main_cache_tree(WRITE_TREE_SILENT);\n-\t\t\tif (write_cache(fd, active_cache, active_nr) ||\n-\t\t\t commit_locked_index(&index_lock))\n+\t\t\tif (write_locked_index(&the_index, &index_lock,\n+\t\t\t\t\t COMMIT_LOCK))\n \t\t\t\tdie(_(\"unable to write new_index file\"));\n \t\t} else {\n \t\t\trollback_lock_file(&index_lock);\n@@ -432,24 +429,22 @@ static char *prepare_index(int argc, const char **argv, const char *prefix,\n \tif (read_cache() < 0)\n \t\tdie(_(\"cannot read the index\"));\n \n-\tfd = hold_locked_index(&index_lock, 1);\n+\thold_locked_index(&index_lock, 1);\n \tadd_remove_files(&partial);\n \trefresh_cache(REFRESH_QUIET);\n-\tif (write_cache(fd, active_cache, active_nr) ||\n-\t close_lock_file(&index_lock))\n+\tif (write_locked_index(&the_index, &index_lock, CLOSE_LOCK))\n \t\tdie(_(\"unable to write new_index file\"));\n \n-\tfd = hold_lock_file_for_update(&false_lock,\n-\t\t\t\t git_path(\"next-index-%\"PRIuMAX,\n-\t\t\t\t\t\t(uintmax_t) getpid()),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n+\thold_lock_file_for_update(&false_lock,\n+\t\t\t\t git_path(\"next-index-%\"PRIuMAX,\n+\t\t\t\t\t (uintmax_t) getpid()),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n \n \tcreate_base_index(current_head);\n \tadd_remove_files(&partial);\n \trefresh_cache(REFRESH_QUIET);\n \n-\tif (write_cache(fd, active_cache, active_nr) ||\n-\t close_lock_file(&false_lock))\n+\tif (write_locked_index(&the_index, &false_lock, CLOSE_LOCK))\n \t\tdie(_(\"unable to write temporary index file\"));\n \n \tdiscard_cache();\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 66d8843..bf770b6 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -657,14 +657,12 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,\n \t\t\t struct commit_list *remoteheads,\n \t\t\t struct commit *head, const char *head_arg)\n {\n-\tint index_fd;\n \tstruct lock_file *lock = xcalloc(1, sizeof(struct lock_file));\n \n-\tindex_fd = hold_locked_index(lock, 1);\n+\thold_locked_index(lock, 1);\n \trefresh_cache(REFRESH_QUIET);\n \tif (active_cache_changed &&\n-\t\t\t(write_cache(index_fd, active_cache, active_nr) ||\n-\t\t\t commit_locked_index(lock)))\n+\t write_locked_index(&the_index, lock, COMMIT_LOCK))\n \t\treturn error(_(\"Unable to write index.\"));\n \trollback_lock_file(lock);\n \n@@ -672,7 +670,6 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,\n \t\tint clean, x;\n \t\tstruct commit *result;\n \t\tstruct lock_file *lock = xcalloc(1, sizeof(struct lock_file));\n-\t\tint index_fd;\n \t\tstruct commit_list *reversed = NULL;\n \t\tstruct merge_options o;\n \t\tstruct commit_list *j;\n@@ -700,12 +697,11 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,\n \t\tfor (j = common; j; j = j->next)\n \t\t\tcommit_list_insert(j->item, &reversed);\n \n-\t\tindex_fd = hold_locked_index(lock, 1);\n+\t\thold_locked_index(lock, 1);\n \t\tclean = merge_recursive(&o, head,\n \t\t\t\tremoteheads->item, reversed, &result);\n \t\tif (active_cache_changed &&\n-\t\t\t\t(write_cache(index_fd, active_cache, active_nr) ||\n-\t\t\t\t commit_locked_index(lock)))\n+\t\t write_locked_index(&the_index, lock, COMMIT_LOCK))\n \t\t\tdie (_(\"unable to write %s\"), get_index_file());\n \t\trollback_lock_file(lock);\n \t\treturn clean ? 0 : 1;\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex 2a7243f..db40777 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -63,7 +63,7 @@ static struct lock_file lock_file;\n \n int cmd_mv(int argc, const char **argv, const char *prefix)\n {\n-\tint i, newfd, gitmodules_modified = 0;\n+\tint i, gitmodules_modified = 0;\n \tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n \tstruct option builtin_mv_options[] = {\n \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n@@ -85,7 +85,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \tif (--argc < 1)\n \t\tusage_with_options(builtin_mv_usage, builtin_mv_options);\n \n-\tnewfd = hold_locked_index(&lock_file, 1);\n+\thold_locked_index(&lock_file, 1);\n \tif (read_cache() < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n@@ -275,8 +275,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tstage_updated_gitmodules();\n \n \tif (active_cache_changed) {\n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(&lock_file))\n+\t\tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\t\tdie(_(\"Unable to write new index file\"));\n \t}\n \ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 0d7ef84..f26d90f 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -99,7 +99,7 @@ static struct lock_file lock_file;\n \n int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n {\n-\tint i, newfd, stage = 0;\n+\tint i, stage = 0;\n \tunsigned char sha1[20];\n \tstruct tree_desc t[MAX_UNPACK_TREES];\n \tstruct unpack_trees_options opts;\n@@ -149,7 +149,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n \targc = parse_options(argc, argv, unused_prefix, read_tree_options,\n \t\t\t read_tree_usage, 0);\n \n-\tnewfd = hold_locked_index(&lock_file, 1);\n+\thold_locked_index(&lock_file, 1);\n \n \tprefix_set = opts.prefix ? 1 : 0;\n \tif (1 < opts.merge + opts.reset + prefix_set)\n@@ -233,8 +233,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n \tif (nr_trees == 1 && !opts.prefix)\n \t\tprime_cache_tree(&active_cache_tree, trees[0]);\n \n-\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t commit_locked_index(&lock_file))\n+\tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(\"unable to write new index file\");\n \treturn 0;\n }\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex f4e0875..0c56d28 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -351,7 +351,7 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \n \tif (reset_type != SOFT) {\n \t\tstruct lock_file *lock = xcalloc(1, sizeof(*lock));\n-\t\tint newfd = hold_locked_index(lock, 1);\n+\t\thold_locked_index(lock, 1);\n \t\tif (reset_type == MIXED) {\n \t\t\tint flags = quiet ? REFRESH_QUIET : REFRESH_IN_PORCELAIN;\n \t\t\tif (read_from_tree(&pathspec, sha1, intent_to_add))\n@@ -367,8 +367,7 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \t\t\t\tdie(_(\"Could not reset index file to revision '%s'.\"), rev);\n \t\t}\n \n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(lock))\n+\t\tif (write_locked_index(&the_index, lock, COMMIT_LOCK))\n \t\t\tdie(_(\"Could not write new index file.\"));\n \t}\n \ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 960634d..bc6490b 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -278,7 +278,7 @@ static struct option builtin_rm_options[] = {\n \n int cmd_rm(int argc, const char **argv, const char *prefix)\n {\n-\tint i, newfd;\n+\tint i;\n \tstruct pathspec pathspec;\n \tchar *seen;\n \n@@ -293,7 +293,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tif (!index_only)\n \t\tsetup_work_tree();\n \n-\tnewfd = hold_locked_index(&lock_file, 1);\n+\thold_locked_index(&lock_file, 1);\n \n \tif (read_cache() < 0)\n \t\tdie(_(\"index file corrupt\"));\n@@ -427,8 +427,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (active_cache_changed) {\n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(&lock_file))\n+\t\tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\t\tdie(_(\"Unable to write new index file\"));\n \t}\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex ba54e19..42cbe4b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -921,8 +921,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t\texit(128);\n \t\t\tunable_to_lock_index_die(get_index_file(), lock_error);\n \t\t}\n-\t\tif (write_cache(newfd, active_cache, active_nr) ||\n-\t\t commit_locked_index(lock_file))\n+\t\tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\t\tdie(\"Unable to write new index file\");\n \t}\n \ndiff --git a/cache-tree.c b/cache-tree.c\nindex 7fa524a..52f8692 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -595,8 +595,7 @@ int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix)\n \t\t\t\t active_nr, flags) < 0)\n \t\t\treturn WRITE_TREE_UNMERGED_INDEX;\n \t\tif (0 <= newfd) {\n-\t\t\tif (!write_cache(newfd, active_cache, active_nr) &&\n-\t\t\t !commit_lock_file(lock_file))\n+\t\t\tif (!write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\t\t\tnewfd = -1;\n \t\t}\n \t\t/* Not being able to write is fine -- we are only interested\ndiff --git a/cache.h b/cache.h\nindex 107ac61..9cc2b97 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -301,7 +301,6 @@ extern void free_name_hash(struct index_state *istate);\n #define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec))\n #define is_cache_unborn() is_index_unborn(&the_index)\n #define read_cache_unmerged() read_index_unmerged(&the_index)\n-#define write_cache(newfd, cache, entries) write_index(&the_index, (newfd))\n #define discard_cache() discard_index(&the_index)\n #define unmerged_cache() unmerged_index(&the_index)\n #define cache_name_pos(name, namelen) index_name_pos(&the_index,(name),(namelen))\n@@ -456,12 +455,15 @@ extern int daemonize(void);\n \t} while (0)\n \n /* Initialize and use the cache information */\n+struct lock_file;\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n-extern int write_index(struct index_state *, int newfd);\n+#define COMMIT_LOCK\t\t(1 << 0)\n+#define CLOSE_LOCK\t\t(1 << 1)\n+extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 4177092..442c1ec 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -1986,7 +1986,7 @@ int merge_recursive_generic(struct merge_options *o,\n \t\t\t const unsigned char **base_list,\n \t\t\t struct commit **result)\n {\n-\tint clean, index_fd;\n+\tint clean;\n \tstruct lock_file *lock = xcalloc(1, sizeof(struct lock_file));\n \tstruct commit *head_commit = get_ref(head, o->branch1);\n \tstruct commit *next_commit = get_ref(merge, o->branch2);\n@@ -2003,12 +2003,11 @@ int merge_recursive_generic(struct merge_options *o,\n \t\t}\n \t}\n \n-\tindex_fd = hold_locked_index(lock, 1);\n+\thold_locked_index(lock, 1);\n \tclean = merge_recursive(o, head_commit, next_commit, ca,\n \t\t\tresult);\n \tif (active_cache_changed &&\n-\t\t\t(write_cache(index_fd, active_cache, active_nr) ||\n-\t\t\t commit_locked_index(lock)))\n+\t write_locked_index(&the_index, lock, COMMIT_LOCK))\n \t\treturn error(_(\"Unable to write index.\"));\n \n \treturn clean ? 0 : 1;\ndiff --git a/merge.c b/merge.c\nindex 70f1000..610725c 100644\n--- a/merge.c\n+++ b/merge.c\n@@ -66,13 +66,13 @@ int checkout_fast_forward(const unsigned char *head,\n \tstruct tree *trees[MAX_UNPACK_TREES];\n \tstruct unpack_trees_options opts;\n \tstruct tree_desc t[MAX_UNPACK_TREES];\n-\tint i, fd, nr_trees = 0;\n+\tint i, nr_trees = 0;\n \tstruct dir_struct dir;\n \tstruct lock_file *lock_file = xcalloc(1, sizeof(struct lock_file));\n \n \trefresh_cache(REFRESH_QUIET);\n \n-\tfd = hold_locked_index(lock_file, 1);\n+\thold_locked_index(lock_file, 1);\n \n \tmemset(&trees, 0, sizeof(trees));\n \tmemset(&opts, 0, sizeof(opts));\n@@ -105,8 +105,7 @@ int checkout_fast_forward(const unsigned char *head,\n \t}\n \tif (unpack_trees(nr_trees, t, &opts))\n \t\treturn -1;\n-\tif (write_cache(fd, active_cache, active_nr) ||\n-\t\tcommit_locked_index(lock_file))\n+\tif (write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n \t\tdie(_(\"unable to write new index file\"));\n \treturn 0;\n }\ndiff --git a/read-cache.c b/read-cache.c\nindex ba13353..44d4732 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1779,13 +1779,11 @@ static int has_racy_timestamp(struct index_state *istate)\n void update_index_if_able(struct index_state *istate, struct lock_file *lockfile)\n {\n \tif ((istate->cache_changed || has_racy_timestamp(istate)) &&\n-\t !write_index(istate, lockfile->fd))\n-\t\tcommit_locked_index(lockfile);\n-\telse\n+\t write_locked_index(istate, lockfile, COMMIT_LOCK))\n \t\trollback_lock_file(lockfile);\n }\n \n-int write_index(struct index_state *istate, int newfd)\n+static int do_write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n@@ -1877,6 +1875,28 @@ int write_index(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+static int do_write_locked_index(struct index_state *istate, struct lock_file *lock,\n+\t\t\t\t unsigned flags)\n+{\n+\tint ret = do_write_index(istate, lock->fd);\n+\tif (ret)\n+\t\treturn ret;\n+\tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n+\t (COMMIT_LOCK | CLOSE_LOCK));\n+\tif (flags & COMMIT_LOCK)\n+\t\treturn commit_locked_index(lock);\n+\telse if (flags & CLOSE_LOCK)\n+\t\treturn close_lock_file(lock);\n+\telse\n+\t\treturn ret;\n+}\n+\n+int write_locked_index(struct index_state *istate, struct lock_file *lock,\n+\t\t unsigned flags)\n+{\n+\treturn do_write_locked_index(istate, lock, flags);\n+}\n+\n /*\n * Read the index file that is potentially unmerged into given\n * index_state, dropping any unmerged entries. Returns true if\ndiff --git a/rerere.c b/rerere.c\nindex d55aa8a..ffc6a5b 100644\n--- a/rerere.c\n+++ b/rerere.c\n@@ -492,8 +492,7 @@ static int update_paths(struct string_list *update)\n \t}\n \n \tif (!status && active_cache_changed) {\n-\t\tif (write_cache(fd, active_cache, active_nr) ||\n-\t\t commit_locked_index(&index_lock))\n+\t\tif (write_locked_index(&the_index, &index_lock, COMMIT_LOCK))\n \t\t\tdie(\"Unable to write new index file\");\n \t} else if (fd >= 0)\n \t\trollback_lock_file(&index_lock);\ndiff --git a/sequencer.c b/sequencer.c\nindex 7b886a6..4fb0774 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -294,11 +294,11 @@ static int do_recursive_merge(struct commit *base, struct commit *next,\n {\n \tstruct merge_options o;\n \tstruct tree *result, *next_tree, *base_tree, *head_tree;\n-\tint clean, index_fd;\n+\tint clean;\n \tconst char **xopt;\n \tstatic struct lock_file index_lock;\n \n-\tindex_fd = hold_locked_index(&index_lock, 1);\n+\thold_locked_index(&index_lock, 1);\n \n \tread_cache();\n \n@@ -319,8 +319,7 @@ static int do_recursive_merge(struct commit *base, struct commit *next,\n \t\t\t next_tree, base_tree, &result);\n \n \tif (active_cache_changed &&\n-\t (write_cache(index_fd, active_cache, active_nr) ||\n-\t commit_locked_index(&index_lock)))\n+\t write_locked_index(&the_index, &index_lock, COMMIT_LOCK))\n \t\t/* TRANSLATORS: %s will be \"revert\" or \"cherry-pick\" */\n \t\tdie(_(\"%s: Unable to write new index file\"), action_name(opts));\n \trollback_lock_file(&index_lock);\n@@ -675,13 +674,12 @@ static void prepare_revs(struct replay_opts *opts)\n static void read_and_refresh_cache(struct replay_opts *opts)\n {\n \tstatic struct lock_file index_lock;\n-\tint index_fd = hold_locked_index(&index_lock, 0);\n+\thold_locked_index(&index_lock, 0);\n \tif (read_index_preload(&the_index, NULL) < 0)\n \t\tdie(_(\"git %s: failed to read the index\"), action_name(opts));\n \trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, NULL, NULL, NULL);\n \tif (the_index.cache_changed && index_fd >= 0) {\n-\t\tif (write_index(&the_index, index_fd) ||\n-\t\t commit_locked_index(&index_lock))\n+\t\tif (write_locked_index(&the_index, &index_lock, COMMIT_LOCK))\n \t\t\tdie(_(\"git %s: failed to refresh the index\"), action_name(opts));\n \t}\n \trollback_lock_file(&index_lock);\ndiff --git a/test-scrap-cache-tree.c b/test-scrap-cache-tree.c\nindex 4728013..9ebcbca 100644\n--- a/test-scrap-cache-tree.c\n+++ b/test-scrap-cache-tree.c\n@@ -6,12 +6,11 @@ static struct lock_file index_lock;\n \n int main(int ac, char **av)\n {\n-\tint fd = hold_locked_index(&index_lock, 1);\n+\thold_locked_index(&index_lock, 1);\n \tif (read_cache() < 0)\n \t\tdie(\"unable to read index file\");\n \tactive_cache_tree = NULL;\n-\tif (write_cache(fd, active_cache, active_nr)\n-\t || commit_lock_file(&index_lock))\n+\tif (write_locked_index(&the_index, &index_lock, COMMIT_LOCK))\n \t\tdie(\"unable to write index file\");\n \treturn 0;\n }\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239868","messageId":"1398682553-11634-6-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 05/32] read-cache: relocate and unexport commit_locked_index()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:26Z","receivedAt":"2014-04-28T10:55:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This function is now only used by write_locked_index(). Move it to\nread-cache.c (because read-cache.c will need to be aware of\nalternate_index_output later) and unexport it.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 -\n lockfile.c | 20 --------------------\n read-cache.c | 20 ++++++++++++++++++++\n 3 files changed, 20 insertions(+), 21 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9cc2b97..e44048c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -552,7 +552,6 @@ extern int commit_lock_file(struct lock_file *);\n extern void update_index_if_able(struct index_state *, struct lock_file *);\n \n extern int hold_locked_index(struct lock_file *, int);\n-extern int commit_locked_index(struct lock_file *);\n extern void set_alternate_index_output(const char *);\n extern int close_lock_file(struct lock_file *);\n extern void rollback_lock_file(struct lock_file *);\ndiff --git a/lockfile.c b/lockfile.c\nindex 8fbcb6a..b706614 100644\n--- a/lockfile.c\n+++ b/lockfile.c\n@@ -5,7 +5,6 @@\n #include \"sigchain.h\"\n \n static struct lock_file *lock_file_list;\n-static const char *alternate_index_output;\n \n static void remove_lock_file(void)\n {\n@@ -252,25 +251,6 @@ int hold_locked_index(struct lock_file *lk, int die_on_error)\n \t\t\t\t\t : 0);\n }\n \n-void set_alternate_index_output(const char *name)\n-{\n-\talternate_index_output = name;\n-}\n-\n-int commit_locked_index(struct lock_file *lk)\n-{\n-\tif (alternate_index_output) {\n-\t\tif (lk->fd >= 0 && close_lock_file(lk))\n-\t\t\treturn -1;\n-\t\tif (rename(lk->filename, alternate_index_output))\n-\t\t\treturn -1;\n-\t\tlk->filename[0] = 0;\n-\t\treturn 0;\n-\t}\n-\telse\n-\t\treturn commit_lock_file(lk);\n-}\n-\n void rollback_lock_file(struct lock_file *lk)\n {\n \tif (lk->filename[0]) {\ndiff --git a/read-cache.c b/read-cache.c\nindex 44d4732..576e506 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -36,6 +36,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n \n struct index_state the_index;\n+static const char *alternate_index_output;\n \n static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n@@ -1875,6 +1876,25 @@ static int do_write_index(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+void set_alternate_index_output(const char *name)\n+{\n+\talternate_index_output = name;\n+}\n+\n+static int commit_locked_index(struct lock_file *lk)\n+{\n+\tif (alternate_index_output) {\n+\t\tif (lk->fd >= 0 && close_lock_file(lk))\n+\t\t\treturn -1;\n+\t\tif (rename(lk->filename, alternate_index_output))\n+\t\t\treturn -1;\n+\t\tlk->filename[0] = 0;\n+\t\treturn 0;\n+\t}\n+\telse\n+\t\treturn commit_lock_file(lk);\n+}\n+\n static int do_write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t\t\t unsigned flags)\n {\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239869","messageId":"1398682553-11634-7-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 06/32] read-cache: store in-memory flags in the first 12 bits of ce_flags","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:27Z","receivedAt":"2014-04-28T10:55:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We're running out of room for in-memory flags. But since b60e188\n(Strip namelen out of ce_flags into a ce_namelen field - 2012-07-11),\nwe copy the namelen (first 12 bits) to ce_namelen field. So those bits\nare free to use. Just make sure we do not accidentally write any\nin-memory flags back.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 2 +-\n read-cache.c | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e44048c..57ad318 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -145,7 +145,7 @@ struct cache_entry {\n #define CE_STAGESHIFT 12\n \n /*\n- * Range 0xFFFF0000 in ce_flags is divided into\n+ * Range 0xFFFF0FFF in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n * Flags in CE_EXTENDED_FLAGS will get saved on-disk\n * if you want to save a new flag, add it in\ndiff --git a/read-cache.c b/read-cache.c\nindex 576e506..5761b1f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1702,7 +1702,7 @@ static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n \tondisk->size = htonl(ce->ce_stat_data.sd_size);\n \thashcpy(ondisk->sha1, ce->sha1);\n \n-\tflags = ce->ce_flags;\n+\tflags = ce->ce_flags & ~CE_NAMEMASK;\n \tflags |= (ce_namelen(ce) >= CE_NAMEMASK ? CE_NAMEMASK : ce_namelen(ce));\n \tondisk->flags = htons(flags);\n \tif (ce->ce_flags & CE_EXTENDED) {\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239870","messageId":"1398682553-11634-8-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 07/32] read-cache: be strict about \"changed\" in remove_marked_cache_entries()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:28Z","receivedAt":"2014-04-28T10:55:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"remove_marked_cache_entries() deletes entries marked with\nCE_REMOVE. But if there is no such entry, do not mark the index as\n\"changed\" because that could trigger an index update unnecessarily.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 5761b1f..9819363 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -510,6 +510,8 @@ void remove_marked_cache_entries(struct index_state *istate)\n \t\telse\n \t\t\tce_array[j++] = ce_array[i];\n \t}\n+\tif (j == istate->cache_nr)\n+\t\treturn;\n \tistate->cache_changed = 1;\n \tistate->cache_nr = j;\n }\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239871","messageId":"1398682553-11634-9-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 08/32] read-cache: be specific what part of the index has changed","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:29Z","receivedAt":"2014-04-28T10:55:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 4 ++++\n read-cache.c | 11 ++++++-----\n 2 files changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 57ad318..d692b74 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -268,6 +268,10 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n \n+#define CE_ENTRY_CHANGED\t(1 << 0)\n+#define CE_ENTRY_REMOVED\t(1 << 1)\n+#define CE_ENTRY_ADDED\t\t(1 << 2)\n+\n struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int version;\ndiff --git a/read-cache.c b/read-cache.c\nindex 9819363..6971fc4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -51,7 +51,7 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \tremove_name_hash(istate, old);\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n-\tistate->cache_changed = 1;\n+\tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n void rename_index_entry_at(struct index_state *istate, int nr, const char *new_name)\n@@ -482,7 +482,7 @@ int remove_index_entry_at(struct index_state *istate, int pos)\n \trecord_resolve_undo(istate, ce);\n \tremove_name_hash(istate, ce);\n \tfree(ce);\n-\tistate->cache_changed = 1;\n+\tistate->cache_changed |= CE_ENTRY_REMOVED;\n \tistate->cache_nr--;\n \tif (pos >= istate->cache_nr)\n \t\treturn 0;\n@@ -512,7 +512,7 @@ void remove_marked_cache_entries(struct index_state *istate)\n \t}\n \tif (j == istate->cache_nr)\n \t\treturn;\n-\tistate->cache_changed = 1;\n+\tistate->cache_changed |= CE_ENTRY_REMOVED;\n \tistate->cache_nr = j;\n }\n \n@@ -1002,7 +1002,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\t\tistate->cache + pos,\n \t\t\t(istate->cache_nr - pos - 1) * sizeof(ce));\n \tset_index_entry(istate, pos, ce);\n-\tistate->cache_changed = 1;\n+\tistate->cache_changed |= CE_ENTRY_ADDED;\n \treturn 0;\n }\n \n@@ -1101,6 +1101,7 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate,\n \t !(ce->ce_flags & CE_VALID))\n \t\tupdated->ce_flags &= ~CE_VALID;\n \n+\t/* istate->cache_changed is updated in the caller */\n \treturn updated;\n }\n \n@@ -1182,7 +1183,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t * means the index is not valid anymore.\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n-\t\t\t\tistate->cache_changed = 1;\n+\t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\n \t\t\t\tcontinue;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239872","messageId":"1398682553-11634-10-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 09/32] update-index: be specific what part of the index has changed","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:30Z","receivedAt":"2014-04-28T10:55:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/update-index.c | 6 +++---\n cache.h | 1 +\n 2 files changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 42cbe4b..e0e881b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -56,7 +56,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n \t\tcache_tree_invalidate_path(active_cache_tree, path);\n-\t\tactive_cache_changed = 1;\n+\t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n \t}\n \treturn -1;\n@@ -268,7 +268,7 @@ static void chmod_path(int flip, const char *path)\n \t\tgoto fail;\n \t}\n \tcache_tree_invalidate_path(active_cache_tree, path);\n-\tactive_cache_changed = 1;\n+\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \treport(\"chmod %cx '%s'\", flip, path);\n \treturn;\n fail:\n@@ -889,7 +889,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t INDEX_FORMAT_LB, INDEX_FORMAT_UB);\n \n \t\tif (the_index.version != preferred_index_format)\n-\t\t\tactive_cache_changed = 1;\n+\t\t\tactive_cache_changed |= SOMETHING_CHANGED;\n \t\tthe_index.version = preferred_index_format;\n \t}\n \ndiff --git a/cache.h b/cache.h\nindex d692b74..4133797 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -271,6 +271,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CE_ENTRY_CHANGED\t(1 << 0)\n #define CE_ENTRY_REMOVED\t(1 << 1)\n #define CE_ENTRY_ADDED\t\t(1 << 2)\n+#define SOMETHING_CHANGED\t(1 << 3) /* unclassified changes go here */\n \n struct index_state {\n \tstruct cache_entry **cache;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239873","messageId":"1398682553-11634-11-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 10/32] resolve-undo: be specific what part of the index has changed","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:31Z","receivedAt":"2014-04-28T10:55:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n resolve-undo.c | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4133797..7155052 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -272,6 +272,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CE_ENTRY_REMOVED\t(1 << 1)\n #define CE_ENTRY_ADDED\t\t(1 << 2)\n #define SOMETHING_CHANGED\t(1 << 3) /* unclassified changes go here */\n+#define RESOLVE_UNDO_CHANGED\t(1 << 4)\n \n struct index_state {\n \tstruct cache_entry **cache;\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 44c697c..468a2eb 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -110,7 +110,7 @@ void resolve_undo_clear_index(struct index_state *istate)\n \tstring_list_clear(resolve_undo, 1);\n \tfree(resolve_undo);\n \tistate->resolve_undo = NULL;\n-\tistate->cache_changed = 1;\n+\tistate->cache_changed |= RESOLVE_UNDO_CHANGED;\n }\n \n int unmerge_index_entry_at(struct index_state *istate, int pos)\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239874","messageId":"1398682553-11634-12-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 11/32] unpack-trees: be specific what part of the index has changed","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:32Z","receivedAt":"2014-04-28T10:55:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n unpack-trees.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 97fc995..a722685 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -246,7 +246,9 @@ static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types,\n \t\t\t\tstruct unpack_trees_options *o);\n \n-static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_options *o)\n+static int apply_sparse_checkout(struct index_state *istate,\n+\t\t\t\t struct cache_entry *ce,\n+\t\t\t\t struct unpack_trees_options *o)\n {\n \tint was_skip_worktree = ce_skip_worktree(ce);\n \n@@ -254,6 +256,8 @@ static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_opt\n \t\tce->ce_flags |= CE_SKIP_WORKTREE;\n \telse\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n+\tif (was_skip_worktree != ce_skip_worktree(ce))\n+\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \n \t/*\n \t * if (!was_skip_worktree && !ce_skip_worktree()) {\n@@ -1131,7 +1135,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t\tret = -1;\n \t\t\t}\n \n-\t\t\tif (apply_sparse_checkout(ce, o)) {\n+\t\t\tif (apply_sparse_checkout(&o->result, ce, o)) {\n \t\t\t\tif (!o->show_all_errors)\n \t\t\t\t\tgoto return_failed;\n \t\t\t\tret = -1;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239875","messageId":"1398682553-11634-13-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 12/32] cache-tree: mark istate->cache_changed on cache tree invalidation","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:33Z","receivedAt":"2014-04-28T10:55:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/blame.c | 2 +-\n builtin/update-index.c | 4 ++--\n cache-tree.c | 15 +++++++++++----\n cache-tree.h | 2 +-\n cache.h | 1 +\n read-cache.c | 6 +++---\n unpack-trees.c | 2 +-\n 7 files changed, 20 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/blame.c b/builtin/blame.c\nindex 88cb799..914d919 100644\n--- a/builtin/blame.c\n+++ b/builtin/blame.c\n@@ -2126,7 +2126,7 @@ static struct commit *fake_working_tree_commit(struct diff_options *opt,\n \t * right now, but someday we might optimize diff-index --cached\n \t * with cache-tree information.\n \t */\n-\tcache_tree_invalidate_path(active_cache_tree, path);\n+\tcache_tree_invalidate_path(&the_index, path);\n \n \treturn commit;\n }\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e0e881b..fa3c441 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -55,7 +55,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n-\t\tcache_tree_invalidate_path(active_cache_tree, path);\n+\t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n \t}\n@@ -267,7 +267,7 @@ static void chmod_path(int flip, const char *path)\n \tdefault:\n \t\tgoto fail;\n \t}\n-\tcache_tree_invalidate_path(active_cache_tree, path);\n+\tcache_tree_invalidate_path(&the_index, path);\n \tactive_cache_changed |= CE_ENTRY_CHANGED;\n \treport(\"chmod %cx '%s'\", flip, path);\n \treturn;\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 52f8692..23ddc73 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -98,7 +98,7 @@ struct cache_tree_sub *cache_tree_sub(struct cache_tree *it, const char *path)\n \treturn find_subtree(it, path, pathlen, 1);\n }\n \n-void cache_tree_invalidate_path(struct cache_tree *it, const char *path)\n+static int do_invalidate_path(struct cache_tree *it, const char *path)\n {\n \t/* a/b/c\n \t * ==> invalidate self\n@@ -116,7 +116,7 @@ void cache_tree_invalidate_path(struct cache_tree *it, const char *path)\n #endif\n \n \tif (!it)\n-\t\treturn;\n+\t\treturn 0;\n \tslash = strchrnul(path, '/');\n \tnamelen = slash - path;\n \tit->entry_count = -1;\n@@ -137,11 +137,18 @@ void cache_tree_invalidate_path(struct cache_tree *it, const char *path)\n \t\t\t\t(it->subtree_nr - pos - 1));\n \t\t\tit->subtree_nr--;\n \t\t}\n-\t\treturn;\n+\t\treturn 1;\n \t}\n \tdown = find_subtree(it, path, namelen, 0);\n \tif (down)\n-\t\tcache_tree_invalidate_path(down->cache_tree, slash + 1);\n+\t\tdo_invalidate_path(down->cache_tree, slash + 1);\n+\treturn 1;\n+}\n+\n+void cache_tree_invalidate_path(struct index_state *istate, const char *path)\n+{\n+\tif (do_invalidate_path(istate->cache_tree, path))\n+\t\tistate->cache_changed |= CACHE_TREE_CHANGED;\n }\n \n static int verify_cache(const struct cache_entry * const *cache,\ndiff --git a/cache-tree.h b/cache-tree.h\nindex f1923ad..dfbcfab 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -23,7 +23,7 @@ struct cache_tree {\n \n struct cache_tree *cache_tree(void);\n void cache_tree_free(struct cache_tree **);\n-void cache_tree_invalidate_path(struct cache_tree *, const char *);\n+void cache_tree_invalidate_path(struct index_state *, const char *);\n struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n \n void cache_tree_write(struct strbuf *, struct cache_tree *root);\ndiff --git a/cache.h b/cache.h\nindex 7155052..4c288e8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -273,6 +273,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define CE_ENTRY_ADDED\t\t(1 << 2)\n #define SOMETHING_CHANGED\t(1 << 3) /* unclassified changes go here */\n #define RESOLVE_UNDO_CHANGED\t(1 << 4)\n+#define CACHE_TREE_CHANGED\t(1 << 5)\n \n struct index_state {\n \tstruct cache_entry **cache;\ndiff --git a/read-cache.c b/read-cache.c\nindex 6971fc4..f1265d4 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -65,7 +65,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tnew->ce_namelen = namelen;\n \tmemcpy(new->name, new_name, namelen + 1);\n \n-\tcache_tree_invalidate_path(istate->cache_tree, old->name);\n+\tcache_tree_invalidate_path(istate, old->name);\n \tremove_index_entry_at(istate, nr);\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n@@ -521,7 +521,7 @@ int remove_file_from_index(struct index_state *istate, const char *path)\n \tint pos = index_name_pos(istate, path, strlen(path));\n \tif (pos < 0)\n \t\tpos = -pos-1;\n-\tcache_tree_invalidate_path(istate->cache_tree, path);\n+\tcache_tree_invalidate_path(istate, path);\n \twhile (pos < istate->cache_nr && !strcmp(istate->cache[pos]->name, path))\n \t\tremove_index_entry_at(istate, pos);\n \treturn 0;\n@@ -939,7 +939,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \tint skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n \tint new_only = option & ADD_CACHE_NEW_ONLY;\n \n-\tcache_tree_invalidate_path(istate->cache_tree, ce->name);\n+\tcache_tree_invalidate_path(istate, ce->name);\n \tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n \n \t/* existing match? Just replace it. */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a722685..3beff8a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1263,7 +1263,7 @@ static void invalidate_ce_path(const struct cache_entry *ce,\n \t\t\t struct unpack_trees_options *o)\n {\n \tif (ce)\n-\t\tcache_tree_invalidate_path(o->src_index->cache_tree, ce->name);\n+\t\tcache_tree_invalidate_path(o->src_index, ce->name);\n }\n \n /*\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239876","messageId":"1398682553-11634-14-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 13/32] cache-tree: mark istate->cache_changed on cache tree update","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:34Z","receivedAt":"2014-04-28T10:55:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache-tree.c | 25 +++++++++++--------------\n cache-tree.h | 2 +-\n merge-recursive.c | 4 +---\n sequencer.c | 4 +---\n test-dump-cache-tree.c | 7 ++++---\n 5 files changed, 18 insertions(+), 24 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 23ddc73..18055f1 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -151,7 +151,7 @@ void cache_tree_invalidate_path(struct index_state *istate, const char *path)\n \t\tistate->cache_changed |= CACHE_TREE_CHANGED;\n }\n \n-static int verify_cache(const struct cache_entry * const *cache,\n+static int verify_cache(struct cache_entry **cache,\n \t\t\tint entries, int flags)\n {\n \tint i, funny;\n@@ -236,7 +236,7 @@ int cache_tree_fully_valid(struct cache_tree *it)\n }\n \n static int update_one(struct cache_tree *it,\n-\t\t const struct cache_entry * const *cache,\n+\t\t struct cache_entry **cache,\n \t\t int entries,\n \t\t const char *base,\n \t\t int baselen,\n@@ -398,18 +398,19 @@ static int update_one(struct cache_tree *it,\n \treturn i;\n }\n \n-int cache_tree_update(struct cache_tree *it,\n-\t\t const struct cache_entry * const *cache,\n-\t\t int entries,\n-\t\t int flags)\n+int cache_tree_update(struct index_state *istate, int flags)\n {\n-\tint i, skip;\n-\ti = verify_cache(cache, entries, flags);\n+\tstruct cache_tree *it = istate->cache_tree;\n+\tstruct cache_entry **cache = istate->cache;\n+\tint entries = istate->cache_nr;\n+\tint skip, i = verify_cache(cache, entries, flags);\n+\n \tif (i)\n \t\treturn i;\n \ti = update_one(it, cache, entries, \"\", 0, &skip, flags);\n \tif (i < 0)\n \t\treturn i;\n+\tistate->cache_changed |= CACHE_TREE_CHANGED;\n \treturn 0;\n }\n \n@@ -597,9 +598,7 @@ int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix)\n \n \twas_valid = cache_tree_fully_valid(active_cache_tree);\n \tif (!was_valid) {\n-\t\tif (cache_tree_update(active_cache_tree,\n-\t\t\t\t (const struct cache_entry * const *)active_cache,\n-\t\t\t\t active_nr, flags) < 0)\n+\t\tif (cache_tree_update(&the_index, flags) < 0)\n \t\t\treturn WRITE_TREE_UNMERGED_INDEX;\n \t\tif (0 <= newfd) {\n \t\t\tif (!write_locked_index(&the_index, lock_file, COMMIT_LOCK))\n@@ -698,7 +697,5 @@ int update_main_cache_tree(int flags)\n {\n \tif (!the_index.cache_tree)\n \t\tthe_index.cache_tree = cache_tree();\n-\treturn cache_tree_update(the_index.cache_tree,\n-\t\t\t\t (const struct cache_entry * const *)the_index.cache,\n-\t\t\t\t the_index.cache_nr, flags);\n+\treturn cache_tree_update(&the_index, flags);\n }\ndiff --git a/cache-tree.h b/cache-tree.h\nindex dfbcfab..154b357 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -30,7 +30,7 @@ void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n \n int cache_tree_fully_valid(struct cache_tree *);\n-int cache_tree_update(struct cache_tree *, const struct cache_entry * const *, int, int);\n+int cache_tree_update(struct index_state *, int);\n \n int update_main_cache_tree(int);\n \ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 442c1ec..0b5d34d 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -265,9 +265,7 @@ struct tree *write_tree_from_memory(struct merge_options *o)\n \t\tactive_cache_tree = cache_tree();\n \n \tif (!cache_tree_fully_valid(active_cache_tree) &&\n-\t cache_tree_update(active_cache_tree,\n-\t\t\t (const struct cache_entry * const *)active_cache,\n-\t\t\t active_nr, 0) < 0)\n+\t cache_tree_update(&the_index, 0) < 0)\n \t\tdie(_(\"error building trees\"));\n \n \tresult = lookup_tree(active_cache_tree->sha1);\ndiff --git a/sequencer.c b/sequencer.c\nindex 4fb0774..377c877 100644\n--- a/sequencer.c\n+++ b/sequencer.c\n@@ -371,9 +371,7 @@ static int is_index_unchanged(void)\n \t\tactive_cache_tree = cache_tree();\n \n \tif (!cache_tree_fully_valid(active_cache_tree))\n-\t\tif (cache_tree_update(active_cache_tree,\n-\t\t\t\t (const struct cache_entry * const *)active_cache,\n-\t\t\t\t active_nr, 0))\n+\t\tif (cache_tree_update(&the_index, 0))\n \t\t\treturn error(_(\"Unable to update cache tree\\n\"));\n \n \treturn !hashcmp(active_cache_tree->sha1, head_commit->tree->object.sha1);\ndiff --git a/test-dump-cache-tree.c b/test-dump-cache-tree.c\nindex 47eab97..330ba4f 100644\n--- a/test-dump-cache-tree.c\n+++ b/test-dump-cache-tree.c\n@@ -56,11 +56,12 @@ static int dump_cache_tree(struct cache_tree *it,\n \n int main(int ac, char **av)\n {\n+\tstruct index_state istate;\n \tstruct cache_tree *another = cache_tree();\n \tif (read_cache() < 0)\n \t\tdie(\"unable to read index file\");\n-\tcache_tree_update(another,\n-\t\t\t (const struct cache_entry * const *)active_cache,\n-\t\t\t active_nr, WRITE_TREE_DRY_RUN);\n+\tistate = the_index;\n+\tistate.cache_tree = another;\n+\tcache_tree_update(&istate, WRITE_TREE_DRY_RUN);\n \treturn dump_cache_tree(active_cache_tree, another, \"\");\n }\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239877","messageId":"1398682553-11634-15-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 14/32] cache-tree: mark istate->cache_changed on prime_cache_tree()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:35Z","receivedAt":"2014-04-28T10:55:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/read-tree.c | 2 +-\n builtin/reset.c | 2 +-\n cache-tree.c | 9 +++++----\n cache-tree.h | 2 +-\n 4 files changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex f26d90f..3204c62 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -231,7 +231,7 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n \t * what came from the tree.\n \t */\n \tif (nr_trees == 1 && !opts.prefix)\n-\t\tprime_cache_tree(&active_cache_tree, trees[0]);\n+\t\tprime_cache_tree(&the_index, trees[0]);\n \n \tif (write_locked_index(&the_index, &lock_file, COMMIT_LOCK))\n \t\tdie(\"unable to write new index file\");\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 0c56d28..234b2eb 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -84,7 +84,7 @@ static int reset_index(const unsigned char *sha1, int reset_type, int quiet)\n \n \tif (reset_type == MIXED || reset_type == HARD) {\n \t\ttree = parse_tree_indirect(sha1);\n-\t\tprime_cache_tree(&active_cache_tree, tree);\n+\t\tprime_cache_tree(&the_index, tree);\n \t}\n \n \treturn 0;\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 18055f1..c53f7de 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -654,11 +654,12 @@ static void prime_cache_tree_rec(struct cache_tree *it, struct tree *tree)\n \tit->entry_count = cnt;\n }\n \n-void prime_cache_tree(struct cache_tree **it, struct tree *tree)\n+void prime_cache_tree(struct index_state *istate, struct tree *tree)\n {\n-\tcache_tree_free(it);\n-\t*it = cache_tree();\n-\tprime_cache_tree_rec(*it, tree);\n+\tcache_tree_free(&istate->cache_tree);\n+\tistate->cache_tree = cache_tree();\n+\tprime_cache_tree_rec(istate->cache_tree, tree);\n+\tistate->cache_changed |= CACHE_TREE_CHANGED;\n }\n \n /*\ndiff --git a/cache-tree.h b/cache-tree.h\nindex 154b357..b47ccec 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -46,7 +46,7 @@ int update_main_cache_tree(int);\n #define WRITE_TREE_PREFIX_ERROR (-3)\n \n int write_cache_as_tree(unsigned char *sha1, int flags, const char *prefix);\n-void prime_cache_tree(struct cache_tree **, struct tree *);\n+void prime_cache_tree(struct index_state *, struct tree *);\n \n extern int cache_tree_matches_traversal(struct cache_tree *, struct name_entry *ent, struct traverse_info *info);\n \n-- \n1.9.1.346.ga2b5940\n"},{"id":"239893","messageId":"1398682553-11634-16-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 15/32] entry.c: update cache_changed if refresh_cache is set in checkout_entry()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:36Z","receivedAt":"2014-04-28T10:55:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Other fill_stat_cache_info() is on new entries, which should set\nCE_ENTRY_ADDED in cache_changed, so we're safe.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/apply.c | 8 +++++---\n builtin/checkout-index.c | 1 +\n builtin/checkout.c | 1 +\n cache.h | 1 +\n entry.c | 2 ++\n unpack-trees.c | 1 +\n 6 files changed, 11 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/apply.c b/builtin/apply.c\nindex 5e13444..adca035 100644\n--- a/builtin/apply.c\n+++ b/builtin/apply.c\n@@ -3084,13 +3084,15 @@ static void prepare_fn_table(struct patch *patch)\n \t}\n }\n \n-static int checkout_target(struct cache_entry *ce, struct stat *st)\n+static int checkout_target(struct index_state *istate,\n+\t\t\t struct cache_entry *ce, struct stat *st)\n {\n \tstruct checkout costate;\n \n \tmemset(&costate, 0, sizeof(costate));\n \tcostate.base_dir = \"\";\n \tcostate.refresh_cache = 1;\n+\tcostate.istate = istate;\n \tif (checkout_entry(ce, &costate, NULL) || lstat(ce->name, st))\n \t\treturn error(_(\"cannot checkout %s\"), ce->name);\n \treturn 0;\n@@ -3257,7 +3259,7 @@ static int load_current(struct image *image, struct patch *patch)\n \tif (lstat(name, &st)) {\n \t\tif (errno != ENOENT)\n \t\t\treturn error(_(\"%s: %s\"), name, strerror(errno));\n-\t\tif (checkout_target(ce, &st))\n+\t\tif (checkout_target(&the_index, ce, &st))\n \t\t\treturn -1;\n \t}\n \tif (verify_index_match(ce, &st))\n@@ -3411,7 +3413,7 @@ static int check_preimage(struct patch *patch, struct cache_entry **ce, struct s\n \t\t}\n \t\t*ce = active_cache[pos];\n \t\tif (stat_ret < 0) {\n-\t\t\tif (checkout_target(*ce, st))\n+\t\t\tif (checkout_target(&the_index, *ce, st))\n \t\t\t\treturn -1;\n \t\t}\n \t\tif (!cached && verify_index_match(*ce, st))\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 9e49bf2..05edd9e 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -135,6 +135,7 @@ static int option_parse_u(const struct option *opt,\n \tint *newfd = opt->value;\n \n \tstate.refresh_cache = 1;\n+\tstate.istate = &the_index;\n \tif (*newfd < 0)\n \t\t*newfd = hold_locked_index(&lock_file, 1);\n \treturn 0;\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 944a634..146ab91 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -336,6 +336,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \tmemset(&state, 0, sizeof(state));\n \tstate.force = 1;\n \tstate.refresh_cache = 1;\n+\tstate.istate = &the_index;\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\ndiff --git a/cache.h b/cache.h\nindex 4c288e8..9bbebab 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1063,6 +1063,7 @@ extern int split_ident_line(struct ident_split *, const char *, int);\n extern int ident_cmp(const struct ident_split *, const struct ident_split *);\n \n struct checkout {\n+\tstruct index_state *istate;\n \tconst char *base_dir;\n \tint base_dir_len;\n \tunsigned force:1,\ndiff --git a/entry.c b/entry.c\nindex 77c6882..d913c1d 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -210,9 +210,11 @@ static int write_entry(struct cache_entry *ce,\n \n finish:\n \tif (state->refresh_cache) {\n+\t\tassert(state->istate);\n \t\tif (!fstat_done)\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n+\t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\n }\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3beff8a..26f65c7 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1029,6 +1029,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tstate.force = 1;\n \tstate.quiet = 1;\n \tstate.refresh_cache = 1;\n+\tstate.istate = &o->result;\n \n \tmemset(&el, 0, sizeof(el));\n \tif (!core_apply_sparse_checkout || !o->update)\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239884","messageId":"1398682553-11634-17-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 16/32] read-cache: save index SHA-1 after reading","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:37Z","receivedAt":"2014-04-28T10:55:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Also update SHA-1 after writing. If we do not do that, the second\nread_index() will see \"initialized\" variable already set and not read\n.git/index again, which is fine, except istate->sha1 now has a stale\nvalue.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 6 ++++--\n unpack-trees.c | 1 +\n 3 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9bbebab..0f6247c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -286,6 +286,7 @@ struct index_state {\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n+\tunsigned char sha1[20];\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex f1265d4..723d769 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1481,6 +1481,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \tif (verify_hdr(hdr, mmap_size) < 0)\n \t\tgoto unmap;\n \n+\thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n \tistate->cache_nr = ntohl(hdr->hdr_entries);\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n@@ -1616,7 +1617,7 @@ static int write_index_ext_header(git_SHA_CTX *context, int fd,\n \t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n }\n \n-static int ce_flush(git_SHA_CTX *context, int fd)\n+static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n {\n \tunsigned int left = write_buffer_len;\n \n@@ -1634,6 +1635,7 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \n \t/* Append the SHA1 signature at the end */\n \tgit_SHA1_Final(write_buffer + left, context);\n+\thashcpy(sha1, write_buffer + left);\n \tleft += 20;\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n@@ -1872,7 +1874,7 @@ static int do_write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 26f65c7..f594932 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1046,6 +1046,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->result.timestamp.sec = o->src_index->timestamp.sec;\n \to->result.timestamp.nsec = o->src_index->timestamp.nsec;\n \to->result.version = o->src_index->version;\n+\thashcpy(o->result.sha1, o->src_index->sha1);\n \to->merge_size = len;\n \tmark_all_ce_unused(o->src_index);\n \n-- \n1.9.1.346.ga2b5940\n"},{"id":"239878","messageId":"1398682553-11634-18-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 17/32] read-cache: split-index mode","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:38Z","receivedAt":"2014-04-28T10:55:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This split-index mode is designed to keep write cost proportional to\nthe number of changes the user has made, not the size of the work\ntree. (Read cost is another matter, to be dealt separately.)\n\nThis mode stores index info in a pair of $GIT_DIR/index and\n$GIT_DIR/sharedindex.. sharedindex is large and unchanged over\ntime while \"index\" is smaller and updated often. Format details are in\nindex-format.txt, although not everything is implemented in this\npatch.\n\nShared indexes are not automatically removed, because it's unclear if\nthe shared index is needed by any (even temporary) indexes by just\nlooking at it. After a while you'll collect stale shared indexes. The\ngood news is one shared index is useable for long, until\n$GIT_DIR/index becomes too big and sluggish that the new shared index\nmust be created.\n\nThe safest way to clean shared indexes is to turn off split index\nmode, so shared files are all garbage, delete them all, then turn on\nsplit index mode again.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/gitrepository-layout.txt | 4 ++\n Documentation/technical/index-format.txt | 35 ++++++++++++\n Makefile | 1 +\n cache.h | 3 +\n read-cache.c | 96 ++++++++++++++++++++++++++++++--\n split-index.c (new) | 90 ++++++++++++++++++++++++++++++\n split-index.h (new) | 25 +++++++++\n unpack-trees.c | 4 ++\n 8 files changed, 253 insertions(+), 5 deletions(-)\n create mode 100644 split-index.c\n create mode 100644 split-index.h\n\ndiff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\nindex 17d2ea6..79653f3 100644\n--- a/Documentation/gitrepository-layout.txt\n+++ b/Documentation/gitrepository-layout.txt\n@@ -155,6 +155,10 @@ index::\n \tThe current index file for the repository. It is\n \tusually not found in a bare repository.\n \n+sharedindex.::\n+\tThe shared index part, to be referenced by $GIT_DIR/index and\n+\tother temporary index files. Only valid in split index mode.\n+\n info::\n \tAdditional information about the repository is recorded\n \tin this directory.\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex f352a9b..fe6f316 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -129,6 +129,9 @@ Git index format\n (Version 4) In version 4, the padding after the pathname does not\n exist.\n \n+ Interpretation of index entries in split index mode is completely\n+ different. See below for details.\n+\n == Extensions\n \n === Cached tree\n@@ -198,3 +201,35 @@ Git index format\n - At most three 160-bit object names of the entry in stages from 1 to 3\n (nothing is written for a missing stage).\n \n+=== Split index\n+\n+ In split index mode, the majority of index entries could be stored\n+ in a separate file. This extension records the changes to be made on\n+ top of that to produce the final index.\n+\n+ The signature for this extension is { 'l', 'i, 'n', 'k' }.\n+\n+ The extension consists of:\n+\n+ - 160-bit SHA-1 of the shared index file. The shared index file path\n+ is $GIT_DIR/sharedindex.. If all 160 bits are zero, the\n+ index does not require a shared index file.\n+\n+ - An ewah-encoded delete bitmap, each bit represents an entry in the\n+ shared index. If a bit is set, its corresponding entry in the\n+ shared index will be removed from the final index. Note, because\n+ a delete operation changes index entry positions, but we do need\n+ original positions in replace phase, it's best to just mark\n+ entries for removal, then do a mass deletion after replacement.\n+\n+ - An ewah-encoded replace bitmap, each bit represents an entry in\n+ the shared index. If a bit is set, its corresponding entry in the\n+ shared index will be replaced with an entry in this index\n+ file. All replaced entries are stored in sorted order in this\n+ index. The first \"1\" bit in the replace bitmap corresponds to the\n+ first index entry, the second \"1\" bit to the second entry and so\n+ on. Replaced entries may have empty path names to save space.\n+\n+ The remaining index entries after replaced ones will be added to the\n+ final index. These added entries are also sorted by entry namme then\n+ stage.\ndiff --git a/Makefile b/Makefile\nindex 74a929b..c3957bb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -884,6 +884,7 @@ LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += split-index.o\n LIB_OBJS += strbuf.o\n LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\ndiff --git a/cache.h b/cache.h\nindex 0f6247c..90a5998 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -135,6 +135,7 @@ struct cache_entry {\n \tunsigned int ce_mode;\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n+\tunsigned int index;\t/* for link extension */\n \tunsigned char sha1[20];\n \tchar name[FLEX_ARRAY]; /* more */\n };\n@@ -275,12 +276,14 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define RESOLVE_UNDO_CHANGED\t(1 << 4)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n \n+struct split_index;\n struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int version;\n \tunsigned int cache_nr, cache_alloc, cache_changed;\n \tstruct string_list *resolve_undo;\n \tstruct cache_tree *cache_tree;\n+\tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n \t\t initialized : 1;\ndiff --git a/read-cache.c b/read-cache.c\nindex 723d769..ff889ad 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -14,6 +14,7 @@\n #include \"resolve-undo.h\"\n #include \"strbuf.h\"\n #include \"varint.h\"\n+#include \"split-index.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -34,6 +35,10 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+#define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n+\n+/* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n+#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -63,6 +68,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tcopy_cache_entry(new, old);\n \tnew->ce_flags &= ~CE_HASHED;\n \tnew->ce_namelen = namelen;\n+\tnew->index = 0;\n \tmemcpy(new->name, new_name, namelen + 1);\n \n \tcache_tree_invalidate_path(istate, old->name);\n@@ -1335,6 +1341,10 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_LINK:\n+\t\tif (read_link_extension(istate, data, sz))\n+\t\t\treturn -1;\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1369,6 +1379,7 @@ static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *on\n \tce->ce_stat_data.sd_size = get_be32(&ondisk->size);\n \tce->ce_flags = flags & ~CE_NAMEMASK;\n \tce->ce_namelen = len;\n+\tce->index = 0;\n \thashcpy(ce->sha1, ondisk->sha1);\n \tmemcpy(ce->name, name, len);\n \tce->name[len] = '\\0';\n@@ -1443,7 +1454,8 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n }\n \n /* remember to discard_cache() before reading a different cache! */\n-int read_index_from(struct index_state *istate, const char *path)\n+static int do_read_index(struct index_state *istate, const char *path,\n+\t\t\t int must_exist)\n {\n \tint fd, i;\n \tstruct stat st;\n@@ -1460,9 +1472,9 @@ int read_index_from(struct index_state *istate, const char *path)\n \tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n \tif (fd < 0) {\n-\t\tif (errno == ENOENT)\n+\t\tif (!must_exist && errno == ENOENT)\n \t\t\treturn 0;\n-\t\tdie_errno(\"index file open failed\");\n+\t\tdie_errno(\"%s: index file open failed\", path);\n \t}\n \n \tif (fstat(fd, &st))\n@@ -1535,6 +1547,42 @@ unmap:\n \tdie(\"index file corrupt\");\n }\n \n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\tstruct split_index *split_index;\n+\tint ret;\n+\n+\t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n+\tif (istate->initialized)\n+\t\treturn istate->cache_nr;\n+\n+\tret = do_read_index(istate, path, 0);\n+\tsplit_index = istate->split_index;\n+\tif (!split_index)\n+\t\treturn ret;\n+\n+\tif (is_null_sha1(split_index->base_sha1))\n+\t\treturn ret;\n+\tif (istate->cache_nr)\n+\t\tdie(\"index in split-index mode must contain no entries\");\n+\n+\tif (split_index->base)\n+\t\tdiscard_index(split_index->base);\n+\telse\n+\t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tret = do_read_index(split_index->base,\n+\t\t\t git_path(\"sharedindex.%s\",\n+\t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n+\tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n+\t\tdie(\"broken index, expect %s in %s, got %s\",\n+\t\t sha1_to_hex(split_index->base_sha1),\n+\t\t git_path(\"sharedindex.%s\",\n+\t\t\t\t sha1_to_hex(split_index->base_sha1)),\n+\t\t sha1_to_hex(split_index->base->sha1));\n+\tmerge_base_index(istate);\n+\treturn ret;\n+}\n+\n int is_index_unborn(struct index_state *istate)\n {\n \treturn (!istate->cache_nr && !istate->timestamp.sec);\n@@ -1544,8 +1592,15 @@ int discard_index(struct index_state *istate)\n {\n \tint i;\n \n-\tfor (i = 0; i < istate->cache_nr; i++)\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (istate->cache[i]->index &&\n+\t\t istate->split_index &&\n+\t\t istate->split_index->base &&\n+\t\t istate->cache[i]->index <= istate->split_index->base->cache_nr &&\n+\t\t istate->cache[i] == istate->split_index->base->cache[istate->cache[i]->index - 1])\n+\t\t\tcontinue;\n \t\tfree(istate->cache[i]);\n+\t}\n \tresolve_undo_clear_index(istate);\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n@@ -1557,6 +1612,7 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tdiscard_split_index(istate);\n \treturn 0;\n }\n \n@@ -1852,6 +1908,17 @@ static int do_write_index(struct index_state *istate, int newfd)\n \tstrbuf_release(&previous_name_buf);\n \n \t/* Write extension data here */\n+\tif (istate->split_index) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\terr = write_link_extension(&sb, istate) < 0 ||\n+\t\t\twrite_index_ext_header(&c, newfd, CACHE_EXT_LINK,\n+\t\t\t\t\t sb.len) < 0 ||\n+\t\t\tce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \tif (istate->cache_tree) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n@@ -1916,10 +1983,29 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n }\n \n+static int write_split_index(struct index_state *istate,\n+\t\t\t struct lock_file *lock,\n+\t\t\t unsigned flags)\n+{\n+\tint ret;\n+\tprepare_to_write_split_index(istate);\n+\tret = do_write_locked_index(istate, lock, flags);\n+\tfinish_writing_split_index(istate);\n+\treturn ret;\n+}\n+\n int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t unsigned flags)\n {\n-\treturn do_write_locked_index(istate, lock, flags);\n+\tstruct split_index *si = istate->split_index;\n+\n+\tif (!si || (istate->cache_changed & ~EXTMASK)) {\n+\t\tif (si)\n+\t\t\thashclr(si->base_sha1);\n+\t\treturn do_write_locked_index(istate, lock, flags);\n+\t}\n+\n+\treturn write_split_index(istate, lock, flags);\n }\n \n /*\ndiff --git a/split-index.c b/split-index.c\nnew file mode 100644\nindex 0000000..63b52bb\n--- /dev/null\n+++ b/split-index.c\n@@ -0,0 +1,90 @@\n+#include \"cache.h\"\n+#include \"split-index.h\"\n+\n+struct split_index *init_split_index(struct index_state *istate)\n+{\n+\tif (!istate->split_index) {\n+\t\tistate->split_index = xcalloc(1, sizeof(*istate->split_index));\n+\t\tistate->split_index->refcount = 1;\n+\t}\n+\treturn istate->split_index;\n+}\n+\n+int read_link_extension(struct index_state *istate,\n+\t\t\t const void *data_, unsigned long sz)\n+{\n+\tconst unsigned char *data = data_;\n+\tstruct split_index *si;\n+\tif (sz < 20)\n+\t\treturn error(\"corrupt link extension (too short)\");\n+\tsi = init_split_index(istate);\n+\thashcpy(si->base_sha1, data);\n+\tdata += 20;\n+\tsz -= 20;\n+\tif (sz)\n+\t\treturn error(\"garbage at the end of link extension\");\n+\treturn 0;\n+}\n+\n+int write_link_extension(struct strbuf *sb,\n+\t\t\t struct index_state *istate)\n+{\n+\tstruct split_index *si = istate->split_index;\n+\tstrbuf_add(sb, si->base_sha1, 20);\n+\treturn 0;\n+}\n+\n+static void mark_base_index_entries(struct index_state *base)\n+{\n+\tint i;\n+\t/*\n+\t * To keep track of the shared entries between\n+\t * istate->base->cache[] and istate->cache[], base entry\n+\t * position is stored in each base entry. All positions start\n+\t * from 1 instead of 0, which is resrved to say \"this is a new\n+\t * entry\".\n+\t */\n+\tfor (i = 0; i < base->cache_nr; i++)\n+\t\tbase->cache[i]->index = i + 1;\n+}\n+\n+void merge_base_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = istate->split_index;\n+\n+\tmark_base_index_entries(si->base);\n+\tistate->cache_nr = si->base->cache_nr;\n+\tALLOC_GROW(istate->cache, istate->cache_nr, istate->cache_alloc);\n+\tmemcpy(istate->cache, si->base->cache,\n+\t sizeof(*istate->cache) * istate->cache_nr);\n+}\n+\n+void prepare_to_write_split_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = init_split_index(istate);\n+\t/* take cache[] out temporarily */\n+\tsi->saved_cache_nr = istate->cache_nr;\n+\tistate->cache_nr = 0;\n+}\n+\n+void finish_writing_split_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = init_split_index(istate);\n+\tistate->cache_nr = si->saved_cache_nr;\n+}\n+\n+void discard_split_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = istate->split_index;\n+\tif (!si)\n+\t\treturn;\n+\tistate->split_index = NULL;\n+\tsi->refcount--;\n+\tif (si->refcount)\n+\t\treturn;\n+\tif (si->base) {\n+\t\tdiscard_index(si->base);\n+\t\tfree(si->base);\n+\t}\n+\tfree(si);\n+}\ndiff --git a/split-index.h b/split-index.h\nnew file mode 100644\nindex 0000000..8d74041\n--- /dev/null\n+++ b/split-index.h\n@@ -0,0 +1,25 @@\n+#ifndef SPLIT_INDEX_H\n+#define SPLIT_INDEX_H\n+\n+struct index_state;\n+struct strbuf;\n+\n+struct split_index {\n+\tunsigned char base_sha1[20];\n+\tstruct index_state *base;\n+\tunsigned int saved_cache_nr;\n+\tint refcount;\n+};\n+\n+struct split_index *init_split_index(struct index_state *istate);\n+int read_link_extension(struct index_state *istate,\n+\t\t\tconst void *data, unsigned long sz);\n+int write_link_extension(struct strbuf *sb,\n+\t\t\t struct index_state *istate);\n+void move_cache_to_base_index(struct index_state *istate);\n+void merge_base_index(struct index_state *istate);\n+void prepare_to_write_split_index(struct index_state *istate);\n+void finish_writing_split_index(struct index_state *istate);\n+void discard_split_index(struct index_state *istate);\n+\n+#endif\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f594932..a941f7c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -8,6 +8,7 @@\n #include \"progress.h\"\n #include \"refs.h\"\n #include \"attr.h\"\n+#include \"split-index.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -1046,6 +1047,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->result.timestamp.sec = o->src_index->timestamp.sec;\n \to->result.timestamp.nsec = o->src_index->timestamp.nsec;\n \to->result.version = o->src_index->version;\n+\to->result.split_index = o->src_index->split_index;\n+\tif (o->result.split_index)\n+\t\to->result.split_index->refcount++;\n \thashcpy(o->result.sha1, o->src_index->sha1);\n \to->merge_size = len;\n \tmark_all_ce_unused(o->src_index);\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239895","messageId":"1398682553-11634-19-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 18/32] read-cache: mark new entries for split index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:39Z","receivedAt":"2014-04-28T10:55:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Make sure entry addition does not lead to unifying the index. We don't\nneed to explicitly keep track of new entries. If ce->index is zero,\nthey're new. Otherwise it's unlikely that they are new, but we'll do a\nthrough check later at writing time.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex ff889ad..2f2e0c1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -38,7 +38,8 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_LINK 0x6c696e6b\t /* \"link\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n-#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED)\n+#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n+\t\t CE_ENTRY_ADDED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239879","messageId":"1398682553-11634-20-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 19/32] read-cache: save deleted entries in split index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:40Z","receivedAt":"2014-04-28T10:55:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Entries that belong to the base index should not be freed. Mark\nCE_REMOVE to track them.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 14 ++++++++------\n split-index.c | 12 ++++++++++++\n split-index.h | 1 +\n 3 files changed, 21 insertions(+), 6 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 2f2e0c1..7cdb171 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -39,7 +39,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n-\t\t CE_ENTRY_ADDED)\n+\t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -488,7 +488,7 @@ int remove_index_entry_at(struct index_state *istate, int pos)\n \n \trecord_resolve_undo(istate, ce);\n \tremove_name_hash(istate, ce);\n-\tfree(ce);\n+\tsave_or_free_index_entry(istate, ce);\n \tistate->cache_changed |= CE_ENTRY_REMOVED;\n \tistate->cache_nr--;\n \tif (pos >= istate->cache_nr)\n@@ -512,7 +512,7 @@ void remove_marked_cache_entries(struct index_state *istate)\n \tfor (i = j = 0; i < istate->cache_nr; i++) {\n \t\tif (ce_array[i]->ce_flags & CE_REMOVE) {\n \t\t\tremove_name_hash(istate, ce_array[i]);\n-\t\t\tfree(ce_array[i]);\n+\t\t\tsave_or_free_index_entry(istate, ce_array[i]);\n \t\t}\n \t\telse\n \t\t\tce_array[j++] = ce_array[i];\n@@ -577,7 +577,9 @@ static int different_name(struct cache_entry *ce, struct cache_entry *alias)\n * So we use the CE_ADDED flag to verify that the alias was an old\n * one before we accept it as\n */\n-static struct cache_entry *create_alias_ce(struct cache_entry *ce, struct cache_entry *alias)\n+static struct cache_entry *create_alias_ce(struct index_state *istate,\n+\t\t\t\t\t struct cache_entry *ce,\n+\t\t\t\t\t struct cache_entry *alias)\n {\n \tint len;\n \tstruct cache_entry *new;\n@@ -590,7 +592,7 @@ static struct cache_entry *create_alias_ce(struct cache_entry *ce, struct cache_\n \tnew = xcalloc(1, cache_entry_size(len));\n \tmemcpy(new->name, alias->name, len);\n \tcopy_cache_entry(new, ce);\n-\tfree(ce);\n+\tsave_or_free_index_entry(istate, ce);\n \treturn new;\n }\n \n@@ -683,7 +685,7 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,\n \t\tset_object_name_for_intent_to_add_entry(ce);\n \n \tif (ignore_case && alias && different_name(ce, alias))\n-\t\tce = create_alias_ce(ce, alias);\n+\t\tce = create_alias_ce(istate, ce, alias);\n \tce->ce_flags |= CE_ADDED;\n \n \t/* It was suspected to be racily clean, but it turns out to be Ok */\ndiff --git a/split-index.c b/split-index.c\nindex 63b52bb..2bb5d55 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -88,3 +88,15 @@ void discard_split_index(struct index_state *istate)\n \t}\n \tfree(si);\n }\n+\n+void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce)\n+{\n+\tif (ce->index &&\n+\t istate->split_index &&\n+\t istate->split_index->base &&\n+\t ce->index <= istate->split_index->base->cache_nr &&\n+\t ce == istate->split_index->base->cache[ce->index - 1])\n+\t\tce->ce_flags |= CE_REMOVE;\n+\telse\n+\t\tfree(ce);\n+}\ndiff --git a/split-index.h b/split-index.h\nindex 8d74041..5302118 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -12,6 +12,7 @@ struct split_index {\n };\n \n struct split_index *init_split_index(struct index_state *istate);\n+void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce);\n int read_link_extension(struct index_state *istate,\n \t\t\tconst void *data, unsigned long sz);\n int write_link_extension(struct strbuf *sb,\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239880","messageId":"1398682553-11634-21-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 20/32] read-cache: mark updated entries for split index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:41Z","receivedAt":"2014-04-28T10:55:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The large part of this patch just follows CE_ENTRY_CHANGED\nmarks. replace_index_entry() is updated to update\nsplit_index->base->cache[] as well so base->cache[] does not reference\nto a freed entry.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/update-index.c | 2 ++\n cache.h | 2 ++\n entry.c | 1 +\n read-cache.c | 5 ++++-\n split-index.c | 15 +++++++++++++++\n split-index.h | 3 +++\n unpack-trees.c | 4 +++-\n 7 files changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex fa3c441..f7a19c4 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -55,6 +55,7 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n \t\t\tactive_cache[pos]->ce_flags &= ~flag;\n+\t\tactive_cache[pos]->ce_flags |= CE_UPDATE_IN_BASE;\n \t\tcache_tree_invalidate_path(&the_index, path);\n \t\tactive_cache_changed |= CE_ENTRY_CHANGED;\n \t\treturn 0;\n@@ -268,6 +269,7 @@ static void chmod_path(int flip, const char *path)\n \t\tgoto fail;\n \t}\n \tcache_tree_invalidate_path(&the_index, path);\n+\tce->ce_flags |= CE_UPDATE_IN_BASE;\n \tactive_cache_changed |= CE_ENTRY_CHANGED;\n \treport(\"chmod %cx '%s'\", flip, path);\n \treturn;\ndiff --git a/cache.h b/cache.h\nindex 90a5998..127804e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -169,6 +169,8 @@ struct cache_entry {\n /* used to temporarily mark paths matched by pathspecs */\n #define CE_MATCHED (1 << 26)\n \n+#define CE_UPDATE_IN_BASE (1 << 27)\n+\n /*\n * Extended on-disk flags\n */\ndiff --git a/entry.c b/entry.c\nindex d913c1d..1eda8e9 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -214,6 +214,7 @@ finish:\n \t\tif (!fstat_done)\n \t\t\tlstat(ce->name, &st);\n \t\tfill_stat_cache_info(ce, &st);\n+\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n \t\tstate->istate->cache_changed |= CE_ENTRY_CHANGED;\n \t}\n \treturn 0;\ndiff --git a/read-cache.c b/read-cache.c\nindex 7cdb171..a717171 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -39,7 +39,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n-\t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED)\n+\t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -54,9 +54,11 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n {\n \tstruct cache_entry *old = istate->cache[nr];\n \n+\treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n \tfree(old);\n \tset_index_entry(istate, nr, ce);\n+\tce->ce_flags |= CE_UPDATE_IN_BASE;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n }\n \n@@ -1192,6 +1194,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\t\t\t * means the index is not valid anymore.\n \t\t\t\t */\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n+\t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n \t\t\t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n \t\t\t}\n \t\t\tif (quiet)\ndiff --git a/split-index.c b/split-index.c\nindex 2bb5d55..b36c73b 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -100,3 +100,18 @@ void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce\n \telse\n \t\tfree(ce);\n }\n+\n+void replace_index_entry_in_base(struct index_state *istate,\n+\t\t\t\t struct cache_entry *old,\n+\t\t\t\t struct cache_entry *new)\n+{\n+\tif (old->index &&\n+\t istate->split_index &&\n+\t istate->split_index->base &&\n+\t old->index <= istate->split_index->base->cache_nr) {\n+\t\tnew->index = old->index;\n+\t\tif (old != istate->split_index->base->cache[new->index - 1])\n+\t\t\tfree(istate->split_index->base->cache[new->index - 1]);\n+\t\tistate->split_index->base->cache[new->index - 1] = new;\n+\t}\n+}\ndiff --git a/split-index.h b/split-index.h\nindex 5302118..812e510 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -13,6 +13,9 @@ struct split_index {\n \n struct split_index *init_split_index(struct index_state *istate);\n void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce);\n+void replace_index_entry_in_base(struct index_state *istate,\n+\t\t\t\t struct cache_entry *old,\n+\t\t\t\t struct cache_entry *new);\n int read_link_extension(struct index_state *istate,\n \t\t\tconst void *data, unsigned long sz);\n int write_link_extension(struct strbuf *sb,\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a941f7c..4a9cdf2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -257,8 +257,10 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags |= CE_SKIP_WORKTREE;\n \telse\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n-\tif (was_skip_worktree != ce_skip_worktree(ce))\n+\tif (was_skip_worktree != ce_skip_worktree(ce)) {\n+\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n \t\tistate->cache_changed |= CE_ENTRY_CHANGED;\n+\t}\n \n \t/*\n \t * if (!was_skip_worktree && !ce_skip_worktree()) {\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239881","messageId":"1398682553-11634-22-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 21/32] split-index: the writing part","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:42Z","receivedAt":"2014-04-28T10:55:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"prepare_to_write_split_index() does the major work, classifying\ndeleted, updated and added entries. write_link_extension() then just\nwrites it down.\n\nAn observation is, deleting an entry, then adding it back is recorded\nas \"entry X is deleted, entry X is added\", not \"entry X is replaced\".\nThis is simpler, with small overhead: a replaced entry is stored\nwithout its path, a new entry is store with its path.\n\nA note about unpack_trees() and the deduplication code inside\nprepare_to_write_split_index(). Usually tracking updated/removed\nentries via read-cache API is enough. unpack_trees() manipulates the\nindex in a different way: it throws the entire source index out,\nbuilds up a new one, copying/duplicating entries (using dup_entry)\nfrom the source index over if necessary, then returns the new index.\n\nA naive solution would be marking the entire source index \"deleted\"\nand add their duplicates as new. That could bring $GIT_DIR/index back\nto the original size. So we try harder and memcmp() between the\noriginal and the duplicate to see if it needs updating.\n\nWe could avoid memcmp() too, by avoiding duplicating the original\nentry in dup_entry(). The performance gain this way is within noise\nlevel and it complicates unpack-trees.c. So memcmp() is the preferred\nway to deal with deduplication.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n split-index.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n split-index.h | 4 +++\n 2 files changed, 103 insertions(+), 2 deletions(-)\n\ndiff --git a/split-index.c b/split-index.c\nindex b36c73b..5708807 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"split-index.h\"\n+#include \"ewah/ewok.h\"\n \n struct split_index *init_split_index(struct index_state *istate)\n {\n@@ -26,11 +27,22 @@ int read_link_extension(struct index_state *istate,\n \treturn 0;\n }\n \n+static int write_strbuf(void *user_data, const void *data, size_t len)\n+{\n+\tstruct strbuf *sb = user_data;\n+\tstrbuf_add(sb, data, len);\n+\treturn len;\n+}\n+\n int write_link_extension(struct strbuf *sb,\n \t\t\t struct index_state *istate)\n {\n \tstruct split_index *si = istate->split_index;\n \tstrbuf_add(sb, si->base_sha1, 20);\n+\tif (!si->delete_bitmap && !si->replace_bitmap)\n+\t\treturn 0;\n+\tewah_serialize_to(si->delete_bitmap, write_strbuf, sb);\n+\tewah_serialize_to(si->replace_bitmap, write_strbuf, sb);\n \treturn 0;\n }\n \n@@ -62,14 +74,99 @@ void merge_base_index(struct index_state *istate)\n void prepare_to_write_split_index(struct index_state *istate)\n {\n \tstruct split_index *si = init_split_index(istate);\n-\t/* take cache[] out temporarily */\n+\tstruct cache_entry **entries = NULL, *ce;\n+\tint i, nr_entries = 0, nr_alloc = 0;\n+\n+\tsi->delete_bitmap = ewah_new();\n+\tsi->replace_bitmap = ewah_new();\n+\n+\tif (si->base) {\n+\t\t/* Go through istate->cache[] and mark CE_MATCHED to\n+\t\t * entry with positive index. We'll go through\n+\t\t * base->cache[] later to delete all entries in base\n+\t\t * that are not marked eith either CE_MATCHED or\n+\t\t * CE_UPDATE_IN_BASE. If istate->cache[i] is a\n+\t\t * duplicate, deduplicate it.\n+\t\t */\n+\t\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\t\tstruct cache_entry *base;\n+\t\t\t/* namelen is checked separately */\n+\t\t\tconst unsigned int ondisk_flags =\n+\t\t\t\tCE_STAGEMASK | CE_VALID | CE_EXTENDED_FLAGS;\n+\t\t\tunsigned int ce_flags, base_flags, ret;\n+\t\t\tce = istate->cache[i];\n+\t\t\tif (!ce->index)\n+\t\t\t\tcontinue;\n+\t\t\tif (ce->index > si->base->cache_nr) {\n+\t\t\t\tce->index = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tce->ce_flags |= CE_MATCHED; /* or \"shared\" */\n+\t\t\tbase = si->base->cache[ce->index - 1];\n+\t\t\tif (ce == base)\n+\t\t\t\tcontinue;\n+\t\t\tif (ce->ce_namelen != base->ce_namelen ||\n+\t\t\t strcmp(ce->name, base->name)) {\n+\t\t\t\tce->index = 0;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tce_flags = ce->ce_flags;\n+\t\t\tbase_flags = base->ce_flags;\n+\t\t\t/* only on-disk flags matter */\n+\t\t\tce->ce_flags &= ondisk_flags;\n+\t\t\tbase->ce_flags &= ondisk_flags;\n+\t\t\tret = memcmp(&ce->ce_stat_data, &base->ce_stat_data,\n+\t\t\t\t offsetof(struct cache_entry, name) -\n+\t\t\t\t offsetof(struct cache_entry, ce_stat_data));\n+\t\t\tce->ce_flags = ce_flags;\n+\t\t\tbase->ce_flags = base_flags;\n+\t\t\tif (ret)\n+\t\t\t\tce->ce_flags |= CE_UPDATE_IN_BASE;\n+\t\t\tfree(base);\n+\t\t\tsi->base->cache[ce->index - 1] = ce;\n+\t\t}\n+\t\tfor (i = 0; i < si->base->cache_nr; i++) {\n+\t\t\tce = si->base->cache[i];\n+\t\t\tif ((ce->ce_flags & CE_REMOVE) ||\n+\t\t\t !(ce->ce_flags & CE_MATCHED))\n+\t\t\t\tewah_set(si->delete_bitmap, i);\n+\t\t\telse if (ce->ce_flags & CE_UPDATE_IN_BASE) {\n+\t\t\t\tewah_set(si->replace_bitmap, i);\n+\t\t\t\tALLOC_GROW(entries, nr_entries+1, nr_alloc);\n+\t\t\t\tentries[nr_entries++] = ce;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tce = istate->cache[i];\n+\t\tif ((!si->base || !ce->index) && !(ce->ce_flags & CE_REMOVE)) {\n+\t\t\tALLOC_GROW(entries, nr_entries+1, nr_alloc);\n+\t\t\tentries[nr_entries++] = ce;\n+\t\t}\n+\t\tce->ce_flags &= ~CE_MATCHED;\n+\t}\n+\n+\t/*\n+\t * take cache[] out temporarily, put entries[] in its place\n+\t * for writing\n+\t */\n+\tsi->saved_cache = istate->cache;\n \tsi->saved_cache_nr = istate->cache_nr;\n-\tistate->cache_nr = 0;\n+\tistate->cache = entries;\n+\tistate->cache_nr = nr_entries;\n }\n \n void finish_writing_split_index(struct index_state *istate)\n {\n \tstruct split_index *si = init_split_index(istate);\n+\n+\tewah_free(si->delete_bitmap);\n+\tewah_free(si->replace_bitmap);\n+\tsi->delete_bitmap = NULL;\n+\tsi->replace_bitmap = NULL;\n+\tfree(istate->cache);\n+\tistate->cache = si->saved_cache;\n \tistate->cache_nr = si->saved_cache_nr;\n }\n \ndiff --git a/split-index.h b/split-index.h\nindex 812e510..53b778f 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -3,10 +3,14 @@\n \n struct index_state;\n struct strbuf;\n+struct ewah_bitmap;\n \n struct split_index {\n \tunsigned char base_sha1[20];\n \tstruct index_state *base;\n+\tstruct ewah_bitmap *delete_bitmap;\n+\tstruct ewah_bitmap *replace_bitmap;\n+\tstruct cache_entry **saved_cache;\n \tunsigned int saved_cache_nr;\n \tint refcount;\n };\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239882","messageId":"1398682553-11634-23-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 22/32] split-index: the reading part","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:43Z","receivedAt":"2014-04-28T10:55:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"CE_REMOVE'd entries are removed here because only parts of the code\nbase (unpack_trees in fact) test this bit when they look for the\npresence of an entry. Leaving them may confuse the code ignores this\nbit and expects to see a real entry.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 2 --\n split-index.c | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n split-index.h | 2 ++\n 3 files changed, 84 insertions(+), 4 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex a717171..a5517bf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1569,8 +1569,6 @@ int read_index_from(struct index_state *istate, const char *path)\n \n \tif (is_null_sha1(split_index->base_sha1))\n \t\treturn ret;\n-\tif (istate->cache_nr)\n-\t\tdie(\"index in split-index mode must contain no entries\");\n \n \tif (split_index->base)\n \t\tdiscard_index(split_index->base);\ndiff --git a/split-index.c b/split-index.c\nindex 5708807..b03a250 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -16,13 +16,27 @@ int read_link_extension(struct index_state *istate,\n {\n \tconst unsigned char *data = data_;\n \tstruct split_index *si;\n+\tint ret;\n+\n \tif (sz < 20)\n \t\treturn error(\"corrupt link extension (too short)\");\n \tsi = init_split_index(istate);\n \thashcpy(si->base_sha1, data);\n \tdata += 20;\n \tsz -= 20;\n-\tif (sz)\n+\tif (!sz)\n+\t\treturn 0;\n+\tsi->delete_bitmap = ewah_new();\n+\tret = ewah_read_mmap(si->delete_bitmap, data, sz);\n+\tif (ret < 0)\n+\t\treturn error(\"corrupt delete bitmap in link extension\");\n+\tdata += ret;\n+\tsz -= ret;\n+\tsi->replace_bitmap = ewah_new();\n+\tret = ewah_read_mmap(si->replace_bitmap, data, sz);\n+\tif (ret < 0)\n+\t\treturn error(\"corrupt replace bitmap in link extension\");\n+\tif (ret != sz)\n \t\treturn error(\"garbage at the end of link extension\");\n \treturn 0;\n }\n@@ -60,15 +74,81 @@ static void mark_base_index_entries(struct index_state *base)\n \t\tbase->cache[i]->index = i + 1;\n }\n \n+static void mark_entry_for_delete(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tif (pos >= istate->cache_nr)\n+\t\tdie(\"position for delete %d exceeds base index size %d\",\n+\t\t (int)pos, istate->cache_nr);\n+\tistate->cache[pos]->ce_flags |= CE_REMOVE;\n+\tistate->split_index->nr_deletions = 1;\n+}\n+\n+static void replace_entry(size_t pos, void *data)\n+{\n+\tstruct index_state *istate = data;\n+\tstruct split_index *si = istate->split_index;\n+\tstruct cache_entry *dst, *src;\n+\tif (pos >= istate->cache_nr)\n+\t\tdie(\"position for replacement %d exceeds base index size %d\",\n+\t\t (int)pos, istate->cache_nr);\n+\tif (si->nr_replacements >= si->saved_cache_nr)\n+\t\tdie(\"too many replacements (%d vs %d)\",\n+\t\t si->nr_replacements, si->saved_cache_nr);\n+\tdst = istate->cache[pos];\n+\tif (dst->ce_flags & CE_REMOVE)\n+\t\tdie(\"entry %d is marked as both replaced and deleted\",\n+\t\t (int)pos);\n+\tsrc = si->saved_cache[si->nr_replacements];\n+\tsrc->index = pos + 1;\n+\tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n+\tfree(dst);\n+\tdst = src;\n+\tsi->nr_replacements++;\n+}\n+\n void merge_base_index(struct index_state *istate)\n {\n \tstruct split_index *si = istate->split_index;\n+\tunsigned int i;\n \n \tmark_base_index_entries(si->base);\n-\tistate->cache_nr = si->base->cache_nr;\n+\n+\tsi->saved_cache\t = istate->cache;\n+\tsi->saved_cache_nr = istate->cache_nr;\n+\tistate->cache_nr = si->base->cache_nr;\n+\tistate->cache\t = NULL;\n+\tistate->cache_alloc = 0;\n \tALLOC_GROW(istate->cache, istate->cache_nr, istate->cache_alloc);\n \tmemcpy(istate->cache, si->base->cache,\n \t sizeof(*istate->cache) * istate->cache_nr);\n+\n+\tsi->nr_deletions = 0;\n+\tsi->nr_replacements = 0;\n+\tewah_each_bit(si->replace_bitmap, replace_entry, istate);\n+\tewah_each_bit(si->delete_bitmap, mark_entry_for_delete, istate);\n+\tif (si->nr_deletions)\n+\t\tremove_marked_cache_entries(istate);\n+\n+\tfor (i = si->nr_replacements; i < si->saved_cache_nr; i++) {\n+\t\tadd_index_entry(istate, si->saved_cache[i],\n+\t\t\t\tADD_CACHE_OK_TO_ADD |\n+\t\t\t\t/*\n+\t\t\t\t * we may have to replay what\n+\t\t\t\t * merge-recursive.c:update_stages()\n+\t\t\t\t * does, which has this flag on\n+\t\t\t\t */\n+\t\t\t\tADD_CACHE_SKIP_DFCHECK);\n+\t\tsi->saved_cache[i] = NULL;\n+\t}\n+\n+\tewah_free(si->delete_bitmap);\n+\tewah_free(si->replace_bitmap);\n+\tfree(si->saved_cache);\n+\tsi->delete_bitmap = NULL;\n+\tsi->replace_bitmap = NULL;\n+\tsi->saved_cache\t = NULL;\n+\tsi->saved_cache_nr = 0;\n }\n \n void prepare_to_write_split_index(struct index_state *istate)\ndiff --git a/split-index.h b/split-index.h\nindex 53b778f..c1324f5 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -12,6 +12,8 @@ struct split_index {\n \tstruct ewah_bitmap *replace_bitmap;\n \tstruct cache_entry **saved_cache;\n \tunsigned int saved_cache_nr;\n+\tunsigned int nr_deletions;\n+\tunsigned int nr_replacements;\n \tint refcount;\n };\n \n-- \n1.9.1.346.ga2b5940\n"},{"id":"239883","messageId":"1398682553-11634-24-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 23/32] split-index: do not invalidate cache-tree at read time","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:44Z","receivedAt":"2014-04-28T10:55:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We are sure that after merge_base_index() is done. cache-tree can\nstill be used with the final index. So don't destroy cache tree.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 3 ++-\n split-index.c | 1 +\n 3 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 127804e..be95b84 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -488,6 +488,7 @@ extern int index_name_pos(const struct index_state *, const char *name, int name\n #define ADD_CACHE_SKIP_DFCHECK 4\t/* Ok to skip DF conflict checks */\n #define ADD_CACHE_JUST_APPEND 8\t\t/* Append only; tree.c::read_tree() */\n #define ADD_CACHE_NEW_ONLY 16\t\t/* Do not replace existing ones */\n+#define ADD_CACHE_KEEP_CACHE_TREE 32\t/* Do not invalidate cache-tree */\n extern int add_index_entry(struct index_state *, struct cache_entry *ce, int option);\n extern void rename_index_entry_at(struct index_state *, int pos, const char *new_name);\n extern int remove_index_entry_at(struct index_state *, int pos);\ndiff --git a/read-cache.c b/read-cache.c\nindex a5517bf..43a61d3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -950,7 +950,8 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \tint skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n \tint new_only = option & ADD_CACHE_NEW_ONLY;\n \n-\tcache_tree_invalidate_path(istate, ce->name);\n+\tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n+\t\tcache_tree_invalidate_path(istate, ce->name);\n \tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n \n \t/* existing match? Just replace it. */\ndiff --git a/split-index.c b/split-index.c\nindex b03a250..33c0c4b 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -133,6 +133,7 @@ void merge_base_index(struct index_state *istate)\n \tfor (i = si->nr_replacements; i < si->saved_cache_nr; i++) {\n \t\tadd_index_entry(istate, si->saved_cache[i],\n \t\t\t\tADD_CACHE_OK_TO_ADD |\n+\t\t\t\tADD_CACHE_KEEP_CACHE_TREE |\n \t\t\t\t/*\n \t\t\t\t * we may have to replay what\n \t\t\t\t * merge-recursive.c:update_stages()\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239885","messageId":"1398682553-11634-25-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 24/32] split-index: strip pathname of on-disk replaced entries","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:45Z","receivedAt":"2014-04-28T10:55:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"We know the positions of replaced entries via the replace bitmap in\n\"link\" extension, so the \"name\" path does not have to be stored (it's\nstill in the shared index). With this, we also have a way to\ndistinguish additions vs replacements at load time and can catch\nbroken \"link\" extensions.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 10 ++++++++++\n split-index.c | 14 ++++++++++++--\n 3 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex be95b84..604328b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -170,6 +170,7 @@ struct cache_entry {\n #define CE_MATCHED (1 << 26)\n \n #define CE_UPDATE_IN_BASE (1 << 27)\n+#define CE_STRIP_NAME (1 << 28)\n \n /*\n * Extended on-disk flags\ndiff --git a/read-cache.c b/read-cache.c\nindex 43a61d3..81835a6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1787,9 +1787,15 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n {\n \tint size;\n \tstruct ondisk_cache_entry *ondisk;\n+\tint saved_namelen;\n \tchar *name;\n \tint result;\n \n+\tif (ce->ce_flags & CE_STRIP_NAME) {\n+\t\tsaved_namelen = ce_namelen(ce);\n+\t\tce->ce_namelen = 0;\n+\t}\n+\n \tif (!previous_name) {\n \t\tsize = ondisk_ce_size(ce);\n \t\tondisk = xcalloc(1, size);\n@@ -1821,6 +1827,10 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n \t\tstrbuf_splice(previous_name, common, to_remove,\n \t\t\t ce->name + common, ce_namelen(ce) - common);\n \t}\n+\tif (ce->ce_flags & CE_STRIP_NAME) {\n+\t\tce->ce_namelen = saved_namelen;\n+\t\tce->ce_flags &= ~CE_STRIP_NAME;\n+\t}\n \n \tresult = ce_write(c, fd, ondisk, size);\n \tfree(ondisk);\ndiff --git a/split-index.c b/split-index.c\nindex 33c0c4b..ee3246f 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -89,6 +89,7 @@ static void replace_entry(size_t pos, void *data)\n \tstruct index_state *istate = data;\n \tstruct split_index *si = istate->split_index;\n \tstruct cache_entry *dst, *src;\n+\n \tif (pos >= istate->cache_nr)\n \t\tdie(\"position for replacement %d exceeds base index size %d\",\n \t\t (int)pos, istate->cache_nr);\n@@ -100,10 +101,14 @@ static void replace_entry(size_t pos, void *data)\n \t\tdie(\"entry %d is marked as both replaced and deleted\",\n \t\t (int)pos);\n \tsrc = si->saved_cache[si->nr_replacements];\n+\tif (ce_namelen(src))\n+\t\tdie(\"corrupt link extension, entry %d should have \"\n+\t\t \"zero length name\", (int)pos);\n \tsrc->index = pos + 1;\n \tsrc->ce_flags |= CE_UPDATE_IN_BASE;\n-\tfree(dst);\n-\tdst = src;\n+\tsrc->ce_namelen = dst->ce_namelen;\n+\tcopy_cache_entry(dst, src);\n+\tfree(src);\n \tsi->nr_replacements++;\n }\n \n@@ -131,6 +136,9 @@ void merge_base_index(struct index_state *istate)\n \t\tremove_marked_cache_entries(istate);\n \n \tfor (i = si->nr_replacements; i < si->saved_cache_nr; i++) {\n+\t\tif (!ce_namelen(si->saved_cache[i]))\n+\t\t\tdie(\"corrupt link extension, entry %d should \"\n+\t\t\t \"have non-zero length name\", i);\n \t\tadd_index_entry(istate, si->saved_cache[i],\n \t\t\t\tADD_CACHE_OK_TO_ADD |\n \t\t\t\tADD_CACHE_KEEP_CACHE_TREE |\n@@ -213,6 +221,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\t\tewah_set(si->delete_bitmap, i);\n \t\t\telse if (ce->ce_flags & CE_UPDATE_IN_BASE) {\n \t\t\t\tewah_set(si->replace_bitmap, i);\n+\t\t\t\tce->ce_flags |= CE_STRIP_NAME;\n \t\t\t\tALLOC_GROW(entries, nr_entries+1, nr_alloc);\n \t\t\t\tentries[nr_entries++] = ce;\n \t\t\t}\n@@ -222,6 +231,7 @@ void prepare_to_write_split_index(struct index_state *istate)\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tce = istate->cache[i];\n \t\tif ((!si->base || !ce->index) && !(ce->ce_flags & CE_REMOVE)) {\n+\t\t\tassert(!(ce->ce_flags & CE_STRIP_NAME));\n \t\t\tALLOC_GROW(entries, nr_entries+1, nr_alloc);\n \t\t\tentries[nr_entries++] = ce;\n \t\t}\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239886","messageId":"1398682553-11634-26-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 25/32] update-index: new options to enable/disable split index mode","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:46Z","receivedAt":"2014-04-28T10:55:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If you have a large work tree but only make changes in a subset, then\n$GIT_DIR/index's size should be stable after a while. If you change\nbranches that touch something else, $GIT_DIR/index's size may grow\nlarge that it becomes as slow as the unified index. Do --split-index\nagain occasionally to force all changes back to the shared index and\nkeep $GIT_DIR/index small.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-update-index.txt | 11 +++++++\n builtin/update-index.c | 18 ++++++++++\n cache.h | 1 +\n read-cache.c | 67 ++++++++++++++++++++++++++++++++++----\n split-index.c | 23 +++++++++++++\n 5 files changed, 114 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex d6de4a0..dfc09d9 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -161,6 +161,17 @@ may not support it yet.\n \tOnly meaningful with `--stdin` or `--index-info`; paths are\n \tseparated with NUL character instead of LF.\n \n+--split-index::\n+--no-split-index::\n+\tEnable or disable split index mode. If enabled, the index is\n+\tsplit into two files, $GIT_DIR/index and $GIT_DIR/sharedindex..\n+\tChanges are accumulated in $GIT_DIR/index while the shared\n+\tindex file contains all index entries stays unchanged. If\n+\tsplit-index mode is already enabled and `--split-index` is\n+\tgiven again, all changes in $GIT_DIR/index are pushed back to\n+\tthe shared index file. This mode is designed for very large\n+\tindexes that take a signficant amount of time to read or write.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex f7a19c4..b0503f4 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -13,6 +13,7 @@\n #include \"parse-options.h\"\n #include \"pathspec.h\"\n #include \"dir.h\"\n+#include \"split-index.h\"\n \n /*\n * Default to not allowing changes to the list of files. The\n@@ -742,6 +743,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tchar set_executable_bit = 0;\n \tstruct refresh_params refresh_args = {0, &has_errors};\n \tint lock_error = 0;\n+\tint split_index = -1;\n \tstruct lock_file *lock_file;\n \tstruct parse_opt_ctx_t ctx;\n \tint parseopt_state = PARSE_OPT_UNKNOWN;\n@@ -824,6 +826,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tresolve_undo_clear_callback},\n \t\tOPT_INTEGER(0, \"index-version\", &preferred_index_format,\n \t\t\tN_(\"write index in this format\")),\n+\t\tOPT_BOOL(0, \"split-index\", &split_index,\n+\t\t\tN_(\"enable or disable split index\")),\n \t\tOPT_END()\n \t};\n \n@@ -917,6 +921,20 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_release(&buf);\n \t}\n \n+\tif (split_index > 0) {\n+\t\tinit_split_index(&the_index);\n+\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n+\t} else if (!split_index && the_index.split_index) {\n+\t\t/*\n+\t\t * can't discard_split_index(&the_index); because that\n+\t\t * will destroy split_index->base->cache[], which may\n+\t\t * be shared with the_index.cache[]. So yeah we're\n+\t\t * leaking a bit here.\n+\t\t */\n+\t\tthe_index.split_index = NULL;\n+\t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n+\t}\n+\n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\ndiff --git a/cache.h b/cache.h\nindex 604328b..42cdfe6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -278,6 +278,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define SOMETHING_CHANGED\t(1 << 3) /* unclassified changes go here */\n #define RESOLVE_UNDO_CHANGED\t(1 << 4)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n+#define SPLIT_INDEX_ORDERED\t(1 << 6)\n \n struct split_index;\n struct index_state {\ndiff --git a/read-cache.c b/read-cache.c\nindex 81835a6..a6c9407 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -15,6 +15,7 @@\n #include \"strbuf.h\"\n #include \"varint.h\"\n #include \"split-index.h\"\n+#include \"sigchain.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -39,7 +40,8 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n-\t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED)\n+\t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n+\t\t SPLIT_INDEX_ORDERED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n@@ -1860,7 +1862,8 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \t\trollback_lock_file(lockfile);\n }\n \n-static int do_write_index(struct index_state *istate, int newfd)\n+static int do_write_index(struct index_state *istate, int newfd,\n+\t\t\t int strip_extensions)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n@@ -1923,7 +1926,7 @@ static int do_write_index(struct index_state *istate, int newfd)\n \tstrbuf_release(&previous_name_buf);\n \n \t/* Write extension data here */\n-\tif (istate->split_index) {\n+\tif (!strip_extensions && istate->split_index) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\terr = write_link_extension(&sb, istate) < 0 ||\n@@ -1934,7 +1937,7 @@ static int do_write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-\tif (istate->cache_tree) {\n+\tif (!strip_extensions && istate->cache_tree) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tcache_tree_write(&sb, istate->cache_tree);\n@@ -1944,7 +1947,7 @@ static int do_write_index(struct index_state *istate, int newfd)\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-\tif (istate->resolve_undo) {\n+\tif (!strip_extensions && istate->resolve_undo) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tresolve_undo_write(&sb, istate->resolve_undo);\n@@ -1985,7 +1988,7 @@ static int commit_locked_index(struct lock_file *lk)\n static int do_write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t\t\t unsigned flags)\n {\n-\tint ret = do_write_index(istate, lock->fd);\n+\tint ret = do_write_index(istate, lock->fd, 0);\n \tif (ret)\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n@@ -2009,6 +2012,52 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n+static char *temporary_sharedindex;\n+\n+static void remove_temporary_sharedindex(void)\n+{\n+\tif (temporary_sharedindex) {\n+\t\tunlink_or_warn(temporary_sharedindex);\n+\t\tfree(temporary_sharedindex);\n+\t\ttemporary_sharedindex = NULL;\n+\t}\n+}\n+\n+static void remove_temporary_sharedindex_on_signal(int signo)\n+{\n+\tremove_temporary_sharedindex();\n+\tsigchain_pop(signo);\n+\traise(signo);\n+}\n+\n+static int write_shared_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = istate->split_index;\n+\tstatic int installed_handler;\n+\tint fd, ret;\n+\n+\ttemporary_sharedindex = git_pathdup(\"sharedindex_XXXXXX\");\n+\tfd = xmkstemp(temporary_sharedindex);\n+\tif (!installed_handler) {\n+\t\tatexit(remove_temporary_sharedindex);\n+\t\tsigchain_push_common(remove_temporary_sharedindex_on_signal);\n+\t}\n+\tmove_cache_to_base_index(istate);\n+\tret = do_write_index(si->base, fd, 1);\n+\tclose(fd);\n+\tif (ret) {\n+\t\tremove_temporary_sharedindex();\n+\t\treturn ret;\n+\t}\n+\tret = rename(temporary_sharedindex,\n+\t\t git_path(\"sharedindex.%s\", sha1_to_hex(si->base->sha1)));\n+\tfree(temporary_sharedindex);\n+\ttemporary_sharedindex = NULL;\n+\tif (!ret)\n+\t\thashcpy(si->base_sha1, si->base->sha1);\n+\treturn ret;\n+}\n+\n int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t unsigned flags)\n {\n@@ -2020,6 +2069,12 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\treturn do_write_locked_index(istate, lock, flags);\n \t}\n \n+\tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n+\t\tint ret = write_shared_index(istate);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\t}\n+\n \treturn write_split_index(istate, lock, flags);\n }\n \ndiff --git a/split-index.c b/split-index.c\nindex ee3246f..21485e2 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -74,6 +74,29 @@ static void mark_base_index_entries(struct index_state *base)\n \t\tbase->cache[i]->index = i + 1;\n }\n \n+void move_cache_to_base_index(struct index_state *istate)\n+{\n+\tstruct split_index *si = istate->split_index;\n+\tint i;\n+\n+\t/*\n+\t * do not delete old si->base, its index entries may be shared\n+\t * with istate->cache[]. Accept a bit of leaking here because\n+\t * this code is only used by short-lived update-index.\n+\t */\n+\tsi->base = xcalloc(1, sizeof(*si->base));\n+\tsi->base->version = istate->version;\n+\t/* zero timestamp disables racy test in ce_write_index() */\n+\tsi->base->timestamp = istate->timestamp;\n+\tALLOC_GROW(si->base->cache, istate->cache_nr, si->base->cache_alloc);\n+\tsi->base->cache_nr = istate->cache_nr;\n+\tmemcpy(si->base->cache, istate->cache,\n+\t sizeof(*istate->cache) * istate->cache_nr);\n+\tmark_base_index_entries(si->base);\n+\tfor (i = 0; i < si->base->cache_nr; i++)\n+\t\tsi->base->cache[i]->ce_flags &= ~CE_UPDATE_IN_BASE;\n+}\n+\n static void mark_entry_for_delete(size_t pos, void *data)\n {\n \tstruct index_state *istate = data;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239892","messageId":"1398682553-11634-27-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 26/32] update-index --split-index: do not split if $GIT_DIR is read only","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:47Z","receivedAt":"2014-04-28T10:55:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If $GIT_DIR is read only, we can't write $GIT_DIR/sharedindex. This\ncould happen when $GIT_INDEX_FILE is set to somehwere outside\n$GIT_DIR.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 13 ++++++++++---\n 1 file changed, 10 insertions(+), 3 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex a6c9407..f9fc3a5 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2030,14 +2030,21 @@ static void remove_temporary_sharedindex_on_signal(int signo)\n \traise(signo);\n }\n \n-static int write_shared_index(struct index_state *istate)\n+static int write_shared_index(struct index_state *istate,\n+\t\t\t struct lock_file *lock, unsigned flags)\n {\n \tstruct split_index *si = istate->split_index;\n \tstatic int installed_handler;\n \tint fd, ret;\n \n \ttemporary_sharedindex = git_pathdup(\"sharedindex_XXXXXX\");\n-\tfd = xmkstemp(temporary_sharedindex);\n+\tfd = mkstemp(temporary_sharedindex);\n+\tif (fd < 0) {\n+\t\tfree(temporary_sharedindex);\n+\t\ttemporary_sharedindex = NULL;\n+\t\thashclr(si->base_sha1);\n+\t\treturn do_write_locked_index(istate, lock, flags);\n+\t}\n \tif (!installed_handler) {\n \t\tatexit(remove_temporary_sharedindex);\n \t\tsigchain_push_common(remove_temporary_sharedindex_on_signal);\n@@ -2070,7 +2077,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t}\n \n \tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n-\t\tint ret = write_shared_index(istate);\n+\t\tint ret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239894","messageId":"1398682553-11634-28-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 27/32] rev-parse: add --shared-index-path to get shared index path","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:48Z","receivedAt":"2014-04-28T10:55:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Normally scripts do not have to be aware about split indexes because\nall shared indexes are in $GIT_DIR. A simple \"mv $tmp_index\n$GIT_DIR/somewhere\" is enough. Scripts that generate temporary indexes\nand move them across repos must be aware about split index and copy\nthe shared file as well. This option enables that.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-rev-parse.txt | 4 ++++\n builtin/rev-parse.c | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt\nindex 987395d..9bd76a5 100644\n--- a/Documentation/git-rev-parse.txt\n+++ b/Documentation/git-rev-parse.txt\n@@ -245,6 +245,10 @@ print a message to stderr and exit with nonzero status.\n --show-toplevel::\n \tShow the absolute path of the top-level directory.\n \n+--shared-index-path::\n+\tShow the path to the shared index file in split index mode, or\n+\tempty if not in split-index mode.\n+\n Other Options\n ~~~~~~~~~~~~~\n \ndiff --git a/builtin/rev-parse.c b/builtin/rev-parse.c\nindex 1a6122d..8102aaa 100644\n--- a/builtin/rev-parse.c\n+++ b/builtin/rev-parse.c\n@@ -11,6 +11,7 @@\n #include \"parse-options.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"split-index.h\"\n \n #define DO_REVS\t\t1\n #define DO_NOREV\t2\n@@ -775,6 +776,15 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t\t: \"false\");\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--shared-index-path\")) {\n+\t\t\t\tif (read_cache() < 0)\n+\t\t\t\t\tdie(_(\"Could not read the index\"));\n+\t\t\t\tif (the_index.split_index) {\n+\t\t\t\t\tconst unsigned char *sha1 = the_index.split_index->base_sha1;\n+\t\t\t\t\tputs(git_path(\"sharedindex.%s\", sha1_to_hex(sha1)));\n+\t\t\t\t}\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (starts_with(arg, \"--since=\")) {\n \t\t\t\tshow_datestring(\"--max-age=\", arg+8);\n \t\t\t\tcontinue;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239889","messageId":"1398682553-11634-29-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 28/32] read-tree: force split-index mode off on --index-output","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:49Z","receivedAt":"2014-04-28T10:55:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Just a (paranoid?) safety measure..\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex f9fc3a5..568bc20 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2070,7 +2070,8 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n {\n \tstruct split_index *si = istate->split_index;\n \n-\tif (!si || (istate->cache_changed & ~EXTMASK)) {\n+\tif (!si || alternate_index_output ||\n+\t (istate->cache_changed & ~EXTMASK)) {\n \t\tif (si)\n \t\t\thashclr(si->base_sha1);\n \t\treturn do_write_locked_index(istate, lock, flags);\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239887","messageId":"1398682553-11634-30-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 29/32] read-tree: note about dropping split-index mode or index version","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:50Z","receivedAt":"2014-04-28T10:55:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/read-tree.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 3204c62..e7e1c33 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -155,6 +155,15 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n+\t/*\n+\t * NEEDSWORK\n+\t *\n+\t * The old index should be read anyway even if we're going to\n+\t * destroy all index entries because we still need to preserve\n+\t * certain information such as index version or split-index\n+\t * mode.\n+\t */\n+\n \tif (opts.reset || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n \t\t\tdie(\"You need to resolve your current index first\");\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239888","messageId":"1398682553-11634-31-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 30/32] read-cache: force split index mode with GIT_TEST_SPLIT_INDEX","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:51Z","receivedAt":"2014-04-28T10:55:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This could be used to run the whole test suite with split\nindexes. Index splitting is carried out at random. \"git read-tree\"\nalso resets the index and forces splitting at the next update.\n\nI had a lot of headaches with the test suite, which proves it\nexercises split index pretty good.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 568bc20..831b67e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1885,8 +1885,11 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\t}\n \t}\n \n-\tif (!istate->version)\n+\tif (!istate->version) {\n \t\tistate->version = get_index_format_default();\n+\t\tif (getenv(\"GIT_TEST_SPLIT_INDEX\"))\n+\t\t\tinit_split_index(istate);\n+\t}\n \n \t/* demote version 3 to version 2 when the latter suffices */\n \tif (istate->version == 3 || istate->version == 2)\n@@ -2077,6 +2080,11 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\treturn do_write_locked_index(istate, lock, flags);\n \t}\n \n+\tif (getenv(\"GIT_TEST_SPLIT_INDEX\")) {\n+\t\tint v = si->base_sha1[0];\n+\t\tif ((v & 15) < 6)\n+\t\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n+\t}\n \tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n \t\tint ret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\n-- \n1.9.1.346.ga2b5940\n"},{"id":"239890","messageId":"1398682553-11634-32-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 31/32] t2104: make sure split index mode is off for the version test","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-28T10:55:52Z","receivedAt":"2014-04-28T10:55:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Version tests only make sense when all entries are in the same file,\nso we can see if version is downgraded to 2 if 3 is not required.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t2104-update-index-skip-worktree.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh\nindex 29c1fb1..cc830da 100755\n--- a/t/t2104-update-index-skip-worktree.sh\n+++ b/t/t2104-update-index-skip-worktree.sh\n@@ -7,6 +7,8 @@ test_description='skip-worktree bit test'\n \n . ./test-lib.sh\n \n+sane_unset GIT_TEST_SPLIT_INDEX\n+\n test_set_index_version 3\n \n cat >expect.full <\n---\n .gitignore | 1 +\n Makefile | 1 +\n cache.h | 2 +\n read-cache.c | 3 +-\n t/t1700-split-index.sh (new +x) | 194 ++++++++++++++++++++++++++++++++++++++++\n test-dump-split-index.c (new) | 34 +++++++\n 6 files changed, 233 insertions(+), 2 deletions(-)\n create mode 100755 t/t1700-split-index.sh\n create mode 100644 test-dump-split-index.c\n\ndiff --git a/.gitignore b/.gitignore\nindex dc600f9..70992a4 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -180,6 +180,7 @@\n /test-date\n /test-delta\n /test-dump-cache-tree\n+/test-dump-split-index\n /test-scrap-cache-tree\n /test-genrandom\n /test-hashmap\ndiff --git a/Makefile b/Makefile\nindex c3957bb..e39a4f5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -562,6 +562,7 @@ TEST_PROGRAMS_NEED_X += test-ctype\n TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n+TEST_PROGRAMS_NEED_X += test-dump-split-index\n TEST_PROGRAMS_NEED_X += test-genrandom\n TEST_PROGRAMS_NEED_X += test-hashmap\n TEST_PROGRAMS_NEED_X += test-index-version\ndiff --git a/cache.h b/cache.h\nindex 42cdfe6..6ad2595 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -473,6 +473,8 @@ extern int daemonize(void);\n struct lock_file;\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n+extern int do_read_index(struct index_state *istate, const char *path,\n+\t\t\t int must_exist); /* for testting only! */\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\ndiff --git a/read-cache.c b/read-cache.c\nindex 831b67e..159c3e8 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1463,8 +1463,7 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n }\n \n /* remember to discard_cache() before reading a different cache! */\n-static int do_read_index(struct index_state *istate, const char *path,\n-\t\t\t int must_exist)\n+int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n \tint fd, i;\n \tstruct stat st;\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nnew file mode 100755\nindex 0000000..94fb473\n--- /dev/null\n+++ b/t/t1700-split-index.sh\n@@ -0,0 +1,194 @@\n+#!/bin/sh\n+\n+test_description='split index mode tests'\n+\n+. ./test-lib.sh\n+\n+# We need total control of index splitting here\n+sane_unset GIT_TEST_SPLIT_INDEX\n+\n+test_expect_success 'enable split index' '\n+\tgit update-index --split-index &&\n+\ttest-dump-split-index .git/index >actual &&\n+\tcat >expect <one &&\n+\tgit update-index --add one &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <one &&\n+\tgit update-index one &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tq_to_tab >expect <two &&\n+\tgit update-index --add two &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tq_to_tab >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tq_to_tab >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <one &&\n+\tgit update-index --add one &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <two &&\n+\tgit update-index --add two &&\n+\tgit ls-files --stage >actual &&\n+\tcat >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <base_sha1));\n+\tfor (i = 0; i< the_index.cache_nr; i++) {\n+\t\tstruct cache_entry *ce = the_index.cache[i];\n+\t\tprintf(\"%06o %s %d\\t%s\\n\", ce->ce_mode,\n+\t\t sha1_to_hex(ce->sha1), ce_stage(ce), ce->name);\n+\t}\n+\tprintf(\"replacements:\");\n+\tewah_each_bit(si->replace_bitmap, show_bit, NULL);\n+\tprintf(\"\\ndeletions:\");\n+\tewah_each_bit(si->delete_bitmap, show_bit, NULL);\n+\tprintf(\"\\n\");\n+\treturn 0;\n+}\n-- \n1.9.1.346.ga2b5940\n"},{"id":"240049","messageId":"CAJo=hJtgijOOMPbFjvTUaENw=hr0YixYmy1UkdqEd=CpLZ5L2A@mail.gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2014-04-28T21:18:44Z","receivedAt":"2014-04-28T21:18:44Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy wrote:\n> I hinted about it earlier [1]. It now passes the test suite and with a\n> design that I'm happy with (thanks to Junio for a suggestion about the\n> rename problem).\n>\n> From the user point of view, this reduces the writable size of index\n> down to the number of updated files. For example my webkit index v4 is\n> 14MB. With a fresh split, I only have to update an index of 200KB.\n> Every file I touch will add about 80 bytes to that. As long as I don't\n> touch every single tracked file in my worktree, I should not pay\n> penalty for writing 14MB index file on every operation.\n\nThis is a very welcome type of improvement.\n\nI am however concerned about the complexity of the format employed.\nWhy do we need two EWAH bitmaps in the new index? Why isn't this just\na pair of sorted files that are merge-joined at read, with records in\n$GIT_DIR/index taking priority over same-named records in\n$GIT_DIR/sharedindex.$SHA1? Deletes could be marked with a bit or an\n\"all zero\" metadata record.\n\n> The read penalty is not addressed here, so I still pay 14MB hashing\n> cost. But that's an easy problem. We could cache the validated index\n> in a daemon. Whenever git needs to load an index, it pokes the daemon.\n> The daemon verifies that the on-disk index still has the same\n> signature, then sends the in-mem index to git. When git updates the\n> index, it pokes the daemon again to update in-mem index. Next time git\n> reads the index, it does not have to pay I/O cost any more (actually\n> it does but the cost is hidden away when you do not have to read it\n> yet).\n\nIf we are going this far, maybe it is worthwhile building a mmap()\nregion the daemon exports to the git client that holds the \"in memory\"\nformat of the index. Clients would mmap this PROT_READ, MAP_PRIVATE\nand can then quickly access the base file information without doing\nfurther validation, or copying the large(ish) data over a pipe.\n\n\nJunio had some other great ideas for improving the index on really\nlarge trees. Maybe I should let him comment since they are really his\nideas. Something about not even checking out most files, storing most\nsubtrees as just a \"tree\" entry in the index. E.g. if you are a bad\ndeveloper and never touch the \"t/\" subdirectory then that is stored as\njust \"t\" and the SHA-1 of the \"t\" tree, rather than the recursively\nexploded list of the test directory.\n"},{"id":"240055","messageId":"xmqq7g69nldy.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-28T22:23:53Z","receivedAt":"2014-04-28T22:23:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> The read penalty is not addressed here, so I still pay 14MB hashing\n> cost.\n\nHmm, yeah, the cost for verify_hdr() would still matter, and\npresumably you would be hashing the additional 200kB to validate the\nsmaller \"changes since the base\" file to give users the same level\nof protection against corruption.\n\n> Doing this in other implementations should be easy (at least the\n> reading part) and with small code change. The whole index format is\n> retained. All you need is to read a new extension that contains two\n> ewah-bitmaps and apply the changes to create the final index.\n\nWhy bitmaps, though? Naïvely I would have expected you to read from\ntwo sorted streams and have the transaction log override the base.\n\nIntrigued to find it out...\n"},{"id":"240056","messageId":"xmqq1twhnkbz.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"1398682553-11634-18-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 17/32] read-cache: split-index mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-28T22:46:40Z","receivedAt":"2014-04-28T22:46:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> diff --git a/cache.h b/cache.h\n> index 0f6247c..90a5998 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -135,6 +135,7 @@ struct cache_entry {\n> \tunsigned int ce_mode;\n> \tunsigned int ce_flags;\n> \tunsigned int ce_namelen;\n> +\tunsigned int index;\t/* for link extension */\n> \tunsigned char sha1[20];\n> \tchar name[FLEX_ARRAY]; /* more */\n> };\n\nI am not sure if we want to keep an otherwise unused 8-byte around\nper cache entry (especially for a large project where the split\nindex mode should matter) after we read the index.\n\nI expected to see some code where entries in this incremental index\nare used to override the entries from the base/shared index, but\nmerge_base_index() seems to do just memcpy() to discard the former\nand replace them with the latter. Is this step meant to work at\nall, or is it a smaller step meant to be completed in later patches?\n\nI do think it is sensible to keep two arrays of \"struct cache_entry\"\naround (one for base and one for incremental changes) inside\nindex_state, and the patch seems to do so via \"struct split_index\"\nthat does have a copy of saved_cache. If the write-out codepath\nwalks these two sorted arrays in parallel, shouldn't it be able to\nfigure out which entry is added, deleted and modified without\nfattening this structure?\n\nMaybe it is too early for me to be asking these questions and it may\nbe better if I read the whole series twice and wait it to become\nclear to me why this field is necessary. I dunno.\n"},{"id":"240101","messageId":"CACsJy8BEJdA+d5DL94n4o3FRphs9bNPhX9oKK3MmOfTTBY=f9Q@mail.gmail.com","threadId":"36517","inReplyTo":"xmqq1twhnkbz.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 17/32] read-cache: split-index mode","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-29T01:43:36Z","receivedAt":"2014-04-29T01:43:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 29, 2014 at 5:46 AM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> diff --git a/cache.h b/cache.h\n>> index 0f6247c..90a5998 100644\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -135,6 +135,7 @@ struct cache_entry {\n>> unsigned int ce_mode;\n>> unsigned int ce_flags;\n>> unsigned int ce_namelen;\n>> + unsigned int index; /* for link extension */\n>> unsigned char sha1[20];\n>> char name[FLEX_ARRAY]; /* more */\n>> };\n>\n> I am not sure if we want to keep an otherwise unused 8-byte around\n> per cache entry (especially for a large project where the split\n> index mode should matter) after we read the index.\n>\n> I expected to see some code where entries in this incremental index\n> are used to override the entries from the base/shared index, but\n> merge_base_index() seems to do just memcpy() to discard the former\n> and replace them with the latter. Is this step meant to work at\n> all, or is it a smaller step meant to be completed in later patches?\n\nThis field only matters at write time, not read time. It's to quickly\ndetect if an entry is shared, see prepare_to_write_split_index().\n\n> I do think it is sensible to keep two arrays of \"struct cache_entry\"\n> around (one for base and one for incremental changes) inside\n> index_state, and the patch seems to do so via \"struct split_index\"\n> that does have a copy of saved_cache. If the write-out codepath\n> walks these two sorted arrays in parallel, shouldn't it be able to\n> figure out which entry is added, deleted and modified without\n> fattening this structure?\n\nSo far without that \"index\" field I would have to resort to hasing\nentries in both arrays to find the shared paths. But ideas are\nwelcome.\n-- \nDuy\n"},{"id":"240102","messageId":"CACsJy8B7P0Un0__9mS2j81PimPZbz9oTKZHKx8yo_smOMck1qA@mail.gmail.com","threadId":"36517","inReplyTo":"CAJo=hJtgijOOMPbFjvTUaENw=hr0YixYmy1UkdqEd=CpLZ5L2A@mail.gmail.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-29T01:52:59Z","receivedAt":"2014-04-29T01:52:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Apr 29, 2014 at 4:18 AM, Shawn Pearce wrote:\n> On Mon, Apr 28, 2014 at 3:55 AM, Nguyễn Thái Ngọc Duy wrote:\n>> I hinted about it earlier [1]. It now passes the test suite and with a\n>> design that I'm happy with (thanks to Junio for a suggestion about the\n>> rename problem).\n>>\n>> From the user point of view, this reduces the writable size of index\n>> down to the number of updated files. For example my webkit index v4 is\n>> 14MB. With a fresh split, I only have to update an index of 200KB.\n>> Every file I touch will add about 80 bytes to that. As long as I don't\n>> touch every single tracked file in my worktree, I should not pay\n>> penalty for writing 14MB index file on every operation.\n>\n> This is a very welcome type of improvement.\n>\n> I am however concerned about the complexity of the format employed.\n> Why do we need two EWAH bitmaps in the new index? Why isn't this just\n> a pair of sorted files that are merge-joined at read, with records in\n> $GIT_DIR/index taking priority over same-named records in\n> $GIT_DIR/sharedindex.$SHA1? Deletes could be marked with a bit or an\n> \"all zero\" metadata record.\n\nWith the bitmaps, I know the exact position to replace or delete an\nentry. Merge sort works, but I would need to walk through all entries\nin both indexes to compare entry name and stage, a bit costly in my\nopinion. And if you look at the format description in patch 0017, I\nstore the replaced entries without their names to save a bit more\nspace. \"EWAH\" is just an implementation detail. A straightforward\nbitmap should work fine (25kb for 200k entries seem reasonable).\n-- \nDuy\n"},{"id":"240188","messageId":"xmqqiopskq1o.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"CACsJy8BEJdA+d5DL94n4o3FRphs9bNPhX9oKK3MmOfTTBY=f9Q@mail.gmail.com","subject":"Re: [PATCH 17/32] read-cache: split-index mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-29T17:23:47Z","receivedAt":"2014-04-29T17:23:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n>> I do think it is sensible to keep two arrays of \"struct cache_entry\"\n>> around (one for base and one for incremental changes) inside\n>> index_state, and the patch seems to do so via \"struct split_index\"\n>> that does have a copy of saved_cache. If the write-out codepath\n>> walks these two sorted arrays in parallel, shouldn't it be able to\n>> figure out which entry is added, deleted and modified without\n>> fattening this structure?\n>\n> So far without that \"index\" field I would have to resort to hasing\n> entries in both arrays to find the shared paths. But ideas are\n> welcome.\n\nHmm, why do you need to hash, when both arrays are sorted? Wouldn't\nit be just the matter of walking these two arrays in parallel,\nwith one scanning index for each array initialized to the beginning,\ncomparing the elements pointed by these indices, noting the side\nthat comes earlier in the sort order and advancing the index on that\nside (or if they compare equal then advance both), ...?\n"},{"id":"240201","messageId":"xmqqlhunkhm9.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"1398682553-11634-25-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 24/32] split-index: strip pathname of on-disk replaced entries","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-29T20:25:50Z","receivedAt":"2014-04-29T20:25:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This triggers \"saved_namelen may be used uninitialized\" for me, even\nthough it looks clear that it is used under CE_STRIP_NAME and it is\nassigned under that condition. Sigh to a stupid compiler...\n"},{"id":"240254","messageId":"CACsJy8Da=n+kh8_bUkJ11-dK09A6zOMnhBQJCsPCWcCJ2mY0pw@mail.gmail.com","threadId":"36517","inReplyTo":"xmqqiopskq1o.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 17/32] read-cache: split-index mode","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-04-29T22:45:26Z","receivedAt":"2014-04-29T22:45:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Apr 30, 2014 at 12:23 AM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>>> I do think it is sensible to keep two arrays of \"struct cache_entry\"\n>>> around (one for base and one for incremental changes) inside\n>>> index_state, and the patch seems to do so via \"struct split_index\"\n>>> that does have a copy of saved_cache. If the write-out codepath\n>>> walks these two sorted arrays in parallel, shouldn't it be able to\n>>> figure out which entry is added, deleted and modified without\n>>> fattening this structure?\n>>\n>> So far without that \"index\" field I would have to resort to hasing\n>> entries in both arrays to find the shared paths. But ideas are\n>> welcome.\n>\n> Hmm, why do you need to hash, when both arrays are sorted? Wouldn't\n> it be just the matter of walking these two arrays in parallel,\n> with one scanning index for each array initialized to the beginning,\n> comparing the elements pointed by these indices, noting the side\n> that comes earlier in the sort order and advancing the index on that\n> side (or if they compare equal then advance both), ...?\n\nAnd compare all names and stages (especially in the unpack-trees case,\nwhen no entry is reused). I kinda hope to avoid that. Speaking about\nreusing cache_entry, we won't be able to share cache_entry because\nwhen it's freed in replace_index_entry, or remove_index_entry_at in\nthe main index, we need to locate the same entry in the shared index\nas well and remove that stale pointer. Without sharing, we nearly\ndouble memory usage from the beginning.\n-- \nDuy\n"},{"id":"240303","messageId":"xmqq38guj4xk.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"CACsJy8Da=n+kh8_bUkJ11-dK09A6zOMnhBQJCsPCWcCJ2mY0pw@mail.gmail.com","subject":"Re: [PATCH 17/32] read-cache: split-index mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-04-30T13:57:27Z","receivedAt":"2014-04-30T13:57:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> when no entry is reused). I kinda hope to avoid that.\n\nI see.\n\n> Speaking about\n> reusing cache_entry, we won't be able to share cache_entry because\n> when it's freed in replace_index_entry, or remove_index_entry_at in\n> the main index, we need to locate the same entry in the shared index\n> as well and remove that stale pointer. Without sharing, we nearly\n> double memory usage from the beginning.\n\nYeah, the point being to have most of the entries come from the base\none, it is expected the real one and the saved base one will be\nmostly the same, so sharing is really necessary.\n"},{"id":"240369","messageId":"CAPig+cStwY3Evy8036Qetmz4Oz0t948=d9C3neBMsi6PbJjOGQ@mail.gmail.com","threadId":"36517","inReplyTo":"1398682553-11634-19-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 18/32] read-cache: mark new entries for split index","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-04-30T20:35:54Z","receivedAt":"2014-04-30T20:35:54Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 28, 2014 at 6:55 AM, Nguyễn Thái Ngọc Duy wrote:\n> Make sure entry addition does not lead to unifying the index. We don't\n> need to explicitly keep track of new entries. If ce->index is zero,\n> they're new. Otherwise it's unlikely that they are new, but we'll do a\n> through check later at writing time.\n\ns/through/thorough/\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> read-cache.c | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n>\n> diff --git a/read-cache.c b/read-cache.c\n> index ff889ad..2f2e0c1 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -38,7 +38,8 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n> #define CACHE_EXT_LINK 0x6c696e6b /* \"link\" */\n>\n> /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n> -#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED)\n> +#define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n> + CE_ENTRY_ADDED)\n>\n> struct index_state the_index;\n> static const char *alternate_index_output;\n> --\n> 1.9.1.346.ga2b5940\n"},{"id":"240370","messageId":"53616185.9000208@bbn.com","threadId":"36517","inReplyTo":"1398682553-11634-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Richard Hansen","fromEmail":"rhansen@bbn.com","sentAt":"2014-04-30T20:48:05Z","receivedAt":"2014-04-30T20:48:05Z","isPatch":true,"sender":{"key":"rhansen@rhansen.org","avatar":null},"body":"On 2014-04-28 06:55, Nguyễn Thái Ngọc Duy wrote:\n> From the user point of view, this reduces the writable size of index\n> down to the number of updated files. For example my webkit index v4 is\n> 14MB. With a fresh split, I only have to update an index of 200KB.\n> Every file I touch will add about 80 bytes to that. As long as I don't\n> touch every single tracked file in my worktree, I should not pay\n> penalty for writing 14MB index file on every operation.\n\nI played around with these changes a bit and have some questions:\n\n * These changes should only affect performance when the index is\n updated, right? In other words, if I do \"git status; git status\"\n the second \"git status\" shouldn't update the index and therefore\n shouldn't have a noticeable performance improvement relative to Git\n without these patches. Right?\n\n * Do you have any before/after benchmark results you can share?\n\n * Are there any benchmark scripts I can use to test it out in my own\n repositories?\n\n * Is there a debug utility I can use to examine the contents of the\n index and sharedindex.* files in a more human-readable way?\n\nI'm asking because in my (very basic) tests I noticed that with the\nfollowing command:\n\n git status; time git status\n\nthe second \"git status\" had an unexpected ~20% performance improvement\nin my repo relative to a build without your patches. The second \"git\nstatus\" in the following command also had about a ~20% performance\nimprovement:\n\n git status; touch file-in-index; time git status\n\nSo it seems like the patches did improve performance somewhat, but in\nways I wasn't expecting. (I'm not entirely certain my benchmark method\nis sound.)\n\nThanks,\nRichard\n"},{"id":"240393","messageId":"20140501000905.GA2107@lanh","threadId":"36517","inReplyTo":"53616185.9000208@bbn.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-01T00:09:05Z","receivedAt":"2014-05-01T00:09:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Apr 30, 2014 at 04:48:05PM -0400, Richard Hansen wrote:\n> I played around with these changes a bit and have some questions:\n> \n> * These changes should only affect performance when the index is\n> updated, right? In other words, if I do \"git status; git status\"\n> the second \"git status\" shouldn't update the index and therefore\n> shouldn't have a noticeable performance improvement relative to Git\n> without these patches. Right?\n\nYes, provided that other factors in \"git status\" give stable numbers\ntoo.\n\n> * Do you have any before/after benchmark results you can share?\n\nI did not pay much attention to benchmarking because the index size is\npretty much the deciding factor to write performance (of course too\nmuch computation could add up to that..). But here it is with the 14MB\nwebkit index, about 182k files. The lines of interest are the first\none (read_cache) and forth (update_index_if_able).\n\nWith normal index\n\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 278.382ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 489.275ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 6.191ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 12.321ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 12.733ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 98.043ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 915.331ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m1.727s\nuser 0m0.809s\nsys 0m0.915s\npclouds@lanh ~/d/webkit $ touch wscript\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 276.307ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 504.034ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 356.122ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 11.870ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 10.077ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 96.205ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 899.425ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m2.071s\nuser 0m1.115s\nsys 0m0.953s\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 279.424ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 484.303ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 5.288ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 13.927ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 12.507ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 98.220ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 920.985ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m1.731s\nuser 0m0.830s\nsys 0m0.897s\n\nAnd with split index:\n\npclouds@lanh ~/d/webkit $ time ~/w/git/git update-index --split-index\n\nreal 0m0.660s\nuser 0m0.601s\nsys 0m0.058s\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 281.211ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.003ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 479.629ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 5.489ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 12.611ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 11.235ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 96.086ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 894.489ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m1.697s\nuser 0m0.813s\nsys 0m0.881s\npclouds@lanh ~/d/webkit $ touch wscript\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 291.411ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.003ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 475.144ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 24.348ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 12.440ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 10.400ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 97.147ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 907.240ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m1.734s\nuser 0m0.842s\nsys 0m0.888s\npclouds@lanh ~/d/webkit $ time ~/w/git/git status >/dev/null\n 281.119ms gitmodules_config:199 if (read_cache() < 0) die(\"index file\n 0.004ms cmd_status:1294 read_cache_preload(&s.pathspec);\n 479.702ms cmd_status:1295 refresh_index(&the_index, REFRESH_QUIE\n 5.061ms cmd_status:1299 update_index_if_able(&the_index, &inde\n 12.220ms wt_status_collect:616 wt_status_collect_changes_worktree(s)\n 11.408ms wt_status_collect:621 wt_status_collect_changes_index(s)\n 95.374ms lazy_init_name_hash:136 { int nr; if (istate->name_hash_initia\n 896.931ms wt_status_collect:622 wt_status_collect_untracked(s)\n\nreal 0m1.700s\nuser 0m0.809s\nsys 0m0.888s\n\n> * Are there any benchmark scripts I can use to test it out in my own\n> repositories?\n\nYou could use the patch I used to generate the timing above. Patch at\nthe bottom of this mail.\n\n> * Is there a debug utility I can use to examine the contents of the\n> index and sharedindex.* files in a more human-readable way?\n\ntest-dump-split-index \n\nwill show you the content of \"index\". Entries without names are\nreplaced entries. Entries with them are added. They are followed by\nreplace/delete bitmaps printed out. So pretty much everything stored\nin $GIT_DIR/index in human-readable format.\n\ngit rev-parse --shared-index-path will give you the path to\nsharedindex. ls-files --stage can be used to show that.\n\n> I'm asking because in my (very basic) tests I noticed that with the\n> following command:\n> \n> git status; time git status\n> \n> the second \"git status\" had an unexpected ~20% performance improvement\n> in my repo relative to a build without your patches. The second \"git\n> status\" in the following command also had about a ~20% performance\n> improvement:\n> \n> git status; touch file-in-index; time git status\n> \n> So it seems like the patches did improve performance somewhat, but in\n> ways I wasn't expecting. (I'm not entirely certain my benchmark method\n> is sound.)\n\ngit-status is a complex operation. If you want to focus on this\nwriting part only, \"git update-index\" may be better.\n\nAnd the timing patch:\n\n-- 8< --\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 243b0c3..d680a44 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1291,12 +1291,12 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \t\t PATHSPEC_PREFER_FULL,\n \t\t prefix, argv);\n \n-\tread_cache_preload(&s.pathspec);\n-\trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);\n+\tTIME(read_cache_preload(&s.pathspec););\n+\tTIME(refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL););\n \n \tfd = hold_locked_index(&index_lock, 0);\n \tif (0 <= fd)\n-\t\tupdate_index_if_able(&the_index, &index_lock);\n+\t\tTIME(update_index_if_able(&the_index, &index_lock));\n \n \ts.is_initial = get_sha1(s.reference, sha1) ? 1 : 0;\n \ts.ignore_submodule_arg = ignore_submodule_arg;\ndiff --git a/cache.h b/cache.h\nindex 6ad2595..99854c7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1480,4 +1480,21 @@ void stat_validity_update(struct stat_validity *sv, int fd);\n \n int versioncmp(const char *s1, const char *s2);\n \n+#define TIME(x) {\t\t\t\t\t\t\\\n+\textern int time_level__;\t\t\t\t\\\n+\tstruct timeval\ttv1, tv2;\t\t\t\t\\\n+\tint current_level = time_level__;\t\t\\\n+\tgettimeofday(&tv1, NULL);\t\t\t\t\\\n+\ttime_level__++;\t\t\t\t\t\t\\\n+\tx;\t\t\t\t\t\t\t\\\n+\ttime_level__--;\t\t\t\t\t\t\\\n+\tgettimeofday(&tv2, NULL);\t\t\t\t\\\n+\tfprintf(stderr, \"% 10.3fms%*s%s:%d %.*s\\n\",\t\t\\\n+\t\ttv2.tv_sec * 1000.0 + tv2.tv_usec / 1000.0 -\t\\\n+\t\ttv1.tv_sec * 1000.0 - tv1.tv_usec / 1000.0,\t\\\n+\t\tcurrent_level, \" \",\t\t\t\t\\\n+\t\t__FUNCTION__, __LINE__, 38, #x\t\t\t\\\n+\t\t);\t\t\t\t\t\t\\\n+\t}\n+\n #endif /* CACHE_H */\ndiff --git a/environment.c b/environment.c\nindex 5c4815d..5dcbc6b 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -63,6 +63,7 @@ int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+int time_level__;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/name-hash.c b/name-hash.c\nindex 97444d0..b9121d1 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -122,7 +122,7 @@ static int cache_entry_cmp(const struct cache_entry *ce1,\n }\n \n static void lazy_init_name_hash(struct index_state *istate)\n-{\n+\tTIME({\n \tint nr;\n \n \tif (istate->name_hash_initialized)\n@@ -133,7 +133,7 @@ static void lazy_init_name_hash(struct index_state *istate)\n \tfor (nr = 0; nr < istate->cache_nr; nr++)\n \t\thash_index_entry(istate, istate->cache[nr]);\n \tistate->name_hash_initialized = 1;\n-}\n+\t\t})\n \n void add_name_hash(struct index_state *istate, struct cache_entry *ce)\n {\ndiff --git a/submodule.c b/submodule.c\nindex b80ecac..1947f20 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -195,8 +195,8 @@ void gitmodules_config(void)\n \t\tint pos;\n \t\tstrbuf_addstr(&gitmodules_path, work_tree);\n \t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n-\t\tif (read_cache() < 0)\n-\t\t\tdie(\"index file corrupt\");\n+\t\tTIME(if (read_cache() < 0)\n+\t\t\tdie(\"index file corrupt\"););\n \t\tpos = cache_name_pos(\".gitmodules\", 11);\n \t\tif (pos < 0) { /* .gitmodules not found or isn't merged */\n \t\t\tpos = -1 - pos;\ndiff --git a/wt-status.c b/wt-status.c\nindex ec7344e..0141856 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -613,13 +613,13 @@ static void wt_status_collect_untracked(struct wt_status *s)\n \n void wt_status_collect(struct wt_status *s)\n {\n-\twt_status_collect_changes_worktree(s);\n+\tTIME(wt_status_collect_changes_worktree(s));\n \n \tif (s->is_initial)\n \t\twt_status_collect_changes_initial(s);\n \telse\n-\t\twt_status_collect_changes_index(s);\n-\twt_status_collect_untracked(s);\n+\t\tTIME(wt_status_collect_changes_index(s));\n+\tTIME(wt_status_collect_untracked(s));\n }\n \n static void wt_status_print_unmerged(struct wt_status *s)\n-- 8< --\n"},{"id":"241134","messageId":"20140509102744.GA4939@lanh","threadId":"36517","inReplyTo":"CAJo=hJtgijOOMPbFjvTUaENw=hr0YixYmy1UkdqEd=CpLZ5L2A@mail.gmail.com","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-09T10:27:44Z","receivedAt":"2014-05-09T10:27:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote:\n> > The read penalty is not addressed here, so I still pay 14MB hashing\n> > cost. But that's an easy problem. We could cache the validated index\n> > in a daemon. Whenever git needs to load an index, it pokes the daemon.\n> > The daemon verifies that the on-disk index still has the same\n> > signature, then sends the in-mem index to git. When git updates the\n> > index, it pokes the daemon again to update in-mem index. Next time git\n> > reads the index, it does not have to pay I/O cost any more (actually\n> > it does but the cost is hidden away when you do not have to read it\n> > yet).\n> \n> If we are going this far, maybe it is worthwhile building a mmap()\n> region the daemon exports to the git client that holds the \"in memory\"\n> format of the index. Clients would mmap this PROT_READ, MAP_PRIVATE\n> and can then quickly access the base file information without doing\n> further validation, or copying the large(ish) data over a pipe.\n\nThe below patch implements such a daemon to cache the index. It takes\n91ms and 377ms to load a 25MB index with and without the daemon. I use\nshare memory instead of pipe, but the format is still \"on disk\" not\n\"in memory\" for simplicity. I think we're good even without in memory\nformat.\n\nThe daemon should work on Windows after shm_open and unix socket are\nreplaced with the equivalents. Then we could cache name-hash in it\ntoo.\n\nWith all improvements on (index v4, split index, preload index,\nuntracked cache, read-cache daemon), \"git status\" goes from 1.8s to\n0.6s on webkit.git (the CPU is clocked at 800 MHz so this is close to\n\"poor machine\" case).\n\nTime to convince Junio it's good and push bit by bit to master :)\n\n-- 8< --\ndiff --git a/.gitignore b/.gitignore\nindex 70992a4..07e0cb6 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -110,6 +110,7 @@\n /git-pull\n /git-push\n /git-quiltimport\n+/git-read-cache--daemon\n /git-read-tree\n /git-rebase\n /git-rebase--am\ndiff --git a/Makefile b/Makefile\nindex 028749b..98d22de 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1502,6 +1502,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS += -DHAVE_SHM\n+\tEXTLIBS += -lrt\n+\tPROGRAM_OBJS += read-cache--daemon.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\ndiff --git a/cache.h b/cache.h\nindex 99854c7..5251bda 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -290,10 +290,14 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n+\t\t poke_daemon : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 23a8803..b6a37e5 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f6d3a46..b2116ab 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -723,4 +723,12 @@ struct tm *git_gmtime_r(const time_t *, struct tm *);\n #define gmtime_r git_gmtime_r\n #endif\n \n+#ifndef HAVE_SHM\n+static inline int shm_open(const char *path, int flags, int mode)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+#endif\n+\n #endif\ndiff --git a/read-cache--daemon.c b/read-cache--daemon.c\nnew file mode 100644\nindex 0000000..32a5aa7\n--- /dev/null\n+++ b/read-cache--daemon.c\n@@ -0,0 +1,168 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+#include \"unix-socket.h\"\n+#include \"split-index.h\"\n+#include \"pkt-line.h\"\n+\n+static char *socket_path;\n+static struct strbuf shm_index = STRBUF_INIT;\n+static struct strbuf shm_sharedindex = STRBUF_INIT;\n+\n+static void cleanup_socket(void)\n+{\n+\tif (socket_path)\n+\t\tunlink(socket_path);\n+\tif (shm_index.len)\n+\t\tshm_unlink(shm_index.buf);\n+\tif (shm_sharedindex.len)\n+\t\tshm_unlink(shm_sharedindex.buf);\n+}\n+\n+static void cleanup_socket_on_signal(int sig)\n+{\n+\tcleanup_socket();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static void share_index(struct index_state *istate, struct strbuf *shm_path)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *map;\n+\tint fd;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s\", sha1_to_hex(istate->sha1));\n+\tif (shm_path->len && strcmp(sb.buf, shm_path->buf)) {\n+\t\tshm_unlink(shm_path->buf);\n+\t\tstrbuf_reset(shm_path);\n+\t}\n+\tfd = shm_open(sb.buf, O_RDWR | O_CREAT | O_TRUNC, 0700);\n+\tif (fd < 0)\n+\t\treturn;\n+\t/*\n+\t * We \"lock\" the shm in preparation by set its size larger\n+\t * than expected. The reader is supposed to check the size and\n+\t * ignore if shm size is different than the actual file size\n+\t */\n+\tif (ftruncate(fd, istate->mmap_size + 1)) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tstrbuf_addbuf(shm_path, &sb);\n+\tmap = xmmap(NULL, istate->mmap_size, PROT_READ | PROT_WRITE,\n+\t\t MAP_SHARED, fd, 0);\n+\tif (map == MAP_FAILED) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tmemcpy(map, istate->mmap, istate->mmap_size);\n+\tmunmap(map, istate->mmap_size);\n+\t/* Now \"unlock\" it */\n+\tif (ftruncate(fd, istate->mmap_size)) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tclose(fd);\n+}\n+\n+static void refresh()\n+{\n+\tthe_index.keep_mmap = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(\"could not read index\");\n+\tshare_index(&the_index, &shm_index);\n+\tif (the_index.split_index &&\n+\t the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_sharedindex);\n+\tdiscard_index(&the_index);\n+\tfprintf(stderr, \"refreshed\\n\");\n+}\n+\n+static unsigned long next;\n+static int serve_cache_loop(int fd)\n+{\n+\tstruct pollfd pfd;\n+\tunsigned long now = time(NULL);\n+\n+\tif (now > next)\n+\t\treturn 0;\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLIN;\n+\tif (poll(&pfd, 1, 1000 * (next - now)) < 0) {\n+\t\tif (errno != EINTR)\n+\t\t\tdie_errno(\"poll failed\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (pfd.revents & POLLIN) {\n+\t\tint client = accept(fd, NULL, NULL);\n+\t\tif (client < 0) {\n+\t\t\twarning(\"accept failed: %s\", strerror(errno));\n+\t\t\treturn 1;\n+\t\t}\n+\t\trefresh();\n+\t\tclose(client);\n+\t\tnext = now + 600;\n+\t}\n+\treturn 1;\n+}\n+\n+static void serve_cache(const char *socket_path)\n+{\n+\tint fd;\n+\n+\tfd = unix_stream_listen(socket_path);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n+\n+\trefresh();\n+\n+\tprintf(\"ok\\n\");\n+\tfclose(stdout);\n+\n+\tnext = time(NULL) + 600;\n+\twhile (serve_cache_loop(fd))\n+\t\t; /* nothing */\n+\n+\tclose(fd);\n+\tunlink(socket_path);\n+}\n+\n+static void check_socket_directory(const char *path)\n+{\n+\tstruct stat st;\n+\tchar *path_copy = xstrdup(path);\n+\tchar *dir = dirname(path_copy);\n+\n+\tif (!stat(dir, &st)) {\n+\t\tfree(path_copy);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * We must be sure to create the directory with the correct mode,\n+\t * not just chmod it after the fact; otherwise, there is a race\n+\t * condition in which somebody can chdir to it, sleep, then try to open\n+\t * our protected socket.\n+\t */\n+\tif (safe_create_leading_directories_const(dir) < 0)\n+\t\tdie_errno(\"unable to create directories for '%s'\", dir);\n+\tif (mkdir(dir, 0700) < 0)\n+\t\tdie_errno(\"unable to mkdir '%s'\", dir);\n+\tfree(path_copy);\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tsetup_git_directory();\n+\tsocket_path = git_pathdup(\"daemon/index\");\n+\tcheck_socket_directory(socket_path);\n+\tatexit(cleanup_socket);\n+\tsigchain_push_common(cleanup_socket_on_signal);\n+\tserve_cache(socket_path);\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex 42eac62..7ab6fb5 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -16,6 +16,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"sigchain.h\"\n+#include \"unix-socket.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -1332,6 +1333,8 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n+\tif (!size)\n+\t\treturn 0;\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, hdr, size - 20);\n \tgit_SHA1_Final(sha1, &c);\n@@ -1462,6 +1465,35 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void *try_shm(void *mmap, size_t mmap_size)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *new_mmap;\n+\tstruct stat st;\n+\tint fd;\n+\n+\tif (mmap_size <= 20)\n+\t\treturn mmap;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s\",\n+\t\t sha1_to_hex((unsigned char *)mmap + mmap_size - 20));\n+\tfd = shm_open(sb.buf, O_RDONLY, 0777);\n+\tstrbuf_release(&sb);\n+\tif (fd < 0)\n+\t\treturn mmap;\n+\tif (fstat(fd, &st) || st.st_size != mmap_size) {\n+\t\tclose(fd);\n+\t\treturn mmap;\n+\t}\n+\tnew_mmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tif (new_mmap == MAP_FAILED)\n+\t\treturn mmap;\n+\tmunmap(mmap, mmap_size);\n+\treturn new_mmap;\n+}\n+\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1469,7 +1501,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tstruct stat st;\n \tunsigned long src_offset;\n \tstruct cache_header *hdr;\n-\tvoid *mmap;\n+\tvoid *mmap, *old_mmap;\n \tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n@@ -1495,11 +1527,23 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n+\told_mmap = mmap;\n+\tmmap = try_shm(old_mmap, mmap_size);\n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n-\t\tgoto unmap;\n+\tif (old_mmap == mmap) {\n+\t\tif (verify_hdr(hdr, mmap_size) < 0)\n+\t\t\tgoto unmap;\n+\t} else {\n+\t\tif (verify_hdr(hdr, 0) < 0)\n+\t\t\tgoto unmap;\n+\t\tistate->poke_daemon = 1;\n+\t}\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n@@ -1547,10 +1591,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n+\tistate->mmap = NULL;\n \tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n@@ -1576,6 +1622,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1618,6 +1665,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \treturn 0;\n }\n@@ -2071,12 +2122,14 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t unsigned flags)\n {\n \tstruct split_index *si = istate->split_index;\n+\tint ret;\n \n \tif (!si || alternate_index_output ||\n \t (istate->cache_changed & ~EXTMASK)) {\n \t\tif (si)\n \t\t\thashclr(si->base_sha1);\n-\t\treturn do_write_locked_index(istate, lock, flags);\n+\t\tret = do_write_locked_index(istate, lock, flags);\n+\t\tgoto done;\n \t}\n \n \tif (getenv(\"GIT_TEST_SPLIT_INDEX\")) {\n@@ -2090,7 +2143,13 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t\treturn ret;\n \t}\n \n-\treturn write_split_index(istate, lock, flags);\n+\tret = write_split_index(istate, lock, flags);\n+done:\n+\tif (!ret && istate->poke_daemon) {\n+\t\tint fd = unix_stream_connect(git_path(\"daemon/index\"));\n+\t\tclose(fd);\n+\t}\n+\treturn ret;\n }\n \n /*\n-- 8< --\n"},{"id":"241187","messageId":"xmqqha4yzvjp.fsf@gitster.dls.corp.google.com","threadId":"36517","inReplyTo":"20140509102744.GA4939@lanh","subject":"Re: [PATCH 00/32] Split index mode for very large indexes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-05-09T17:55:22Z","receivedAt":"2014-05-09T17:55:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Mon, Apr 28, 2014 at 02:18:44PM -0700, Shawn Pearce wrote:\n>> > The read penalty is not addressed here, so I still pay 14MB hashing\n>> > cost. But that's an easy problem. We could cache the validated index\n>> > in a daemon. Whenever git needs to load an index, it pokes the daemon.\n>> > The daemon verifies that the on-disk index still has the same\n>> > signature, then sends the in-mem index to git. When git updates the\n>> > index, it pokes the daemon again to update in-mem index. Next time git\n>> > reads the index, it does not have to pay I/O cost any more (actually\n>> > it does but the cost is hidden away when you do not have to read it\n>> > yet).\n>> \n>> If we are going this far, maybe it is worthwhile building a mmap()\n>> region the daemon exports to the git client that holds the \"in memory\"\n>> format of the index. Clients would mmap this PROT_READ, MAP_PRIVATE\n>> and can then quickly access the base file information without doing\n>> further validation, or copying the large(ish) data over a pipe.\n>\n> The below patch implements such a daemon to cache the index. It takes\n> 91ms and 377ms to load a 25MB index with and without the daemon. I use\n> share memory instead of pipe, but the format is still \"on disk\" not\n> \"in memory\" for simplicity. I think we're good even without in memory\n> format.\n\nInteresting ;-).\n"},{"id":"241345","messageId":"1399979737-8577-1-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"CAJo=hJtgijOOMPbFjvTUaENw=hr0YixYmy1UkdqEd=CpLZ5L2A@mail.gmail.com","subject":"[PATCH 0/8] Speed up cache loading time","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:27Z","receivedAt":"2014-05-13T11:15:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, May 9, 2014 at 5:27 PM, Duy Nguyen wrote:\n> The below patch implements such a daemon to cache the index. It takes\n> 91ms and 377ms to load a 25MB index with and without the daemon. I use\n> share memory instead of pipe, but the format is still \"on disk\" not\n> \"in memory\" for simplicity. I think we're good even without in memory\n> format.\n\nHere is a better version (on top of split-index). I duplicated webkit\nindex 8 times to get its size to 199MB (version 2), close to what\nFacebook tried last time [1]. read_cache() on index v2, v4, with the\ndaemon caching v2 and v4 respectively is 2994.861ms (199MB index\nfile), 2245.113ms (118MB) and 663.399ms and 880.935ms. The best number\nis 4.5 times better the worst.\n\nThat is clocked at 800 MHz. A repository at this size deserves a\nbetter CPU. At 2.5 GHz we spend 183.228ms on loading the index. A\nreasonable number to me. If we scale other parts of git-status as well\nas this, we should be able to make \"git status\" within 1 or 2 seconds.\n\nThe tested index does not have fully populated cache-tree so real\nworld numbers could be a bit higher.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/189776/focus=190156\n\nNguyễn Thái Ngọc Duy (8):\n read-cache: allow to keep mmap'd memory after reading\n unix-socket: stub impl. for platforms with no unix socket support\n daemonize: set a flag before exiting the main process\n Add read-cache--daemon for caching index and related stuff\n read-cache: try index data from shared memory\n read-cache--daemon: do not read index from shared memory\n read-cache: skip verifying trailing SHA-1 on cached index\n read-cache: inform the daemon that the index has been updated\n\n .gitignore | 1 +\n Documentation/config.txt | 4 +\n Documentation/git-read-cache--daemon.txt (new) | 27 ++++\n Makefile | 8 +\n builtin/gc.c | 2 +-\n cache.h | 7 +-\n config.c | 12 ++\n config.mak.uname | 1 +\n daemon.c | 2 +-\n environment.c | 1 +\n read-cache--daemon.c (new) | 208 +++++++++++++++++++++++++\n read-cache.c | 116 +++++++++++++-\n setup.c | 4 +-\n submodule.c | 1 +\n unix-socket.h | 18 +++\n wrapper.c | 14 ++\n 16 files changed, 414 insertions(+), 12 deletions(-)\n create mode 100644 Documentation/git-read-cache--daemon.txt\n create mode 100644 read-cache--daemon.c\n\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241346","messageId":"1399979737-8577-2-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/8] read-cache: allow to keep mmap'd memory after reading","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:28Z","receivedAt":"2014-05-13T11:15:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 3 +++\n read-cache.c | 13 ++++++++++++-\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex c6b7770..6549e02 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -290,10 +290,13 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 342fe52..a5031f3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1495,6 +1495,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n \thdr = mmap;\n@@ -1547,10 +1551,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n+\tistate->mmap = NULL;\n \tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n@@ -1576,6 +1582,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t git_path(\"sharedindex.%s\",\n \t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1618,6 +1625,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \treturn 0;\n }\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241347","messageId":"1399979737-8577-3-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/3] Add read-cache--daemon","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:29Z","receivedAt":"2014-05-13T11:15:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"---\n .gitignore | 1 +\n Makefile | 6 ++\n config.mak.uname | 1 +\n git-compat-util.h | 8 +++\n read-cache--daemon.c (new) | 167 +++++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 183 insertions(+)\n create mode 100644 read-cache--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 70992a4..07e0cb6 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -110,6 +110,7 @@\n /git-pull\n /git-push\n /git-quiltimport\n+/git-read-cache--daemon\n /git-read-tree\n /git-rebase\n /git-rebase--am\ndiff --git a/Makefile b/Makefile\nindex 028749b..98d22de 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1502,6 +1502,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS += -DHAVE_SHM\n+\tEXTLIBS += -lrt\n+\tPROGRAM_OBJS += read-cache--daemon.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 23a8803..b6a37e5 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f6d3a46..b2116ab 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -723,4 +723,12 @@ struct tm *git_gmtime_r(const time_t *, struct tm *);\n #define gmtime_r git_gmtime_r\n #endif\n \n+#ifndef HAVE_SHM\n+static inline int shm_open(const char *path, int flags, int mode)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+#endif\n+\n #endif\ndiff --git a/read-cache--daemon.c b/read-cache--daemon.c\nnew file mode 100644\nindex 0000000..52b4067\n--- /dev/null\n+++ b/read-cache--daemon.c\n@@ -0,0 +1,167 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+#include \"unix-socket.h\"\n+#include \"split-index.h\"\n+#include \"pkt-line.h\"\n+\n+static char *socket_path;\n+static struct strbuf shm_index = STRBUF_INIT;\n+static struct strbuf shm_sharedindex = STRBUF_INIT;\n+\n+static void cleanup_socket(void)\n+{\n+\tif (socket_path)\n+\t\tunlink(socket_path);\n+\tif (shm_index.len)\n+\t\tshm_unlink(shm_index.buf);\n+\tif (shm_sharedindex.len)\n+\t\tshm_unlink(shm_sharedindex.buf);\n+}\n+\n+static void cleanup_socket_on_signal(int sig)\n+{\n+\tcleanup_socket();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static void share_index(struct index_state *istate, struct strbuf *shm_path)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *map;\n+\tint fd;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s\", sha1_to_hex(istate->sha1));\n+\tif (shm_path->len && strcmp(sb.buf, shm_path->buf)) {\n+\t\tshm_unlink(shm_path->buf);\n+\t\tstrbuf_reset(shm_path);\n+\t}\n+\tfd = shm_open(sb.buf, O_RDWR | O_CREAT | O_TRUNC, 0700);\n+\tif (fd < 0)\n+\t\treturn;\n+\t/*\n+\t * We \"lock\" the shm in preparation by set its size larger\n+\t * than expected. The reader is supposed to check the size and\n+\t * ignore if shm size is different than the actual file size\n+\t */\n+\tif (ftruncate(fd, istate->mmap_size + 1)) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tstrbuf_addbuf(shm_path, &sb);\n+\tmap = xmmap(NULL, istate->mmap_size, PROT_READ | PROT_WRITE,\n+\t\t MAP_SHARED, fd, 0);\n+\tif (map == MAP_FAILED) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tmemcpy(map, istate->mmap, istate->mmap_size);\n+\tmunmap(map, istate->mmap_size);\n+\t/* Now \"unlock\" it */\n+\tif (ftruncate(fd, istate->mmap_size)) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn;\n+\t}\n+\tclose(fd);\n+}\n+\n+static void refresh()\n+{\n+\tthe_index.keep_mmap = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(\"could not read index\");\n+\tshare_index(&the_index, &shm_index);\n+\tif (the_index.split_index &&\n+\t the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_sharedindex);\n+\tdiscard_index(&the_index);\n+}\n+\n+static unsigned long next;\n+static int serve_cache_loop(int fd)\n+{\n+\tstruct pollfd pfd;\n+\tunsigned long now = time(NULL);\n+\n+\tif (now > next)\n+\t\treturn 0;\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLIN;\n+\tif (poll(&pfd, 1, 1000 * (next - now)) < 0) {\n+\t\tif (errno != EINTR)\n+\t\t\tdie_errno(\"poll failed\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (pfd.revents & POLLIN) {\n+\t\tint client = accept(fd, NULL, NULL);\n+\t\tif (client < 0) {\n+\t\t\twarning(\"accept failed: %s\", strerror(errno));\n+\t\t\treturn 1;\n+\t\t}\n+\t\trefresh();\n+\t\tclose(client);\n+\t\tnext = now + 600;\n+\t}\n+\treturn 1;\n+}\n+\n+static void serve_cache(const char *socket_path)\n+{\n+\tint fd;\n+\n+\tfd = unix_stream_listen(socket_path);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n+\n+\trefresh();\n+\n+\tprintf(\"ok\\n\");\n+\tfclose(stdout);\n+\n+\tnext = time(NULL) + 600;\n+\twhile (serve_cache_loop(fd))\n+\t\t; /* nothing */\n+\n+\tclose(fd);\n+\tunlink(socket_path);\n+}\n+\n+static void check_socket_directory(const char *path)\n+{\n+\tstruct stat st;\n+\tchar *path_copy = xstrdup(path);\n+\tchar *dir = dirname(path_copy);\n+\n+\tif (!stat(dir, &st)) {\n+\t\tfree(path_copy);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * We must be sure to create the directory with the correct mode,\n+\t * not just chmod it after the fact; otherwise, there is a race\n+\t * condition in which somebody can chdir to it, sleep, then try to open\n+\t * our protected socket.\n+\t */\n+\tif (safe_create_leading_directories_const(dir) < 0)\n+\t\tdie_errno(\"unable to create directories for '%s'\", dir);\n+\tif (mkdir(dir, 0700) < 0)\n+\t\tdie_errno(\"unable to mkdir '%s'\", dir);\n+\tfree(path_copy);\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tsetup_git_directory();\n+\tsocket_path = git_pathdup(\"daemon/index\");\n+\tcheck_socket_directory(socket_path);\n+\tatexit(cleanup_socket);\n+\tsigchain_push_common(cleanup_socket_on_signal);\n+\tserve_cache(socket_path);\n+\treturn 0;\n+}\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241348","messageId":"1399979737-8577-4-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/8] unix-socket: stub impl. for platforms with no unix socket support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:30Z","receivedAt":"2014-05-13T11:15:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"With this we can make unix_stream_* calls without #ifdef.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Makefile | 2 ++\n unix-socket.h | 18 ++++++++++++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 028749b..d0a2b4b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1417,6 +1417,8 @@ ifndef NO_UNIX_SOCKETS\n \tLIB_H += unix-socket.h\n \tPROGRAM_OBJS += credential-cache.o\n \tPROGRAM_OBJS += credential-cache--daemon.o\n+else\n+\tBASIC_CFLAGS += -DNO_UNIX_SOCKETS\n endif\n \n ifdef NO_ICONV\ndiff --git a/unix-socket.h b/unix-socket.h\nindex e271aee..f1cba70 100644\n--- a/unix-socket.h\n+++ b/unix-socket.h\n@@ -1,7 +1,25 @@\n #ifndef UNIX_SOCKET_H\n #define UNIX_SOCKET_H\n \n+#ifndef NO_UNIX_SOCKETS\n+\n int unix_stream_connect(const char *path);\n int unix_stream_listen(const char *path);\n \n+#else\n+\n+static inline int unix_stream_connect(const char *path)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+static inline int unix_stream_listen(const char *path)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+#endif\n+\n #endif /* UNIX_SOCKET_H */\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241349","messageId":"1399979737-8577-5-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/8] daemonize: set a flag before exiting the main process","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:31Z","receivedAt":"2014-05-13T11:15:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/gc.c | 2 +-\n cache.h | 2 +-\n daemon.c | 2 +-\n setup.c | 4 +++-\n 4 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 85f5c2b..50275af 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -325,7 +325,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t * failure to daemonize is ok, we'll continue\n \t\t\t * in foreground\n \t\t\t */\n-\t\t\tdaemonize();\n+\t\t\tdaemonize(NULL);\n \t} else\n \t\tadd_repack_all_option();\n \ndiff --git a/cache.h b/cache.h\nindex 6549e02..d0ff11c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -450,7 +450,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n-extern int daemonize(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex eba1255..2650504 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1311,7 +1311,7 @@ int main(int argc, char **argv)\n \t\treturn execute();\n \n \tif (detach) {\n-\t\tif (daemonize())\n+\t\tif (daemonize(NULL))\n \t\t\tdie(\"--detach not supported on this platform\");\n \t} else\n \t\tsanitize_stdfds();\ndiff --git a/setup.c b/setup.c\nindex 613e3b3..e8e129a 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -842,7 +842,7 @@ void sanitize_stdfds(void)\n \t\tclose(fd);\n }\n \n-int daemonize(void)\n+int daemonize(int *daemonized)\n {\n #ifdef NO_POSIX_GOODIES\n \terrno = ENOSYS;\n@@ -854,6 +854,8 @@ int daemonize(void)\n \t\tcase -1:\n \t\t\tdie_errno(\"fork failed\");\n \t\tdefault:\n+\t\t\tif (daemonized)\n+\t\t\t\t*daemonized = 1;\n \t\t\texit(0);\n \t}\n \tif (setsid() == -1)\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241350","messageId":"1399979737-8577-6-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/3] read-cache: try index data from shared memory","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:32Z","receivedAt":"2014-05-13T11:15:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"---\n read-cache.c | 30 ++++++++++++++++++++++++++++++\n 1 file changed, 30 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9e742c7..3100a59 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1462,6 +1462,35 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void *try_shm(void *mmap, size_t mmap_size)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *new_mmap;\n+\tstruct stat st;\n+\tint fd;\n+\n+\tif (mmap_size <= 20)\n+\t\treturn mmap;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s\",\n+\t\t sha1_to_hex((unsigned char *)mmap + mmap_size - 20));\n+\tfd = shm_open(sb.buf, O_RDONLY, 0777);\n+\tstrbuf_release(&sb);\n+\tif (fd < 0)\n+\t\treturn mmap;\n+\tif (fstat(fd, &st) || st.st_size != mmap_size) {\n+\t\tclose(fd);\n+\t\treturn mmap;\n+\t}\n+\tnew_mmap = xmmap(NULL, mmap_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tif (new_mmap == MAP_FAILED)\n+\t\treturn mmap;\n+\tmunmap(mmap, mmap_size);\n+\treturn new_mmap;\n+}\n+\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1501,6 +1530,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t}\n \tclose(fd);\n \n+\tmmap = try_shm(mmap, mmap_size);\n \thdr = mmap;\n \tif (verify_hdr(hdr, mmap_size) < 0)\n \t\tgoto unmap;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241351","messageId":"1399979737-8577-7-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/8] Add read-cache--daemon for caching index and related stuff","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:33Z","receivedAt":"2014-05-13T11:15:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The name of the shared memory folows the template \"/git-index-\"\nwhere is the trailing SHA-1 of the index file. If such a shared\nmemory exists, it contains the same index content as on disk. The\ncontent is already validated by the daemon. Note that it does not\nnecessarily use the same format as the on-disk version. The content\ncould be in a format that can be parsed much faster, or even reused\nwithout parsing).\n\nWhile preparing the shm object, the daemon would keep the shm object\n\"/git-index-.lock\". After \"git-index-\" is ready, the\n\".lock\" object is removed. A shared object must not be updated\nafterwards. So if \".lock\" does not exist, it's safe to assume that the\nassociated shm object is ready.\n\nOther info could also by cached if it's tied to the index. For\nexample, name hash could be stored in \"/git-namehash-\"..\n\nAfter Git writes a new index down, it may want to ask the daemon to\npreload the new index so next time Git runs the index is already\nvalidated and in memory. It does so by send a command to a UNIX socket\nin $GIT_DIR/daemon/index.\n\nWindows can use its named shared memory instead of POSIX shared memory\nand probably named pipe in place of UNIX socket.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n Documentation/git-read-cache--daemon.txt (new) | 27 ++++\n Makefile | 6 +\n config.mak.uname | 1 +\n read-cache--daemon.c (new) | 207 +++++++++++++++++++++++++\n wrapper.c | 14 ++\n 6 files changed, 256 insertions(+)\n create mode 100644 Documentation/git-read-cache--daemon.txt\n create mode 100644 read-cache--daemon.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 70992a4..07e0cb6 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -110,6 +110,7 @@\n /git-pull\n /git-push\n /git-quiltimport\n+/git-read-cache--daemon\n /git-read-tree\n /git-rebase\n /git-rebase--am\ndiff --git a/Documentation/git-read-cache--daemon.txt b/Documentation/git-read-cache--daemon.txt\nnew file mode 100644\nindex 0000000..1b05be4\n--- /dev/null\n+++ b/Documentation/git-read-cache--daemon.txt\n@@ -0,0 +1,27 @@\n+git-read-cache--daemon(1)\n+=============\n+\n+NAME\n+----\n+git-daemon - A simple cache server for speeding up index file access\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git daemon' [--detach]\n+\n+DESCRIPTION\n+-----------\n+Keep the index file in memory for faster access. This daemon is per\n+repository. Note that core.useReadCacheDaemon must be set for Git to\n+contact the daemon. This daemon is only available on POSIX system with\n+shared memory support (e.g. Linux)\n+\n+OPTIONS\n+-------\n+--detach::\n+\tDetach from the shell.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex d0a2b4b..a44ab0b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1504,6 +1504,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS += -DHAVE_SHM\n+\tEXTLIBS += -lrt\n+\tPROGRAM_OBJS += read-cache--daemon.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 23a8803..b6a37e5 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -33,6 +33,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/read-cache--daemon.c b/read-cache--daemon.c\nnew file mode 100644\nindex 0000000..4531978\n--- /dev/null\n+++ b/read-cache--daemon.c\n@@ -0,0 +1,207 @@\n+#include \"cache.h\"\n+#include \"sigchain.h\"\n+#include \"unix-socket.h\"\n+#include \"split-index.h\"\n+#include \"pkt-line.h\"\n+\n+static char *socket_path;\n+static struct strbuf shm_index = STRBUF_INIT;\n+static struct strbuf shm_sharedindex = STRBUF_INIT;\n+static struct strbuf shm_lock = STRBUF_INIT;\n+static int lock_fd = -1;\n+static int daemonized;\n+\n+static void cleanup_socket(void)\n+{\n+\tif (daemonized)\n+\t\treturn;\n+\tif (socket_path)\n+\t\tunlink(socket_path);\n+\tif (shm_index.len)\n+\t\tshm_unlink(shm_index.buf);\n+\tif (shm_sharedindex.len)\n+\t\tshm_unlink(shm_sharedindex.buf);\n+\tif (lock_fd != -1)\n+\t\tclose(lock_fd);\n+\tif (shm_lock.len)\n+\t\tshm_unlink(shm_lock.buf);\n+}\n+\n+static void cleanup_socket_on_signal(int sig)\n+{\n+\tcleanup_socket();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static int do_share_index(struct index_state *istate, struct strbuf *shm_path)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tvoid *map;\n+\tint fd;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s\", sha1_to_hex(istate->sha1));\n+\tfd = shm_open(sb.buf, O_RDWR | O_CREAT | O_EXCL, 0700);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (shm_path->len) {\n+\t\tshm_unlink(shm_path->buf);\n+\t\tstrbuf_reset(shm_path);\n+\t}\n+\tif (ftruncate(fd, istate->mmap_size)) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn -1;\n+\t}\n+\tstrbuf_addbuf(shm_path, &sb);\n+\tmap = xmmap(NULL, istate->mmap_size, PROT_READ | PROT_WRITE,\n+\t\t MAP_SHARED, fd, 0);\n+\tif (map == MAP_FAILED) {\n+\t\tclose(fd);\n+\t\tshm_unlink(shm_path->buf);\n+\t\treturn -1;\n+\t}\n+\tmemcpy(map, istate->mmap, istate->mmap_size);\n+\tmunmap(map, istate->mmap_size);\n+\tfchmod(fd, 0400);\n+\tclose(fd);\n+\treturn 0;\n+}\n+\n+static void share_index(struct index_state *istate, struct strbuf *shm_path)\n+{\n+\tif (shm_lock.len)\n+\t\treturn;\n+\n+\tstrbuf_addf(&shm_lock, \"/git-index-%s.lock\", sha1_to_hex(istate->sha1));\n+\tlock_fd = shm_open(shm_lock.buf, O_CREAT | O_EXCL, 0700);\n+\tif (lock_fd < 0) {\n+\t\tstrbuf_reset(&shm_lock);\n+\t\treturn;\n+\t}\n+\tdo_share_index(istate, shm_path);\n+\tclose(lock_fd);\n+\tlock_fd = -1;\n+\tshm_unlink(shm_lock.buf);\n+\tstrbuf_reset(&shm_lock);\n+}\n+\n+static void refresh()\n+{\n+\tthe_index.keep_mmap = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(\"could not read index\");\n+\tshare_index(&the_index, &shm_index);\n+\tif (the_index.split_index &&\n+\t the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_sharedindex);\n+\tdiscard_index(&the_index);\n+}\n+\n+static void serve_one_client(int fd)\n+{\n+\tchar *buf = packet_read_line(fd, NULL);\n+\tif (!strcmp(buf, \"refresh\"))\n+\t\trefresh();\n+\telse\n+\t\tfprintf(stderr, \"unrecognized command %s\\n\", buf);\n+}\n+\n+static unsigned long next;\n+static int serve_cache_loop(int fd)\n+{\n+\tstruct pollfd pfd;\n+\tunsigned long now = time(NULL);\n+\n+\tif (now > next)\n+\t\treturn 0;\n+\n+\tpfd.fd = fd;\n+\tpfd.events = POLLIN;\n+\tif (poll(&pfd, 1, 1000 * (next - now)) < 0) {\n+\t\tif (errno != EINTR)\n+\t\t\tdie_errno(\"poll failed\");\n+\t\treturn 1;\n+\t}\n+\n+\tif (pfd.revents & POLLIN) {\n+\t\tint client = accept(fd, NULL, NULL);\n+\t\tif (client < 0) {\n+\t\t\twarning(\"accept failed: %s\", strerror(errno));\n+\t\t\treturn 1;\n+\t\t}\n+\t\tserve_one_client(client);\n+\t\tclose(client);\n+\t\tnext = now + 600;\n+\t}\n+\treturn 1;\n+}\n+\n+static void serve_cache(const char *socket_path, int detach)\n+{\n+\tint fd;\n+\n+\tfd = unix_stream_listen(socket_path);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n+\n+\trefresh();\n+\tif (detach && daemonize(&daemonized))\n+\t\tdie_errno(\"unable to detach\");\n+\n+\tnext = time(NULL) + 600;\n+\twhile (serve_cache_loop(fd))\n+\t\t; /* nothing */\n+\n+\tclose(fd);\n+\tunlink(socket_path);\n+}\n+\n+static void check_socket_directory(const char *path)\n+{\n+\tstruct stat st;\n+\tchar *path_copy = xstrdup(path);\n+\tchar *dir = dirname(path_copy);\n+\n+\tif (!stat(dir, &st)) {\n+\t\tfree(path_copy);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * We must be sure to create the directory with the correct mode,\n+\t * not just chmod it after the fact; otherwise, there is a race\n+\t * condition in which somebody can chdir to it, sleep, then try to open\n+\t * our protected socket.\n+\t */\n+\tif (safe_create_leading_directories_const(dir) < 0)\n+\t\tdie_errno(\"unable to create directories for '%s'\", dir);\n+\tif (mkdir(dir, 0700) < 0)\n+\t\tdie_errno(\"unable to mkdir '%s'\", dir);\n+\tfree(path_copy);\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tint detach = 0;\n+\tswitch (argc) {\n+\tcase 1:\n+\t\tbreak;\n+\tcase 2:\n+\t\tif (!strcmp(argv[1], \"--detach\"))\n+\t\t\tdetach = 1;\n+\t\telse\n+\t\t\tdie(\"unknown option %s\", argv[1]);\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"unexpected argument number %d\\n\", argc);\n+\t}\n+\n+\tsetup_git_directory();\n+\tsocket_path = git_pathdup(\"daemon/index\");\n+\tcheck_socket_directory(socket_path);\n+\tatexit(cleanup_socket);\n+\tsigchain_push_common(cleanup_socket_on_signal);\n+\tserve_cache(socket_path, detach);\n+\treturn 0;\n+}\ndiff --git a/wrapper.c b/wrapper.c\nindex 0cc5636..4cd7415 100644\n--- a/wrapper.c\n+++ b/wrapper.c\n@@ -455,3 +455,17 @@ struct passwd *xgetpwuid_self(void)\n \t\t errno ? strerror(errno) : _(\"no such user\"));\n \treturn pw;\n }\n+\n+#ifndef HAVE_SHM\n+int shm_open(const char *path, int flags, mode_t mode)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+\n+int shm_unlink(const char *path)\n+{\n+\terrno = ENOSYS;\n+\treturn -1;\n+}\n+#endif\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241352","messageId":"1399979737-8577-8-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/8] read-cache: try index data from shared memory","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:34Z","receivedAt":"2014-05-13T11:15:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 4 ++++\n cache.h | 1 +\n config.c | 12 ++++++++++++\n environment.c | 1 +\n read-cache.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n submodule.c | 1 +\n 6 files changed, 62 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d8b6cc9..ccbe00b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -617,6 +617,10 @@ relatively high IO latencies. With this set to 'true', Git will do the\n index comparison to the filesystem data in parallel, allowing\n overlapping IO's.\n \n+core.useReadCacheDaemon::\n+\tUse `git read-cache--daemon` to speed up index reading. See\n+\tlinkgit:git-read-cache--daemon for more information.\n+\n core.createObject::\n \tYou can set this to 'link', in which case a hardlink followed by\n \ta delete of the source are used to make sure that object creation\ndiff --git a/cache.h b/cache.h\nindex d0ff11c..fb29c7e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -603,6 +603,7 @@ extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n extern unsigned long pack_size_limit_cfg;\n+extern int use_read_cache_daemon;\n \n /*\n * Do replace refs need to be checked this run? This variable is\ndiff --git a/config.c b/config.c\nindex a30cb5c..5c832ad 100644\n--- a/config.c\n+++ b/config.c\n@@ -874,6 +874,18 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+#ifdef HAVE_SHM\n+\t/*\n+\t * Currently git-read-cache--daemon is only built when\n+\t * HAVE_SHM is set. Ignore user settings if HAVE_SHM is not\n+\t * defined.\n+\t */\n+\tif (!strcmp(var, \"core.usereadcachedaemon\")) {\n+\t\tuse_read_cache_daemon = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+#endif\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 5c4815d..b76a414 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -63,6 +63,7 @@ int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+int use_read_cache_daemon;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/read-cache.c b/read-cache.c\nindex a5031f3..0e46523 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1462,6 +1462,48 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void *try_shm(void *mmap, size_t *mmap_size)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tsize_t old_size = *mmap_size;\n+\tvoid *new_mmap;\n+\tstruct stat st;\n+\tint fd;\n+\n+\tif (old_size <= 20 || !use_read_cache_daemon)\n+\t\treturn mmap;\n+\n+\tstrbuf_addf(&sb, \"/git-index-%s.lock\",\n+\t\t sha1_to_hex((unsigned char *)mmap + old_size - 20));\n+\tfd = shm_open(sb.buf, O_RDONLY, 0777);\n+\tif (fd >= 0) {\n+\t\tclose(fd);\n+\t\treturn mmap;\n+\t}\n+\tstrbuf_setlen(&sb, sb.len - 5); /* no \".lock\" */\n+\tfd = shm_open(sb.buf, O_RDONLY, 0777);\n+\tstrbuf_release(&sb);\n+\tif (fd < 0)\n+\t\t/*\n+\t\t * git-read-cache--daemon is probably not started yet. For\n+\t\t * simplicity, only start it at the next index update, which\n+\t\t * should happen often.\n+\t\t */\n+\t\treturn mmap;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn mmap;\n+\t}\n+\tnew_mmap = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tif (new_mmap == MAP_FAILED)\n+\t\treturn mmap;\n+\tmunmap(mmap, old_size);\n+\t*mmap_size = st.st_size;\n+\treturn new_mmap;\n+}\n+\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1501,6 +1543,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t}\n \tclose(fd);\n \n+\tmmap = try_shm(mmap, &mmap_size);\n \thdr = mmap;\n \tif (verify_hdr(hdr, mmap_size) < 0)\n \t\tgoto unmap;\ndiff --git a/submodule.c b/submodule.c\nindex b80ecac..9872928 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -195,6 +195,7 @@ void gitmodules_config(void)\n \t\tint pos;\n \t\tstrbuf_addstr(&gitmodules_path, work_tree);\n \t\tstrbuf_addstr(&gitmodules_path, \"/.gitmodules\");\n+\t\tgit_config(git_default_config, NULL);\n \t\tif (read_cache() < 0)\n \t\t\tdie(\"index file corrupt\");\n \t\tpos = cache_name_pos(\".gitmodules\", 11);\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241353","messageId":"1399979737-8577-9-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 6/8] read-cache--daemon: do not read index from shared memory","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:35Z","receivedAt":"2014-05-13T11:15:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It does not hurt doing that. But it does not help anybody either.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache--daemon.c | 1 +\n read-cache.c | 2 +-\n 2 files changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache--daemon.c b/read-cache--daemon.c\nindex 4531978..bd6d84f 100644\n--- a/read-cache--daemon.c\n+++ b/read-cache--daemon.c\n@@ -145,6 +145,7 @@ static void serve_cache(const char *socket_path, int detach)\n \tif (fd < 0)\n \t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n \n+\tuse_read_cache_daemon = -1;\n \trefresh();\n \tif (detach && daemonize(&daemonized))\n \t\tdie_errno(\"unable to detach\");\ndiff --git a/read-cache.c b/read-cache.c\nindex 0e46523..4041485 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1470,7 +1470,7 @@ static void *try_shm(void *mmap, size_t *mmap_size)\n \tstruct stat st;\n \tint fd;\n \n-\tif (old_size <= 20 || !use_read_cache_daemon)\n+\tif (old_size <= 20 || use_read_cache_daemon <= 0)\n \t\treturn mmap;\n \n \tstrbuf_addf(&sb, \"/git-index-%s.lock\",\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241354","messageId":"1399979737-8577-10-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 7/8] read-cache: skip verifying trailing SHA-1 on cached index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:36Z","receivedAt":"2014-05-13T11:15:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The daemon is responsible for verifying the index before putting it in\nthe shared memory. No need to redo it again.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 4041485..e98521f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1332,6 +1332,8 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n+\tif (!size)\n+\t\treturn 0;\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, hdr, size - 20);\n \tgit_SHA1_Final(sha1, &c);\n@@ -1511,7 +1513,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tstruct stat st;\n \tunsigned long src_offset;\n \tstruct cache_header *hdr;\n-\tvoid *mmap;\n+\tvoid *mmap, *old_mmap;\n \tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n@@ -1543,9 +1545,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t}\n \tclose(fd);\n \n+\told_mmap = mmap;\n \tmmap = try_shm(mmap, &mmap_size);\n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tif (verify_hdr(hdr, old_mmap != mmap ? 0 : mmap_size) < 0)\n \t\tgoto unmap;\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241355","messageId":"1399979737-8577-11-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 8/8] read-cache: inform the daemon that the index has been updated","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:15:37Z","receivedAt":"2014-05-13T11:15:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The daemon would immediately load the new index in memory in\nbackground. Next time Git needs to read the index again, everything is\nready.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n read-cache.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++-----\n 2 files changed, 49 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex fb29c7e..3115b86 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -483,6 +483,7 @@ extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n+#define REFRESH_DAEMON\t\t(1 << 2)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n extern int unmerged_index(const struct index_state *);\ndiff --git a/read-cache.c b/read-cache.c\nindex e98521f..d5c9247 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -16,6 +16,9 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"sigchain.h\"\n+#include \"unix-socket.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t unsigned int options);\n@@ -2030,6 +2033,32 @@ void set_alternate_index_output(const char *name)\n \talternate_index_output = name;\n }\n \n+static void refresh_daemon(struct index_state *istate)\n+{\n+\tint fd;\n+\tfd = unix_stream_connect(git_path(\"daemon/index\"));\n+\tif (fd < 0) {\n+\t\tstruct child_process cp;\n+\t\tconst char *av[] = {\"read-cache--daemon\", \"--detach\", NULL };\n+\t\tmemset(&cp, 0, sizeof(cp));\n+\t\tcp.argv = av;\n+\t\tcp.git_cmd = 1;\n+\t\tcp.no_stdin = 1;\n+\t\tif (run_command(&cp))\n+\t\t\twarning(_(\"failed to start read-cache--daemon: %s\"),\n+\t\t\t\tstrerror(errno));\n+\t\treturn;\n+\t}\n+\t/*\n+\t * packet_write() could die() but unless this is from\n+\t * update_index_if_able(), we're about to exit anyway,\n+\t * probably ok to die (for now). Blocking mode is another\n+\t * problem to deal with later.\n+\t */\n+\tpacket_write(fd, \"refresh\");\n+\tclose(fd);\n+}\n+\n static int commit_locked_index(struct lock_file *lk)\n {\n \tif (alternate_index_output) {\n@@ -2052,9 +2081,22 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n \t (COMMIT_LOCK | CLOSE_LOCK));\n-\tif (flags & COMMIT_LOCK)\n-\t\treturn commit_locked_index(lock);\n-\telse if (flags & CLOSE_LOCK)\n+\tif (flags & COMMIT_LOCK) {\n+\t\tint ret;\n+\t\tint len = strlen(lock->filename) - 5; /* .lock */\n+\t\tif (!use_read_cache_daemon || len < 6 ||\n+\t\t /*\n+\t\t * do not wake the daemon when we update a temporary\n+\t\t * index. This is not a perfect test for this, but good\n+\t\t * enough.\n+\t\t */\n+\t\t strncmp(lock->filename + len - 6, \"/index\", 6))\n+\t\t\tflags &= ~REFRESH_DAEMON;\n+\t\tret = commit_locked_index(lock);\n+\t\tif (!ret && use_read_cache_daemon)\n+\t\t\trefresh_daemon(istate);\n+\t\treturn ret;\n+\t} else if (flags & CLOSE_LOCK)\n \t\treturn close_lock_file(lock);\n \telse\n \t\treturn ret;\n@@ -2066,7 +2108,7 @@ static int write_split_index(struct index_state *istate,\n {\n \tint ret;\n \tprepare_to_write_split_index(istate);\n-\tret = do_write_locked_index(istate, lock, flags);\n+\tret = do_write_locked_index(istate, lock, flags | REFRESH_DAEMON);\n \tfinish_writing_split_index(istate);\n \treturn ret;\n }\n@@ -2133,7 +2175,8 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t (istate->cache_changed & ~EXTMASK)) {\n \t\tif (si)\n \t\t\thashclr(si->base_sha1);\n-\t\treturn do_write_locked_index(istate, lock, flags);\n+\t\treturn do_write_locked_index(istate, lock,\n+\t\t\t\t\t flags | REFRESH_DAEMON);\n \t}\n \n \tif (getenv(\"GIT_TEST_SPLIT_INDEX\")) {\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241356","messageId":"1399980019-8706-1-git-send-email-pclouds@gmail.com","threadId":"36517","inReplyTo":"CAJo=hJtgijOOMPbFjvTUaENw=hr0YixYmy1UkdqEd=CpLZ5L2A@mail.gmail.com","subject":"[PATCH 9/8] even faster loading time with index version 254","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T11:20:19Z","receivedAt":"2014-05-13T11:20:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This dirty (and likely buggy) patch shows a direction of lowering load\ntime even more. Basically the shared memory now contains a clean\nmemory dump that a git process could use with little preparation\n(which also means it's tied to C Git, other implementations can't use\nthis)\n\nMemory is actually shared, git won't malloc and copy over, so even if\nthe v254 is 235 MB (larger than v2 199MB), we use less memory.\n\nWith this patch, we can get as low as 256.442ms (compared to 663ms in\n0/8) at 800 MHz, or 91ms at 2.5 GHz. Index load time should be a\nsolved problem.\n\nBut I'm not going to polish this patch and try to get it merged. I'd\nrather see a real world repository of this size first to justify\nmessing up read-cache.c even more.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 2 +\n read-cache--daemon.c | 31 +++++------\n read-cache.c | 154 ++++++++++++++++++++++++++++++++++++++++++---------\n split-index.c | 3 +\n 4 files changed, 149 insertions(+), 41 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex c246dee..7f0ef1e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -297,6 +297,8 @@ struct index_state {\n \tunsigned char sha1[20];\n \tvoid *mmap;\n \tsize_t mmap_size;\n+\tint mmap_fd;\n+\tvoid *(*allocate_254)(struct index_state *, size_t);\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache--daemon.c b/read-cache--daemon.c\nindex bd6d84f..a44bd09 100644\n--- a/read-cache--daemon.c\n+++ b/read-cache--daemon.c\n@@ -34,10 +34,19 @@ static void cleanup_socket_on_signal(int sig)\n \traise(sig);\n }\n \n+static void *allocate_254(struct index_state *istate, unsigned long size)\n+{\n+\tftruncate(istate->mmap_fd, size);\n+\tistate->mmap_size = size;\n+\tistate->mmap = xmmap(NULL, istate->mmap_size, PROT_READ | PROT_WRITE,\n+\t\t\t MAP_SHARED, istate->mmap_fd, 0);\n+\treturn istate->mmap != MAP_FAILED ? istate->mmap : NULL;\n+}\n+\n+extern int do_write_index(struct index_state *istate, int newfd, int strip_extensions);\n static int do_share_index(struct index_state *istate, struct strbuf *shm_path)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tvoid *map;\n \tint fd;\n \n \tstrbuf_addf(&sb, \"/git-index-%s\", sha1_to_hex(istate->sha1));\n@@ -48,21 +57,16 @@ static int do_share_index(struct index_state *istate, struct strbuf *shm_path)\n \t\tshm_unlink(shm_path->buf);\n \t\tstrbuf_reset(shm_path);\n \t}\n-\tif (ftruncate(fd, istate->mmap_size)) {\n-\t\tclose(fd);\n-\t\tshm_unlink(shm_path->buf);\n-\t\treturn -1;\n-\t}\n+\tistate->version = 254;\n+\tistate->allocate_254 = allocate_254;\n+\tistate->mmap_fd = fd;\n+\tdo_write_index(istate, -1, 0);\n \tstrbuf_addbuf(shm_path, &sb);\n-\tmap = xmmap(NULL, istate->mmap_size, PROT_READ | PROT_WRITE,\n-\t\t MAP_SHARED, fd, 0);\n-\tif (map == MAP_FAILED) {\n+\tif (istate->mmap == MAP_FAILED) {\n \t\tclose(fd);\n \t\tshm_unlink(shm_path->buf);\n \t\treturn -1;\n \t}\n-\tmemcpy(map, istate->mmap, istate->mmap_size);\n-\tmunmap(map, istate->mmap_size);\n \tfchmod(fd, 0400);\n \tclose(fd);\n \treturn 0;\n@@ -88,13 +92,9 @@ static void share_index(struct index_state *istate, struct strbuf *shm_path)\n \n static void refresh()\n {\n-\tthe_index.keep_mmap = 1;\n \tif (read_cache() < 0)\n \t\tdie(\"could not read index\");\n \tshare_index(&the_index, &shm_index);\n-\tif (the_index.split_index &&\n-\t the_index.split_index->base)\n-\t\tshare_index(the_index.split_index->base, &shm_sharedindex);\n \tdiscard_index(&the_index);\n }\n \n@@ -145,7 +145,6 @@ static void serve_cache(const char *socket_path, int detach)\n \tif (fd < 0)\n \t\tdie_errno(\"unable to bind to '%s'\", socket_path);\n \n-\tuse_read_cache_daemon = -1;\n \trefresh();\n \tif (detach && daemonize(&daemonized))\n \t\tdie_errno(\"unable to detach\");\ndiff --git a/read-cache.c b/read-cache.c\nindex d5c9247..4db1c30 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -61,7 +61,8 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n \n \treplace_index_entry_in_base(istate, old, ce);\n \tremove_name_hash(istate, old);\n-\tfree(old);\n+\tif (old->index != 0xffffffff) /* special mark by v254 entry writing code */\n+\t\tfree(old);\n \tset_index_entry(istate, nr, ce);\n \tce->ce_flags |= CE_UPDATE_IN_BASE;\n \tistate->cache_changed |= CE_ENTRY_CHANGED;\n@@ -1333,9 +1334,11 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n \t\treturn error(\"bad signature\");\n \thdr_version = ntohl(hdr->hdr_version);\n-\tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n+\tif (!size && hdr_version == 254)\n+\t\tfprintf(stderr, \"yeah\\n\");\t\t/* go on */\n+\telse if (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n-\tif (!size)\n+\tif (!size || hdr_version == 254)\n \t\treturn 0;\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, hdr, size - 20);\n@@ -1499,7 +1502,8 @@ static void *try_shm(void *mmap, size_t *mmap_size)\n \t\tclose(fd);\n \t\treturn mmap;\n \t}\n-\tnew_mmap = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tnew_mmap = xmmap(NULL, st.st_size, PROT_READ | PROT_WRITE,\n+\t\t\t MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \tif (new_mmap == MAP_FAILED)\n \t\treturn mmap;\n@@ -1519,6 +1523,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tvoid *mmap, *old_mmap;\n \tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\tint ver_254 = 0;\n \n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n@@ -1561,7 +1566,13 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n \n-\tif (istate->version == 4)\n+\tif (istate->version == 254) {\n+\t\tistate->version = 4;\n+\t\tver_254 = 1;\n+\t\tistate->keep_mmap = 1;\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t} else if (istate->version == 4)\n \t\tprevious_name = &previous_name_buf;\n \telse\n \t\tprevious_name = NULL;\n@@ -1573,7 +1584,14 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tunsigned long consumed;\n \n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tif (ver_254) {\n+\t\t\tce = mmap + src_offset;\n+\t\t\tconsumed =\n+\t\t\t\toffsetof(struct cache_entry, name) +\n+\t\t\t\tce_namelen(ce) + 1;\n+\t\t\tconsumed = (consumed + 7) & ~7;\n+\t\t} else\n+\t\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n \n \t\tsrc_offset += consumed;\n@@ -1655,6 +1673,8 @@ int discard_index(struct index_state *istate)\n \tint i;\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (istate->cache[i]->index == 0xffffffff)\n+\t\t\tcontinue;\n \t\tif (istate->cache[i]->index &&\n \t\t istate->split_index &&\n \t\t istate->split_index->base &&\n@@ -1696,13 +1716,51 @@ int unmerged_index(const struct index_state *istate)\n static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n static unsigned long write_buffer_len;\n \n+struct file_block {\n+\tstruct file_block *next;\n+\tchar buf[1];\n+};\n+static struct file_block *start, *end;\n+static unsigned long file_block_size;\n+#define FB_ALLOC_SIZE 65536\n+#define FB_USABLE_SIZE (FB_ALLOC_SIZE - sizeof(struct file_block *))\n+\n+static void fill_file_block(const unsigned char *buffer, unsigned int len)\n+{\n+\tif (!start) {\n+\t\tstart = end = xmalloc(FB_ALLOC_SIZE);\n+\t\tstart->next = NULL;\n+\t}\n+\n+\twhile (len) {\n+\t\tunsigned long used = file_block_size % FB_USABLE_SIZE;\n+\t\tunsigned long remaining = FB_USABLE_SIZE - used;\n+\t\tif (len < remaining) {\n+\t\t\tmemcpy(end->buf + used, buffer, len);\n+\t\t\tfile_block_size += len;\n+\t\t\treturn;\n+\t\t}\n+\t\tmemcpy(end->buf + used, buffer, remaining);\n+\t\tfile_block_size += remaining;\n+\t\tbuffer\t\t+= remaining;\n+\t\tlen\t\t-= remaining;\n+\t\tend->next\t = xmalloc(FB_ALLOC_SIZE);\n+\t\tend\t\t = end->next;\n+\t\tend->next\t = NULL;\n+\t}\n+}\n+\n static int ce_write_flush(git_SHA_CTX *context, int fd)\n {\n \tunsigned int buffered = write_buffer_len;\n \tif (buffered) {\n-\t\tgit_SHA1_Update(context, write_buffer, buffered);\n-\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n-\t\t\treturn -1;\n+\t\tif (context) {\n+\t\t\tgit_SHA1_Update(context, write_buffer, buffered);\n+\t\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\tfill_file_block(write_buffer, buffered);\n+\t\t}\n \t\twrite_buffer_len = 0;\n \t}\n \treturn 0;\n@@ -1745,7 +1803,8 @@ static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n \n \tif (left) {\n \t\twrite_buffer_len = 0;\n-\t\tgit_SHA1_Update(context, write_buffer, left);\n+\t\tif (context)\n+\t\t\tgit_SHA1_Update(context, write_buffer, left);\n \t}\n \n \t/* Flush first if not enough space for SHA1 signature */\n@@ -1756,10 +1815,18 @@ static int ce_flush(git_SHA_CTX *context, int fd, unsigned char *sha1)\n \t}\n \n \t/* Append the SHA1 signature at the end */\n-\tgit_SHA1_Final(write_buffer + left, context);\n+\tif (context)\n+\t\tgit_SHA1_Final(write_buffer + left, context);\n+\telse\n+\t\thashclr(write_buffer + left);\n \thashcpy(sha1, write_buffer + left);\n \tleft += 20;\n-\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n+\tif (context)\n+\t\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n+\telse {\n+\t\tfill_file_block(write_buffer, left);\n+\t\treturn 0;\n+\t}\n }\n \n static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n@@ -1921,10 +1988,9 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \t\trollback_lock_file(lockfile);\n }\n \n-static int do_write_index(struct index_state *istate, int newfd,\n-\t\t\t int strip_extensions)\n+int do_write_index(struct index_state *istate, int newfd, int strip_extensions)\n {\n-\tgit_SHA_CTX c;\n+\tgit_SHA_CTX c, *c_p;\n \tstruct cache_header hdr;\n \tint i, err, removed, extended, hdr_version;\n \tstruct cache_entry **cache = istate->cache;\n@@ -1960,8 +2026,14 @@ static int do_write_index(struct index_state *istate, int newfd,\n \thdr.hdr_version = htonl(hdr_version);\n \thdr.hdr_entries = htonl(entries - removed);\n \n-\tgit_SHA1_Init(&c);\n-\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n+\tif (istate->version == 254)\n+\t\tc_p = NULL;\n+\telse {\n+\t\tc_p = &c;\n+\t\tgit_SHA1_Init(c_p);\n+\t}\n+\n+\tif (ce_write(c_p, newfd, &hdr, sizeof(hdr)) < 0)\n \t\treturn -1;\n \n \tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n@@ -1982,7 +2054,21 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\t\telse\n \t\t\t\treturn error(msg, ce->name);\n \t\t}\n-\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n+\t\tif (!c_p) {\n+\t\t\tstatic unsigned padding[8];\n+\t\t\tunsigned sz = offsetof(struct cache_entry, name) + ce_namelen(ce) + 1;\n+\t\t\tunsigned int ce_flags = ce->ce_flags;\n+\t\t\tstruct hashmap_entry he = ce->ent;\n+\t\t\tce->index = 0xffffffff;\n+\t\t\tmemset(&ce->ent, 0, sizeof(ce->ent));\n+\t\t\tce->ce_flags &= CE_VALID | CE_EXTENDED_FLAGS;\n+\t\t\tce_write(NULL, 0, ce, sz);\n+\t\t\tce->ce_flags = ce_flags;\n+\t\t\tmemcpy(&ce->ent, &he, sizeof(he));\n+\t\t\tce->index = 0;\n+\t\t\tif (sz % 8)\n+\t\t\t\tce_write(NULL, 0, padding, 8 - (sz % 8));\n+\t\t} else if (ce_write_entry(c_p, newfd, ce, previous_name) < 0)\n \t\t\treturn -1;\n \t}\n \tstrbuf_release(&previous_name_buf);\n@@ -1992,9 +2078,9 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\terr = write_link_extension(&sb, istate) < 0 ||\n-\t\t\twrite_index_ext_header(&c, newfd, CACHE_EXT_LINK,\n+\t\t\twrite_index_ext_header(c_p, newfd, CACHE_EXT_LINK,\n \t\t\t\t\t sb.len) < 0 ||\n-\t\t\tce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\t\tce_write(c_p, newfd, sb.buf, sb.len) < 0;\n \t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n@@ -2003,8 +2089,8 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tcache_tree_write(&sb, istate->cache_tree);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\terr = write_index_ext_header(c_p, newfd, CACHE_EXT_TREE, sb.len) < 0\n+\t\t\t|| ce_write(c_p, newfd, sb.buf, sb.len) < 0;\n \t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n@@ -2013,16 +2099,34 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tresolve_undo_write(&sb, istate->resolve_undo);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n+\t\terr = write_index_ext_header(c_p, newfd, CACHE_EXT_RESOLVE_UNDO,\n \t\t\t\t\t sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\t\t|| ce_write(c_p, newfd, sb.buf, sb.len) < 0;\n \t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n \n-\tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n+\tif (ce_flush(c_p, newfd, istate->sha1) || (c_p && fstat(newfd, &st)))\n \t\treturn -1;\n+\tif (!c_p) {\n+\t\tunsigned char *p = NULL;\n+\t\tif (istate->allocate_254)\n+\t\t\tp = istate->allocate_254(istate, file_block_size);\n+\t\twhile (file_block_size) {\n+\t\t\tstruct file_block *to_free = start;\n+\t\t\tint len = file_block_size > FB_USABLE_SIZE ? FB_USABLE_SIZE : file_block_size;\n+\t\t\tif (p) {\n+\t\t\t\tmemcpy(p, start->buf, len);\n+\t\t\t\tp += len;\n+\t\t\t} else\n+\t\t\t\twrite_or_die(newfd, start->buf, len);\n+\t\t\tfile_block_size -= len;\n+\t\t\tstart = start->next;\n+\t\t\tfree(to_free);\n+\t\t}\n+\t\tstart = end = NULL;\n+\t}\n \tistate->timestamp.sec = (unsigned int)st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \treturn 0;\ndiff --git a/split-index.c b/split-index.c\nindex 21485e2..a47f805 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -302,6 +302,9 @@ void discard_split_index(struct index_state *istate)\n \n void save_or_free_index_entry(struct index_state *istate, struct cache_entry *ce)\n {\n+\tif (ce->index == 0xffffffff)\n+\t\treturn;\n+\n \tif (ce->index &&\n \t istate->split_index &&\n \t istate->split_index->base &&\n-- \n1.9.1.346.ga2b5940\n"},{"id":"241357","messageId":"CABPQNSaSDRhuQYey0ad6J0cXrCzrK1CYZQar5GgUM8g3JVmRSA@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T11:52:50Z","receivedAt":"2014-05-13T11:52:50Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n> diff --git a/Makefile b/Makefile\n> index 028749b..98d22de 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1502,6 +1502,12 @@ ifdef HAVE_DEV_TTY\n> BASIC_CFLAGS += -DHAVE_DEV_TTY\n> endif\n>\n> +ifdef HAVE_SHM\n> + BASIC_CFLAGS += -DHAVE_SHM\n> + EXTLIBS += -lrt\n> + PROGRAM_OBJS += read-cache--daemon.o\n> +endif\n> +\n\nI think read-cache--daemon will fail in case of NO_UNIX_SOCKETS.\n\nBut, read-cache--daemon.c only gets compiled if we have shm_open...\n\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index f6d3a46..b2116ab 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -723,4 +723,12 @@ struct tm *git_gmtime_r(const time_t *, struct tm *);\n> #define gmtime_r git_gmtime_r\n> #endif\n>\n> +#ifndef HAVE_SHM\n> +static inline int shm_open(const char *path, int flags, int mode)\n> +{\n> + errno = ENOSYS;\n> + return -1;\n> +}\n> +#endif\n\n\n...yet, you introduce a compatibility-shim...\n\n> diff --git a/read-cache--daemon.c b/read-cache--daemon.c\n> new file mode 100644\n> index 0000000..52b4067\n> --- /dev/null\n> +++ b/read-cache--daemon.c\n> @@ -0,0 +1,167 @@\n> +#include \"cache.h\"\n> +#include \"sigchain.h\"\n> +#include \"unix-socket.h\"\n> +#include \"split-index.h\"\n> +#include \"pkt-line.h\"\n> +\n> +static char *socket_path;\n> +static struct strbuf shm_index = STRBUF_INIT;\n> +static struct strbuf shm_sharedindex = STRBUF_INIT;\n> +\n> +static void cleanup_socket(void)\n> +{\n> + if (socket_path)\n> + unlink(socket_path);\n> + if (shm_index.len)\n> + shm_unlink(shm_index.buf);\n> + if (shm_sharedindex.len)\n> + shm_unlink(shm_sharedindex.buf);\n> +}\n> +\n> +static void cleanup_socket_on_signal(int sig)\n> +{\n> + cleanup_socket();\n> + sigchain_pop(sig);\n> + raise(sig);\n> +}\n> +\n> +static void share_index(struct index_state *istate, struct strbuf *shm_path)\n> +{\n> + struct strbuf sb = STRBUF_INIT;\n> + void *map;\n> + int fd;\n> +\n> + strbuf_addf(&sb, \"/git-index-%s\", sha1_to_hex(istate->sha1));\n> + if (shm_path->len && strcmp(sb.buf, shm_path->buf)) {\n> + shm_unlink(shm_path->buf);\n> + strbuf_reset(shm_path);\n> + }\n> + fd = shm_open(sb.buf, O_RDWR | O_CREAT | O_TRUNC, 0700);\n> + if (fd < 0)\n> + return;\n\n...that only gets called from read-cache--daemon.c, which already only\ngets compiled if we have open?\n"},{"id":"241358","messageId":"CABPQNSatvV7X97j_+V52-cQ4t5qHx-Ho+R1bq63CGorg4gPhAg@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-7-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 4/8] Add read-cache--daemon for caching index and related stuff","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T11:56:14Z","receivedAt":"2014-05-13T11:56:14Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n> diff --git a/Documentation/git-read-cache--daemon.txt b/Documentation/git-read-cache--daemon.txt\n> new file mode 100644\n> index 0000000..1b05be4\n> --- /dev/null\n> +++ b/Documentation/git-read-cache--daemon.txt\n> @@ -0,0 +1,27 @@\n> +git-read-cache--daemon(1)\n> +=============\n> +\n> +NAME\n> +----\n> +git-daemon - A simple cache server for speeding up index file access\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git daemon' [--detach]\n> +\n\nHuh, \"git daemon\" can't be right...\n\n> diff --git a/Makefile b/Makefile\n> index d0a2b4b..a44ab0b 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1504,6 +1504,12 @@ ifdef HAVE_DEV_TTY\n> BASIC_CFLAGS += -DHAVE_DEV_TTY\n> endif\n>\n> +ifdef HAVE_SHM\n> + BASIC_CFLAGS += -DHAVE_SHM\n> + EXTLIBS += -lrt\n> + PROGRAM_OBJS += read-cache--daemon.o\n> +endif\n\nI think this also needs to be protected against NO_UNIX_SOCKETS\n"},{"id":"241359","messageId":"CABPQNSZAxyWJucyJbO0ND2X5V+cnCFCGLZPfFhygi2tUVQCXYg@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/8] unix-socket: stub impl. for platforms with no unix socket support","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T11:59:53Z","receivedAt":"2014-05-13T11:59:53Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n> With this we can make unix_stream_* calls without #ifdef.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Makefile | 2 ++\n> unix-socket.h | 18 ++++++++++++++++++\n> 2 files changed, 20 insertions(+)\n>\n> diff --git a/Makefile b/Makefile\n> index 028749b..d0a2b4b 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1417,6 +1417,8 @@ ifndef NO_UNIX_SOCKETS\n> LIB_H += unix-socket.h\n> PROGRAM_OBJS += credential-cache.o\n> PROGRAM_OBJS += credential-cache--daemon.o\n> +else\n> + BASIC_CFLAGS += -DNO_UNIX_SOCKETS\n> endif\n>\n> ifdef NO_ICONV\n> diff --git a/unix-socket.h b/unix-socket.h\n> index e271aee..f1cba70 100644\n> --- a/unix-socket.h\n> +++ b/unix-socket.h\n> @@ -1,7 +1,25 @@\n> #ifndef UNIX_SOCKET_H\n> #define UNIX_SOCKET_H\n>\n> +#ifndef NO_UNIX_SOCKETS\n> +\n> int unix_stream_connect(const char *path);\n> int unix_stream_listen(const char *path);\n>\n> +#else\n> +\n> +static inline int unix_stream_connect(const char *path)\n> +{\n> + errno = ENOSYS;\n> + return -1;\n> +}\n> +\n> +static inline int unix_stream_listen(const char *path)\n> +{\n> + errno = ENOSYS;\n> + return -1;\n> +}\n> +\n> +#endif\n> +\n> #endif /* UNIX_SOCKET_H */\n\nOK, so I missed this before my other two comments. But still... in\nwhat way does errno=ENOSYS make this *work*? Won't we end up compiling\nlots of non-functional tools on Windows in this case?\n"},{"id":"241360","messageId":"CACsJy8DD3c+bsGfUBm3Lm7okqRzwV2oSbZ5k0u92MXS2KjYr_A@mail.gmail.com","threadId":"36517","inReplyTo":"CABPQNSaSDRhuQYey0ad6J0cXrCzrK1CYZQar5GgUM8g3JVmRSA@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T12:01:29Z","receivedAt":"2014-05-13T12:01:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 13, 2014 at 6:52 PM, Erik Faye-Lund wrote:\n> On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n>> diff --git a/Makefile b/Makefile\n>> index 028749b..98d22de 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -1502,6 +1502,12 @@ ifdef HAVE_DEV_TTY\n>> BASIC_CFLAGS += -DHAVE_DEV_TTY\n>> endif\n>>\n>> +ifdef HAVE_SHM\n>> + BASIC_CFLAGS += -DHAVE_SHM\n>> + EXTLIBS += -lrt\n>> + PROGRAM_OBJS += read-cache--daemon.o\n>> +endif\n>> +\n>\n> I think read-cache--daemon will fail in case of NO_UNIX_SOCKETS.\n>\n> But, read-cache--daemon.c only gets compiled if we have shm_open...\n\nPortability is something to be sorted out.Ideally we should not build\nthis unless we have both unix socket and shared memory support. On\nWindows, I'm not sure how much code can be shared, or it'll be a\ncompletely different program. In that case maybe this program should\nbe read-cache--daemon-posix (at least the .c file name, the binary may\nbe still git-read-cache--daemon) or something and the Windows version\nread-cache--daemon-windows..\n-- \nDuy\n"},{"id":"241361","messageId":"CABPQNSayUdjgby7Pp7rGD0Vn9qb7rGOXfC8neM-4g-Lu7aAQzw@mail.gmail.com","threadId":"36517","inReplyTo":"CABPQNSZAxyWJucyJbO0ND2X5V+cnCFCGLZPfFhygi2tUVQCXYg@mail.gmail.com","subject":"Re: [PATCH 2/8] unix-socket: stub impl. for platforms with no unix socket support","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T12:03:27Z","receivedAt":"2014-05-13T12:03:27Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:59 PM, Erik Faye-Lund wrote:\n> On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n>> With this we can make unix_stream_* calls without #ifdef.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>> Makefile | 2 ++\n>> unix-socket.h | 18 ++++++++++++++++++\n>> 2 files changed, 20 insertions(+)\n>>\n>> diff --git a/Makefile b/Makefile\n>> index 028749b..d0a2b4b 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -1417,6 +1417,8 @@ ifndef NO_UNIX_SOCKETS\n>> LIB_H += unix-socket.h\n>> PROGRAM_OBJS += credential-cache.o\n>> PROGRAM_OBJS += credential-cache--daemon.o\n>> +else\n>> + BASIC_CFLAGS += -DNO_UNIX_SOCKETS\n>> endif\n>>\n>> ifdef NO_ICONV\n>> diff --git a/unix-socket.h b/unix-socket.h\n>> index e271aee..f1cba70 100644\n>> --- a/unix-socket.h\n>> +++ b/unix-socket.h\n>> @@ -1,7 +1,25 @@\n>> #ifndef UNIX_SOCKET_H\n>> #define UNIX_SOCKET_H\n>>\n>> +#ifndef NO_UNIX_SOCKETS\n>> +\n>> int unix_stream_connect(const char *path);\n>> int unix_stream_listen(const char *path);\n>>\n>> +#else\n>> +\n>> +static inline int unix_stream_connect(const char *path)\n>> +{\n>> + errno = ENOSYS;\n>> + return -1;\n>> +}\n>> +\n>> +static inline int unix_stream_listen(const char *path)\n>> +{\n>> + errno = ENOSYS;\n>> + return -1;\n>> +}\n>> +\n>> +#endif\n>> +\n>> #endif /* UNIX_SOCKET_H */\n>\n> OK, so I missed this before my other two comments. But still... in\n> what way does errno=ENOSYS make this *work*? Won't we end up compiling\n> lots of non-functional tools on Windows in this case?\n\nENOSYS makes git-credential-cache.c just die with the message \"unable\nto start cache daemon\", and git-credential--daemon.c die with \"unable\nto bind to \".\n"},{"id":"241362","messageId":"CABPQNSa+QyKjhG=355aCVoPj_U09TU6Ep9djHD0kXX62od2zhA@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-8-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 5/8] read-cache: try index data from shared memory","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T12:13:49Z","receivedAt":"2014-05-13T12:13:49Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/config.txt | 4 ++++\n> cache.h | 1 +\n> config.c | 12 ++++++++++++\n> environment.c | 1 +\n> read-cache.c | 43 +++++++++++++++++++++++++++++++++++++++++++\n> submodule.c | 1 +\n> 6 files changed, 62 insertions(+)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index d8b6cc9..ccbe00b 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -617,6 +617,10 @@ relatively high IO latencies. With this set to 'true', Git will do the\n> index comparison to the filesystem data in parallel, allowing\n> overlapping IO's.\n>\n> +core.useReadCacheDaemon::\n> + Use `git read-cache--daemon` to speed up index reading. See\n> + linkgit:git-read-cache--daemon for more information.\n> +\n> core.createObject::\n> You can set this to 'link', in which case a hardlink followed by\n> a delete of the source are used to make sure that object creation\n> diff --git a/cache.h b/cache.h\n> index d0ff11c..fb29c7e 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -603,6 +603,7 @@ extern size_t packed_git_limit;\n> extern size_t delta_base_cache_limit;\n> extern unsigned long big_file_threshold;\n> extern unsigned long pack_size_limit_cfg;\n> +extern int use_read_cache_daemon;\n>\n> /*\n> * Do replace refs need to be checked this run? This variable is\n> diff --git a/config.c b/config.c\n> index a30cb5c..5c832ad 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -874,6 +874,18 @@ static int git_default_core_config(const char *var, const char *value)\n> return 0;\n> }\n>\n> +#ifdef HAVE_SHM\n> + /*\n> + * Currently git-read-cache--daemon is only built when\n> + * HAVE_SHM is set. Ignore user settings if HAVE_SHM is not\n> + * defined.\n> + */\n> + if (!strcmp(var, \"core.usereadcachedaemon\")) {\n> + use_read_cache_daemon = git_config_bool(var, value);\n> + return 0;\n> + }\n> +#endif\n> +\n> /* Add other config variables here and to Documentation/config.txt. */\n> return 0;\n> }\n> diff --git a/environment.c b/environment.c\n> index 5c4815d..b76a414 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -63,6 +63,7 @@ int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> struct startup_info *startup_info;\n> unsigned long pack_size_limit_cfg;\n> +int use_read_cache_daemon;\n>\n> /*\n> * The character that begins a commented line in user-editable file\n> diff --git a/read-cache.c b/read-cache.c\n> index a5031f3..0e46523 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1462,6 +1462,48 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n> return ce;\n> }\n>\n> +static void *try_shm(void *mmap, size_t *mmap_size)\n> +{\n> + struct strbuf sb = STRBUF_INIT;\n> + size_t old_size = *mmap_size;\n> + void *new_mmap;\n> + struct stat st;\n> + int fd;\n> +\n> + if (old_size <= 20 || !use_read_cache_daemon)\n> + return mmap;\n> +\n> + strbuf_addf(&sb, \"/git-index-%s.lock\",\n> + sha1_to_hex((unsigned char *)mmap + old_size - 20));\n> + fd = shm_open(sb.buf, O_RDONLY, 0777);\n\nOK, so *here* you do an unguarded use of shm_open, but the code never\ngets executed because of the HAVE_SHM guard in\ngit_default_core_config. Perhaps not introduce the compatibility shim\nuntil this patch, then?\n"},{"id":"241363","messageId":"CABPQNSY67gNZnDf27D8xLyiLrG+g13Zau2zY0vz_XgGJfj3cKw@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-11-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 8/8] read-cache: inform the daemon that the index has been updated","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T12:17:13Z","receivedAt":"2014-05-13T12:17:13Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 1:15 PM, Nguyễn Thái Ngọc Duy wrote:\n> The daemon would immediately load the new index in memory in\n> background. Next time Git needs to read the index again, everything is\n> ready.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> cache.h | 1 +\n> read-cache.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++-----\n> 2 files changed, 49 insertions(+), 5 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index fb29c7e..3115b86 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -483,6 +483,7 @@ extern int is_index_unborn(struct index_state *);\n> extern int read_index_unmerged(struct index_state *);\n> #define COMMIT_LOCK (1 << 0)\n> #define CLOSE_LOCK (1 << 1)\n> +#define REFRESH_DAEMON (1 << 2)\n> extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n> extern int discard_index(struct index_state *);\n> extern int unmerged_index(const struct index_state *);\n> diff --git a/read-cache.c b/read-cache.c\n> index e98521f..d5c9247 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -16,6 +16,9 @@\n> #include \"varint.h\"\n> #include \"split-index.h\"\n> #include \"sigchain.h\"\n> +#include \"unix-socket.h\"\n> +#include \"pkt-line.h\"\n> +#include \"run-command.h\"\n>\n> static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n> unsigned int options);\n> @@ -2030,6 +2033,32 @@ void set_alternate_index_output(const char *name)\n> alternate_index_output = name;\n> }\n>\n> +static void refresh_daemon(struct index_state *istate)\n> +{\n> + int fd;\n> + fd = unix_stream_connect(git_path(\"daemon/index\"));\n> + if (fd < 0) {\n> + struct child_process cp;\n> + const char *av[] = {\"read-cache--daemon\", \"--detach\", NULL };\n> + memset(&cp, 0, sizeof(cp));\n> + cp.argv = av;\n> + cp.git_cmd = 1;\n> + cp.no_stdin = 1;\n> + if (run_command(&cp))\n> + warning(_(\"failed to start read-cache--daemon: %s\"),\n> + strerror(errno));\n> + return;\n> + }\n> + /*\n> + * packet_write() could die() but unless this is from\n> + * update_index_if_able(), we're about to exit anyway,\n> + * probably ok to die (for now). Blocking mode is another\n> + * problem to deal with later.\n> + */\n> + packet_write(fd, \"refresh\");\n> + close(fd);\n> +}\n> +\n\nSeems the argument 'istate' isn't used.\n"},{"id":"241364","messageId":"CACsJy8BFEtXQc7mO1pvdeQ9GMofcW0H2uJF=E6yfN4SwQWgRbw@mail.gmail.com","threadId":"36517","inReplyTo":"CABPQNSaSDRhuQYey0ad6J0cXrCzrK1CYZQar5GgUM8g3JVmRSA@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T13:01:31Z","receivedAt":"2014-05-13T13:01:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"What do you think is a good replacement for unix socket on Windows?\nIt's only used to refresh the cache in the daemon, no sensitive data\nsent over, so security is not a problem. I'm thinking maybe just\nTCP/IP server, but that's going to be a system-wide daemon.. Perhaps\nthe windows daemon could just monitor $GIT_DIR/index and refresh it?\n-- \nDuy\n"},{"id":"241367","messageId":"CABPQNSbu9uVh_LZqQupG9TJe0_ggA1EmAWnk8H7+xOBvcsrcOg@mail.gmail.com","threadId":"36517","inReplyTo":"CACsJy8BFEtXQc7mO1pvdeQ9GMofcW0H2uJF=E6yfN4SwQWgRbw@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T13:37:44Z","receivedAt":"2014-05-13T13:37:44Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 3:01 PM, Duy Nguyen wrote:\n> What do you think is a good replacement for unix socket on Windows?\n> It's only used to refresh the cache in the daemon, no sensitive data\n> sent over, so security is not a problem. I'm thinking maybe just\n> TCP/IP server, but that's going to be a system-wide daemon.. Perhaps\n> the windows daemon could just monitor $GIT_DIR/index and refresh it?\n\nWindows has support for Named Pipes, which seems like the right kind\nof communication channel. However, the programming model differs quite\na bit from unix-sockets:\n\nhttp://msdn.microsoft.com/en-us/library/windows/desktop/aa365594%28v=vs.85%29.aspx\n"},{"id":"241368","messageId":"CACsJy8Du+tvQ=EhZ0rdumeQwfcenhg6R9pbGcYV9bCMqSfHzfw@mail.gmail.com","threadId":"36517","inReplyTo":"CABPQNSbu9uVh_LZqQupG9TJe0_ggA1EmAWnk8H7+xOBvcsrcOg@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T13:49:56Z","receivedAt":"2014-05-13T13:49:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 13, 2014 at 8:37 PM, Erik Faye-Lund wrote:\n> On Tue, May 13, 2014 at 3:01 PM, Duy Nguyen wrote:\n>> What do you think is a good replacement for unix socket on Windows?\n>> It's only used to refresh the cache in the daemon, no sensitive data\n>> sent over, so security is not a problem. I'm thinking maybe just\n>> TCP/IP server, but that's going to be a system-wide daemon.. Perhaps\n>> the windows daemon could just monitor $GIT_DIR/index and refresh it?\n>\n> Windows has support for Named Pipes, which seems like the right kind\n> of communication channel. However, the programming model differs quite\n> a bit from unix-sockets:\n>\n> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365594%28v=vs.85%29.aspx\n\nYeah that was my first option, but if code cannot be shared to\ndifferences then we probably should go another way. The old\nFindWindow/PostMessage still works with modern Windows, right? Maybe\nwe could create a window with a name derived from the daemon's pid and\nsave the name in the index, then PostMessage can signal the daemon. On\nthe UNIX front, we store pid and send SIGUSR1 instead..The good thing\nhere is the Git side will be very simple (PostMessage vs kill).\n-- \nDuy\n"},{"id":"241370","messageId":"CABPQNSZoo_GD6c6_9my1=+6b2noHWhQSGtYAz-MHE76G-SyyQA@mail.gmail.com","threadId":"36517","inReplyTo":"CACsJy8Du+tvQ=EhZ0rdumeQwfcenhg6R9pbGcYV9bCMqSfHzfw@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T14:06:30Z","receivedAt":"2014-05-13T14:06:30Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 3:49 PM, Duy Nguyen wrote:\n> On Tue, May 13, 2014 at 8:37 PM, Erik Faye-Lund wrote:\n>> On Tue, May 13, 2014 at 3:01 PM, Duy Nguyen wrote:\n>>> What do you think is a good replacement for unix socket on Windows?\n>>> It's only used to refresh the cache in the daemon, no sensitive data\n>>> sent over, so security is not a problem. I'm thinking maybe just\n>>> TCP/IP server, but that's going to be a system-wide daemon.. Perhaps\n>>> the windows daemon could just monitor $GIT_DIR/index and refresh it?\n>>\n>> Windows has support for Named Pipes, which seems like the right kind\n>> of communication channel. However, the programming model differs quite\n>> a bit from unix-sockets:\n>>\n>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365594%28v=vs.85%29.aspx\n>\n> Yeah that was my first option, but if code cannot be shared to\n> differences then we probably should go another way. The old\n> FindWindow/PostMessage still works with modern Windows, right? Maybe\n> we could create a window with a name derived from the daemon's pid and\n> save the name in the index, then PostMessage can signal the daemon. On\n> the UNIX front, we store pid and send SIGUSR1 instead..The good thing\n> here is the Git side will be very simple (PostMessage vs kill).\n\nHmmm.... I'm a bit worried about having to load in USER32.DLL just to\nread the cache that way. But it seems we already do that, thanks to\ncompat/poll/poll.c (it depends on DispatchMessage,\nMsgWaitForMultipleObjects, PeekMessage and TranslateMessage, all from\nthat DLL).\n\nPreferably, we should delay-load USER32.DLL in compat/poll/poll.c, but\nif we start needing it for the reading the index, it'll be loaded by\nthe vast majority of processes anyway.\n"},{"id":"241371","messageId":"CACsJy8CH1AJ=wOCzMN1EKf383=RwXUSrjUVrCW4M3kbBu6sXUA@mail.gmail.com","threadId":"36517","inReplyTo":"CABPQNSZoo_GD6c6_9my1=+6b2noHWhQSGtYAz-MHE76G-SyyQA@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T14:10:33Z","receivedAt":"2014-05-13T14:10:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 13, 2014 at 9:06 PM, Erik Faye-Lund wrote:\n> On Tue, May 13, 2014 at 3:49 PM, Duy Nguyen wrote:\n>> On Tue, May 13, 2014 at 8:37 PM, Erik Faye-Lund wrote:\n>>> On Tue, May 13, 2014 at 3:01 PM, Duy Nguyen wrote:\n>>>> What do you think is a good replacement for unix socket on Windows?\n>>>> It's only used to refresh the cache in the daemon, no sensitive data\n>>>> sent over, so security is not a problem. I'm thinking maybe just\n>>>> TCP/IP server, but that's going to be a system-wide daemon.. Perhaps\n>>>> the windows daemon could just monitor $GIT_DIR/index and refresh it?\n>>>\n>>> Windows has support for Named Pipes, which seems like the right kind\n>>> of communication channel. However, the programming model differs quite\n>>> a bit from unix-sockets:\n>>>\n>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365594%28v=vs.85%29.aspx\n>>\n>> Yeah that was my first option, but if code cannot be shared to\n>> differences then we probably should go another way. The old\n>> FindWindow/PostMessage still works with modern Windows, right? Maybe\n>> we could create a window with a name derived from the daemon's pid and\n>> save the name in the index, then PostMessage can signal the daemon. On\n>> the UNIX front, we store pid and send SIGUSR1 instead..The good thing\n>> here is the Git side will be very simple (PostMessage vs kill).\n>\n> Hmmm.... I'm a bit worried about having to load in USER32.DLL just to\n> read the cache that way. But it seems we already do that, thanks to\n> compat/poll/poll.c (it depends on DispatchMessage,\n> MsgWaitForMultipleObjects, PeekMessage and TranslateMessage, all from\n> that DLL).\n>\n> Preferably, we should delay-load USER32.DLL in compat/poll/poll.c, but\n> if we start needing it for the reading the index, it'll be loaded by\n> the vast majority of processes anyway.\n\nThanks for the info. I'll see if we can still stick to named pipes. If\nwe have to load user32.dll, hopefully the gain will outweigh load time\nfor that dell.\n-- \nDuy\n"},{"id":"241372","messageId":"CABPQNSZM+6LYTyiybe24p4RWWUicWR=uhh1m30UUy0RKdLj6GQ@mail.gmail.com","threadId":"36517","inReplyTo":"CACsJy8CH1AJ=wOCzMN1EKf383=RwXUSrjUVrCW4M3kbBu6sXUA@mail.gmail.com","subject":"Re: [PATCH 2/3] Add read-cache--daemon","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@gmail.com","sentAt":"2014-05-13T14:16:54Z","receivedAt":"2014-05-13T14:16:54Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Tue, May 13, 2014 at 4:10 PM, Duy Nguyen wrote:\n> On Tue, May 13, 2014 at 9:06 PM, Erik Faye-Lund wrote:\n>> On Tue, May 13, 2014 at 3:49 PM, Duy Nguyen wrote:\n>>> On Tue, May 13, 2014 at 8:37 PM, Erik Faye-Lund wrote:\n>>>> On Tue, May 13, 2014 at 3:01 PM, Duy Nguyen wrote:\n>>>>> What do you think is a good replacement for unix socket on Windows?\n>>>>> It's only used to refresh the cache in the daemon, no sensitive data\n>>>>> sent over, so security is not a problem. I'm thinking maybe just\n>>>>> TCP/IP server, but that's going to be a system-wide daemon.. Perhaps\n>>>>> the windows daemon could just monitor $GIT_DIR/index and refresh it?\n>>>>\n>>>> Windows has support for Named Pipes, which seems like the right kind\n>>>> of communication channel. However, the programming model differs quite\n>>>> a bit from unix-sockets:\n>>>>\n>>>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa365594%28v=vs.85%29.aspx\n>>>\n>>> Yeah that was my first option, but if code cannot be shared to\n>>> differences then we probably should go another way. The old\n>>> FindWindow/PostMessage still works with modern Windows, right? Maybe\n>>> we could create a window with a name derived from the daemon's pid and\n>>> save the name in the index, then PostMessage can signal the daemon. On\n>>> the UNIX front, we store pid and send SIGUSR1 instead..The good thing\n>>> here is the Git side will be very simple (PostMessage vs kill).\n>>\n>> Hmmm.... I'm a bit worried about having to load in USER32.DLL just to\n>> read the cache that way. But it seems we already do that, thanks to\n>> compat/poll/poll.c (it depends on DispatchMessage,\n>> MsgWaitForMultipleObjects, PeekMessage and TranslateMessage, all from\n>> that DLL).\n>>\n>> Preferably, we should delay-load USER32.DLL in compat/poll/poll.c, but\n>> if we start needing it for the reading the index, it'll be loaded by\n>> the vast majority of processes anyway.\n>\n> Thanks for the info. I'll see if we can still stick to named pipes. If\n> we have to load user32.dll, hopefully the gain will outweigh load time\n> for that dell.\n\nI just timed it here on my system, and omitting USER32.DLL didn't gain\nanything for \"git --version\", so I suspect I was worrying too soon.\n"},{"id":"241373","messageId":"CALbm-EZEeFv=CX-TJJ3ZAUOxdBrqWfcJ_W-to20XAaB5=oZ0Rg@mail.gmail.com","threadId":"36517","inReplyTo":"1399979737-8577-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 0/8] Speed up cache loading time","fromName":"Stefan Beller","fromEmail":"stefanbeller@gmail.com","sentAt":"2014-05-13T14:24:04Z","receivedAt":"2014-05-13T14:24:04Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> That is clocked at 800 MHz. A repository at this size deserves a\n> better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A\n> reasonable number to me. If we scale other parts of git-status as well\n> as this, we should be able to make \"git status\" within 1 or 2 seconds.\n>\n\nWhich harddrive do you use? Traditional or SSDs?\nDoes have harddrive loading time significant impact here? (just a\nguess/question)\n"},{"id":"241374","messageId":"CACsJy8C9caJam4ZO5GNMym-LRxSkss_06KHq-Ob3Qxz5xYqjxA@mail.gmail.com","threadId":"36517","inReplyTo":"CALbm-EZEeFv=CX-TJJ3ZAUOxdBrqWfcJ_W-to20XAaB5=oZ0Rg@mail.gmail.com","subject":"Re: [PATCH 0/8] Speed up cache loading time","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-05-13T14:35:47Z","receivedAt":"2014-05-13T14:35:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, May 13, 2014 at 9:24 PM, Stefan Beller wrote:\n>> That is clocked at 800 MHz. A repository at this size deserves a\n>> better CPU. At 2.5 GHz we spend 183.228ms on loading the index. A\n>> reasonable number to me. If we scale other parts of git-status as well\n>> as this, we should be able to make \"git status\" within 1 or 2 seconds.\n>>\n>\n> Which harddrive do you use? Traditional or SSDs?\n\nTraditional\n\n> Does have harddrive loading time significant impact here? (just a\n> guess/question)\n\nIn the hot cache case, I assume the index stays in OS cache anyway so\nhard drive should not impact much (the other parts of git-status like\nindex refresh or untracked file listing is a different story and some\nmay fall out of cache). My laptop has 4G ram, with my repeated tests,\nI guess the index (even 200mb) stayed in the cache (but did not really\nverify it).\n-- \nDuy\n"},{"id":"242495","messageId":"1400776690.1933.8.camel@stross","threadId":"36517","inReplyTo":"1399979737-8577-11-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 8/8] read-cache: inform the daemon that the index has been updated","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-05-22T16:38:10Z","receivedAt":"2014-05-22T16:38:10Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2014-05-13 at 18:15 +0700, Nguyễn Thái Ngọc Duy wrote:\n> +\t\tif (run_command(&cp))\n> +\t\t\twarning(_(\"failed to start read-cache--daemon: %s\"),\n> +\t\t\t\tstrerror(errno));\n\nerrno is not always (ever?) set, so if read-cache--daemon is missing,\nyou get:\nwarning: failed to start read-cache--daemon: Success\n"}]}