{"thread":{"id":"36881","subject":"[PATCH v2 0/5] Speed up cache loading time","startedAt":"2014-06-10T13:24:20Z","lastAt":"2014-06-10T18:44:19Z","messageCount":7,"participants":["Nguyễn Thái Ngọc Duy","David Turner"],"isPatch":true,"patchVersion":2,"patchTotal":5},"messages":[{"id":"243727","messageId":"1402406665-27988-1-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":null,"subject":"[PATCH v2 0/5] Speed up cache loading time","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:20Z","receivedAt":"2014-06-10T13:24:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Compared to v1 [1], this is like a new series\n\n - git-read-cache--daemon is renamed to git-index-helper (easier to\n   guess what it's for)\n - simplified locking mechanism on shared memory\n - UNIX signals are used for notification instead of UNIX sockets\n - Windows support (only tested with wine)\n\nI think I'm getting closer to something that can finally have a chance\nof merging. Still don't know how to write tests for this though.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/247268/focus=248760\n\nNguyễn Thái Ngọc Duy (5):\n  read-cache: allow to keep mmap'd memory after reading\n  index-helper: new daemon for caching index and related stuff\n  index-helper: add Windows support\n  daemonize(): set a flag before exiting the main process\n  index-helper: add --detach\n\n .gitignore                               |   1 +\n Documentation/git-index-helper.txt (new) |  26 ++++\n Makefile                                 |   7 ++\n builtin/gc.c                             |   2 +-\n cache.h                                  |   6 +-\n config.mak.uname                         |   3 +\n daemon.c                                 |   2 +-\n git-compat-util.h                        |   1 +\n index-helper.c (new)                     | 198 +++++++++++++++++++++++++++++++\n read-cache.c                             |  97 +++++++++++++--\n setup.c                                  |   4 +-\n shm.c (new)                              | 163 +++++++++++++++++++++++++\n shm.h (new)                              |  23 ++++\n 13 files changed, 521 insertions(+), 12 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243728","messageId":"1402406665-27988-2-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":"1402406665-27988-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/5] read-cache: allow to keep mmap'd memory after reading","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:21Z","receivedAt":"2014-06-10T13:24:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h      |  3 +++\n read-cache.c | 13 ++++++++++++-\n 2 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex c6b7770..6549e02 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -290,10 +290,13 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n+\t\t keep_mmap : 1,\n \t\t initialized : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n };\n \n extern struct index_state the_index;\ndiff --git a/read-cache.c b/read-cache.c\nindex 342fe52..a5031f3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1495,6 +1495,10 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n+\tif (istate->keep_mmap) {\n+\t\tistate->mmap = mmap;\n+\t\tistate->mmap_size = mmap_size;\n+\t}\n \tclose(fd);\n \n \thdr = mmap;\n@@ -1547,10 +1551,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tmunmap(mmap, mmap_size);\n+\tif (!istate->keep_mmap)\n+\t\tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n+\tistate->mmap = NULL;\n \tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n@@ -1576,6 +1582,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n+\tsplit_index->base->keep_mmap = istate->keep_mmap;\n \tret = do_read_index(split_index->base,\n \t\t\t    git_path(\"sharedindex.%s\",\n \t\t\t\t     sha1_to_hex(split_index->base_sha1)), 1);\n@@ -1618,6 +1625,10 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tif (istate->keep_mmap && istate->mmap) {\n+\t\tmunmap(istate->mmap, istate->mmap_size);\n+\t\tistate->mmap = NULL;\n+\t}\n \tdiscard_split_index(istate);\n \treturn 0;\n }\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243729","messageId":"1402406665-27988-3-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":"1402406665-27988-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/5] index-helper: new daemon for caching index and related stuff","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:22Z","receivedAt":"2014-06-10T13:24:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The shared memory's name folows the template \"git-<something>-<SHA1>\"\nwhere <SHA1> is the trailing SHA-1 of the index file. <something> is\n\"index\" for caching index files. If such shared memory exists, it\ncontains the same index content as on disk. The content is already\nvalidated by the daemon and git won't validate it again. Note that it\ndoes not necessarily use the same format as the on-disk version. The\ncontent could be in a format that can be parsed much faster, or even\nreused without parsing).\n\nGit can poke the daemon to tell it to refresh the index cache, or to\nnot exit for another some minutes via UNIX signals. It can't give any\nreal data directly to the daemon. Real data goes to disk first, then\nthe daemon reads and verifies it from there.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n .gitignore                               |   1 +\n Documentation/git-index-helper.txt (new) |  24 +++++\n Makefile                                 |   7 ++\n cache.h                                  |   1 +\n config.mak.uname                         |   1 +\n git-compat-util.h                        |   1 +\n index-helper.c (new)                     | 145 +++++++++++++++++++++++++++++++\n read-cache.c                             |  78 +++++++++++++++--\n shm.c (new)                              |  67 ++++++++++++++\n shm.h (new)                              |  23 +++++\n 10 files changed, 341 insertions(+), 7 deletions(-)\n create mode 100644 Documentation/git-index-helper.txt\n create mode 100644 index-helper.c\n create mode 100644 shm.c\n create mode 100644 shm.h\n\ndiff --git a/.gitignore b/.gitignore\nindex 70992a4..5a829dd 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -71,6 +71,7 @@\n /git-http-fetch\n /git-http-push\n /git-imap-send\n+/git-index-helper\n /git-index-pack\n /git-init\n /git-init-db\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nnew file mode 100644\nindex 0000000..d0b1365\n--- /dev/null\n+++ b/Documentation/git-index-helper.txt\n@@ -0,0 +1,24 @@\n+git-index-helper(1)\n+=============\n+\n+NAME\n+----\n+git-index-helper - A simple cache server for speeding up index file access\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git index-helper\n+\n+DESCRIPTION\n+-----------\n+Keep the index file in memory for faster access. This daemon is per\n+repository. This daemon is only available on POSIX system with\n+shared memory support (e.g. Linux)\n+\n+OPTIONS\n+-------\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f7058a6..d42f3cc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -886,6 +886,7 @@ LIB_OBJS += sha1-lookup.o\n LIB_OBJS += sha1_file.o\n LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n+LIB_OBJS += shm.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += split-index.o\n@@ -1498,6 +1499,12 @@ ifdef HAVE_DEV_TTY\n \tBASIC_CFLAGS += -DHAVE_DEV_TTY\n endif\n \n+ifdef HAVE_SHM\n+\tBASIC_CFLAGS += -DHAVE_SHM\n+\tEXTLIBS += -lrt\n+\tPROGRAM_OBJS += index-helper.o\n+endif\n+\n ifdef DIR_HAS_BSD_GROUP_SEMANTICS\n \tCOMPAT_CFLAGS += -DDIR_HAS_BSD_GROUP_SEMANTICS\n endif\ndiff --git a/cache.h b/cache.h\nindex 6549e02..f05e062 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -483,6 +483,7 @@ extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n #define COMMIT_LOCK\t\t(1 << 0)\n #define CLOSE_LOCK\t\t(1 << 1)\n+#define REFRESH_DAEMON\t\t(1 << 2)\n extern int write_locked_index(struct index_state *, struct lock_file *lock, unsigned flags);\n extern int discard_index(struct index_state *);\n extern int unmerged_index(const struct index_state *);\ndiff --git a/config.mak.uname b/config.mak.uname\nindex eee0fc2..8de61a4 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -39,6 +39,7 @@ ifeq ($(uname_S),Linux)\n \tHAVE_PATHS_H = YesPlease\n \tLIBC_CONTAINS_LIBINTL = YesPlease\n \tHAVE_DEV_TTY = YesPlease\n+\tHAVE_SHM = YesPlease\n endif\n ifeq ($(uname_S),GNU/kFreeBSD)\n \tNO_STRLCPY = YesPlease\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f6d3a46..a6ebecc 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -356,6 +356,7 @@ static inline const char *skip_prefix(const char *str, const char *prefix)\n #define PROT_READ 1\n #define PROT_WRITE 2\n #define MAP_PRIVATE 1\n+#define MAP_SHARED 2\n #endif\n \n #define mmap git_mmap\ndiff --git a/index-helper.c b/index-helper.c\nnew file mode 100644\nindex 0000000..e10d0c3\n--- /dev/null\n+++ b/index-helper.c\n@@ -0,0 +1,145 @@\n+#include \"cache.h\"\n+#include \"parse-options.h\"\n+#include \"sigchain.h\"\n+#include \"split-index.h\"\n+#include \"shm.h\"\n+\n+static unsigned char cached_sha1[20];\n+static unsigned char cached_shared_sha1[20];\n+static void *shm_index, *shm_sharedindex;\n+static size_t size_index, size_sharedindex;\n+static int daemonized;\n+\n+static void cleanup(void)\n+{\n+\tif (daemonized)\n+\t\treturn;\n+\tunlink(git_path(\"read-cache--daemon.pid\"));\n+\tif (shm_index) {\n+\t\tmunmap(shm_index, size_index);\n+\t\tgit_shm_unlink(\"git-index-%s\", sha1_to_hex(cached_sha1));\n+\t}\n+\tif (shm_sharedindex) {\n+\t\tmunmap(shm_sharedindex, size_sharedindex);\n+\t\tgit_shm_unlink(\"git-index-%s\", sha1_to_hex(cached_shared_sha1));\n+\t}\n+}\n+\n+static void cleanup_on_signal(int sig)\n+{\n+\tcleanup();\n+\tsigchain_pop(sig);\n+\traise(sig);\n+}\n+\n+static void share_index(struct index_state *istate, void **mmap,\n+\t\t\tsize_t *length, unsigned char *sha1)\n+{\n+\tvoid *new_mmap;\n+\tif (istate->mmap_size <= 20 ||\n+\t    hashcmp(istate->sha1,\n+\t\t    (unsigned char *)istate->mmap + istate->mmap_size - 20) ||\n+\t    !hashcmp(istate->sha1, sha1) ||\n+\t    git_shm_map(O_CREAT | O_EXCL | O_RDWR, 0700, istate->mmap_size,\n+\t\t\t&new_mmap, PROT_READ | PROT_WRITE, MAP_SHARED,\n+\t\t\t\"git-index-%s\", sha1_to_hex(istate->sha1)) < 0)\n+\t\treturn;\n+\n+\tif (*mmap) {\n+\t\tmunmap(*mmap, *length);\n+\t\tgit_shm_unlink(\"git-index-%s\", sha1_to_hex(sha1));\n+\t}\n+\n+\t*length = istate->mmap_size;\n+\t*mmap = new_mmap;\n+\thashcpy(sha1, istate->sha1);\n+\tmemcpy(new_mmap, istate->mmap, istate->mmap_size - 20);\n+\n+\t/*\n+\t * The trailing hash must be written last after everything is\n+\t * written. It's the indication that the shared memory is now\n+\t * ready.\n+\t */\n+\thashcpy((unsigned char *)new_mmap + istate->mmap_size - 20, sha1);\n+}\n+\n+static void refresh(int sig)\n+{\n+\tthe_index.keep_mmap = 1;\n+\tif (read_cache() < 0)\n+\t\tdie(\"could not read index\");\n+\tshare_index(&the_index, &shm_index, &size_index, cached_sha1);\n+\tif (the_index.split_index && the_index.split_index->base)\n+\t\tshare_index(the_index.split_index->base, &shm_sharedindex,\n+\t\t\t    &size_sharedindex, cached_shared_sha1);\n+\tdiscard_index(&the_index);\n+}\n+\n+#ifdef HAVE_SHM\n+\n+static void do_nothing(int sig)\n+{\n+\t/*\n+\t * what we need is the signal received and interrupts\n+\t * sleep(). We don't need to do anything else when receving\n+\t * the signal\n+\t */\n+}\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+\tsigchain_push(SIGHUP, refresh);\n+\tsigchain_push(SIGUSR1, do_nothing);\n+\trefresh(0);\n+\twhile (sleep(idle_in_seconds))\n+\t\tutime(git_path(\"read-cache--daemon.pid\"), NULL);\n+}\n+\n+#else\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+}\n+\n+#endif\n+\n+static const char * const usage_text[] = {\n+\t\"git read-cache--daemon [options]\",\n+\tNULL\n+};\n+\n+int main(int argc, char **argv)\n+{\n+\tstatic struct lock_file lock;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst char *prefix;\n+\tint fd;\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(usage_text, options);\n+\tprefix = setup_git_directory();\n+\tif (parse_options(argc, (const char **)argv, prefix,\n+\t\t\t  options, usage_text, 0))\n+\t\tdie(\"too many arguments\");\n+\n+\tfd = hold_lock_file_for_update(&lock,\n+\t\t\t\t       git_path(\"read-cache--daemon.pid\"),\n+\t\t\t\t       LOCK_DIE_ON_ERROR);\n+\tstrbuf_addf(&sb,\n+\t\t    \"%\" PRIuMAX, (uintmax_t) getpid());\n+\twrite_in_full(fd, sb.buf, sb.len);\n+\tcommit_lock_file(&lock);\n+\n+\tatexit(cleanup);\n+\tsigchain_push(SIGINT,  cleanup_on_signal);\n+\tsigchain_push(SIGTERM, cleanup_on_signal);\n+\tsigchain_push(SIGQUIT, cleanup_on_signal);\n+\tsigchain_push(SIGPIPE, cleanup_on_signal);\n+\n+\tloop(sb.buf, 600);\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\ndiff --git a/read-cache.c b/read-cache.c\nindex a5031f3..f9df984 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -16,6 +16,7 @@\n #include \"varint.h\"\n #include \"split-index.h\"\n #include \"sigchain.h\"\n+#include \"shm.h\"\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n \t\t\t\t\t       unsigned int options);\n@@ -1332,6 +1333,8 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n+\tif (!size)\n+\t\treturn 0;\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, hdr, size - 20);\n \tgit_SHA1_Final(sha1, &c);\n@@ -1462,6 +1465,58 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static void poke_daemon(struct stat *st, int refresh_cache)\n+{\n+\tint fd;\n+\tstruct strbuf sb;\n+\n+\tif (st->st_mtime + 600 < time(NULL))\n+\t\treturn;\t  /* don't try to read from stale .pid file */\n+\n+\tfd = open(git_path(\"read-cache--daemon.pid\"), O_RDONLY);\n+\tif (fd < 0)\n+\t\treturn;\n+\tstrbuf_init(&sb, st->st_size);\n+\tif (read_in_full(fd, sb.buf, st->st_size) == st->st_size) {\n+#ifdef HAVE_SHM\n+\t\tchar *end = NULL;\n+\t\tpid_t pid = strtoul(sb.buf, &end, 10);\n+\t\tif (end && !*end)\n+\t\t\tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+#endif\n+\t}\n+\tclose(fd);\n+\tstrbuf_release(&sb);\n+}\n+\n+static void *try_shm(struct index_state *istate, void *mmap, size_t *mmap_size)\n+{\n+\tvoid *new_mmap = NULL;\n+\tsize_t old_size = *mmap_size;\n+\tssize_t new_length;\n+\tconst unsigned char *sha1;\n+\tstruct stat st;\n+\n+\tif (old_size <= 20)\n+\t\treturn mmap;\n+\tsha1 = (unsigned char *)mmap + old_size - 20;\n+\tif (stat(git_path(\"read-cache--daemon.pid\"), &st))\n+\t\treturn mmap;\n+\tnew_length = git_shm_map(O_RDONLY, 0700, -1, &new_mmap,\n+\t\t\t\t PROT_READ, MAP_SHARED,\n+\t\t\t\t \"git-index-%s\", sha1_to_hex(sha1));\n+\tif (new_length <= 20 ||\n+\t    hashcmp((unsigned char *)mmap + old_size - 20,\n+\t\t    (unsigned char *)new_mmap + new_length - 20)) {\n+\t\tif (new_mmap)\n+\t\t\tmunmap(new_mmap, new_length);\n+\t\treturn mmap;\n+\t}\n+\tpoke_daemon(&st, 0);\n+\t*mmap_size = new_length;\n+\treturn new_mmap;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int do_read_index(struct index_state *istate, const char *path, int must_exist)\n {\n@@ -1469,8 +1524,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tstruct stat st;\n \tunsigned long src_offset;\n \tstruct cache_header *hdr;\n-\tvoid *mmap;\n-\tsize_t mmap_size;\n+\tvoid *mmap, *old_mmap;\n+\tsize_t mmap_size, old_mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n \tif (istate->initialized)\n@@ -1501,9 +1556,14 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \t}\n \tclose(fd);\n \n-\thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\told_mmap      = mmap;\n+\told_mmap_size = mmap_size;\n+\tmmap\t      = try_shm(istate, mmap, &mmap_size);\n+\thdr\t      = mmap;\n+\tif (mmap == old_mmap && verify_hdr(hdr, mmap_size) < 0)\n \t\tgoto unmap;\n+\tif (mmap != old_mmap)\n+\t\tmunmap(old_mmap, old_mmap_size);\n \n \thashcpy(istate->sha1, (const unsigned char *)hdr + mmap_size - 20);\n \tistate->version = ntohl(hdr->hdr_version);\n@@ -2006,9 +2066,13 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \t\treturn ret;\n \tassert((flags & (COMMIT_LOCK | CLOSE_LOCK)) !=\n \t       (COMMIT_LOCK | CLOSE_LOCK));\n-\tif (flags & COMMIT_LOCK)\n-\t\treturn commit_locked_index(lock);\n-\telse if (flags & CLOSE_LOCK)\n+\tif (flags & COMMIT_LOCK) {\n+\t\tstruct stat st;\n+\t\tif (!commit_locked_index(lock) &&\n+\t\t    !stat(git_path(\"read-cache--daemon.pid\"), &st))\n+\t\t\tpoke_daemon(&st, 1);\n+\t\treturn ret;\n+\t} else if (flags & CLOSE_LOCK)\n \t\treturn close_lock_file(lock);\n \telse\n \t\treturn ret;\ndiff --git a/shm.c b/shm.c\nnew file mode 100644\nindex 0000000..4ec1a00\n--- /dev/null\n+++ b/shm.c\n@@ -0,0 +1,67 @@\n+#include \"git-compat-util.h\"\n+#include \"shm.h\"\n+\n+#ifdef HAVE_SHM\n+\n+#define SHM_PATH_LEN 72\t\t/* we don't create very long paths.. */\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t    int prot, int flags, const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\tint fd;\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tfd = shm_open(path, oflag, perm);\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (length > 0 && ftruncate(fd, length)) {\n+\t\tshm_unlink(path);\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tif (length < 0 && !(oflag & O_CREAT)) {\n+\t\tstruct stat st;\n+\t\tif (fstat(fd, &st))\n+\t\t\tdie_errno(\"unable to stat %s\", path);\n+\t\tlength = st.st_size;\n+\t}\n+\t*mmap = xmmap(NULL, length, prot, flags, fd, 0);\n+\tclose(fd);\n+\tif (*mmap == MAP_FAILED) {\n+\t\t*mmap = NULL;\n+\t\tshm_unlink(path);\n+\t\treturn -1;\n+\t}\n+\treturn length;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\n+\tpath[0] = '/';\n+\tva_start(ap, fmt);\n+\tvsprintf(path + 1, fmt, ap);\n+\tva_end(ap);\n+\tshm_unlink(path);\n+}\n+\n+#else\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t    int prot, int flags, const char *fmt, ...)\n+{\n+\treturn -1;\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+}\n+\n+#endif\ndiff --git a/shm.h b/shm.h\nnew file mode 100644\nindex 0000000..798d3fd\n--- /dev/null\n+++ b/shm.h\n@@ -0,0 +1,23 @@\n+#ifndef SHM_H\n+#define SHM_H\n+\n+/*\n+ * Create or open a shared memory and mmap it. Return mmap size if\n+ * successful, -1 otherwise. If successful mmap contains the mmap'd\n+ * pointer. If oflag does not contain O_CREAT and length is negative,\n+ * the mmap size is retrieved from existing shared memory object.\n+ *\n+ * The mmap could be freed by munmap, even on Windows. Note that on\n+ * Windows, git_shm_unlink() is no-op, so the last unmap will destroy\n+ * the shared memory.\n+ */\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t    int prot, int flags, const char *fmt, ...);\n+\n+/*\n+ * Unlink a shared memory object. Only needed on POSIX platforms. On\n+ * Windows this is no-op.\n+ */\n+void git_shm_unlink(const char *fmt, ...);\n+\n+#endif\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243730","messageId":"1402406665-27988-4-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":"1402406665-27988-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] index-helper: add Windows support","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:23Z","receivedAt":"2014-06-10T13:24:23Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Windows supports shared memory, but a bit different that POSIX\nshm. The most noticeable thing is there's no way to get the shared\nmemory's size from the reader. So the size is added near the end in\nthe shared memory, hidden away from shm users (storing it in headers\nwould cause more problems with munmap, storing it as a separate shm is\neven worse).\n\nPostMessage is used instead of UNIX signals for\nnotification. Lightweight (at least code-wise) on the client side.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-index-helper.txt |  2 +-\n config.mak.uname                   |  2 +\n index-helper.c                     | 49 +++++++++++++++++++\n read-cache.c                       |  6 +++\n shm.c                              | 96 ++++++++++++++++++++++++++++++++++++++\n 5 files changed, 154 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex d0b1365..afeb334 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -14,7 +14,7 @@ DESCRIPTION\n -----------\n Keep the index file in memory for faster access. This daemon is per\n repository. This daemon is only available on POSIX system with\n-shared memory support (e.g. Linux)\n+shared memory support (e.g. Linux or Windows)\n \n OPTIONS\n -------\ndiff --git a/config.mak.uname b/config.mak.uname\nindex 8de61a4..17d35e3 100644\n--- a/config.mak.uname\n+++ b/config.mak.uname\n@@ -376,6 +376,7 @@ ifndef DEBUG\n else\n \tBASIC_CFLAGS += -Zi -MTd\n endif\n+\tPROGRAM_OBJS += index-helper.o\n \tX = .exe\n endif\n ifeq ($(uname_S),Interix)\n@@ -529,6 +530,7 @@ ifneq (,$(wildcard ../THIS_IS_MSYSGIT))\n else\n \tNO_CURL = YesPlease\n endif\n+\tPROGRAM_OBJS += index-helper.o\n endif\n ifeq ($(uname_S),QNX)\n \tCOMPAT_CFLAGS += -DSA_RESTART=0\ndiff --git a/index-helper.c b/index-helper.c\nindex e10d0c3..f92326a 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -95,6 +95,52 @@ static void loop(const char *pid_file, int idle_in_seconds)\n \t\tutime(git_path(\"read-cache--daemon.pid\"), NULL);\n }\n \n+#elif defined(GIT_WINDOWS_NATIVE)\n+\n+static void loop(const char *pid_file, int idle_in_seconds)\n+{\n+\tHWND hwnd;\n+\tUINT_PTR timer = 0;\n+\tMSG msg;\n+\tHINSTANCE hinst = GetModuleHandle(NULL);\n+\tWNDCLASS wc;\n+\n+\t/*\n+\t * Emulate UNIX signals by sending WM_USER+x to a\n+\t * window. Register window class and create a new window to\n+\t * catch these messages.\n+\t */\n+\tmemset(&wc, 0, sizeof(wc));\n+\twc.lpfnWndProc\t = DefWindowProc;\n+\twc.hInstance\t = hinst;\n+\twc.lpszClassName = \"git-read-cache--daemon\";\n+\tif (!RegisterClass(&wc))\n+\t\tdie_errno(\"could not register new window class\");\n+\n+\thwnd = CreateWindow(\"git-read-cache--daemon\", pid_file,\n+\t\t\t    0, 0, 0, 1, 1, NULL, NULL, hinst, NULL);\n+\tif (!hwnd)\n+\t\tdie_errno(\"could not register new window\");\n+\n+\trefresh(0);\n+\twhile (1) {\n+\t\ttimer = SetTimer(hwnd, timer, idle_in_seconds * 1000, NULL);\n+\t\tif (!timer)\n+\t\t\tdie(\"no timer!\");\n+\t\tif (!GetMessage(&msg, hwnd, 0, 0) || msg.message == WM_TIMER)\n+\t\t\tbreak;\n+\t\tswitch (msg.message) {\n+\t\tcase WM_USER:\n+\t\t\trefresh(0);\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* just reset the timer */\n+\t\t\tbreak;\n+\t\t}\n+\t\tutime(git_path(\"read-cache--daemon.pid\"), NULL);\n+\t}\n+}\n+\n #else\n \n static void loop(const char *pid_file, int idle_in_seconds)\n@@ -129,6 +175,9 @@ int main(int argc, char **argv)\n \t\t\t\t       git_path(\"read-cache--daemon.pid\"),\n \t\t\t\t       LOCK_DIE_ON_ERROR);\n \tstrbuf_addf(&sb,\n+#ifdef GIT_WINDOWS_NATIVE\n+\t\t    \"HWND\"\n+#endif\n \t\t    \"%\" PRIuMAX, (uintmax_t) getpid());\n \twrite_in_full(fd, sb.buf, sb.len);\n \tcommit_lock_file(&lock);\ndiff --git a/read-cache.c b/read-cache.c\nindex f9df984..8001c29 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1483,6 +1483,12 @@ static void poke_daemon(struct stat *st, int refresh_cache)\n \t\tpid_t pid = strtoul(sb.buf, &end, 10);\n \t\tif (end && !*end)\n \t\t\tkill(pid, refresh_cache ? SIGHUP : SIGUSR1);\n+#elif defined(GIT_WINDOWS_NATIVE)\n+\t\tif (starts_with(sb.buf, \"HWND\")) {\n+\t\t\tHWND hwnd = FindWindow(\"git-read-cache--daemon\", sb.buf);\n+\t\t\tPostMessage(hwnd, refresh_cache ? WM_USER : WM_USER + 1,\n+\t\t\t\t    0, 0);\n+\t\t}\n #endif\n \t}\n \tclose(fd);\ndiff --git a/shm.c b/shm.c\nindex 4ec1a00..04d8a35 100644\n--- a/shm.c\n+++ b/shm.c\n@@ -52,6 +52,102 @@ void git_shm_unlink(const char *fmt, ...)\n \tshm_unlink(path);\n }\n \n+#elif defined(GIT_WINDOWS_NATIVE)\n+\n+#define SHM_PATH_LEN 82\t/* a little bit longer than POSIX because of \"Local\\\\\" */\n+\n+static ssize_t create_shm_map(int oflag, int perm, ssize_t length,\n+\t\t\t      void **mmap, int prot, int flags,\n+\t\t\t      const char *path, unsigned long page_size)\n+{\n+\tsize_t real_length;\n+\tvoid *last_page;\n+\tHANDLE h;\n+\n+\tassert(perm   == 0700);\n+\tassert(oflag  == (O_CREAT | O_EXCL | O_RDWR));\n+\tassert(prot   == (PROT_READ | PROT_WRITE));\n+\tassert(flags  == MAP_SHARED);\n+\tassert(length >= 0);\n+\n+\treal_length = length;\n+\tif (real_length % page_size)\n+\t\treal_length += page_size - (real_length % page_size);\n+\treal_length += page_size;\n+\th = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0,\n+\t\t\t      real_length, path);\n+\tif (!h)\n+\t\treturn -1;\n+\t*mmap = MapViewOfFile(h, FILE_MAP_ALL_ACCESS, 0, 0, real_length);\n+\tCloseHandle(h);\n+\tif (!*mmap)\n+\t\treturn -1;\n+\tlast_page = (unsigned char *)*mmap + real_length - page_size;\n+\t*(unsigned long *)last_page = length;\n+\treturn length;\n+}\n+\n+static ssize_t open_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t\t    int prot, int flags, const char *path,\n+\t\t\t    unsigned long page_size)\n+{\n+\tvoid *last_page;\n+\tHANDLE h;\n+\n+\tassert(perm   == 0700);\n+\tassert(oflag  == O_RDONLY);\n+\tassert(prot   == PROT_READ);\n+\tassert(flags  == MAP_SHARED);\n+\tassert(length <= 0);\n+\n+\th = OpenFileMapping(FILE_MAP_READ, FALSE, path);\n+\tif (!h)\n+\t\treturn -1;\n+\t*mmap = MapViewOfFile(h, FILE_MAP_READ, 0, 0, 0);\n+\tCloseHandle(h);\n+\tif (!*mmap)\n+\t\treturn -1;\n+\tif (length < 0) {\n+\t\tMEMORY_BASIC_INFORMATION mbi;\n+\t\tif (!VirtualQuery(*mmap, &mbi, sizeof(mbi))) {\n+\t\t\tUnmapViewOfFile(*mmap);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (mbi.RegionSize % page_size)\n+\t\t\tdie(\"expected size %lu to be %lu aligned\",\n+\t\t\t\t    mbi.RegionSize, page_size);\n+\t\tlast_page = (unsigned char *)*mmap + mbi.RegionSize - page_size;\n+\t\tlength = *(unsigned long *)last_page;\n+\t}\n+\treturn length;\n+}\n+\n+ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n+\t\t    int prot, int flags, const char *fmt, ...)\n+{\n+\tSYSTEM_INFO si;\n+\tva_list ap;\n+\tchar path[SHM_PATH_LEN];\n+\n+\tGetSystemInfo(&si);\n+\n+\tstrcpy(path, \"Local\\\\\");\n+\tva_start(ap, fmt);\n+\tvsprintf(path + strlen(path), fmt, ap);\n+\tva_end(ap);\n+\n+\tif (oflag & O_CREAT)\n+\t\treturn create_shm_map(oflag, perm, length, mmap, prot,\n+\t\t\t\t      flags, path, si.dwPageSize);\n+\telse\n+\t\treturn open_shm_map(oflag, perm, length, mmap, prot,\n+\t\t\t\t    flags, path, si.dwPageSize);\n+}\n+\n+void git_shm_unlink(const char *fmt, ...)\n+{\n+}\n+\n #else\n \n ssize_t git_shm_map(int oflag, int perm, ssize_t length, void **mmap,\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243731","messageId":"1402406665-27988-5-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":"1402406665-27988-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/5] daemonize(): set a flag before exiting the main process","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:24Z","receivedAt":"2014-06-10T13:24:24Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This allows signal handlers and atexit functions to realize this\nsituation and not clean up.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/gc.c | 2 +-\n cache.h      | 2 +-\n daemon.c     | 2 +-\n setup.c      | 4 +++-\n 4 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 85f5c2b..50275af 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -325,7 +325,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t * failure to daemonize is ok, we'll continue\n \t\t\t * in foreground\n \t\t\t */\n-\t\t\tdaemonize();\n+\t\t\tdaemonize(NULL);\n \t} else\n \t\tadd_repack_all_option();\n \ndiff --git a/cache.h b/cache.h\nindex f05e062..6f4b863 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -450,7 +450,7 @@ extern int set_git_dir_init(const char *git_dir, const char *real_git_dir, int);\n extern int init_db(const char *template_dir, unsigned int flags);\n \n extern void sanitize_stdfds(void);\n-extern int daemonize(void);\n+extern int daemonize(int *);\n \n #define alloc_nr(x) (((x)+16)*3/2)\n \ndiff --git a/daemon.c b/daemon.c\nindex eba1255..2650504 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1311,7 +1311,7 @@ int main(int argc, char **argv)\n \t\treturn execute();\n \n \tif (detach) {\n-\t\tif (daemonize())\n+\t\tif (daemonize(NULL))\n \t\t\tdie(\"--detach not supported on this platform\");\n \t} else\n \t\tsanitize_stdfds();\ndiff --git a/setup.c b/setup.c\nindex 613e3b3..e8e129a 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -842,7 +842,7 @@ void sanitize_stdfds(void)\n \t\tclose(fd);\n }\n \n-int daemonize(void)\n+int daemonize(int *daemonized)\n {\n #ifdef NO_POSIX_GOODIES\n \terrno = ENOSYS;\n@@ -854,6 +854,8 @@ int daemonize(void)\n \t\tcase -1:\n \t\t\tdie_errno(\"fork failed\");\n \t\tdefault:\n+\t\t\tif (daemonized)\n+\t\t\t\t*daemonized = 1;\n \t\t\texit(0);\n \t}\n \tif (setsid() == -1)\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243732","messageId":"1402406665-27988-6-git-send-email-pclouds@gmail.com","threadId":"36881","inReplyTo":"1402406665-27988-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/5] index-helper: add --detach","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-06-10T13:24:25Z","receivedAt":"2014-06-10T13:24:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-index-helper.txt | 4 +++-\n index-helper.c                     | 6 +++++-\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-index-helper.txt b/Documentation/git-index-helper.txt\nindex afeb334..303cda0 100644\n--- a/Documentation/git-index-helper.txt\n+++ b/Documentation/git-index-helper.txt\n@@ -8,7 +8,7 @@ git-index-helper - A simple cache server for speeding up index file access\n SYNOPSIS\n --------\n [verse]\n-'git index-helper\n+'git index-helper [--detach]\n \n DESCRIPTION\n -----------\n@@ -18,6 +18,8 @@ shared memory support (e.g. Linux or Windows)\n \n OPTIONS\n -------\n+--detach::\n+\tDetach from the shell.\n \n GIT\n ---\ndiff --git a/index-helper.c b/index-helper.c\nindex f92326a..9420591 100644\n--- a/index-helper.c\n+++ b/index-helper.c\n@@ -159,8 +159,9 @@ int main(int argc, char **argv)\n \tstatic struct lock_file lock;\n \tstruct strbuf sb = STRBUF_INIT;\n \tconst char *prefix;\n-\tint fd;\n+\tint fd, detach = 0;\n \tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"detach\", &detach, \"detach the process\"),\n \t\tOPT_END()\n \t};\n \n@@ -188,6 +189,9 @@ int main(int argc, char **argv)\n \tsigchain_push(SIGQUIT, cleanup_on_signal);\n \tsigchain_push(SIGPIPE, cleanup_on_signal);\n \n+\tif (detach && daemonize(&daemonized))\n+\t\tdie_errno(\"unable to detach\");\n+\n \tloop(sb.buf, 600);\n \tstrbuf_release(&sb);\n \treturn 0;\n-- \n1.9.1.346.ga2b5940\n"},{"id":"243752","messageId":"1402425859.18134.207.camel@stross","threadId":"36881","inReplyTo":"1402406665-27988-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/5] index-helper: new daemon for caching index and related stuff","fromName":"David Turner","fromEmail":"dturner@twopensource.com","sentAt":"2014-06-10T18:44:19Z","receivedAt":"2014-06-10T18:44:19Z","isPatch":true,"sender":{"key":"novalis@novalis.org","avatar":"https://avatars.githubusercontent.com/u/77003?v=4"},"body":"On Tue, 2014-06-10 at 20:24 +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> +\tloop(sb.buf, 600);\n...\n> +\tif (st->st_mtime + 600 < time(NULL))\n\ns/600/INDEX_HELPER_TIMEOUT/ or something.\n\n> +\t\treturn;\t  /* don't try to read from stale .pid file */\n> +\n> +\tfd = open(git_path(\"read-cache--daemon.pid\"), O_RDONLY);\n\nshould be index-helper.pid (actually, there are a few other instances of\nthis as well); maybe make it a constant.\n"}]}