git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 5/8] read-cache: try index data from shared memory

From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date
May 13, 2014, 11:15 UTC
Message-ID
<1399979737-8577-8-git-send-email-pclouds@gmail.com>
In-Reply-To
<1399979737-8577-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/config.txt |  4 ++++
 cache.h                  |  1 +
 config.c                 | 12 ++++++++++++
 environment.c            |  1 +
 read-cache.c             | 43 +++++++++++++++++++++++++++++++++++++++++++
 submodule.c              |  1 +
 6 files changed, 62 insertions(+)
diff --git a/Documentation/config.txt b/Documentation/config.txt
index d8b6cc9..ccbe00b 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -617,6 +617,10 @@ relatively high IO latencies.  With this set to 'true', Git will do the
 index comparison to the filesystem data in parallel, allowing
 overlapping IO's.
 
+core.useReadCacheDaemon::
+	Use `git read-cache--daemon` to speed up index reading. See
+	linkgit:git-read-cache--daemon for more information.
+
 core.createObject::
 	You can set this to 'link', in which case a hardlink followed by
 	a delete of the source are used to make sure that object creation
diff --git a/cache.h b/cache.h
index d0ff11c..fb29c7e 100644
--- a/cache.h
+++ b/cache.h
@@ -603,6 +603,7 @@ extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
 extern unsigned long pack_size_limit_cfg;
+extern int use_read_cache_daemon;
 
 /*
  * Do replace refs need to be checked this run?  This variable is
diff --git a/config.c b/config.c
index a30cb5c..5c832ad 100644
--- a/config.c
+++ b/config.c
@@ -874,6 +874,18 @@ static int git_default_core_config(const char *var, const char *value)
 		return 0;
 	}
 
+#ifdef HAVE_SHM
+	/*
+	 * Currently git-read-cache--daemon is only built when
+	 * HAVE_SHM is set. Ignore user settings if HAVE_SHM is not
+	 * defined.
+	 */
+	if (!strcmp(var, "core.usereadcachedaemon")) {
+		use_read_cache_daemon = git_config_bool(var, value);
+		return 0;
+	}
+#endif
+
 	/* Add other config variables here and to Documentation/config.txt. */
 	return 0;
 }
diff --git a/environment.c b/environment.c
index 5c4815d..b76a414 100644
--- a/environment.c
+++ b/environment.c
@@ -63,6 +63,7 @@ int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 struct startup_info *startup_info;
 unsigned long pack_size_limit_cfg;
+int use_read_cache_daemon;
 
 /*
  * The character that begins a commented line in user-editable file
diff --git a/read-cache.c b/read-cache.c
index a5031f3..0e46523 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -1462,6 +1462,48 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,
 	return ce;
 }
 
+static void *try_shm(void *mmap, size_t *mmap_size)
+{
+	struct strbuf sb = STRBUF_INIT;
+	size_t old_size = *mmap_size;
+	void *new_mmap;
+	struct stat st;
+	int fd;
+
+	if (old_size <= 20 || !use_read_cache_daemon)
+		return mmap;
+
+	strbuf_addf(&sb, "/git-index-%s.lock",
+		    sha1_to_hex((unsigned char *)mmap + old_size - 20));
+	fd = shm_open(sb.buf, O_RDONLY, 0777);
+	if (fd >= 0) {
+		close(fd);
+		return mmap;
+	}
+	strbuf_setlen(&sb, sb.len - 5); /* no ".lock" */
+	fd = shm_open(sb.buf, O_RDONLY, 0777);
+	strbuf_release(&sb);
+	if (fd < 0)
+		/*
+		 * git-read-cache--daemon is probably not started yet. For
+		 * simplicity, only start it at the next index update, which
+		 * should happen often.
+		 */
+		return mmap;
+	if (fstat(fd, &st)) {
+		close(fd);
+		return mmap;
+	}
+	new_mmap = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+	close(fd);
+	if (new_mmap == MAP_FAILED)
+		return mmap;
+	munmap(mmap, old_size);
+	*mmap_size = st.st_size;
+	return new_mmap;
+}
+
+
 /* remember to discard_cache() before reading a different cache! */
 int do_read_index(struct index_state *istate, const char *path, int must_exist)
 {
@@ -1501,6 +1543,7 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)
 	}
 	close(fd);
 
+	mmap = try_shm(mmap, &mmap_size);
 	hdr = mmap;
 	if (verify_hdr(hdr, mmap_size) < 0)
 		goto unmap;
diff --git a/submodule.c b/submodule.c
index b80ecac..9872928 100644
--- a/submodule.c
+++ b/submodule.c
@@ -195,6 +195,7 @@ void gitmodules_config(void)
 		int pos;
 		strbuf_addstr(&gitmodules_path, work_tree);
 		strbuf_addstr(&gitmodules_path, "/.gitmodules");
+		git_config(git_default_config, NULL);
 		if (read_cache() < 0)
 			die("index file corrupt");
 		pos = cache_name_pos(".gitmodules", 11);
-- 
1.9.1.346.ga2b5940
Previous: Erik Faye-LundNext: Erik Faye-Lund
Message 63 of 75 in “Split index mode for very large indexes”
  1. 00/32 Split index mode for very large indexesNguyễn Thái Ngọc Duy, Apr 28, 2014
  2. 01/32 ewah: fix constness of ewah_read_mmapNguyễn Thái Ngọc Duy, Apr 28, 2014
  3. 02/32 ewah: delete unused ewah_read_mmap_native declarationNguyễn Thái Ngọc Duy, Apr 28, 2014
  4. 03/32 sequencer: do not update/refresh index if the lock cannot be heldNguyễn Thái Ngọc Duy, Apr 28, 2014
  5. 04/32 read-cache: new API write_locked_index instead of write_index/write_cacheNguyễn Thái Ngọc Duy, Apr 28, 2014
  6. 05/32 read-cache: relocate and unexport commit_locked_index()Nguyễn Thái Ngọc Duy, Apr 28, 2014
  7. 06/32 read-cache: store in-memory flags in the first 12 bits of ce_flagsNguyễn Thái Ngọc Duy, Apr 28, 2014
  8. 07/32 read-cache: be strict about "changed" in remove_marked_cache_entries()Nguyễn Thái Ngọc Duy, Apr 28, 2014
  9. 08/32 read-cache: be specific what part of the index has changedNguyễn Thái Ngọc Duy, Apr 28, 2014
  10. 09/32 update-index: be specific what part of the index has changedNguyễn Thái Ngọc Duy, Apr 28, 2014
  11. 10/32 resolve-undo: be specific what part of the index has changedNguyễn Thái Ngọc Duy, Apr 28, 2014
  12. 11/32 unpack-trees: be specific what part of the index has changedNguyễn Thái Ngọc Duy, Apr 28, 2014
  13. 12/32 cache-tree: mark istate->cache_changed on cache tree invalidationNguyễn Thái Ngọc Duy, Apr 28, 2014
  14. 13/32 cache-tree: mark istate->cache_changed on cache tree updateNguyễn Thái Ngọc Duy, Apr 28, 2014
  15. 14/32 cache-tree: mark istate->cache_changed on prime_cache_tree()Nguyễn Thái Ngọc Duy, Apr 28, 2014
  16. 15/32 entry.c: update cache_changed if refresh_cache is set in checkout_entry()Nguyễn Thái Ngọc Duy, Apr 28, 2014
  17. 16/32 read-cache: save index SHA-1 after readingNguyễn Thái Ngọc Duy, Apr 28, 2014
  18. 17/32 read-cache: split-index modeNguyễn Thái Ngọc Duy, Apr 28, 2014
  19. Junio C HamanoApr 28, 2014
  20. Duy NguyenApr 29, 2014
  21. Junio C HamanoApr 29, 2014
  22. Duy NguyenApr 29, 2014
  23. Junio C HamanoApr 30, 2014
  24. 18/32 read-cache: mark new entries for split indexNguyễn Thái Ngọc Duy, Apr 28, 2014
  25. Eric SunshineApr 30, 2014
  26. 19/32 read-cache: save deleted entries in split indexNguyễn Thái Ngọc Duy, Apr 28, 2014
  27. 20/32 read-cache: mark updated entries for split indexNguyễn Thái Ngọc Duy, Apr 28, 2014
  28. 21/32 split-index: the writing partNguyễn Thái Ngọc Duy, Apr 28, 2014
  29. 22/32 split-index: the reading partNguyễn Thái Ngọc Duy, Apr 28, 2014
  30. 23/32 split-index: do not invalidate cache-tree at read timeNguyễn Thái Ngọc Duy, Apr 28, 2014
  31. 24/32 split-index: strip pathname of on-disk replaced entriesNguyễn Thái Ngọc Duy, Apr 28, 2014
  32. Junio C HamanoApr 29, 2014
  33. 25/32 update-index: new options to enable/disable split index modeNguyễn Thái Ngọc Duy, Apr 28, 2014
  34. 26/32 update-index --split-index: do not split if $GIT_DIR is read onlyNguyễn Thái Ngọc Duy, Apr 28, 2014
  35. 27/32 rev-parse: add --shared-index-path to get shared index pathNguyễn Thái Ngọc Duy, Apr 28, 2014
  36. 28/32 read-tree: force split-index mode off on --index-outputNguyễn Thái Ngọc Duy, Apr 28, 2014
  37. 29/32 read-tree: note about dropping split-index mode or index versionNguyễn Thái Ngọc Duy, Apr 28, 2014
  38. 30/32 read-cache: force split index mode with GIT_TEST_SPLIT_INDEXNguyễn Thái Ngọc Duy, Apr 28, 2014
  39. 31/32 t2104: make sure split index mode is off for the version testNguyễn Thái Ngọc Duy, Apr 28, 2014
  40. 32/32 t1700: new tests for split-index modeNguyễn Thái Ngọc Duy, Apr 28, 2014
  41. Shawn PearceApr 28, 2014
  42. Duy NguyenApr 29, 2014
  43. Duy NguyenMay 9, 2014
  44. Junio C HamanoMay 9, 2014
  45. 0/8 Speed up cache loading timeNguyễn Thái Ngọc Duy, May 13, 2014
  46. 1/8 read-cache: allow to keep mmap'd memory after readingNguyễn Thái Ngọc Duy, May 13, 2014
  47. 2/3 Add read-cache--daemonNguyễn Thái Ngọc Duy, May 13, 2014
  48. Erik Faye-LundMay 13, 2014
  49. Duy NguyenMay 13, 2014
  50. Duy NguyenMay 13, 2014
  51. Erik Faye-LundMay 13, 2014
  52. Duy NguyenMay 13, 2014
  53. Erik Faye-LundMay 13, 2014
  54. Duy NguyenMay 13, 2014
  55. Erik Faye-LundMay 13, 2014
  56. 2/8 unix-socket: stub impl. for platforms with no unix socket supportNguyễn Thái Ngọc Duy, May 13, 2014
  57. Erik Faye-LundMay 13, 2014
  58. Erik Faye-LundMay 13, 2014
  59. 3/8 daemonize: set a flag before exiting the main processNguyễn Thái Ngọc Duy, May 13, 2014
  60. 3/3 read-cache: try index data from shared memoryNguyễn Thái Ngọc Duy, May 13, 2014
  61. 4/8 Add read-cache--daemon for caching index and related stuffNguyễn Thái Ngọc Duy, May 13, 2014
  62. Erik Faye-LundMay 13, 2014
  63. 5/8 read-cache: try index data from shared memoryNguyễn Thái Ngọc Duy, May 13, 2014
  64. Erik Faye-LundMay 13, 2014
  65. 6/8 read-cache--daemon: do not read index from shared memoryNguyễn Thái Ngọc Duy, May 13, 2014
  66. 7/8 read-cache: skip verifying trailing SHA-1 on cached indexNguyễn Thái Ngọc Duy, May 13, 2014
  67. 8/8 read-cache: inform the daemon that the index has been updatedNguyễn Thái Ngọc Duy, May 13, 2014
  68. Erik Faye-LundMay 13, 2014
  69. David TurnerMay 22, 2014
  70. Stefan BellerMay 13, 2014
  71. Duy NguyenMay 13, 2014
  72. 9/8 even faster loading time with index version 254Nguyễn Thái Ngọc Duy, May 13, 2014
  73. Junio C HamanoApr 28, 2014
  74. Richard HansenApr 30, 2014
  75. Duy NguyenMay 1, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.