{"thread":{"id":"17806","subject":"[PATCH/RFC v1 0/2] The ext4 filesystem and racy git","startedAt":"2009-02-15T19:46:13Z","lastAt":"2009-02-15T19:46:15Z","messageCount":3,"participants":["Kjetil Barvik"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"104810","messageId":"cover.1234720401.git.barvik@broadpark.no","threadId":"17806","inReplyTo":null,"subject":"[PATCH/RFC v1 0/2] The ext4 filesystem and racy git","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-02-15T19:46:13Z","receivedAt":"2009-02-15T19:46:13Z","isPatch":true,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"Ext4 is marked stable in Linux v2.6.28, and I have done a very simple\ntest on one disk-partition of my slow laptop, and the numbers look\nvery promising.\n\nWith default created ext3 disk-partition on my harddisk the 'git\ncheckout -q my-v2.6.25/27' test takes around 20 seconds real time for\nthe best cases.  The same test run on a ext4 partition takes around 14\nseconds for the best cases.\n\nAnd, since ext4 supports nanosecond timestamps, when I added patch\n2/2, the checkout time is much more stable and more close to 14-15\nseconds most of the time.\n\nConclusion: for GIT on my laptop the ext4 filesystem is a speedup!\n\n>From '/proc/mounts' here is the mount options:\n\n  /dev/hda10 /home ext3 rw,noatime,errors=continue,data=ordered 0 0\n  /dev/hda12 /opt2 ext4 rw,noatime,barrier=1,journal_async_commit,data=ordered 0 0\n\n/dev/hda10 is formatted with default (gentoo) ext3 parameters, and the\n/dev/hda12 is formatted with:\n\n  /sbin/mkfs -t ext4 -I 256 -G 64 -Oflex_bg,uninit_bg /dev/hda12\n\n\nKjetil Barvik (2):\n  fix compile error when USE_NSEC is defined\n  make USE_NSEC work as expected\n\n builtin-fetch-pack.c |    4 +-\n cache.h              |    6 ++--\n read-cache.c         |   70 ++++++++++++++++++++++++++++++++++++++++----------\n 3 files changed, 61 insertions(+), 19 deletions(-)\n"},{"id":"104811","messageId":"43e621190d3e766ea650a4c13e33d6976dde64ad.1234720401.git.barvik@broadpark.no","threadId":"17806","inReplyTo":"cover.1234720401.git.barvik@broadpark.no","subject":"[PATCH/RFC v1 1/2] fix compile error when USE_NSEC is defined","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-02-15T19:46:14Z","receivedAt":"2009-02-15T19:46:14Z","isPatch":true,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"'struct cache' does not have a 'usec' member, but a 'unsigned int\nnsec' member.  Simmilar 'struct stat' does not have a 'st_mtim.usec'\nmember, and we should instead use 'st_mtim.tv_nsec'.\n\nSigned-off-by: Kjetil Barvik <barvik@broadpark.no>\n---\n builtin-fetch-pack.c |    4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex 67fb80e..3b210c7 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -802,14 +802,14 @@ struct ref *fetch_pack(struct fetch_pack_args *my_args,\n \n \t\tmtime.sec = st.st_mtime;\n #ifdef USE_NSEC\n-\t\tmtime.usec = st.st_mtim.usec;\n+\t\tmtime.nsec = st.st_mtim.tv_nsec;\n #endif\n \t\tif (stat(shallow, &st)) {\n \t\t\tif (mtime.sec)\n \t\t\t\tdie(\"shallow file was removed during fetch\");\n \t\t} else if (st.st_mtime != mtime.sec\n #ifdef USE_NSEC\n-\t\t\t\t|| st.st_mtim.usec != mtime.usec\n+\t\t\t\t|| st.st_mtim.tv_nsec != mtime.nsec\n #endif\n \t\t\t  )\n \t\t\tdie(\"shallow file was changed during fetch\");\n-- \n1.6.1.349.g99fa5\n"},{"id":"104812","messageId":"2a800178e58702215aaa0436221412a49a5b3f66.1234720401.git.barvik@broadpark.no","threadId":"17806","inReplyTo":"cover.1234720401.git.barvik@broadpark.no","subject":"[PATCH/RFC v1 2/2] make USE_NSEC work as expected","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-02-15T19:46:15Z","receivedAt":"2009-02-15T19:46:15Z","isPatch":true,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"Since the filesystem ext4 is now defined as stable in Linux v2.6.28,\nand ext4 supports nanonsecond resolution timestamps natively, it is\ntime to make USE_NSEC work as expected.\n\nThis will make racy git situations less likely to happen.  For 'git\ncheckout' this means it will be less likely that we have to open, read\nthe contents of the file into RAM, and check if file is really\nmodified or not.  The result sould be a litle less used CPU time, less\npagefaults and a litle faster program, at least for 'git checkout'.\n\nSince the number of possible racy git situations would increase when\ndisks gets faster, this patch would be more and more helpfull as times\ngo by.  For a fast Solid State Disk, this patch should be helpfull.\n\nNote that, when file operations starts to take less than 1 nanosecond,\none would again start to get more racy git situations.\n\nFor more info on racy git, see Documentation/technical/racy-git.txt\nFor more info on ext4, see http://kernelnewbies.org/Ext4\n\nSigned-off-by: Kjetil Barvik <barvik@broadpark.no>\n---\n cache.h      |    6 ++--\n read-cache.c |   70 ++++++++++++++++++++++++++++++++++++++++++++++-----------\n 2 files changed, 59 insertions(+), 17 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 37dfb1c..309053d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -140,8 +140,8 @@ struct ondisk_cache_entry_extended {\n };\n \n struct cache_entry {\n-\tunsigned int ce_ctime;\n-\tunsigned int ce_mtime;\n+\tstruct cache_time ce_ctime;\n+\tstruct cache_time ce_mtime;\n \tunsigned int ce_dev;\n \tunsigned int ce_ino;\n \tunsigned int ce_mode;\n@@ -282,7 +282,7 @@ struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int cache_nr, cache_alloc, cache_changed;\n \tstruct cache_tree *cache_tree;\n-\ttime_t timestamp;\n+\tstruct cache_time timestamp;\n \tvoid *alloc;\n \tunsigned name_hash_initialized : 1,\n \t\t initialized : 1;\ndiff --git a/read-cache.c b/read-cache.c\nindex 940ec76..ca4bec2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -67,8 +67,15 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n  */\n void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n {\n-\tce->ce_ctime = st->st_ctime;\n-\tce->ce_mtime = st->st_mtime;\n+\tce->ce_ctime.sec = (unsigned int)st->st_ctime;\n+\tce->ce_mtime.sec = (unsigned int)st->st_mtime;\n+#ifdef USE_NSEC\n+\tce->ce_ctime.nsec = (unsigned int)st->st_ctim.tv_nsec;\n+\tce->ce_mtime.nsec = (unsigned int)st->st_mtim.tv_nsec;\n+#else\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = 0;\n+#endif\n \tce->ce_dev = st->st_dev;\n \tce->ce_ino = st->st_ino;\n \tce->ce_uid = st->st_uid;\n@@ -196,11 +203,18 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n \tdefault:\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n-\tif (ce->ce_mtime != (unsigned int) st->st_mtime)\n+\tif (ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n \t\tchanged |= MTIME_CHANGED;\n-\tif (trust_ctime && ce->ce_ctime != (unsigned int) st->st_ctime)\n+\tif (trust_ctime && ce->ce_ctime.sec != (unsigned int)st->st_ctime)\n \t\tchanged |= CTIME_CHANGED;\n \n+#ifdef USE_NSEC\n+\tif (ce->ce_mtime.nsec != (unsigned int)st->st_mtim.tv_nsec)\n+\t\tchanged |= MTIME_CHANGED;\n+\tif (trust_ctime && ce->ce_ctime.nsec != (unsigned int)st->st_ctim.tv_nsec)\n+\t\tchanged |= CTIME_CHANGED;\n+#endif\n+\n \tif (ce->ce_uid != (unsigned int) st->st_uid ||\n \t    ce->ce_gid != (unsigned int) st->st_gid)\n \t\tchanged |= OWNER_CHANGED;\n@@ -232,8 +246,16 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n static int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce)\n {\n \treturn (!S_ISGITLINK(ce->ce_mode) &&\n-\t\tistate->timestamp &&\n-\t\t((unsigned int)istate->timestamp) <= ce->ce_mtime);\n+\t\tistate->timestamp.sec &&\n+#ifdef USE_NSEC\n+\t\t /* nanosecond timestamped files can also be racy! */\n+\t\t(istate->timestamp.sec < ce->ce_mtime.sec ||\n+\t\t (istate->timestamp.sec == ce->ce_mtime.sec &&\n+\t\t  istate->timestamp.nsec <= ce->ce_mtime.nsec))\n+#else\n+\t\tistate->timestamp.sec <= ce->ce_mtime.sec\n+#endif\n+\t\t );\n }\n \n int ie_match_stat(const struct index_state *istate,\n@@ -1139,8 +1161,15 @@ static void convert_from_disk(struct ondisk_cache_entry *ondisk, struct cache_en\n \tsize_t len;\n \tconst char *name;\n \n-\tce->ce_ctime = ntohl(ondisk->ctime.sec);\n-\tce->ce_mtime = ntohl(ondisk->mtime.sec);\n+\tce->ce_ctime.sec = ntohl(ondisk->ctime.sec);\n+\tce->ce_mtime.sec = ntohl(ondisk->mtime.sec);\n+#ifdef USE_NSEC\n+\tce->ce_ctime.nsec = ntohl(ondisk->ctime.nsec);\n+\tce->ce_mtime.nsec = ntohl(ondisk->mtime.nsec);\n+#else\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = 0;\n+#endif\n \tce->ce_dev   = ntohl(ondisk->dev);\n \tce->ce_ino   = ntohl(ondisk->ino);\n \tce->ce_mode  = ntohl(ondisk->mode);\n@@ -1206,7 +1235,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\treturn istate->cache_nr;\n \n \terrno = ENOENT;\n-\tistate->timestamp = 0;\n+\tistate->timestamp.sec = 0;\n+\tistate->timestamp.nsec = 0;\n \tfd = open(path, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n@@ -1258,7 +1288,13 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += ondisk_ce_size(ce);\n \t\tdst_offset += ce_size(ce);\n \t}\n-\tistate->timestamp = st.st_mtime;\n+\tistate->timestamp.sec = st.st_mtime;\n+#ifdef USE_NSEC\n+\tistate->timestamp.nsec = (unsigned int)st.st_mtim.tv_nsec;\n+#else\n+\tistate->timestamp.nsec = 0;\n+#endif\n+\n \twhile (src_offset <= mmap_size - 20 - 8) {\n \t\t/* After an array of active_nr index entries,\n \t\t * there can be arbitrary number of extended\n@@ -1288,14 +1324,15 @@ unmap:\n \n int is_index_unborn(struct index_state *istate)\n {\n-\treturn (!istate->cache_nr && !istate->alloc && !istate->timestamp);\n+\treturn (!istate->cache_nr && !istate->alloc && !istate->timestamp.sec);\n }\n \n int discard_index(struct index_state *istate)\n {\n \tistate->cache_nr = 0;\n \tistate->cache_changed = 0;\n-\tistate->timestamp = 0;\n+\tistate->timestamp.sec = 0;\n+\tistate->timestamp.nsec = 0;\n \tistate->name_hash_initialized = 0;\n \tfree_hash(&istate->name_hash);\n \tcache_tree_free(&(istate->cache_tree));\n@@ -1441,10 +1478,15 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce)\n \tstruct ondisk_cache_entry *ondisk = xcalloc(1, size);\n \tchar *name;\n \n-\tondisk->ctime.sec = htonl(ce->ce_ctime);\n+\tondisk->ctime.sec = htonl(ce->ce_ctime.sec);\n+\tondisk->mtime.sec = htonl(ce->ce_mtime.sec);\n+#ifdef USE_NSEC\n+\tondisk->ctime.nsec = htonl(ce->ce_ctime.nsec);\n+\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n+#else\n \tondisk->ctime.nsec = 0;\n-\tondisk->mtime.sec = htonl(ce->ce_mtime);\n \tondisk->mtime.nsec = 0;\n+#endif\n \tondisk->dev  = htonl(ce->ce_dev);\n \tondisk->ino  = htonl(ce->ce_ino);\n \tondisk->mode = htonl(ce->ce_mode);\n-- \n1.6.1.349.g99fa5\n"}]}