{"thread":{"id":"38142","subject":"[PATCH v3 00/23] nd/untracked-cache updates","startedAt":"2014-12-08T13:58:17Z","lastAt":"2014-12-15T19:35:37Z","messageCount":35,"participants":["Nguyễn Thái Ngọc Duy","Duy Nguyen","brian m. carlson","Eric Sunshine","Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":23},"messages":[{"id":"253334","messageId":"CACsJy8Aifd_Ly=G=fHXaAMq8p9mdVZWEUpDD=dQeizd1B142wQ@mail.gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] checkout: add --ignore-other-wortrees","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T13:58:17Z","receivedAt":"2014-12-08T13:58:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Oops. This one does not belong to the series. I cleaned up `pwd`, then\njumped to another one for testing and forgot to clean up patches again\n:(\n-- \nDuy\n"},{"id":"253309","messageId":"1418047507-22892-1-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":null,"subject":"[PATCH v3 00/23] nd/untracked-cache updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:43Z","receivedAt":"2014-12-08T14:04:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Compare to v1 (the one on 'pu' as v2 never got to 'pu'):\n\n - New cleanup patch 09/23\n\n - New patch 17/23 allows to ignore untracked cache without destroying\n   it (for comparison and verification)\n\n - New patches 22/23 and 23/23 add some protection against filesystem\n   or operating system changes\n\n - Document UNTR extension\n\n - Reorganize UNTR to avoid saving SHA-1/stat data when it's useless\n\n - Fix writing UNTR to base index in split index mode\n\n - Various review comments since v1\n\n-- 8< --\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex fe6f316..5dc2bee 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -233,3 +233,64 @@ Git index format\n   The remaining index entries after replaced ones will be added to the\n   final index. These added entries are also sorted by entry namme then\n   stage.\n+\n+== Untracked cache\n+\n+  Untracked cache saves the untracked file list and necessary data to\n+  verify the cache. The signature for this extension is { 'U', 'N',\n+  'T', 'R' }.\n+\n+  The extension starts with\n+\n+  - A NUL-terminated string describing the environment when the cache\n+    is created.\n+\n+  - Stat data of $GIT_DIR/info/exclude. See \"Index entry\" section from\n+    ctime field until \"file size\".\n+\n+  - Stat data of core.excludesfile\n+\n+  - 32-bit dir_flags (see struct dir_struct)\n+\n+  - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n+    does not exist.\n+\n+  - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n+    not exist.\n+\n+  - NUL-terminated string of per-dir exclude file name. This usually\n+    is \".gitignore\".\n+\n+  - The number of following directory blocks, variable width\n+    encoding. If this number is zero, the extension ends here with a\n+    following NUL.\n+\n+  - A number of directory blocks in depth-first-search order, each\n+    consists of\n+\n+    - The number of untracked entries, variable witdh encoding.\n+\n+    - The number of sub-directory blocks, variable with encoding.\n+\n+    - The directory name terminated by NUL.\n+\n+    - A number of untrached file/dir names terminated by NUL.\n+\n+The remaining data of each directory block is grouped by type:\n+\n+  - An ewah bitmap, the n-th bit marks whether the n-th directory has\n+    valid untracked cache entries.\n+\n+  - An ewah bitmap, the n-th bit records \"check-only\" bit of\n+    read_directory_recursive() for the n-th directory.\n+\n+  - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n+    is valid for the n-th directory and exists in the next data.\n+\n+  - An array of stat data. The n-th data corresponds with the n-th\n+    \"one\" bit in the previous ewah bitmap.\n+\n+  - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n+    in the previous ewah bitmap.\n+\n+  - One NUL.\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex c1c18db..f23ec83 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -115,6 +115,7 @@ static int test_if_untracked_cache_is_supported(void)\n \tfd = create_file(\"dir-mtime-test/newfile\");\n \txstat(\"dir-mtime-test\", &st);\n \tif (!match_stat_data(&base, &st)) {\n+\t\tclose(fd);\n \t\tfputc('\\n', stderr);\n \t\tfprintf_ln(stderr,_(\"directory stat info does not \"\n \t\t\t\t    \"change after adding a new file\"));\n@@ -127,6 +128,7 @@ static int test_if_untracked_cache_is_supported(void)\n \txmkdir(\"dir-mtime-test/new-dir\");\n \txstat(\"dir-mtime-test\", &st);\n \tif (!match_stat_data(&base, &st)) {\n+\t\tclose(fd);\n \t\tfputc('\\n', stderr);\n \t\tfprintf_ln(stderr, _(\"directory stat info does not change \"\n \t\t\t\t     \"after adding a new directory\"));\n@@ -1094,10 +1096,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t/* should be the same flags used by git-status */\n \t\tuc->dir_flags = DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;\n \t\tthe_index.untracked = uc;\n-\t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n+\t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n \t} else if (!untracked_cache && the_index.untracked) {\n \t\tthe_index.untracked = NULL;\n-\t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n+\t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n \t}\n \n \tif (active_cache_changed) {\ndiff --git a/cache.h b/cache.h\nindex 201b22e..fca979b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -295,7 +295,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define RESOLVE_UNDO_CHANGED\t(1 << 4)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n-#define UNTRACKED_CHANGED       (1 << 7)\n+#define UNTRACKED_CHANGED\t(1 << 7)\n \n struct split_index;\n struct untracked_cache;\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex c5c37e5..88140e4 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2128,3 +2128,14 @@ void mingw_startup()\n \t/* initialize Unicode console */\n \twinansi_init();\n }\n+\n+int uname(struct utsname *buf)\n+{\n+\tDWORD v = GetVersion();\n+\tmemset(buf, 0, sizeof(*buf));\n+\tstrcpy(buf->sysname, \"Windows\");\n+\tsprintf(buf->release, \"%u.%u\", v & 0xff, (v >> 8) & 0xff);\n+\t/* assuming NT variants only.. */\n+\tsprintf(buf->version, \"%u\", (v >> 16) & 0x7fff);\n+\treturn 0;\n+}\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex df0e320..d00ba7a 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -77,6 +77,14 @@ struct itimerval {\n };\n #define ITIMER_REAL 0\n \n+struct utsname {\n+\tchar sysname[16];\n+\tchar nodename[1];\n+\tchar release[16];\n+\tchar version[16];\n+\tchar machine[1];\n+};\n+\n /*\n  * sanitize preprocessor namespace polluted by Windows headers defining\n  * macros which collide with git local versions\n@@ -166,6 +174,7 @@ struct passwd *getpwuid(uid_t uid);\n int setitimer(int type, struct itimerval *in, struct itimerval *out);\n int sigaction(int sig, struct sigaction *in, struct sigaction *out);\n int link(const char *oldpath, const char *newpath);\n+int uname(struct utsname *buf);\n \n /*\n  * replacements of existing functions\ndiff --git a/dir.c b/dir.c\nindex 0ae2188..ef58547 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -13,6 +13,7 @@\n #include \"wildmatch.h\"\n #include \"pathspec.h\"\n #include \"varint.h\"\n+#include \"ewah/ewok.h\"\n \n struct path_simplify {\n \tint len;\n@@ -482,7 +483,7 @@ void add_exclude(const char *string, const char *base,\n }\n \n static void *read_skip_worktree_file_from_index(const char *path, size_t *size,\n-\t\t\t\t\t\tstruct sha1_stat *ss)\n+\t\t\t\t\t\tstruct sha1_stat *sha1_stat)\n {\n \tint pos, len;\n \tunsigned long sz;\n@@ -501,9 +502,9 @@ static void *read_skip_worktree_file_from_index(const char *path, size_t *size,\n \t\treturn NULL;\n \t}\n \t*size = xsize_t(sz);\n-\tif (ss) {\n-\t\tmemset(&ss->stat, 0, sizeof(ss->stat));\n-\t\thashcpy(ss->sha1, active_cache[pos]->sha1);\n+\tif (sha1_stat) {\n+\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n+\t\thashcpy(sha1_stat->sha1, active_cache[pos]->sha1);\n \t}\n \treturn data;\n }\n@@ -584,9 +585,10 @@ static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \t}\n \n \tuc->dir_created++;\n-\td = xmalloc(sizeof(*d) + len);\n-\tmemset(d, 0, sizeof(*d) + len);\n+\td = xmalloc(sizeof(*d) + len + 1);\n+\tmemset(d, 0, sizeof(*d));\n \tmemcpy(d->name, name, len);\n+\td->name[len] = '\\0';\n \n \tALLOC_GROW(dir->dirs, dir->dirs_nr + 1, dir->dirs_alloc);\n \tmemmove(dir->dirs + first + 1, dir->dirs + first,\n@@ -634,7 +636,7 @@ static void invalidate_directory(struct untracked_cache *uc,\n  */\n static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t\tstruct exclude_list *el, int check_index,\n-\t\t\tstruct sha1_stat *ss, int ss_valid)\n+\t\t\tstruct sha1_stat *sha1_stat)\n {\n \tstruct stat st;\n \tint fd, i, lineno = 1;\n@@ -648,7 +650,7 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\tif (0 <= fd)\n \t\t\tclose(fd);\n \t\tif (!check_index ||\n-\t\t    (buf = read_skip_worktree_file_from_index(fname, &size, ss)) == NULL)\n+\t\t    (buf = read_skip_worktree_file_from_index(fname, &size, sha1_stat)) == NULL)\n \t\t\treturn -1;\n \t\tif (size == 0) {\n \t\t\tfree(buf);\n@@ -661,9 +663,10 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t} else {\n \t\tsize = xsize_t(st.st_size);\n \t\tif (size == 0) {\n-\t\t\tif (ss) {\n-\t\t\t\tfill_stat_data(&ss->stat, &st);\n-\t\t\t\thashcpy(ss->sha1, EMPTY_BLOB_SHA1_BIN);\n+\t\t\tif (sha1_stat) {\n+\t\t\t\tfill_stat_data(&sha1_stat->stat, &st);\n+\t\t\t\thashcpy(sha1_stat->sha1, EMPTY_BLOB_SHA1_BIN);\n+\t\t\t\tsha1_stat->valid = 1;\n \t\t\t}\n \t\t\tclose(fd);\n \t\t\treturn 0;\n@@ -676,19 +679,20 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\t}\n \t\tbuf[size++] = '\\n';\n \t\tclose(fd);\n-\t\tif (ss) {\n+\t\tif (sha1_stat) {\n \t\t\tint pos;\n-\t\t\tif (ss_valid &&\n-\t\t\t    !match_stat_data_racy(&the_index, &ss->stat, &st))\n+\t\t\tif (sha1_stat->valid &&\n+\t\t\t    !match_stat_data_racy(&the_index, &sha1_stat->stat, &st))\n \t\t\t\t; /* no content change, ss->sha1 still good */\n \t\t\telse if (check_index &&\n \t\t\t\t (pos = cache_name_pos(fname, strlen(fname))) >= 0 &&\n \t\t\t\t !ce_stage(active_cache[pos]) &&\n \t\t\t\t ce_uptodate(active_cache[pos]))\n-\t\t\t\thashcpy(ss->sha1, active_cache[pos]->sha1);\n+\t\t\t\thashcpy(sha1_stat->sha1, active_cache[pos]->sha1);\n \t\t\telse\n-\t\t\t\thash_sha1_file(buf, size, \"blob\", ss->sha1);\n-\t\t\tfill_stat_data(&ss->stat, &st);\n+\t\t\t\thash_sha1_file(buf, size, \"blob\", sha1_stat->sha1);\n+\t\t\tfill_stat_data(&sha1_stat->stat, &st);\n+\t\t\tsha1_stat->valid = 1;\n \t\t}\n \t}\n \n@@ -712,7 +716,7 @@ int add_excludes_from_file_to_list(const char *fname, const char *base,\n \t\t\t\t   int baselen, struct exclude_list *el,\n \t\t\t\t   int check_index)\n {\n-\treturn add_excludes(fname, base, baselen, el, check_index, NULL, 0);\n+\treturn add_excludes(fname, base, baselen, el, check_index, NULL);\n }\n \n struct exclude_list *add_exclude_list(struct dir_struct *dir,\n@@ -733,7 +737,7 @@ struct exclude_list *add_exclude_list(struct dir_struct *dir,\n  * Used to set up core.excludesfile and .git/info/exclude lists.\n  */\n static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,\n-\t\t\t\t     struct sha1_stat *ss, int ss_valid)\n+\t\t\t\t     struct sha1_stat *sha1_stat)\n {\n \tstruct exclude_list *el;\n \t/*\n@@ -744,14 +748,14 @@ static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,\n \tif (!dir->untracked)\n \t\tdir->unmanaged_exclude_files++;\n \tel = add_exclude_list(dir, EXC_FILE, fname);\n-\tif (add_excludes(fname, \"\", 0, el, 0, ss, ss_valid) < 0)\n+\tif (add_excludes(fname, \"\", 0, el, 0, sha1_stat) < 0)\n \t\tdie(\"cannot use %s as an exclude file\", fname);\n }\n \n void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n {\n \tdir->unmanaged_exclude_files++; /* see validate_untracked_cache() */\n-\tadd_excludes_from_file_1(dir, fname, NULL, 0);\n+\tadd_excludes_from_file_1(dir, fname, NULL);\n }\n \n int match_basename(const char *basename, int basenamelen,\n@@ -974,7 +978,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \twhile (current < baselen) {\n \t\tstruct exclude_stack *stk = xcalloc(1, sizeof(*stk));\n \t\tconst char *cp;\n-\t\tstruct sha1_stat ss;\n+\t\tstruct sha1_stat sha1_stat;\n \n \t\tif (current < 0) {\n \t\t\tcp = base;\n@@ -1015,7 +1019,8 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t}\n \n \t\t/* Try to read per-directory file */\n-\t\thashclr(ss.sha1);\n+\t\thashclr(sha1_stat.sha1);\n+\t\tsha1_stat.valid = 0;\n \t\tif (dir->exclude_per_dir &&\n \t\t    /*\n \t\t     * If we know that no files have been added in\n@@ -1044,7 +1049,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t\tstrbuf_addstr(&sb, dir->exclude_per_dir);\n \t\t\tel->src = strbuf_detach(&sb, NULL);\n \t\t\tadd_excludes(el->src, el->src, stk->baselen, el, 1,\n-\t\t\t\t     untracked ? &ss : NULL, 0);\n+\t\t\t\t     untracked ? &sha1_stat : NULL);\n \t\t}\n \t\t/*\n \t\t * NEEDSWORK: when untracked cache is enabled, prep_exclude()\n@@ -1060,9 +1065,10 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t * last_exclude_matching(). Be careful about ignore rule\n \t\t * order, though, if you do that.\n \t\t */\n-\t\tif (untracked && hashcmp(ss.sha1, untracked->exclude_sha1)) {\n+\t\tif (untracked &&\n+\t\t    hashcmp(sha1_stat.sha1, untracked->exclude_sha1)) {\n \t\t\tinvalidate_gitignore(dir->untracked, untracked);\n-\t\t\thashcpy(untracked->exclude_sha1, ss.sha1);\n+\t\t\thashcpy(untracked->exclude_sha1, sha1_stat.sha1);\n \t\t}\n \t\tdir->exclude_stack = stk;\n \t\tcurrent = stk->baselen;\n@@ -1457,12 +1463,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t   int baselen,\n \t\t\t\t\t   const struct path_simplify *simplify)\n {\n+\tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n-\t\tstrbuf_setlen(path, baselen);\n \t\tstrbuf_addstr(path, cdir->file);\n \t\treturn path_untracked;\n \t}\n-\tstrbuf_setlen(path, baselen);\n \tstrbuf_addstr(path, cdir->ucd->name);\n \t/* treat_one_path() does this before it calls treat_directory() */\n \tif (path->buf[path->len - 1] != '/')\n@@ -1474,7 +1479,6 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t * with check_only set.\n \t\t */\n \t\treturn read_directory_recursive(dir, path->buf, path->len,\n-\n \t\t\t\t\t\tcdir->ucd, 1, simplify);\n \t/*\n \t * We get path_recurse in the first run when\n@@ -1582,7 +1586,7 @@ static int open_cached_dir(struct cached_dir *cdir,\n \treturn 0;\n }\n \n-int read_cached_dir(struct cached_dir *cdir)\n+static int read_cached_dir(struct cached_dir *cdir)\n {\n \tif (cdir->fdir) {\n \t\tcdir->de = readdir(cdir->fdir);\n@@ -1796,7 +1800,7 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \tstruct untracked_cache_dir *root;\n \tint i;\n \n-\tif (!dir->untracked)\n+\tif (!dir->untracked || getenv(\"GIT_DISABLE_UNTRACKED_CACHE\"))\n \t\treturn NULL;\n \n \t/*\n@@ -2101,30 +2105,12 @@ void setup_standard_excludes(struct dir_struct *dir)\n \t\thome_config_paths(NULL, &xdg_path, \"ignore\");\n \t\texcludes_file = xdg_path;\n \t}\n-\tif (!access_or_warn(path, R_OK, 0)) {\n-\t\tstruct sha1_stat *ss = NULL;\n-\t\tint ss_valid = 0;\n-\t\tif (dir->untracked) {\n-\t\t\tss = &dir->ss_info_exclude;\n-\t\t\tif (dir->untracked->loaded) {\n-\t\t\t\t*ss = dir->untracked->ss_info_exclude;\n-\t\t\t\tss_valid = 1;\n-\t\t\t}\n-\t\t}\n-\t\tadd_excludes_from_file_1(dir, path, ss, ss_valid);\n-\t}\n-\tif (excludes_file && !access_or_warn(excludes_file, R_OK, 0)) {\n-\t\tstruct sha1_stat *ss = NULL;\n-\t\tint ss_valid = 0;\n-\t\tif (dir->untracked) {\n-\t\t\tss = &dir->ss_excludes_file;\n-\t\t\tif (dir->untracked->loaded) {\n-\t\t\t\t*ss = dir->untracked->ss_excludes_file;\n-\t\t\t\tss_valid = 1;\n-\t\t\t}\n-\t\t}\n-\t\tadd_excludes_from_file_1(dir, excludes_file, ss, ss_valid);\n-\t}\n+\tif (!access_or_warn(path, R_OK, 0))\n+\t\tadd_excludes_from_file_1(dir, path,\n+\t\t\t\t\t dir->untracked ? &dir->ss_info_exclude : NULL);\n+\tif (excludes_file && !access_or_warn(excludes_file, R_OK, 0))\n+\t\tadd_excludes_from_file_1(dir, excludes_file,\n+\t\t\t\t\t dir->untracked ? &dir->ss_excludes_file : NULL);\n }\n \n int remove_path(const char *name)\n@@ -2186,6 +2172,16 @@ struct ondisk_untracked_cache {\n \tchar exclude_per_dir[1];\n };\n \n+struct write_data {\n+\tint index;\t   /* number of written untracked_cache_dir */\n+\tstruct ewah_bitmap *check_only; /* from untracked_cache_dir */\n+\tstruct ewah_bitmap *valid;\t/* from untracked_cache_dir */\n+\tstruct ewah_bitmap *sha1_valid; /* set if exclude_sha1 is not null */\n+\tstruct strbuf out;\n+\tstruct strbuf sb_stat;\n+\tstruct strbuf sb_sha1;\n+};\n+\n static void stat_data_to_disk(struct stat_data *to, const struct stat_data *from)\n {\n \tto->sd_ctime.sec  = htonl(from->sd_ctime.sec);\n@@ -2199,16 +2195,14 @@ static void stat_data_to_disk(struct stat_data *to, const struct stat_data *from\n \tto->sd_size\t  = htonl(from->sd_size);\n }\n \n-static void write_one_dir(struct strbuf *out, struct untracked_cache_dir *untracked)\n+static void write_one_dir(struct untracked_cache_dir *untracked,\n+\t\t\t  struct write_data *wd)\n {\n \tstruct stat_data stat_data;\n+\tstruct strbuf *out = &wd->out;\n \tunsigned char intbuf[16];\n \tunsigned int intlen, value;\n-\tint i;\n-\n-\tstat_data_to_disk(&stat_data, &untracked->stat_data);\n-\tstrbuf_add(out, &stat_data, sizeof(stat_data));\n-\tstrbuf_add(out, untracked->exclude_sha1, 20);\n+\tint i = wd->index++;\n \n \t/*\n \t * untracked_nr should be reset whenever valid is clear, but\n@@ -2219,10 +2213,19 @@ static void write_one_dir(struct strbuf *out, struct untracked_cache_dir *untrac\n \t\tuntracked->check_only = 0;\n \t}\n \n-\tvalue  = untracked->valid;\n-\tvalue |= untracked->check_only   << 1;\n-\tvalue |= untracked->untracked_nr << 2;\n-\tintlen = encode_varint(value, intbuf);\n+\tif (untracked->check_only)\n+\t\tewah_set(wd->check_only, i);\n+\tif (untracked->valid) {\n+\t\tewah_set(wd->valid, i);\n+\t\tstat_data_to_disk(&stat_data, &untracked->stat_data);\n+\t\tstrbuf_add(&wd->sb_stat, &stat_data, sizeof(stat_data));\n+\t}\n+\tif (!is_null_sha1(untracked->exclude_sha1)) {\n+\t\tewah_set(wd->sha1_valid, i);\n+\t\tstrbuf_add(&wd->sb_sha1, untracked->exclude_sha1, 20);\n+\t}\n+\n+\tintlen = encode_varint(untracked->untracked_nr, intbuf);\n \tstrbuf_add(out, intbuf, intlen);\n \n \t/* skip non-recurse directories */\n@@ -2240,13 +2243,26 @@ static void write_one_dir(struct strbuf *out, struct untracked_cache_dir *untrac\n \n \tfor (i = 0; i < untracked->dirs_nr; i++)\n \t\tif (untracked->dirs[i]->recurse)\n-\t\t\twrite_one_dir(out, untracked->dirs[i]);\n+\t\t\twrite_one_dir(untracked->dirs[i], wd);\n+}\n+\n+static void get_ident_string(struct strbuf *sb)\n+{\n+\tstruct utsname uts;\n+\n+\tif (uname(&uts))\n+\t\tdie_errno(_(\"failed to get kernel name and information\"));\n+\tstrbuf_addf(sb, \"Location %s, system %s %s %s\", get_git_work_tree(),\n+\t\t    uts.sysname, uts.release, uts.version);\n }\n \n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked)\n {\n \tstruct ondisk_untracked_cache *ouc;\n-\tint len = 0;\n+\tstruct write_data wd;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tunsigned char varbuf[16];\n+\tint len = 0, varint_len;\n \tif (untracked->exclude_per_dir)\n \t\tlen = strlen(untracked->exclude_per_dir);\n \touc = xmalloc(sizeof(*ouc) + len);\n@@ -2256,9 +2272,43 @@ void write_untracked_extension(struct strbuf *out, struct untracked_cache *untra\n \thashcpy(ouc->excludes_file_sha1, untracked->ss_excludes_file.sha1);\n \touc->dir_flags = htonl(untracked->dir_flags);\n \tmemcpy(ouc->exclude_per_dir, untracked->exclude_per_dir, len + 1);\n+\n+\tget_ident_string(&sb);\n+\tstrbuf_add(out, sb.buf, sb.len + 1);\n+\tstrbuf_release(&sb);\n+\n \tstrbuf_add(out, ouc, sizeof(*ouc) + len);\n-\tif (untracked->root)\n-\t\twrite_one_dir(out, untracked->root);\n+\tif (!untracked->root) {\n+\t\tvarint_len = encode_varint(0, varbuf);\n+\t\tstrbuf_add(out, varbuf, varint_len);\n+\t\treturn;\n+\t}\n+\n+\twd.index      = 0;\n+\twd.check_only = ewah_new();\n+\twd.valid      = ewah_new();\n+\twd.sha1_valid = ewah_new();\n+\tstrbuf_init(&wd.out, 1024);\n+\tstrbuf_init(&wd.sb_stat, 1024);\n+\tstrbuf_init(&wd.sb_sha1, 1024);\n+\twrite_one_dir(untracked->root, &wd);\n+\n+\tvarint_len = encode_varint(wd.index, varbuf);\n+\tstrbuf_add(out, varbuf, varint_len);\n+\tstrbuf_addbuf(out, &wd.out);\n+\tewah_serialize_strbuf(wd.valid, out);\n+\tewah_serialize_strbuf(wd.check_only, out);\n+\tewah_serialize_strbuf(wd.sha1_valid, out);\n+\tstrbuf_addbuf(out, &wd.sb_stat);\n+\tstrbuf_addbuf(out, &wd.sb_sha1);\n+\tstrbuf_addch(out, '\\0'); /* safe guard for string lists */\n+\n+\tewah_free(wd.valid);\n+\tewah_free(wd.check_only);\n+\tewah_free(wd.sha1_valid);\n+\tstrbuf_release(&wd.out);\n+\tstrbuf_release(&wd.sb_stat);\n+\tstrbuf_release(&wd.sb_sha1);\n }\n \n static void free_untracked(struct untracked_cache_dir *ucd)\n@@ -2282,6 +2332,16 @@ void free_untracked_cache(struct untracked_cache *uc)\n \tfree(uc);\n }\n \n+struct read_data {\n+\tint index;\n+\tstruct untracked_cache_dir **ucd;\n+\tstruct ewah_bitmap *check_only;\n+\tstruct ewah_bitmap *valid;\n+\tstruct ewah_bitmap *sha1_valid;\n+\tconst unsigned char *data;\n+\tconst unsigned char *end;\n+};\n+\n static void stat_data_from_disk(struct stat_data *to, const struct stat_data *from)\n {\n \tto->sd_ctime.sec  = get_be32(&from->sd_ctime.sec);\n@@ -2296,43 +2356,34 @@ static void stat_data_from_disk(struct stat_data *to, const struct stat_data *fr\n }\n \n static int read_one_dir(struct untracked_cache_dir **untracked_,\n-\t\t\tconst unsigned char *data_, unsigned long sz)\n+\t\t\tstruct read_data *rd)\n {\n #define NEXT(x) \\\n \tnext = data + (x); \\\n-\tif (next > data_ + sz) \\\n+\tif (next > rd->end) \\\n \t\treturn -1;\n \n \tstruct untracked_cache_dir ud, *untracked;\n-\tconst unsigned char *next, *data = data_;\n+\tconst unsigned char *next, *data = rd->data, *end = rd->end;\n \tunsigned int value;\n \tint i, len;\n \n \tmemset(&ud, 0, sizeof(ud));\n \n-\tNEXT(sizeof(struct stat_data));\n-\tstat_data_from_disk(&ud.stat_data, (struct stat_data *)data);\n-\tdata = next;\n-\n-\tNEXT(20);\n-\thashcpy(ud.exclude_sha1, data);\n-\tdata = next;\n-\n \tnext = data;\n \tvalue = decode_varint(&next);\n-\tif (next > data_ + sz)\n+\tif (next > end)\n \t\treturn -1;\n-\tud.recurse = 1;\n-\tud.valid = value & 1;\n-\tud.check_only = (value >> 1) & 1;\n-\tud.untracked_alloc = ud.untracked_nr = value >> 2;\n+\tud.recurse\t   = 1;\n+\tud.untracked_alloc = value;\n+\tud.untracked_nr\t   = value;\n \tif (ud.untracked_nr)\n \t\tud.untracked = xmalloc(sizeof(*ud.untracked) * ud.untracked_nr);\n \tdata = next;\n \n \tnext = data;\n \tud.dirs_alloc = ud.dirs_nr = decode_varint(&next);\n-\tif (next > data_ + sz)\n+\tif (next > end)\n \t\treturn -1;\n \tud.dirs = xmalloc(sizeof(*ud.dirs) * ud.dirs_nr);\n \tdata = next;\n@@ -2351,41 +2402,145 @@ static int read_one_dir(struct untracked_cache_dir **untracked_,\n \t\tdata = next;\n \t}\n \n+\trd->ucd[rd->index++] = untracked;\n+\trd->data = data;\n+\n \tfor (i = 0; i < untracked->dirs_nr; i++) {\n-\t\tlen = read_one_dir(untracked->dirs + i, data, sz - (data - data_));\n+\t\tlen = read_one_dir(untracked->dirs + i, rd);\n \t\tif (len < 0)\n \t\t\treturn -1;\n-\t\tdata += len;\n \t}\n-\treturn data - data_;\n+\treturn 0;\n+}\n+\n+static void set_check_only(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tud->check_only = 1;\n+}\n+\n+static void read_stat(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tif (rd->data + sizeof(struct stat_data) > rd->end) {\n+\t\trd->data = rd->end + 1;\n+\t\treturn;\n+\t}\n+\tstat_data_from_disk(&ud->stat_data, (struct stat_data *)rd->data);\n+\trd->data += sizeof(struct stat_data);\n+\tud->valid = 1;\n+}\n+\n+static void read_sha1(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tif (rd->data + 20 > rd->end) {\n+\t\trd->data = rd->end + 1;\n+\t\treturn;\n+\t}\n+\thashcpy(ud->exclude_sha1, rd->data);\n+\trd->data += 20;\n+}\n+\n+static void load_sha1_stat(struct sha1_stat *sha1_stat,\n+\t\t\t   const struct stat_data *stat,\n+\t\t\t   const unsigned char *sha1)\n+{\n+\tstat_data_from_disk(&sha1_stat->stat, stat);\n+\thashcpy(sha1_stat->sha1, sha1);\n+\tsha1_stat->valid = 1;\n }\n \n struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz)\n {\n-\tconst struct ondisk_untracked_cache *ouc = data;\n+\tconst struct ondisk_untracked_cache *ouc;\n \tstruct untracked_cache *uc;\n+\tstruct read_data rd;\n+\tconst unsigned char *next = data, *end = data + sz;\n+\tstruct strbuf sb = STRBUF_INIT;\n \tint len;\n \n-\tif (sz < sizeof(*ouc))\n+\tif (sz <= 1 || end[-1] != '\\0')\n+\t\treturn NULL;\n+\tend--;\n+\n+\tget_ident_string(&sb);\n+\tif (strcmp(sb.buf, (const char *)next)) {\n+\t\twarning(_(\"system identification does not match, untracked cache disabled.\\n\"\n+\t\t\t  \"Stored: %s\\nCurrent: %s\\n\"),\n+\t\t\tnext, sb.buf);\n+\t\tstrbuf_release(&sb);\n+\t\treturn NULL;\n+\t}\n+\tnext += sb.len + 1;\n+\tstrbuf_release(&sb);\n+\n+\touc = (const struct ondisk_untracked_cache *)next;\n+\tif (next + sizeof(*ouc) > end)\n \t\treturn NULL;\n \n \tuc = xcalloc(1, sizeof(*uc));\n-\tstat_data_from_disk(&uc->ss_info_exclude.stat, &ouc->info_exclude_stat);\n-\tstat_data_from_disk(&uc->ss_excludes_file.stat, &ouc->excludes_file_stat);\n-\thashcpy(uc->ss_info_exclude.sha1, ouc->info_exclude_sha1);\n-\thashcpy(uc->ss_excludes_file.sha1, ouc->excludes_file_sha1);\n+\tload_sha1_stat(&uc->ss_info_exclude, &ouc->info_exclude_stat,\n+\t\t       ouc->info_exclude_sha1);\n+\tload_sha1_stat(&uc->ss_excludes_file, &ouc->excludes_file_stat,\n+\t\t       ouc->excludes_file_sha1);\n \tuc->dir_flags = get_be32(&ouc->dir_flags);\n \tuc->exclude_per_dir = xstrdup(ouc->exclude_per_dir);\n-\tuc->loaded = 1;\n-\tlen = sizeof(*ouc) + strlen(ouc->exclude_per_dir);\n-\tif (sz == len)\n-\t\treturn uc;\n-\tif (sz > len &&\n-\t    read_one_dir(&uc->root, (const unsigned char *)data + len,\n-\t\t\t sz - len) == sz - len)\n-\t\treturn uc;\n-\tfree_untracked_cache(uc);\n-\treturn NULL;\n+\t/* NUL after exclude_per_dir is covered by sizeof(*ouc) */\n+\tnext += sizeof(*ouc) + strlen(ouc->exclude_per_dir);\n+\tif (next >= end)\n+\t\tgoto done2;\n+\n+\tlen = decode_varint(&next);\n+\tif (next > end || len == 0)\n+\t\tgoto done2;\n+\n+\trd.valid      = ewah_new();\n+\trd.check_only = ewah_new();\n+\trd.sha1_valid = ewah_new();\n+\trd.data\t      = next;\n+\trd.end\t      = end;\n+\trd.index      = 0;\n+\trd.ucd        = xmalloc(sizeof(*rd.ucd) * len);\n+\n+\tif (read_one_dir(&uc->root, &rd) || rd.index != len)\n+\t\tgoto done;\n+\n+\tnext = rd.data;\n+\tlen = ewah_read_mmap(rd.valid, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tnext += len;\n+\tlen = ewah_read_mmap(rd.check_only, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tnext += len;\n+\tlen = ewah_read_mmap(rd.sha1_valid, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tewah_each_bit(rd.check_only, set_check_only, &rd);\n+\trd.data = next + len;\n+\tewah_each_bit(rd.valid, read_stat, &rd);\n+\tewah_each_bit(rd.sha1_valid, read_sha1, &rd);\n+\tnext = rd.data;\n+\n+done:\n+\tfree(rd.ucd);\n+\tewah_free(rd.valid);\n+\tewah_free(rd.check_only);\n+\tewah_free(rd.sha1_valid);\n+done2:\n+\tif (next != end) {\n+\t\tfree_untracked_cache(uc);\n+\t\tuc = NULL;\n+\t}\n+\treturn uc;\n }\n \n void untracked_cache_invalidate_path(struct index_state *istate,\ndiff --git a/dir.h b/dir.h\nindex 8c29324..2ce7dd3 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -77,6 +77,7 @@ struct exclude_list_group {\n struct sha1_stat {\n \tstruct stat_data stat;\n \tunsigned char sha1[20];\n+\tint valid;\n };\n \n /*\n@@ -85,7 +86,7 @@ struct sha1_stat {\n  *  The following inputs are sufficient to determine what files in a\n  *  directory are excluded:\n  *\n- *   - The list of files and directories of the direction in question\n+ *   - The list of files and directories of the directory in question\n  *   - The $GIT_DIR/index\n  *   - dir_struct flags\n  *   - The content of $GIT_DIR/info/exclude\n@@ -110,16 +111,16 @@ struct sha1_stat {\n struct untracked_cache_dir {\n \tstruct untracked_cache_dir **dirs;\n \tchar **untracked;\n-\t/* null SHA-1 means this directory does not have .gitignore */\n-\tunsigned char exclude_sha1[20];\n \tstruct stat_data stat_data;\n-\tunsigned int recurse : 1;\n+\tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n+\tunsigned int untracked_nr;\n \tunsigned int check_only : 1;\n \t/* all data except 'dirs' in this struct are good */\n \tunsigned int valid : 1;\n-\tunsigned int untracked_nr : 29;\n-\tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n-\tchar name[1];\n+\tunsigned int recurse : 1;\n+\t/* null SHA-1 means this directory does not have .gitignore */\n+\tunsigned char exclude_sha1[20];\n+\tchar name[FLEX_ARRAY];\n };\n \n struct untracked_cache {\n@@ -137,7 +138,6 @@ struct untracked_cache {\n \tint gitignore_invalidated;\n \tint dir_invalidated;\n \tint dir_opened;\n-\tint loaded;\n };\n \n struct dir_struct {\ndiff --git a/ewah/ewah_io.c b/ewah/ewah_io.c\nindex 1c2d7af..43481b9 100644\n--- a/ewah/ewah_io.c\n+++ b/ewah/ewah_io.c\n@@ -19,6 +19,7 @@\n  */\n #include \"git-compat-util.h\"\n #include \"ewok.h\"\n+#include \"strbuf.h\"\n \n int ewah_serialize_native(struct ewah_bitmap *self, int fd)\n {\n@@ -110,6 +111,18 @@ int ewah_serialize(struct ewah_bitmap *self, int fd)\n \treturn ewah_serialize_to(self, write_helper, (void *)(intptr_t)fd);\n }\n \n+static int write_strbuf(void *user_data, const void *data, size_t len)\n+{\n+\tstruct strbuf *sb = user_data;\n+\tstrbuf_add(sb, data, len);\n+\treturn len;\n+}\n+\n+int ewah_serialize_strbuf(struct ewah_bitmap *self, struct strbuf *sb)\n+{\n+\treturn ewah_serialize_to(self, write_strbuf, sb);\n+}\n+\n int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len)\n {\n \tconst uint8_t *ptr = map;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex f6ad190..4d7f5e9 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -30,6 +30,7 @@\n #\tdefine ewah_calloc xcalloc\n #endif\n \n+struct strbuf;\n typedef uint64_t eword_t;\n #define BITS_IN_WORD (sizeof(eword_t) * 8)\n \n@@ -97,6 +98,7 @@ int ewah_serialize_to(struct ewah_bitmap *self,\n \t\t      void *out);\n int ewah_serialize(struct ewah_bitmap *self, int fd);\n int ewah_serialize_native(struct ewah_bitmap *self, int fd);\n+int ewah_serialize_strbuf(struct ewah_bitmap *self, struct strbuf *);\n \n int ewah_deserialize(struct ewah_bitmap *self, int fd);\n int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len);\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f587749..e9502a1 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -132,6 +132,7 @@\n #elif defined(_MSC_VER)\n #include \"compat/msvc.h\"\n #else\n+#include <sys/utsname.h>\n #include <sys/wait.h>\n #include <sys/resource.h>\n #include <sys/socket.h>\ndiff --git a/read-cache.c b/read-cache.c\nindex 21ae963..71d8e20 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2040,7 +2040,7 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n-\tif (istate->untracked) {\n+\tif (!strip_extensions && istate->untracked) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\twrite_untracked_extension(&sb, istate->untracked);\ndiff --git a/split-index.c b/split-index.c\nindex 21485e2..968b780 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -41,13 +41,6 @@ int read_link_extension(struct index_state *istate,\n \treturn 0;\n }\n \n-static int write_strbuf(void *user_data, const void *data, size_t len)\n-{\n-\tstruct strbuf *sb = user_data;\n-\tstrbuf_add(sb, data, len);\n-\treturn len;\n-}\n-\n int write_link_extension(struct strbuf *sb,\n \t\t\t struct index_state *istate)\n {\n@@ -55,8 +48,8 @@ int write_link_extension(struct strbuf *sb,\n \tstrbuf_add(sb, si->base_sha1, 20);\n \tif (!si->delete_bitmap && !si->replace_bitmap)\n \t\treturn 0;\n-\tewah_serialize_to(si->delete_bitmap, write_strbuf, sb);\n-\tewah_serialize_to(si->replace_bitmap, write_strbuf, sb);\n+\tewah_serialize_strbuf(si->delete_bitmap, sb);\n+\tewah_serialize_strbuf(si->replace_bitmap, sb);\n \treturn 0;\n }\n \ndiff --git a/test-dump-untracked-cache.c b/test-dump-untracked-cache.c\nindex 710441e..25d855d 100644\n--- a/test-dump-untracked-cache.c\n+++ b/test-dump-untracked-cache.c\n@@ -44,6 +44,7 @@ int main(int ac, char **av)\n {\n \tstruct untracked_cache *uc;\n \tstruct strbuf base = STRBUF_INIT;\n+\tsetup_git_directory();\n \tif (read_cache() < 0)\n \t\tdie(\"unable to read index file\");\n \tuc = the_index.untracked;\n-- 8< --\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253310","messageId":"1418047507-22892-2-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH] checkout: add --ignore-other-wortrees","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:44Z","receivedAt":"2014-12-08T14:04:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-checkout.txt |  6 ++++++\n builtin/checkout.c             | 19 +++++++++++--------\n t/t2025-checkout-to.sh         |  7 +++++++\n 3 files changed, 24 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex 0c13825..71d9e4e 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -232,6 +232,12 @@ section of linkgit:git-add[1] to learn how to operate the `--patch` mode.\n \tspecific files such as HEAD, index... See \"MULTIPLE WORKING\n \tTREES\" section for more information.\n \n+--ignore-other-worktrees::\n+\t`git checkout` refuses when the wanted ref is already checked out\n+\tby another worktree. This option makes `git checkout` check the\n+\tref out anyway. In other words, the ref is held by more than one\n+\tworktree.\n+\n <branch>::\n \tBranch to checkout; if it refers to a branch (i.e., a name that,\n \twhen prepended with \"refs/heads/\", is a valid ref), then that\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 953b763..8b2bf20 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -37,6 +37,7 @@ struct checkout_opts {\n \tint writeout_stage;\n \tint overwrite_ignore;\n \tint ignore_skipworktree;\n+\tint ignore_other_worktrees;\n \n \tconst char *new_branch;\n \tconst char *new_branch_force;\n@@ -1079,11 +1080,12 @@ static void check_linked_checkouts(struct branch_info *new)\n static int parse_branchname_arg(int argc, const char **argv,\n \t\t\t\tint dwim_new_local_branch_ok,\n \t\t\t\tstruct branch_info *new,\n-\t\t\t\tstruct tree **source_tree,\n-\t\t\t\tunsigned char rev[20],\n-\t\t\t\tconst char **new_branch,\n-\t\t\t\tint force_detach)\n+\t\t\t\tstruct checkout_opts *opts,\n+\t\t\t\tunsigned char rev[20])\n {\n+\tstruct tree **source_tree = &opts->source_tree;\n+\tconst char **new_branch = &opts->new_branch;\n+\tint force_detach = opts->force_detach;\n \tint argcount = 0;\n \tunsigned char branch_rev[20];\n \tconst char *arg;\n@@ -1209,7 +1211,8 @@ static int parse_branchname_arg(int argc, const char **argv,\n \t\tint flag;\n \t\tchar *head_ref = resolve_refdup(\"HEAD\", 0, sha1, &flag);\n \t\tif (head_ref &&\n-\t\t    (!(flag & REF_ISSYMREF) || strcmp(head_ref, new->path)))\n+\t\t    (!(flag & REF_ISSYMREF) || strcmp(head_ref, new->path)) &&\n+\t\t    !opts->ignore_other_worktrees)\n \t\t\tcheck_linked_checkouts(new);\n \t\tfree(head_ref);\n \t}\n@@ -1340,6 +1343,8 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)\n \t\t\t\tN_(\"second guess 'git checkout no-such-branch'\")),\n \t\tOPT_FILENAME(0, \"to\", &opts.new_worktree,\n \t\t\t   N_(\"check a branch out in a separate working directory\")),\n+\t\tOPT_BOOL(0, \"ignore-other-worktrees\", &opts.ignore_other_worktrees,\n+\t\t\t N_(\"do not check if another worktree is holding the given ref\")),\n \t\tOPT_END(),\n \t};\n \n@@ -1420,9 +1425,7 @@ int cmd_checkout(int argc, const char **argv, const char *prefix)\n \t\t\topts.track == BRANCH_TRACK_UNSPECIFIED &&\n \t\t\t!opts.new_branch;\n \t\tint n = parse_branchname_arg(argc, argv, dwim_ok,\n-\t\t\t\t\t     &new, &opts.source_tree,\n-\t\t\t\t\t     rev, &opts.new_branch,\n-\t\t\t\t\t     opts.force_detach);\n+\t\t\t\t\t     &new, &opts, rev);\n \t\targv += n;\n \t\targc -= n;\n \t}\ndiff --git a/t/t2025-checkout-to.sh b/t/t2025-checkout-to.sh\nindex 915b506..f8e4df4 100755\n--- a/t/t2025-checkout-to.sh\n+++ b/t/t2025-checkout-to.sh\n@@ -79,6 +79,13 @@ test_expect_success 'die the same branch is already checked out' '\n \t)\n '\n \n+test_expect_success 'not die the same branch is already checked out' '\n+\t(\n+\t\tcd here &&\n+\t\tgit checkout --ignore-other-worktrees --to anothernewmaster newmaster\n+\t)\n+'\n+\n test_expect_success 'not die on re-checking out current branch' '\n \t(\n \t\tcd there &&\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253311","messageId":"1418047507-22892-3-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 01/23] dir.c: optionally compute sha-1 of a .gitignore file","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:45Z","receivedAt":"2014-12-08T14:04:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is not used anywhere yet. But the goal is to compare quickly if a\n.gitignore file has changed when we have the SHA-1 of both old (cached\nsomewhere) and new (from index or a tree) versions.\n\nHelped-by: Junio C Hamano <gitster@pobox.com>\nHelped-by: Torsten Bögershausen <tboegi@web.de>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-------\n dir.h |  6 ++++++\n 2 files changed, 52 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex fcb6872..4cc936b 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -466,7 +466,8 @@ void add_exclude(const char *string, const char *base,\n \tx->el = el;\n }\n \n-static void *read_skip_worktree_file_from_index(const char *path, size_t *size)\n+static void *read_skip_worktree_file_from_index(const char *path, size_t *size,\n+\t\t\t\t\t\tstruct sha1_stat *sha1_stat)\n {\n \tint pos, len;\n \tunsigned long sz;\n@@ -485,6 +486,10 @@ static void *read_skip_worktree_file_from_index(const char *path, size_t *size)\n \t\treturn NULL;\n \t}\n \t*size = xsize_t(sz);\n+\tif (sha1_stat) {\n+\t\tmemset(&sha1_stat->stat, 0, sizeof(sha1_stat->stat));\n+\t\thashcpy(sha1_stat->sha1, active_cache[pos]->sha1);\n+\t}\n \treturn data;\n }\n \n@@ -529,11 +534,18 @@ static void trim_trailing_spaces(char *buf)\n \t\t*last_space = '\\0';\n }\n \n-int add_excludes_from_file_to_list(const char *fname,\n-\t\t\t\t   const char *base,\n-\t\t\t\t   int baselen,\n-\t\t\t\t   struct exclude_list *el,\n-\t\t\t\t   int check_index)\n+/*\n+ * Given a file with name \"fname\", read it (either from disk, or from\n+ * the index if \"check_index\" is non-zero), parse it and store the\n+ * exclude rules in \"el\".\n+ *\n+ * If \"ss\" is not NULL, compute SHA-1 of the exclude file and fill\n+ * stat data from disk (only valid if add_excludes returns zero). If\n+ * ss_valid is non-zero, \"ss\" must contain good value as input.\n+ */\n+static int add_excludes(const char *fname, const char *base, int baselen,\n+\t\t\tstruct exclude_list *el, int check_index,\n+\t\t\tstruct sha1_stat *sha1_stat)\n {\n \tstruct stat st;\n \tint fd, i, lineno = 1;\n@@ -547,7 +559,7 @@ int add_excludes_from_file_to_list(const char *fname,\n \t\tif (0 <= fd)\n \t\t\tclose(fd);\n \t\tif (!check_index ||\n-\t\t    (buf = read_skip_worktree_file_from_index(fname, &size)) == NULL)\n+\t\t    (buf = read_skip_worktree_file_from_index(fname, &size, sha1_stat)) == NULL)\n \t\t\treturn -1;\n \t\tif (size == 0) {\n \t\t\tfree(buf);\n@@ -560,6 +572,11 @@ int add_excludes_from_file_to_list(const char *fname,\n \t} else {\n \t\tsize = xsize_t(st.st_size);\n \t\tif (size == 0) {\n+\t\t\tif (sha1_stat) {\n+\t\t\t\tfill_stat_data(&sha1_stat->stat, &st);\n+\t\t\t\thashcpy(sha1_stat->sha1, EMPTY_BLOB_SHA1_BIN);\n+\t\t\t\tsha1_stat->valid = 1;\n+\t\t\t}\n \t\t\tclose(fd);\n \t\t\treturn 0;\n \t\t}\n@@ -571,6 +588,21 @@ int add_excludes_from_file_to_list(const char *fname,\n \t\t}\n \t\tbuf[size++] = '\\n';\n \t\tclose(fd);\n+\t\tif (sha1_stat) {\n+\t\t\tint pos;\n+\t\t\tif (sha1_stat->valid &&\n+\t\t\t    !match_stat_data(&sha1_stat->stat, &st))\n+\t\t\t\t; /* no content change, ss->sha1 still good */\n+\t\t\telse if (check_index &&\n+\t\t\t\t (pos = cache_name_pos(fname, strlen(fname))) >= 0 &&\n+\t\t\t\t !ce_stage(active_cache[pos]) &&\n+\t\t\t\t ce_uptodate(active_cache[pos]))\n+\t\t\t\thashcpy(sha1_stat->sha1, active_cache[pos]->sha1);\n+\t\t\telse\n+\t\t\t\thash_sha1_file(buf, size, \"blob\", sha1_stat->sha1);\n+\t\t\tfill_stat_data(&sha1_stat->stat, &st);\n+\t\t\tsha1_stat->valid = 1;\n+\t\t}\n \t}\n \n \tel->filebuf = buf;\n@@ -589,6 +621,13 @@ int add_excludes_from_file_to_list(const char *fname,\n \treturn 0;\n }\n \n+int add_excludes_from_file_to_list(const char *fname, const char *base,\n+\t\t\t\t   int baselen, struct exclude_list *el,\n+\t\t\t\t   int check_index)\n+{\n+\treturn add_excludes(fname, base, baselen, el, check_index, NULL);\n+}\n+\n struct exclude_list *add_exclude_list(struct dir_struct *dir,\n \t\t\t\t      int group_type, const char *src)\n {\ndiff --git a/dir.h b/dir.h\nindex 6c45e9d..cdca71b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -73,6 +73,12 @@ struct exclude_list_group {\n \tstruct exclude_list *el;\n };\n \n+struct sha1_stat {\n+\tstruct stat_data stat;\n+\tunsigned char sha1[20];\n+\tint valid;\n+};\n+\n struct dir_struct {\n \tint nr, alloc;\n \tint ignored_nr, ignored_alloc;\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253312","messageId":"1418047507-22892-4-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 02/23] untracked cache: record .gitignore information and dir hierarchy","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:46Z","receivedAt":"2014-12-08T14:04:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The idea is if we can capture all input and (non-rescursive) output of\nread_directory_recursive(), and can verify later that all the input is\nthe same, then the second r_d_r() should produce the same output as in\nthe first run.\n\nThe requirement for this to work is stat info of a directory MUST\nchange if an entry is added to or removed from that directory (and\nshould not change often otherwise). If your OS and filesystem do not\nmeet this requirement, untracked cache is not for you. Most file\nsystems on *nix should be fine. On Windows, NTFS is fine while FAT may\nnot be [1] even though FAT on Linux seems to be fine.\n\nThe list of input of r_d_r() is in the big comment block in dir.h. In\nshort, the output of a directory (not counting subdirs) mainly depends\non stat info of the directory in question, all .gitignore leading to\nit and the check_only flag when r_d_r() is called recursively. This\npatch records all this info (and the output) as r_d_r() runs.\n\nTwo hash_sha1_file() are required for $GIT_DIR/info/exclude and\ncore.excludesfile unless their stat data matches. hash_sha1_file() is\nonly needed when .gitignore files in the worktree are modified,\notherwise their SHA-1 in index is used (see the previous patch).\n\nWe could store stat data for .gitignore files so we don't have to\nrehash them if their content is different from index, but I think\n.gitignore files are rarely modified, so not worth extra cache data\n(and hashing penalty read-cache.c:verify_hdr(), as we will be storing\nthis as an index extension).\n\nThe implication is, if you change .gitignore, you better add it to the\nindex soon or you lose all the benefit of untracked cache because a\nmodified .gitignore invalidates all subdirs recursively. This is\nespecially bad for .gitignore at root.\n\nThis cached output is about untracked files only, not ignored files\nbecause the number of tracked files is usually small, so small cache\noverhead, while the number of ignored files could go really high\n(e.g. *.o files mixing with source code).\n\n[1] \"Description of NTFS date and time stamps for files and folders\"\n    http://support.microsoft.com/kb/299648\n\nHelped-by: Torsten Bögershausen <tboegi@web.de>\nHelped-by: David Turner <dturner@twopensource.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n dir.h |  60 ++++++++++++++++++++++++++++\n 2 files changed, 183 insertions(+), 19 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 4cc936b..27734f0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -32,7 +32,7 @@ enum path_treatment {\n };\n \n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n-\tconst char *path, int len,\n+\tconst char *path, int len, struct untracked_cache_dir *untracked,\n \tint check_only, const struct path_simplify *simplify);\n static int get_dtype(struct dirent *de, const char *path, int len);\n \n@@ -535,6 +535,54 @@ static void trim_trailing_spaces(char *buf)\n }\n \n /*\n+ * Given a subdirectory name and \"dir\" of the current directory,\n+ * search the subdir in \"dir\" and return it, or create a new one if it\n+ * does not exist in \"dir\".\n+ *\n+ * If \"name\" has the trailing slash, it'll be excluded in the search.\n+ */\n+static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n+\t\t\t\t\t\t    struct untracked_cache_dir *dir,\n+\t\t\t\t\t\t    const char *name, int len)\n+{\n+\tint first, last;\n+\tstruct untracked_cache_dir *d;\n+\tif (!dir)\n+\t\treturn NULL;\n+\tif (len && name[len - 1] == '/')\n+\t\tlen--;\n+\tfirst = 0;\n+\tlast = dir->dirs_nr;\n+\twhile (last > first) {\n+\t\tint cmp, next = (last + first) >> 1;\n+\t\td = dir->dirs[next];\n+\t\tcmp = strncmp(name, d->name, len);\n+\t\tif (!cmp && strlen(d->name) > len)\n+\t\t\tcmp = -1;\n+\t\tif (!cmp)\n+\t\t\treturn d;\n+\t\tif (cmp < 0) {\n+\t\t\tlast = next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tfirst = next+1;\n+\t}\n+\n+\tuc->dir_created++;\n+\td = xmalloc(sizeof(*d) + len + 1);\n+\tmemset(d, 0, sizeof(*d));\n+\tmemcpy(d->name, name, len);\n+\td->name[len] = '\\0';\n+\n+\tALLOC_GROW(dir->dirs, dir->dirs_nr + 1, dir->dirs_alloc);\n+\tmemmove(dir->dirs + first + 1, dir->dirs + first,\n+\t\t(dir->dirs_nr - first) * sizeof(*dir->dirs));\n+\tdir->dirs_nr++;\n+\tdir->dirs[first] = d;\n+\treturn d;\n+}\n+\n+/*\n  * Given a file with name \"fname\", read it (either from disk, or from\n  * the index if \"check_index\" is non-zero), parse it and store the\n  * exclude rules in \"el\".\n@@ -645,14 +693,20 @@ struct exclude_list *add_exclude_list(struct dir_struct *dir,\n /*\n  * Used to set up core.excludesfile and .git/info/exclude lists.\n  */\n-void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n+static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,\n+\t\t\t\t     struct sha1_stat *sha1_stat)\n {\n \tstruct exclude_list *el;\n \tel = add_exclude_list(dir, EXC_FILE, fname);\n-\tif (add_excludes_from_file_to_list(fname, \"\", 0, el, 0) < 0)\n+\tif (add_excludes(fname, \"\", 0, el, 0, sha1_stat) < 0)\n \t\tdie(\"cannot use %s as an exclude file\", fname);\n }\n \n+void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n+{\n+\tadd_excludes_from_file_1(dir, fname, NULL);\n+}\n+\n int match_basename(const char *basename, int basenamelen,\n \t\t   const char *pattern, int prefix, int patternlen,\n \t\t   int flags)\n@@ -827,6 +881,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \tstruct exclude_list_group *group;\n \tstruct exclude_list *el;\n \tstruct exclude_stack *stk = NULL;\n+\tstruct untracked_cache_dir *untracked;\n \tint current;\n \n \tgroup = &dir->exclude_list_group[EXC_DIRS];\n@@ -864,9 +919,15 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t/* Read from the parent directories and push them down. */\n \tcurrent = stk ? stk->baselen : -1;\n \tstrbuf_setlen(&dir->basebuf, current < 0 ? 0 : current);\n+\tif (dir->untracked)\n+\t\tuntracked = stk ? stk->ucd : dir->untracked->root;\n+\telse\n+\t\tuntracked = NULL;\n+\n \twhile (current < baselen) {\n \t\tstruct exclude_stack *stk = xcalloc(1, sizeof(*stk));\n \t\tconst char *cp;\n+\t\tstruct sha1_stat sha1_stat;\n \n \t\tif (current < 0) {\n \t\t\tcp = base;\n@@ -876,10 +937,15 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t\tif (!cp)\n \t\t\t\tdie(\"oops in prep_exclude\");\n \t\t\tcp++;\n+\t\t\tuntracked =\n+\t\t\t\tlookup_untracked(dir->untracked, untracked,\n+\t\t\t\t\t\t base + current,\n+\t\t\t\t\t\t cp - base - current);\n \t\t}\n \t\tstk->prev = dir->exclude_stack;\n \t\tstk->baselen = cp - base;\n \t\tstk->exclude_ix = group->nr;\n+\t\tstk->ucd = untracked;\n \t\tel = add_exclude_list(dir, EXC_DIRS, NULL);\n \t\tstrbuf_add(&dir->basebuf, base + current, stk->baselen - current);\n \t\tassert(stk->baselen == dir->basebuf.len);\n@@ -902,6 +968,8 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t}\n \n \t\t/* Try to read per-directory file */\n+\t\thashclr(sha1_stat.sha1);\n+\t\tsha1_stat.valid = 0;\n \t\tif (dir->exclude_per_dir) {\n \t\t\t/*\n \t\t\t * dir->basebuf gets reused by the traversal, but we\n@@ -915,8 +983,11 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t\tstrbuf_addbuf(&sb, &dir->basebuf);\n \t\t\tstrbuf_addstr(&sb, dir->exclude_per_dir);\n \t\t\tel->src = strbuf_detach(&sb, NULL);\n-\t\t\tadd_excludes_from_file_to_list(el->src, el->src,\n-\t\t\t\t\t\t       stk->baselen, el, 1);\n+\t\t\tadd_excludes(el->src, el->src, stk->baselen, el, 1,\n+\t\t\t\t     untracked ? &sha1_stat : NULL);\n+\t\t}\n+\t\tif (untracked) {\n+\t\t\thashcpy(untracked->exclude_sha1, sha1_stat.sha1);\n \t\t}\n \t\tdir->exclude_stack = stk;\n \t\tcurrent = stk->baselen;\n@@ -1097,6 +1168,7 @@ static enum exist_status directory_exists_in_index(const char *dirname, int len)\n  *  (c) otherwise, we recurse into it.\n  */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n+\tstruct untracked_cache_dir *untracked,\n \tconst char *dirname, int len, int exclude,\n \tconst struct path_simplify *simplify)\n {\n@@ -1124,7 +1196,9 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n \t\treturn exclude ? path_excluded : path_untracked;\n \n-\treturn read_directory_recursive(dir, dirname, len, 1, simplify);\n+\tuntracked = lookup_untracked(dir->untracked, untracked, dirname, len);\n+\treturn read_directory_recursive(dir, dirname, len,\n+\t\t\t\t\tuntracked, 1, simplify);\n }\n \n /*\n@@ -1240,6 +1314,7 @@ static int get_dtype(struct dirent *de, const char *path, int len)\n }\n \n static enum path_treatment treat_one_path(struct dir_struct *dir,\n+\t\t\t\t\t  struct untracked_cache_dir *untracked,\n \t\t\t\t\t  struct strbuf *path,\n \t\t\t\t\t  const struct path_simplify *simplify,\n \t\t\t\t\t  int dtype, struct dirent *de)\n@@ -1292,7 +1367,7 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t\treturn path_none;\n \tcase DT_DIR:\n \t\tstrbuf_addch(path, '/');\n-\t\treturn treat_directory(dir, path->buf, path->len, exclude,\n+\t\treturn treat_directory(dir, untracked, path->buf, path->len, exclude,\n \t\t\tsimplify);\n \tcase DT_REG:\n \tcase DT_LNK:\n@@ -1301,6 +1376,7 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n }\n \n static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t      struct untracked_cache_dir *untracked,\n \t\t\t\t      struct dirent *de,\n \t\t\t\t      struct strbuf *path,\n \t\t\t\t      int baselen,\n@@ -1316,7 +1392,16 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tdtype = DTYPE(de);\n-\treturn treat_one_path(dir, path, simplify, dtype, de);\n+\treturn treat_one_path(dir, untracked, path, simplify, dtype, de);\n+}\n+\n+static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n+{\n+\tif (!dir)\n+\t\treturn;\n+\tALLOC_GROW(dir->untracked, dir->untracked_nr + 1,\n+\t\t   dir->untracked_alloc);\n+\tdir->untracked[dir->untracked_nr++] = xstrdup(name);\n }\n \n /*\n@@ -1332,7 +1417,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n  */\n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t    const char *base, int baselen,\n-\t\t\t\t    int check_only,\n+\t\t\t\t    struct untracked_cache_dir *untracked, int check_only,\n \t\t\t\t    const struct path_simplify *simplify)\n {\n \tDIR *fdir;\n@@ -1346,24 +1431,36 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tif (!fdir)\n \t\tgoto out;\n \n+\tif (untracked)\n+\t\tuntracked->check_only = !!check_only;\n+\n \twhile ((de = readdir(fdir)) != NULL) {\n \t\t/* check how the file or directory should be treated */\n-\t\tstate = treat_path(dir, de, &path, baselen, simplify);\n+\t\tstate = treat_path(dir, untracked, de, &path, baselen, simplify);\n+\n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n \t\tif (state == path_recurse) {\n-\t\t\tsubdir_state = read_directory_recursive(dir, path.buf,\n-\t\t\t\tpath.len, check_only, simplify);\n+\t\t\tstruct untracked_cache_dir *ud;\n+\t\t\tud = lookup_untracked(dir->untracked, untracked,\n+\t\t\t\t\t      path.buf + baselen,\n+\t\t\t\t\t      path.len - baselen);\n+\t\t\tsubdir_state =\n+\t\t\t\tread_directory_recursive(dir, path.buf, path.len,\n+\t\t\t\t\t\t\t ud, check_only, simplify);\n \t\t\tif (subdir_state > dir_state)\n \t\t\t\tdir_state = subdir_state;\n \t\t}\n \n \t\tif (check_only) {\n \t\t\t/* abort early if maximum state has been reached */\n-\t\t\tif (dir_state == path_untracked)\n+\t\t\tif (dir_state == path_untracked) {\n+\t\t\t\tif (untracked)\n+\t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n+\t\t\t}\n \t\t\t/* skip the dir_add_* part */\n \t\t\tcontinue;\n \t\t}\n@@ -1381,8 +1478,11 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tbreak;\n \n \t\tcase path_untracked:\n-\t\t\tif (!(dir->flags & DIR_SHOW_IGNORED))\n-\t\t\t\tdir_add_name(dir, path.buf, path.len);\n+\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\t\tbreak;\n+\t\t\tdir_add_name(dir, path.buf, path.len);\n+\t\t\tif (untracked)\n+\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\tbreak;\n \n \t\tdefault:\n@@ -1459,7 +1559,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\tbreak;\n \t\tif (simplify_away(sb.buf, sb.len, simplify))\n \t\t\tbreak;\n-\t\tif (treat_one_path(dir, &sb, simplify,\n+\t\tif (treat_one_path(dir, NULL, &sb, simplify,\n \t\t\t\t   DT_DIR, NULL) == path_none)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen) {\n@@ -1499,7 +1599,9 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \t */\n \tsimplify = create_simplify(pathspec ? pathspec->_raw : NULL);\n \tif (!len || treat_leading_path(dir, path, len, simplify))\n-\t\tread_directory_recursive(dir, path, len, 0, simplify);\n+\t\tread_directory_recursive(dir, path, len,\n+\t\t\t\t\t dir->untracked ? dir->untracked->root : NULL,\n+\t\t\t\t\t 0, simplify);\n \tfree_simplify(simplify);\n \tqsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);\n \tqsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name);\n@@ -1666,9 +1768,11 @@ void setup_standard_excludes(struct dir_struct *dir)\n \t\texcludes_file = xdg_path;\n \t}\n \tif (!access_or_warn(path, R_OK, 0))\n-\t\tadd_excludes_from_file(dir, path);\n+\t\tadd_excludes_from_file_1(dir, path,\n+\t\t\t\t\t dir->untracked ? &dir->ss_info_exclude : NULL);\n \tif (excludes_file && !access_or_warn(excludes_file, R_OK, 0))\n-\t\tadd_excludes_from_file(dir, excludes_file);\n+\t\tadd_excludes_from_file_1(dir, excludes_file,\n+\t\t\t\t\t dir->untracked ? &dir->ss_excludes_file : NULL);\n }\n \n int remove_path(const char *name)\ndiff --git a/dir.h b/dir.h\nindex cdca71b..9ab74b4 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -66,6 +66,7 @@ struct exclude_stack {\n \tstruct exclude_stack *prev; /* the struct exclude_stack for the parent directory */\n \tint baselen;\n \tint exclude_ix; /* index of exclude_list within EXC_DIRS exclude_list_group */\n+\tstruct untracked_cache_dir *ucd;\n };\n \n struct exclude_list_group {\n@@ -79,6 +80,60 @@ struct sha1_stat {\n \tint valid;\n };\n \n+/*\n+ *  Untracked cache\n+ *\n+ *  The following inputs are sufficient to determine what files in a\n+ *  directory are excluded:\n+ *\n+ *   - The list of files and directories of the directory in question\n+ *   - The $GIT_DIR/index\n+ *   - dir_struct flags\n+ *   - The content of $GIT_DIR/info/exclude\n+ *   - The content of core.excludesfile\n+ *   - The content (or the lack) of .gitignore of all parent directories\n+ *     from $GIT_WORK_TREE\n+ *   - The check_only flag in read_directory_recursive (for\n+ *     DIR_HIDE_EMPTY_DIRECTORIES)\n+ *\n+ *  The first input can be checked using directory mtime. In many\n+ *  filesystems, directory mtime (stat_data field) is updated when its\n+ *  files or direct subdirs are added or removed.\n+ *\n+ *  The second one can be hooked from cache_tree_invalidate_path().\n+ *  Whenever a file (or a submodule) is added or removed from a\n+ *  directory, we invalidate that directory.\n+ *\n+ *  The remaining inputs are easy, their SHA-1 could be used to verify\n+ *  their contents (exclude_sha1[], info_exclude_sha1[] and\n+ *  excludes_file_sha1[])\n+ */\n+struct untracked_cache_dir {\n+\tstruct untracked_cache_dir **dirs;\n+\tchar **untracked;\n+\tstruct stat_data stat_data;\n+\tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n+\tunsigned int untracked_nr;\n+\tunsigned int check_only : 1;\n+\t/* null SHA-1 means this directory does not have .gitignore */\n+\tunsigned char exclude_sha1[20];\n+\tchar name[FLEX_ARRAY];\n+};\n+\n+struct untracked_cache {\n+\tstruct sha1_stat ss_info_exclude;\n+\tstruct sha1_stat ss_excludes_file;\n+\tconst char *exclude_per_dir;\n+\t/*\n+\t * dir_struct#flags must match dir_flags or the untracked\n+\t * cache is ignored.\n+\t */\n+\tunsigned dir_flags;\n+\tstruct untracked_cache_dir *root;\n+\t/* Statistics */\n+\tint dir_created;\n+};\n+\n struct dir_struct {\n \tint nr, alloc;\n \tint ignored_nr, ignored_alloc;\n@@ -126,6 +181,11 @@ struct dir_struct {\n \tstruct exclude_stack *exclude_stack;\n \tstruct exclude *exclude;\n \tstruct strbuf basebuf;\n+\n+\t/* Enable untracked file cache if set */\n+\tstruct untracked_cache *untracked;\n+\tstruct sha1_stat ss_info_exclude;\n+\tstruct sha1_stat ss_excludes_file;\n };\n \n /*\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253313","messageId":"1418047507-22892-5-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 03/23] untracked cache: initial untracked cache validation","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:47Z","receivedAt":"2014-12-08T14:04:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Make sure the starting conditions and all global exclude files are\ngood to go. If not, either disable untracked cache completely, or wipe\nout the cache and start fresh.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n dir.h |   4 +++\n 2 files changed, 114 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 27734f0..44ed9f2 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -582,6 +582,22 @@ static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \treturn d;\n }\n \n+static void do_invalidate_gitignore(struct untracked_cache_dir *dir)\n+{\n+\tint i;\n+\tdir->valid = 0;\n+\tdir->untracked_nr = 0;\n+\tfor (i = 0; i < dir->dirs_nr; i++)\n+\t\tdo_invalidate_gitignore(dir->dirs[i]);\n+}\n+\n+static void invalidate_gitignore(struct untracked_cache *uc,\n+\t\t\t\t struct untracked_cache_dir *dir)\n+{\n+\tuc->gitignore_invalidated++;\n+\tdo_invalidate_gitignore(dir);\n+}\n+\n /*\n  * Given a file with name \"fname\", read it (either from disk, or from\n  * the index if \"check_index\" is non-zero), parse it and store the\n@@ -697,6 +713,13 @@ static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,\n \t\t\t\t     struct sha1_stat *sha1_stat)\n {\n \tstruct exclude_list *el;\n+\t/*\n+\t * catch setup_standard_excludes() that's called before\n+\t * dir->untracked is assigned. That function behaves\n+\t * differently when dir->untracked is non-NULL.\n+\t */\n+\tif (!dir->untracked)\n+\t\tdir->unmanaged_exclude_files++;\n \tel = add_exclude_list(dir, EXC_FILE, fname);\n \tif (add_excludes(fname, \"\", 0, el, 0, sha1_stat) < 0)\n \t\tdie(\"cannot use %s as an exclude file\", fname);\n@@ -704,6 +727,7 @@ static void add_excludes_from_file_1(struct dir_struct *dir, const char *fname,\n \n void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n {\n+\tdir->unmanaged_exclude_files++; /* see validate_untracked_cache() */\n \tadd_excludes_from_file_1(dir, fname, NULL);\n }\n \n@@ -1572,9 +1596,87 @@ static int treat_leading_path(struct dir_struct *dir,\n \treturn rc;\n }\n \n+static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *dir,\n+\t\t\t\t\t\t      int base_len,\n+\t\t\t\t\t\t      const struct pathspec *pathspec)\n+{\n+\tstruct untracked_cache_dir *root;\n+\n+\tif (!dir->untracked)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * We only support $GIT_DIR/info/exclude and core.excludesfile\n+\t * as the global ignore rule files. Any other additions\n+\t * (e.g. from command line) invalidate the cache. This\n+\t * condition also catches running setup_standard_excludes()\n+\t * before setting dir->untracked!\n+\t */\n+\tif (dir->unmanaged_exclude_files)\n+\t\treturn NULL;\n+\n+\t/*\n+\t * Optimize for the main use case only: whole-tree git\n+\t * status. More work involved in treat_leading_path() if we\n+\t * use cache on just a subset of the worktree. pathspec\n+\t * support could make the matter even worse.\n+\t */\n+\tif (base_len || (pathspec && pathspec->nr))\n+\t\treturn NULL;\n+\n+\t/* Different set of flags may produce different results */\n+\tif (dir->flags != dir->untracked->dir_flags ||\n+\t    /*\n+\t     * See treat_directory(), case index_nonexistent. Without\n+\t     * this flag, we may need to also cache .git file content\n+\t     * for the resolve_gitlink_ref() call, which we don't.\n+\t     */\n+\t    !(dir->flags & DIR_SHOW_OTHER_DIRECTORIES) ||\n+\t    /* We don't support collecting ignore files */\n+\t    (dir->flags & (DIR_SHOW_IGNORED | DIR_SHOW_IGNORED_TOO |\n+\t\t\t   DIR_COLLECT_IGNORED)))\n+\t\treturn NULL;\n+\n+\t/*\n+\t * If we use .gitignore in the cache and now you change it to\n+\t * .gitexclude, everything will go wrong.\n+\t */\n+\tif (dir->exclude_per_dir != dir->untracked->exclude_per_dir &&\n+\t    strcmp(dir->exclude_per_dir, dir->untracked->exclude_per_dir))\n+\t\treturn NULL;\n+\n+\t/*\n+\t * EXC_CMDL is not considered in the cache. If people set it,\n+\t * skip the cache.\n+\t */\n+\tif (dir->exclude_list_group[EXC_CMDL].nr)\n+\t\treturn NULL;\n+\n+\tif (!dir->untracked->root) {\n+\t\tconst int len = sizeof(*dir->untracked->root);\n+\t\tdir->untracked->root = xmalloc(len);\n+\t\tmemset(dir->untracked->root, 0, len);\n+\t}\n+\n+\t/* Validate $GIT_DIR/info/exclude and core.excludesfile */\n+\troot = dir->untracked->root;\n+\tif (hashcmp(dir->ss_info_exclude.sha1,\n+\t\t    dir->untracked->ss_info_exclude.sha1)) {\n+\t\tinvalidate_gitignore(dir->untracked, root);\n+\t\tdir->untracked->ss_info_exclude = dir->ss_info_exclude;\n+\t}\n+\tif (hashcmp(dir->ss_excludes_file.sha1,\n+\t\t    dir->untracked->ss_excludes_file.sha1)) {\n+\t\tinvalidate_gitignore(dir->untracked, root);\n+\t\tdir->untracked->ss_excludes_file = dir->ss_excludes_file;\n+\t}\n+\treturn root;\n+}\n+\n int read_directory(struct dir_struct *dir, const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct path_simplify *simplify;\n+\tstruct untracked_cache_dir *untracked;\n \n \t/*\n \t * Check out create_simplify()\n@@ -1598,10 +1700,15 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \t * create_simplify().\n \t */\n \tsimplify = create_simplify(pathspec ? pathspec->_raw : NULL);\n+\tuntracked = validate_untracked_cache(dir, len, pathspec);\n+\tif (!untracked)\n+\t\t/*\n+\t\t * make sure untracked cache code path is disabled,\n+\t\t * e.g. prep_exclude()\n+\t\t */\n+\t\tdir->untracked = NULL;\n \tif (!len || treat_leading_path(dir, path, len, simplify))\n-\t\tread_directory_recursive(dir, path, len,\n-\t\t\t\t\t dir->untracked ? dir->untracked->root : NULL,\n-\t\t\t\t\t 0, simplify);\n+\t\tread_directory_recursive(dir, path, len, untracked, 0, simplify);\n \tfree_simplify(simplify);\n \tqsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);\n \tqsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name);\ndiff --git a/dir.h b/dir.h\nindex 9ab74b4..1d7a958 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -115,6 +115,8 @@ struct untracked_cache_dir {\n \tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n \tunsigned int untracked_nr;\n \tunsigned int check_only : 1;\n+\t/* all data in this struct are good */\n+\tunsigned int valid : 1;\n \t/* null SHA-1 means this directory does not have .gitignore */\n \tunsigned char exclude_sha1[20];\n \tchar name[FLEX_ARRAY];\n@@ -132,6 +134,7 @@ struct untracked_cache {\n \tstruct untracked_cache_dir *root;\n \t/* Statistics */\n \tint dir_created;\n+\tint gitignore_invalidated;\n };\n \n struct dir_struct {\n@@ -186,6 +189,7 @@ struct dir_struct {\n \tstruct untracked_cache *untracked;\n \tstruct sha1_stat ss_info_exclude;\n \tstruct sha1_stat ss_excludes_file;\n+\tunsigned unmanaged_exclude_files;\n };\n \n /*\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253314","messageId":"1418047507-22892-6-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 04/23] untracked cache: invalidate dirs recursively if .gitignore changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:48Z","receivedAt":"2014-12-08T14:04:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"It's easy to see that if an existing .gitignore changes, its SHA-1\nwould be different and invalidate_gitignore() is called.\n\nIf .gitignore is removed, add_excludes() will treat it like an empty\n.gitignore, which again should invalidate the cached directory data.\n\nif .gitignore is added, lookup_untracked() already fills initial\n.gitignore SHA-1 as \"empty file\", so again invalidate_gitignore() is\ncalled.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 18 +++++++++++++++++-\n 1 file changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 44ed9f2..6e91315 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1010,7 +1010,23 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t\tadd_excludes(el->src, el->src, stk->baselen, el, 1,\n \t\t\t\t     untracked ? &sha1_stat : NULL);\n \t\t}\n-\t\tif (untracked) {\n+\t\t/*\n+\t\t * NEEDSWORK: when untracked cache is enabled, prep_exclude()\n+\t\t * will first be called in valid_cached_dir() then maybe many\n+\t\t * times more in last_exclude_matching(). When the cache is\n+\t\t * used, last_exclude_matching() will not be called and\n+\t\t * reading .gitignore content will be a waste.\n+\t\t *\n+\t\t * So when it's called by valid_cached_dir() and we can get\n+\t\t * .gitignore SHA-1 from the index (i.e. .gitignore is not\n+\t\t * modified on work tree), we could delay reading the\n+\t\t * .gitignore content until we absolutely need it in\n+\t\t * last_exclude_matching(). Be careful about ignore rule\n+\t\t * order, though, if you do that.\n+\t\t */\n+\t\tif (untracked &&\n+\t\t    hashcmp(sha1_stat.sha1, untracked->exclude_sha1)) {\n+\t\t\tinvalidate_gitignore(dir->untracked, untracked);\n \t\t\thashcpy(untracked->exclude_sha1, sha1_stat.sha1);\n \t\t}\n \t\tdir->exclude_stack = stk;\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253315","messageId":"1418047507-22892-7-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 05/23] untracked cache: make a wrapper around {open,read,close}dir()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:49Z","receivedAt":"2014-12-08T14:04:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This allows us to feed different info to read_directory_recursive()\nbased on untracked cache in the next patch.\n\nHelped-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++--------\n 1 file changed, 47 insertions(+), 8 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 6e91315..fb6ed86 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -31,6 +31,15 @@ enum path_treatment {\n \tpath_untracked\n };\n \n+/*\n+ * Support data structure for our opendir/readdir/closedir wrappers\n+ */\n+struct cached_dir {\n+\tDIR *fdir;\n+\tstruct untracked_cache_dir *untracked;\n+\tstruct dirent *de;\n+};\n+\n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tconst char *path, int len, struct untracked_cache_dir *untracked,\n \tint check_only, const struct path_simplify *simplify);\n@@ -1417,12 +1426,13 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \n static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t      struct untracked_cache_dir *untracked,\n-\t\t\t\t      struct dirent *de,\n+\t\t\t\t      struct cached_dir *cdir,\n \t\t\t\t      struct strbuf *path,\n \t\t\t\t      int baselen,\n \t\t\t\t      const struct path_simplify *simplify)\n {\n \tint dtype;\n+\tstruct dirent *de = cdir->de;\n \n \tif (is_dot_or_dotdot(de->d_name) || !strcmp(de->d_name, \".git\"))\n \t\treturn path_none;\n@@ -1444,6 +1454,37 @@ static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n \tdir->untracked[dir->untracked_nr++] = xstrdup(name);\n }\n \n+static int open_cached_dir(struct cached_dir *cdir,\n+\t\t\t   struct dir_struct *dir,\n+\t\t\t   struct untracked_cache_dir *untracked,\n+\t\t\t   struct strbuf *path,\n+\t\t\t   int check_only)\n+{\n+\tmemset(cdir, 0, sizeof(*cdir));\n+\tcdir->untracked = untracked;\n+\tcdir->fdir = opendir(path->len ? path->buf : \".\");\n+\tif (!cdir->fdir)\n+\t\treturn -1;\n+\treturn 0;\n+}\n+\n+static int read_cached_dir(struct cached_dir *cdir)\n+{\n+\tif (cdir->fdir) {\n+\t\tcdir->de = readdir(cdir->fdir);\n+\t\tif (!cdir->de)\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n+\treturn -1;\n+}\n+\n+static void close_cached_dir(struct cached_dir *cdir)\n+{\n+\tif (cdir->fdir)\n+\t\tclosedir(cdir->fdir);\n+}\n+\n /*\n  * Read a directory tree. We currently ignore anything but\n  * directories, regular files and symlinks. That's because git\n@@ -1460,23 +1501,21 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t    struct untracked_cache_dir *untracked, int check_only,\n \t\t\t\t    const struct path_simplify *simplify)\n {\n-\tDIR *fdir;\n+\tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n-\tstruct dirent *de;\n \tstruct strbuf path = STRBUF_INIT;\n \n \tstrbuf_add(&path, base, baselen);\n \n-\tfdir = opendir(path.len ? path.buf : \".\");\n-\tif (!fdir)\n+\tif (open_cached_dir(&cdir, dir, untracked, &path, check_only))\n \t\tgoto out;\n \n \tif (untracked)\n \t\tuntracked->check_only = !!check_only;\n \n-\twhile ((de = readdir(fdir)) != NULL) {\n+\twhile (!read_cached_dir(&cdir)) {\n \t\t/* check how the file or directory should be treated */\n-\t\tstate = treat_path(dir, untracked, de, &path, baselen, simplify);\n+\t\tstate = treat_path(dir, untracked, &cdir, &path, baselen, simplify);\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n@@ -1529,7 +1568,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tbreak;\n \t\t}\n \t}\n-\tclosedir(fdir);\n+\tclose_cached_dir(&cdir);\n  out:\n \tstrbuf_release(&path);\n \n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253316","messageId":"1418047507-22892-8-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 06/23] untracked cache: record/validate dir mtime and reuse cached output","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:50Z","receivedAt":"2014-12-08T14:04:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The main readdir loop in read_directory_recursive() is replaced with a\nnew one that checks if cached results of a directory is still valid.\n\nIf a file is added or removed from the index, the containing directory\nis invalidated (but not its subdirs). If directory's mtime is changed,\nthe same happens. If a .gitignore is updated, the containing directory\nand all subdirs are invalidated recursively. If dir_struct#flags or\nother conditions change, the cache is ignored.\n\nIf a directory is invalidated, we opendir/readdir/closedir and run the\nexclude machinery on that directory listing as usual. If untracked\ncache is also enabled, we'll update the cache along the way. If a\ndirectory is validated, we simply pull the untracked listing out from\nthe cache. The cache also records the list of direct subdirs that we\nhave to recurse in. Fully excluded directories are seen as \"untracked\nfiles\".\n\nIn the best case when no dirs are invalidated, read_directory()\nbecomes a series of\n\n  stat(dir), open(.gitignore), fstat(), read(), close() and optionally\n  hash_sha1_file()\n\nFor comparison, standard read_directory() is a sequence of\n\n  opendir(), readdir(), open(.gitignore), fstat(), read(), close(), the\n  expensive last_exclude_matching() and closedir().\n\nWe already try not to open(.gitignore) if we know it does not exist,\nso open/fstat/read/close sequence does not apply to every\ndirectory. The sequence could be reduced further, as noted in\nprep_exclude() in another patch. So in theory, the entire best-case\nread_directory sequence could be reduced to a series of stat() and\nnothing else.\n\nThis is not a silver bullet approach. When you compile a C file, for\nexample, the old .o file is removed and a new one with the same name\ncreated, effectively invalidating the containing directory's cache\n(but not its subdirectories). If your build process touches every\ndirectory, this cache adds extra overhead for nothing, so it's a good\nidea to separate generated files from tracked files.. Editors may use\nthe same strategy for saving files. And of course you're out of luck\nrunning your repo on an unsupported filesystem and/or operating system.\n\nHelped-by: Eric Sunshine <sunshine@sunshineco.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n dir.h |   2 ++\n 2 files changed, 121 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex fb6ed86..8c989e3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -37,7 +37,12 @@ enum path_treatment {\n struct cached_dir {\n \tDIR *fdir;\n \tstruct untracked_cache_dir *untracked;\n+\tint nr_files;\n+\tint nr_dirs;\n+\n \tstruct dirent *de;\n+\tconst char *file;\n+\tstruct untracked_cache_dir *ucd;\n };\n \n static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n@@ -607,6 +612,14 @@ static void invalidate_gitignore(struct untracked_cache *uc,\n \tdo_invalidate_gitignore(dir);\n }\n \n+static void invalidate_directory(struct untracked_cache *uc,\n+\t\t\t\t struct untracked_cache_dir *dir)\n+{\n+\tuc->dir_invalidated++;\n+\tdir->valid = 0;\n+\tdir->untracked_nr = 0;\n+}\n+\n /*\n  * Given a file with name \"fname\", read it (either from disk, or from\n  * the index if \"check_index\" is non-zero), parse it and store the\n@@ -1424,6 +1437,39 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t   struct untracked_cache_dir *untracked,\n+\t\t\t\t\t   struct cached_dir *cdir,\n+\t\t\t\t\t   struct strbuf *path,\n+\t\t\t\t\t   int baselen,\n+\t\t\t\t\t   const struct path_simplify *simplify)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tif (path->buf[path->len - 1] != '/')\n+\t\tstrbuf_addch(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, simplify);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t      struct untracked_cache_dir *untracked,\n \t\t\t\t      struct cached_dir *cdir,\n@@ -1434,6 +1480,9 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \tint dtype;\n \tstruct dirent *de = cdir->de;\n \n+\tif (!de)\n+\t\treturn treat_path_fast(dir, untracked, cdir, path,\n+\t\t\t\t       baselen, simplify);\n \tif (is_dot_or_dotdot(de->d_name) || !strcmp(de->d_name, \".git\"))\n \t\treturn path_none;\n \tstrbuf_setlen(path, baselen);\n@@ -1454,6 +1503,52 @@ static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n \tdir->untracked[dir->untracked_nr++] = xstrdup(name);\n }\n \n+static int valid_cached_dir(struct dir_struct *dir,\n+\t\t\t    struct untracked_cache_dir *untracked,\n+\t\t\t    struct strbuf *path,\n+\t\t\t    int check_only)\n+{\n+\tstruct stat st;\n+\n+\tif (!untracked)\n+\t\treturn 0;\n+\n+\tif (stat(path->len ? path->buf : \".\", &st)) {\n+\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\tmemset(&untracked->stat_data, 0, sizeof(untracked->stat_data));\n+\t\treturn 0;\n+\t}\n+\tif (!untracked->valid ||\n+\t    match_stat_data(&untracked->stat_data, &st)) {\n+\t\tif (untracked->valid)\n+\t\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\tfill_stat_data(&untracked->stat_data, &st);\n+\t\treturn 0;\n+\t}\n+\n+\tif (untracked->check_only != !!check_only) {\n+\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\treturn 0;\n+\t}\n+\n+\t/*\n+\t * prep_exclude will be called eventually on this directory,\n+\t * but it's called much later in last_exclude_matching(). We\n+\t * need it now to determine the validity of the cache for this\n+\t * path. The next calls will be nearly no-op, the way\n+\t * prep_exclude() is designed.\n+\t */\n+\tif (path->len && path->buf[path->len - 1] != '/') {\n+\t\tstrbuf_addch(path, '/');\n+\t\tprep_exclude(dir, path->buf, path->len);\n+\t\tstrbuf_setlen(path, path->len - 1);\n+\t} else\n+\t\tprep_exclude(dir, path->buf, path->len);\n+\n+\t/* hopefully prep_exclude() haven't invalidated this entry... */\n+\treturn untracked->valid;\n+}\n+\n static int open_cached_dir(struct cached_dir *cdir,\n \t\t\t   struct dir_struct *dir,\n \t\t\t   struct untracked_cache_dir *untracked,\n@@ -1462,7 +1557,11 @@ static int open_cached_dir(struct cached_dir *cdir,\n {\n \tmemset(cdir, 0, sizeof(*cdir));\n \tcdir->untracked = untracked;\n+\tif (valid_cached_dir(dir, untracked, path, check_only))\n+\t\treturn 0;\n \tcdir->fdir = opendir(path->len ? path->buf : \".\");\n+\tif (dir->untracked)\n+\t\tdir->untracked->dir_opened++;\n \tif (!cdir->fdir)\n \t\treturn -1;\n \treturn 0;\n@@ -1476,6 +1575,18 @@ static int read_cached_dir(struct cached_dir *cdir)\n \t\t\treturn -1;\n \t\treturn 0;\n \t}\n+\twhile (cdir->nr_dirs < cdir->untracked->dirs_nr) {\n+\t\tstruct untracked_cache_dir *d = cdir->untracked->dirs[cdir->nr_dirs];\n+\t\tcdir->ucd = d;\n+\t\tcdir->nr_dirs++;\n+\t\treturn 0;\n+\t}\n+\tcdir->ucd = NULL;\n+\tif (cdir->nr_files < cdir->untracked->untracked_nr) {\n+\t\tstruct untracked_cache_dir *d = cdir->untracked;\n+\t\tcdir->file = d->untracked[cdir->nr_files++];\n+\t\treturn 0;\n+\t}\n \treturn -1;\n }\n \n@@ -1483,6 +1594,12 @@ static void close_cached_dir(struct cached_dir *cdir)\n {\n \tif (cdir->fdir)\n \t\tclosedir(cdir->fdir);\n+\t/*\n+\t * We have gone through this directory and found no untracked\n+\t * entries. Mark it valid.\n+\t */\n+\tif (cdir->untracked)\n+\t\tcdir->untracked->valid = 1;\n }\n \n /*\n@@ -1536,7 +1653,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\tif (check_only) {\n \t\t\t/* abort early if maximum state has been reached */\n \t\t\tif (dir_state == path_untracked) {\n-\t\t\t\tif (untracked)\n+\t\t\t\tif (cdir.fdir)\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n@@ -1560,7 +1677,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n \t\t\t\tbreak;\n \t\t\tdir_add_name(dir, path.buf, path.len);\n-\t\t\tif (untracked)\n+\t\t\tif (cdir.fdir)\n \t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\tbreak;\n \ndiff --git a/dir.h b/dir.h\nindex 1d7a958..ff3d99b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -135,6 +135,8 @@ struct untracked_cache {\n \t/* Statistics */\n \tint dir_created;\n \tint gitignore_invalidated;\n+\tint dir_invalidated;\n+\tint dir_opened;\n };\n \n struct dir_struct {\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253317","messageId":"1418047507-22892-9-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 07/23] untracked cache: mark what dirs should be recursed/saved","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:51Z","receivedAt":"2014-12-08T14:04:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If we redo this thing in a functional style, we would have one struct\nuntracked_dir as input tree and another as output. The input is used\nfor verification. The output is a brand new tree, reflecting current\nworktree.\n\nBut that means recreate a lot of dir nodes even if a lot could be\nshared between input and output trees in good cases. So we go with the\nmessy but efficient way, combining both input and output trees into\none. We need a way to know which node in this combined tree belongs to\nthe output. This is the purpose of this \"recurse\" flag.\n\n\"valid\" bit can't be used for this because it's about data of the node\nexcept the subdirs. When we invalidate a directory, we want to keep\ncached data of the subdirs intact even though we don't really know\nwhat subdir still exists (yet). Then we check worktree to see what\nactual subdir remains on disk. Those will have 'recurse' bit set\nagain. If cached data for those are still valid, we may be able to\navoid computing exclude files for them. Those subdirs that are deleted\nwill have 'recurse' remained clear and their 'valid' bits do not\nmatter.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 14 +++++++++++++-\n dir.h |  3 ++-\n 2 files changed, 15 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 8c989e3..02cdc26 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -615,9 +615,12 @@ static void invalidate_gitignore(struct untracked_cache *uc,\n static void invalidate_directory(struct untracked_cache *uc,\n \t\t\t\t struct untracked_cache_dir *dir)\n {\n+\tint i;\n \tuc->dir_invalidated++;\n \tdir->valid = 0;\n \tdir->untracked_nr = 0;\n+\tfor (i = 0; i < dir->dirs_nr; i++)\n+\t\tdir->dirs[i]->recurse = 0;\n }\n \n /*\n@@ -1577,6 +1580,10 @@ static int read_cached_dir(struct cached_dir *cdir)\n \t}\n \twhile (cdir->nr_dirs < cdir->untracked->dirs_nr) {\n \t\tstruct untracked_cache_dir *d = cdir->untracked->dirs[cdir->nr_dirs];\n+\t\tif (!d->recurse) {\n+\t\t\tcdir->nr_dirs++;\n+\t\t\tcontinue;\n+\t\t}\n \t\tcdir->ucd = d;\n \t\tcdir->nr_dirs++;\n \t\treturn 0;\n@@ -1598,8 +1605,10 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t * We have gone through this directory and found no untracked\n \t * entries. Mark it valid.\n \t */\n-\tif (cdir->untracked)\n+\tif (cdir->untracked) {\n \t\tcdir->untracked->valid = 1;\n+\t\tcdir->untracked->recurse = 1;\n+\t}\n }\n \n /*\n@@ -1842,6 +1851,9 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \t\tinvalidate_gitignore(dir->untracked, root);\n \t\tdir->untracked->ss_excludes_file = dir->ss_excludes_file;\n \t}\n+\n+\t/* Make sure this directory is not dropped out at saving phase */\n+\troot->recurse = 1;\n \treturn root;\n }\n \ndiff --git a/dir.h b/dir.h\nindex ff3d99b..95baf01 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -115,8 +115,9 @@ struct untracked_cache_dir {\n \tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n \tunsigned int untracked_nr;\n \tunsigned int check_only : 1;\n-\t/* all data in this struct are good */\n+\t/* all data except 'dirs' in this struct are good */\n \tunsigned int valid : 1;\n+\tunsigned int recurse : 1;\n \t/* null SHA-1 means this directory does not have .gitignore */\n \tunsigned char exclude_sha1[20];\n \tchar name[FLEX_ARRAY];\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253319","messageId":"1418047507-22892-10-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 08/23] untracked cache: don't open non-existent .gitignore","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:52Z","receivedAt":"2014-12-08T14:04:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This cuts down a signficant number of open(.gitignore) because most\ndirectories usually don't have .gitignore files.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 26 +++++++++++++++++++++++++-\n 1 file changed, 25 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 02cdc26..95a0f0a 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1019,7 +1019,21 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\t/* Try to read per-directory file */\n \t\thashclr(sha1_stat.sha1);\n \t\tsha1_stat.valid = 0;\n-\t\tif (dir->exclude_per_dir) {\n+\t\tif (dir->exclude_per_dir &&\n+\t\t    /*\n+\t\t     * If we know that no files have been added in\n+\t\t     * this directory (i.e. valid_cached_dir() has\n+\t\t     * been executed and set untracked->valid) ..\n+\t\t     */\n+\t\t    (!untracked || !untracked->valid ||\n+\t\t     /*\n+\t\t      * .. and .gitignore does not exist before\n+\t\t      * (i.e. null exclude_sha1 and skip_worktree is\n+\t\t      * not set). Then we can skip loading .gitignore,\n+\t\t      * which would result in ENOENT anyway.\n+\t\t      * skip_worktree is taken care in read_directory()\n+\t\t      */\n+\t\t     !is_null_sha1(untracked->exclude_sha1))) {\n \t\t\t/*\n \t\t\t * dir->basebuf gets reused by the traversal, but we\n \t\t\t * need fname to remain unchanged to ensure the src\n@@ -1782,6 +1796,7 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \t\t\t\t\t\t      const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *root;\n+\tint i;\n \n \tif (!dir->untracked)\n \t\treturn NULL;\n@@ -1833,6 +1848,15 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \tif (dir->exclude_list_group[EXC_CMDL].nr)\n \t\treturn NULL;\n \n+\t/*\n+\t * An optimization in prep_exclude() does not play well with\n+\t * CE_SKIP_WORKTREE. It's a rare case anyway, if a single\n+\t * entry has that bit set, disable the whole untracked cache.\n+\t */\n+\tfor (i = 0; i < active_nr; i++)\n+\t\tif (ce_skip_worktree(active_cache[i]))\n+\t\t\treturn NULL;\n+\n \tif (!dir->untracked->root) {\n \t\tconst int len = sizeof(*dir->untracked->root);\n \t\tdir->untracked->root = xmalloc(len);\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253318","messageId":"1418047507-22892-11-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 09/23] ewah: add convenient wrapper ewah_serialize_strbuf()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:53Z","receivedAt":"2014-12-08T14:04:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n ewah/ewah_io.c | 13 +++++++++++++\n ewah/ewok.h    |  2 ++\n split-index.c  | 11 ++---------\n 3 files changed, 17 insertions(+), 9 deletions(-)\n\ndiff --git a/ewah/ewah_io.c b/ewah/ewah_io.c\nindex 1c2d7af..43481b9 100644\n--- a/ewah/ewah_io.c\n+++ b/ewah/ewah_io.c\n@@ -19,6 +19,7 @@\n  */\n #include \"git-compat-util.h\"\n #include \"ewok.h\"\n+#include \"strbuf.h\"\n \n int ewah_serialize_native(struct ewah_bitmap *self, int fd)\n {\n@@ -110,6 +111,18 @@ int ewah_serialize(struct ewah_bitmap *self, int fd)\n \treturn ewah_serialize_to(self, write_helper, (void *)(intptr_t)fd);\n }\n \n+static int write_strbuf(void *user_data, const void *data, size_t len)\n+{\n+\tstruct strbuf *sb = user_data;\n+\tstrbuf_add(sb, data, len);\n+\treturn len;\n+}\n+\n+int ewah_serialize_strbuf(struct ewah_bitmap *self, struct strbuf *sb)\n+{\n+\treturn ewah_serialize_to(self, write_strbuf, sb);\n+}\n+\n int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len)\n {\n \tconst uint8_t *ptr = map;\ndiff --git a/ewah/ewok.h b/ewah/ewok.h\nindex f6ad190..4d7f5e9 100644\n--- a/ewah/ewok.h\n+++ b/ewah/ewok.h\n@@ -30,6 +30,7 @@\n #\tdefine ewah_calloc xcalloc\n #endif\n \n+struct strbuf;\n typedef uint64_t eword_t;\n #define BITS_IN_WORD (sizeof(eword_t) * 8)\n \n@@ -97,6 +98,7 @@ int ewah_serialize_to(struct ewah_bitmap *self,\n \t\t      void *out);\n int ewah_serialize(struct ewah_bitmap *self, int fd);\n int ewah_serialize_native(struct ewah_bitmap *self, int fd);\n+int ewah_serialize_strbuf(struct ewah_bitmap *self, struct strbuf *);\n \n int ewah_deserialize(struct ewah_bitmap *self, int fd);\n int ewah_read_mmap(struct ewah_bitmap *self, const void *map, size_t len);\ndiff --git a/split-index.c b/split-index.c\nindex 21485e2..968b780 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -41,13 +41,6 @@ int read_link_extension(struct index_state *istate,\n \treturn 0;\n }\n \n-static int write_strbuf(void *user_data, const void *data, size_t len)\n-{\n-\tstruct strbuf *sb = user_data;\n-\tstrbuf_add(sb, data, len);\n-\treturn len;\n-}\n-\n int write_link_extension(struct strbuf *sb,\n \t\t\t struct index_state *istate)\n {\n@@ -55,8 +48,8 @@ int write_link_extension(struct strbuf *sb,\n \tstrbuf_add(sb, si->base_sha1, 20);\n \tif (!si->delete_bitmap && !si->replace_bitmap)\n \t\treturn 0;\n-\tewah_serialize_to(si->delete_bitmap, write_strbuf, sb);\n-\tewah_serialize_to(si->replace_bitmap, write_strbuf, sb);\n+\tewah_serialize_strbuf(si->delete_bitmap, sb);\n+\tewah_serialize_strbuf(si->replace_bitmap, sb);\n \treturn 0;\n }\n \n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253320","messageId":"1418047507-22892-12-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 10/23] untracked cache: save to an index extension","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:54Z","receivedAt":"2014-12-08T14:04:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/technical/index-format.txt |  58 +++++++++++++\n cache.h                                  |   3 +\n dir.c                                    | 134 +++++++++++++++++++++++++++++++\n dir.h                                    |   1 +\n read-cache.c                             |  12 +++\n 5 files changed, 208 insertions(+)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex fe6f316..b97ac8d 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -233,3 +233,61 @@ Git index format\n   The remaining index entries after replaced ones will be added to the\n   final index. These added entries are also sorted by entry namme then\n   stage.\n+\n+== Untracked cache\n+\n+  Untracked cache saves the untracked file list and necessary data to\n+  verify the cache. The signature for this extension is { 'U', 'N',\n+  'T', 'R' }.\n+\n+  The extension starts with\n+\n+  - Stat data of $GIT_DIR/info/exclude. See \"Index entry\" section from\n+    ctime field until \"file size\".\n+\n+  - Stat data of core.excludesfile\n+\n+  - 32-bit dir_flags (see struct dir_struct)\n+\n+  - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n+    does not exist.\n+\n+  - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n+    not exist.\n+\n+  - NUL-terminated string of per-dir exclude file name. This usually\n+    is \".gitignore\".\n+\n+  - The number of following directory blocks, variable width\n+    encoding. If this number is zero, the extension ends here with a\n+    following NUL.\n+\n+  - A number of directory blocks in depth-first-search order, each\n+    consists of\n+\n+    - The number of untracked entries, variable witdh encoding.\n+\n+    - The number of sub-directory blocks, variable with encoding.\n+\n+    - The directory name terminated by NUL.\n+\n+    - A number of untrached file/dir names terminated by NUL.\n+\n+The remaining data of each directory block is grouped by type:\n+\n+  - An ewah bitmap, the n-th bit marks whether the n-th directory has\n+    valid untracked cache entries.\n+\n+  - An ewah bitmap, the n-th bit records \"check-only\" bit of\n+    read_directory_recursive() for the n-th directory.\n+\n+  - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n+    is valid for the n-th directory and exists in the next data.\n+\n+  - An array of stat data. The n-th data corresponds with the n-th\n+    \"one\" bit in the previous ewah bitmap.\n+\n+  - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n+    in the previous ewah bitmap.\n+\n+  - One NUL.\ndiff --git a/cache.h b/cache.h\nindex dcf3a2a..b14d6e2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -297,6 +297,8 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n \n struct split_index;\n+struct untracked_cache;\n+\n struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int version;\n@@ -310,6 +312,7 @@ struct index_state {\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\n+\tstruct untracked_cache *untracked;\n };\n \n extern struct index_state the_index;\ndiff --git a/dir.c b/dir.c\nindex 95a0f0a..a0a7330 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -12,6 +12,8 @@\n #include \"refs.h\"\n #include \"wildmatch.h\"\n #include \"pathspec.h\"\n+#include \"varint.h\"\n+#include \"ewah/ewok.h\"\n \n struct path_simplify {\n \tint len;\n@@ -2139,3 +2141,135 @@ void clear_directory(struct dir_struct *dir)\n \t}\n \tstrbuf_release(&dir->basebuf);\n }\n+\n+struct ondisk_untracked_cache {\n+\tstruct stat_data info_exclude_stat;\n+\tstruct stat_data excludes_file_stat;\n+\tuint32_t dir_flags;\n+\tunsigned char info_exclude_sha1[20];\n+\tunsigned char excludes_file_sha1[20];\n+\tchar exclude_per_dir[1];\n+};\n+\n+struct write_data {\n+\tint index;\t   /* number of written untracked_cache_dir */\n+\tstruct ewah_bitmap *check_only; /* from untracked_cache_dir */\n+\tstruct ewah_bitmap *valid;\t/* from untracked_cache_dir */\n+\tstruct ewah_bitmap *sha1_valid; /* set if exclude_sha1 is not null */\n+\tstruct strbuf out;\n+\tstruct strbuf sb_stat;\n+\tstruct strbuf sb_sha1;\n+};\n+\n+static void stat_data_to_disk(struct stat_data *to, const struct stat_data *from)\n+{\n+\tto->sd_ctime.sec  = htonl(from->sd_ctime.sec);\n+\tto->sd_ctime.nsec = htonl(from->sd_ctime.nsec);\n+\tto->sd_mtime.sec  = htonl(from->sd_mtime.sec);\n+\tto->sd_mtime.nsec = htonl(from->sd_mtime.nsec);\n+\tto->sd_dev\t  = htonl(from->sd_dev);\n+\tto->sd_ino\t  = htonl(from->sd_ino);\n+\tto->sd_uid\t  = htonl(from->sd_uid);\n+\tto->sd_gid\t  = htonl(from->sd_gid);\n+\tto->sd_size\t  = htonl(from->sd_size);\n+}\n+\n+static void write_one_dir(struct untracked_cache_dir *untracked,\n+\t\t\t  struct write_data *wd)\n+{\n+\tstruct stat_data stat_data;\n+\tstruct strbuf *out = &wd->out;\n+\tunsigned char intbuf[16];\n+\tunsigned int intlen, value;\n+\tint i = wd->index++;\n+\n+\t/*\n+\t * untracked_nr should be reset whenever valid is clear, but\n+\t * for safety..\n+\t */\n+\tif (!untracked->valid) {\n+\t\tuntracked->untracked_nr = 0;\n+\t\tuntracked->check_only = 0;\n+\t}\n+\n+\tif (untracked->check_only)\n+\t\tewah_set(wd->check_only, i);\n+\tif (untracked->valid) {\n+\t\tewah_set(wd->valid, i);\n+\t\tstat_data_to_disk(&stat_data, &untracked->stat_data);\n+\t\tstrbuf_add(&wd->sb_stat, &stat_data, sizeof(stat_data));\n+\t}\n+\tif (!is_null_sha1(untracked->exclude_sha1)) {\n+\t\tewah_set(wd->sha1_valid, i);\n+\t\tstrbuf_add(&wd->sb_sha1, untracked->exclude_sha1, 20);\n+\t}\n+\n+\tintlen = encode_varint(untracked->untracked_nr, intbuf);\n+\tstrbuf_add(out, intbuf, intlen);\n+\n+\t/* skip non-recurse directories */\n+\tfor (i = 0, value = 0; i < untracked->dirs_nr; i++)\n+\t\tif (untracked->dirs[i]->recurse)\n+\t\t\tvalue++;\n+\tintlen = encode_varint(value, intbuf);\n+\tstrbuf_add(out, intbuf, intlen);\n+\n+\tstrbuf_add(out, untracked->name, strlen(untracked->name) + 1);\n+\n+\tfor (i = 0; i < untracked->untracked_nr; i++)\n+\t\tstrbuf_add(out, untracked->untracked[i],\n+\t\t\t   strlen(untracked->untracked[i]) + 1);\n+\n+\tfor (i = 0; i < untracked->dirs_nr; i++)\n+\t\tif (untracked->dirs[i]->recurse)\n+\t\t\twrite_one_dir(untracked->dirs[i], wd);\n+}\n+\n+void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked)\n+{\n+\tstruct ondisk_untracked_cache *ouc;\n+\tstruct write_data wd;\n+\tunsigned char varbuf[16];\n+\tint len = 0, varint_len;\n+\tif (untracked->exclude_per_dir)\n+\t\tlen = strlen(untracked->exclude_per_dir);\n+\touc = xmalloc(sizeof(*ouc) + len);\n+\tstat_data_to_disk(&ouc->info_exclude_stat, &untracked->ss_info_exclude.stat);\n+\tstat_data_to_disk(&ouc->excludes_file_stat, &untracked->ss_excludes_file.stat);\n+\thashcpy(ouc->info_exclude_sha1, untracked->ss_info_exclude.sha1);\n+\thashcpy(ouc->excludes_file_sha1, untracked->ss_excludes_file.sha1);\n+\touc->dir_flags = htonl(untracked->dir_flags);\n+\tmemcpy(ouc->exclude_per_dir, untracked->exclude_per_dir, len + 1);\n+\tstrbuf_add(out, ouc, sizeof(*ouc) + len);\n+\tif (!untracked->root) {\n+\t\tvarint_len = encode_varint(0, varbuf);\n+\t\tstrbuf_add(out, varbuf, varint_len);\n+\t\treturn;\n+\t}\n+\n+\twd.index      = 0;\n+\twd.check_only = ewah_new();\n+\twd.valid      = ewah_new();\n+\twd.sha1_valid = ewah_new();\n+\tstrbuf_init(&wd.out, 1024);\n+\tstrbuf_init(&wd.sb_stat, 1024);\n+\tstrbuf_init(&wd.sb_sha1, 1024);\n+\twrite_one_dir(untracked->root, &wd);\n+\n+\tvarint_len = encode_varint(wd.index, varbuf);\n+\tstrbuf_add(out, varbuf, varint_len);\n+\tstrbuf_addbuf(out, &wd.out);\n+\tewah_serialize_strbuf(wd.valid, out);\n+\tewah_serialize_strbuf(wd.check_only, out);\n+\tewah_serialize_strbuf(wd.sha1_valid, out);\n+\tstrbuf_addbuf(out, &wd.sb_stat);\n+\tstrbuf_addbuf(out, &wd.sb_sha1);\n+\tstrbuf_addch(out, '\\0'); /* safe guard for string lists */\n+\n+\tewah_free(wd.valid);\n+\tewah_free(wd.check_only);\n+\tewah_free(wd.sha1_valid);\n+\tstrbuf_release(&wd.out);\n+\tstrbuf_release(&wd.sb_stat);\n+\tstrbuf_release(&wd.sb_sha1);\n+}\ndiff --git a/dir.h b/dir.h\nindex 95baf01..dc3ee0b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -298,4 +298,5 @@ static inline int dir_path_match(const struct dir_entry *ent,\n \t\t\t      has_trailing_dir);\n }\n \n+void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n #endif\ndiff --git a/read-cache.c b/read-cache.c\nindex 6f0057f..baf3057 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -37,6 +37,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n #define CACHE_EXT_LINK 0x6c696e6b\t  /* \"link\" */\n+#define CACHE_EXT_UNTRACKED 0x554E5452\t  /* \"UNTR\" */\n \n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n@@ -2016,6 +2017,17 @@ static int do_write_index(struct index_state *istate, int newfd,\n \t\tif (err)\n \t\t\treturn -1;\n \t}\n+\tif (!strip_extensions && istate->untracked) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\twrite_untracked_extension(&sb, istate->untracked);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_UNTRACKED,\n+\t\t\t\t\t     sb.len) < 0 ||\n+\t\t\tce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \n \tif (ce_flush(&c, newfd, istate->sha1) || fstat(newfd, &st))\n \t\treturn -1;\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253321","messageId":"1418047507-22892-13-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 11/23] untracked cache: load from UNTR index extension","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:55Z","receivedAt":"2014-12-08T14:04:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c        | 220 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n dir.h        |   2 +\n read-cache.c |   5 ++\n 3 files changed, 227 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex a0a7330..0126a0d 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2273,3 +2273,223 @@ void write_untracked_extension(struct strbuf *out, struct untracked_cache *untra\n \tstrbuf_release(&wd.sb_stat);\n \tstrbuf_release(&wd.sb_sha1);\n }\n+\n+static void free_untracked(struct untracked_cache_dir *ucd)\n+{\n+\tint i;\n+\tif (!ucd)\n+\t\treturn;\n+\tfor (i = 0; i < ucd->dirs_nr; i++)\n+\t\tfree_untracked(ucd->dirs[i]);\n+\tfor (i = 0; i < ucd->untracked_nr; i++)\n+\t\tfree(ucd->untracked[i]);\n+\tfree(ucd->untracked);\n+\tfree(ucd->dirs);\n+\tfree(ucd);\n+}\n+\n+void free_untracked_cache(struct untracked_cache *uc)\n+{\n+\tif (uc)\n+\t\tfree_untracked(uc->root);\n+\tfree(uc);\n+}\n+\n+struct read_data {\n+\tint index;\n+\tstruct untracked_cache_dir **ucd;\n+\tstruct ewah_bitmap *check_only;\n+\tstruct ewah_bitmap *valid;\n+\tstruct ewah_bitmap *sha1_valid;\n+\tconst unsigned char *data;\n+\tconst unsigned char *end;\n+};\n+\n+static void stat_data_from_disk(struct stat_data *to, const struct stat_data *from)\n+{\n+\tto->sd_ctime.sec  = get_be32(&from->sd_ctime.sec);\n+\tto->sd_ctime.nsec = get_be32(&from->sd_ctime.nsec);\n+\tto->sd_mtime.sec  = get_be32(&from->sd_mtime.sec);\n+\tto->sd_mtime.nsec = get_be32(&from->sd_mtime.nsec);\n+\tto->sd_dev\t  = get_be32(&from->sd_dev);\n+\tto->sd_ino\t  = get_be32(&from->sd_ino);\n+\tto->sd_uid\t  = get_be32(&from->sd_uid);\n+\tto->sd_gid\t  = get_be32(&from->sd_gid);\n+\tto->sd_size\t  = get_be32(&from->sd_size);\n+}\n+\n+static int read_one_dir(struct untracked_cache_dir **untracked_,\n+\t\t\tstruct read_data *rd)\n+{\n+#define NEXT(x) \\\n+\tnext = data + (x); \\\n+\tif (next > rd->end) \\\n+\t\treturn -1;\n+\n+\tstruct untracked_cache_dir ud, *untracked;\n+\tconst unsigned char *next, *data = rd->data, *end = rd->end;\n+\tunsigned int value;\n+\tint i, len;\n+\n+\tmemset(&ud, 0, sizeof(ud));\n+\n+\tnext = data;\n+\tvalue = decode_varint(&next);\n+\tif (next > end)\n+\t\treturn -1;\n+\tud.recurse\t   = 1;\n+\tud.untracked_alloc = value;\n+\tud.untracked_nr\t   = value;\n+\tif (ud.untracked_nr)\n+\t\tud.untracked = xmalloc(sizeof(*ud.untracked) * ud.untracked_nr);\n+\tdata = next;\n+\n+\tnext = data;\n+\tud.dirs_alloc = ud.dirs_nr = decode_varint(&next);\n+\tif (next > end)\n+\t\treturn -1;\n+\tud.dirs = xmalloc(sizeof(*ud.dirs) * ud.dirs_nr);\n+\tdata = next;\n+\n+\tlen = strlen((const char *)data);\n+\tNEXT(len + 1);\n+\t*untracked_ = untracked = xmalloc(sizeof(*untracked) + len);\n+\tmemcpy(untracked, &ud, sizeof(ud));\n+\tmemcpy(untracked->name, data, len + 1);\n+\tdata = next;\n+\n+\tfor (i = 0; i < untracked->untracked_nr; i++) {\n+\t\tlen = strlen((const char *)data);\n+\t\tNEXT(len + 1);\n+\t\tuntracked->untracked[i] = xstrdup((const char*)data);\n+\t\tdata = next;\n+\t}\n+\n+\trd->ucd[rd->index++] = untracked;\n+\trd->data = data;\n+\n+\tfor (i = 0; i < untracked->dirs_nr; i++) {\n+\t\tlen = read_one_dir(untracked->dirs + i, rd);\n+\t\tif (len < 0)\n+\t\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\n+\n+static void set_check_only(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tud->check_only = 1;\n+}\n+\n+static void read_stat(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tif (rd->data + sizeof(struct stat_data) > rd->end) {\n+\t\trd->data = rd->end + 1;\n+\t\treturn;\n+\t}\n+\tstat_data_from_disk(&ud->stat_data, (struct stat_data *)rd->data);\n+\trd->data += sizeof(struct stat_data);\n+\tud->valid = 1;\n+}\n+\n+static void read_sha1(size_t pos, void *cb)\n+{\n+\tstruct read_data *rd = cb;\n+\tstruct untracked_cache_dir *ud = rd->ucd[pos];\n+\tif (rd->data + 20 > rd->end) {\n+\t\trd->data = rd->end + 1;\n+\t\treturn;\n+\t}\n+\thashcpy(ud->exclude_sha1, rd->data);\n+\trd->data += 20;\n+}\n+\n+static void load_sha1_stat(struct sha1_stat *sha1_stat,\n+\t\t\t   const struct stat_data *stat,\n+\t\t\t   const unsigned char *sha1)\n+{\n+\tstat_data_from_disk(&sha1_stat->stat, stat);\n+\thashcpy(sha1_stat->sha1, sha1);\n+\tsha1_stat->valid = 1;\n+}\n+\n+struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz)\n+{\n+\tconst struct ondisk_untracked_cache *ouc;\n+\tstruct untracked_cache *uc;\n+\tstruct read_data rd;\n+\tconst unsigned char *next = data, *end = data + sz;\n+\tint len;\n+\n+\tif (sz <= 1 || end[-1] != '\\0')\n+\t\treturn NULL;\n+\tend--;\n+\n+\touc = (const struct ondisk_untracked_cache *)next;\n+\tif (next + sizeof(*ouc) > end)\n+\t\treturn NULL;\n+\n+\tuc = xcalloc(1, sizeof(*uc));\n+\tload_sha1_stat(&uc->ss_info_exclude, &ouc->info_exclude_stat,\n+\t\t       ouc->info_exclude_sha1);\n+\tload_sha1_stat(&uc->ss_excludes_file, &ouc->excludes_file_stat,\n+\t\t       ouc->excludes_file_sha1);\n+\tuc->dir_flags = get_be32(&ouc->dir_flags);\n+\tuc->exclude_per_dir = xstrdup(ouc->exclude_per_dir);\n+\t/* NUL after exclude_per_dir is covered by sizeof(*ouc) */\n+\tnext += sizeof(*ouc) + strlen(ouc->exclude_per_dir);\n+\tif (next >= end)\n+\t\tgoto done2;\n+\n+\tlen = decode_varint(&next);\n+\tif (next > end || len == 0)\n+\t\tgoto done2;\n+\n+\trd.valid      = ewah_new();\n+\trd.check_only = ewah_new();\n+\trd.sha1_valid = ewah_new();\n+\trd.data\t      = next;\n+\trd.end\t      = end;\n+\trd.index      = 0;\n+\trd.ucd        = xmalloc(sizeof(*rd.ucd) * len);\n+\n+\tif (read_one_dir(&uc->root, &rd) || rd.index != len)\n+\t\tgoto done;\n+\n+\tnext = rd.data;\n+\tlen = ewah_read_mmap(rd.valid, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tnext += len;\n+\tlen = ewah_read_mmap(rd.check_only, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tnext += len;\n+\tlen = ewah_read_mmap(rd.sha1_valid, next, end - next);\n+\tif (len < 0)\n+\t\tgoto done;\n+\n+\tewah_each_bit(rd.check_only, set_check_only, &rd);\n+\trd.data = next + len;\n+\tewah_each_bit(rd.valid, read_stat, &rd);\n+\tewah_each_bit(rd.sha1_valid, read_sha1, &rd);\n+\tnext = rd.data;\n+\n+done:\n+\tfree(rd.ucd);\n+\tewah_free(rd.valid);\n+\tewah_free(rd.check_only);\n+\tewah_free(rd.sha1_valid);\n+done2:\n+\tif (next != end) {\n+\t\tfree_untracked_cache(uc);\n+\t\tuc = NULL;\n+\t}\n+\treturn uc;\n+}\ndiff --git a/dir.h b/dir.h\nindex dc3ee0b..40a679a 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -298,5 +298,7 @@ static inline int dir_path_match(const struct dir_entry *ent,\n \t\t\t      has_trailing_dir);\n }\n \n+void free_untracked_cache(struct untracked_cache *);\n+struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz);\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\n #endif\ndiff --git a/read-cache.c b/read-cache.c\nindex baf3057..3736a56 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1366,6 +1366,9 @@ static int read_index_extension(struct index_state *istate,\n \t\tif (read_link_extension(istate, data, sz))\n \t\t\treturn -1;\n \t\tbreak;\n+\tcase CACHE_EXT_UNTRACKED:\n+\t\tistate->untracked = read_untracked_extension(data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1631,6 +1634,8 @@ int discard_index(struct index_state *istate)\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n \tdiscard_split_index(istate);\n+\tfree_untracked_cache(istate->untracked);\n+\tistate->untracked = NULL;\n \treturn 0;\n }\n \n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253322","messageId":"1418047507-22892-14-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 12/23] untracked cache: invalidate at index addition or removal","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:56Z","receivedAt":"2014-12-08T14:04:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Ideally we should implement untracked_cache_remove_from_index() and\nuntracked_cache_add_to_index() so that they update untracked cache\nright away instead of invalidating it and wait for read_directory()\nnext time to deal with it. But that may need some more work in\nunpack-trees.c. So stay simple as the first step.\n\nThe new call in add_index_entry_with_check() may look strange because\nnew calls usually stay close to cache_tree_invalidate_path(). We do it\na bit later than c_t_i_p() in this function because if it's about\nreplacing the entry with the same name, we don't care (but cache-tree\ndoes).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c          | 31 +++++++++++++++++++++++++++++++\n dir.h          |  4 ++++\n read-cache.c   |  4 ++++\n unpack-trees.c |  7 +++++--\n 4 files changed, 44 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0126a0d..a14b48f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2493,3 +2493,34 @@ done2:\n \t}\n \treturn uc;\n }\n+\n+void untracked_cache_invalidate_path(struct index_state *istate,\n+\t\t\t\t     const char *path)\n+{\n+\tconst char *sep;\n+\tstruct untracked_cache_dir *d;\n+\tif (!istate->untracked || !istate->untracked->root)\n+\t\treturn;\n+\tsep = strrchr(path, '/');\n+\tif (sep)\n+\t\td = lookup_untracked(istate->untracked,\n+\t\t\t\t     istate->untracked->root,\n+\t\t\t\t     path, sep - path);\n+\telse\n+\t\td = istate->untracked->root;\n+\tistate->untracked->dir_invalidated++;\n+\td->valid = 0;\n+\td->untracked_nr = 0;\n+}\n+\n+void untracked_cache_remove_from_index(struct index_state *istate,\n+\t\t\t\t       const char *path)\n+{\n+\tuntracked_cache_invalidate_path(istate, path);\n+}\n+\n+void untracked_cache_add_to_index(struct index_state *istate,\n+\t\t\t\t  const char *path)\n+{\n+\tuntracked_cache_invalidate_path(istate, path);\n+}\ndiff --git a/dir.h b/dir.h\nindex 40a679a..2ce7dd3 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -298,6 +298,10 @@ static inline int dir_path_match(const struct dir_entry *ent,\n \t\t\t      has_trailing_dir);\n }\n \n+void untracked_cache_invalidate_path(struct index_state *, const char *);\n+void untracked_cache_remove_from_index(struct index_state *, const char *);\n+void untracked_cache_add_to_index(struct index_state *, const char *);\n+\n void free_untracked_cache(struct untracked_cache *);\n struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz);\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);\ndiff --git a/read-cache.c b/read-cache.c\nindex 3736a56..d643a3f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -78,6 +78,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tmemcpy(new->name, new_name, namelen + 1);\n \n \tcache_tree_invalidate_path(istate, old->name);\n+\tuntracked_cache_remove_from_index(istate, old->name);\n \tremove_index_entry_at(istate, nr);\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n@@ -537,6 +538,7 @@ int remove_file_from_index(struct index_state *istate, const char *path)\n \tif (pos < 0)\n \t\tpos = -pos-1;\n \tcache_tree_invalidate_path(istate, path);\n+\tuntracked_cache_remove_from_index(istate, path);\n \twhile (pos < istate->cache_nr && !strcmp(istate->cache[pos]->name, path))\n \t\tremove_index_entry_at(istate, pos);\n \treturn 0;\n@@ -968,6 +970,8 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t}\n \tpos = -pos-1;\n \n+\tuntracked_cache_add_to_index(istate, ce->name);\n+\n \t/*\n \t * Inserting a merged entry (\"stage 0\") into the index\n \t * will always replace all non-merged entries..\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 629c658..e5ddb0c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -9,6 +9,7 @@\n #include \"refs.h\"\n #include \"attr.h\"\n #include \"split-index.h\"\n+#include \"dir.h\"\n \n /*\n  * Error messages expected by scripts out of plumbing commands such as\n@@ -1255,8 +1256,10 @@ static int verify_uptodate_sparse(const struct cache_entry *ce,\n static void invalidate_ce_path(const struct cache_entry *ce,\n \t\t\t       struct unpack_trees_options *o)\n {\n-\tif (ce)\n-\t\tcache_tree_invalidate_path(o->src_index, ce->name);\n+\tif (!ce)\n+\t\treturn;\n+\tcache_tree_invalidate_path(o->src_index, ce->name);\n+\tuntracked_cache_invalidate_path(o->src_index, ce->name);\n }\n \n /*\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253323","messageId":"1418047507-22892-15-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 13/23] read-cache.c: split racy stat test to a separate function","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:57Z","receivedAt":"2014-12-08T14:04:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n read-cache.c | 24 +++++++++++++++---------\n 1 file changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d643a3f..f12a185 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -270,20 +270,26 @@ static int ce_match_stat_basic(const struct cache_entry *ce, struct stat *st)\n \treturn changed;\n }\n \n-static int is_racy_timestamp(const struct index_state *istate,\n-\t\t\t     const struct cache_entry *ce)\n+static int is_racy_stat(const struct index_state *istate,\n+\t\t\tconst struct stat_data *sd)\n {\n-\treturn (!S_ISGITLINK(ce->ce_mode) &&\n-\t\tistate->timestamp.sec &&\n+\treturn (istate->timestamp.sec &&\n #ifdef USE_NSEC\n \t\t /* nanosecond timestamped files can also be racy! */\n-\t\t(istate->timestamp.sec < ce->ce_stat_data.sd_mtime.sec ||\n-\t\t (istate->timestamp.sec == ce->ce_stat_data.sd_mtime.sec &&\n-\t\t  istate->timestamp.nsec <= ce->ce_stat_data.sd_mtime.nsec))\n+\t\t(istate->timestamp.sec < sd->sd_mtime.sec ||\n+\t\t (istate->timestamp.sec == sd->sd_mtime.sec &&\n+\t\t  istate->timestamp.nsec <= sd->sd_mtime.nsec))\n #else\n-\t\tistate->timestamp.sec <= ce->ce_stat_data.sd_mtime.sec\n+\t\tistate->timestamp.sec <= sd->sd_mtime.sec\n #endif\n-\t\t );\n+\t\t);\n+}\n+\n+static int is_racy_timestamp(const struct index_state *istate,\n+\t\t\t     const struct cache_entry *ce)\n+{\n+\treturn (!S_ISGITLINK(ce->ce_mode) &&\n+\t\tis_racy_stat(istate, &ce->ce_stat_data));\n }\n \n int ie_match_stat(const struct index_state *istate,\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253324","messageId":"1418047507-22892-16-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 14/23] untracked cache: avoid racy timestamps","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:58Z","receivedAt":"2014-12-08T14:04:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When a directory is updated within the same second that its timestamp\nis last saved, we cannot realize the directory has been updated by\nchecking timestamps. Assume the worst (something is update). See\n29e4d36 (Racy GIT - 2005-12-20) for more information.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h      | 2 ++\n dir.c        | 4 ++--\n read-cache.c | 8 ++++++++\n 3 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b14d6e2..f8b3dc5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -561,6 +561,8 @@ extern void fill_stat_data(struct stat_data *sd, struct stat *st);\n  * INODE_CHANGED, and DATA_CHANGED.\n  */\n extern int match_stat_data(const struct stat_data *sd, struct stat *st);\n+extern int match_stat_data_racy(const struct index_state *istate,\n+\t\t\t\tconst struct stat_data *sd, struct stat *st);\n \n extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n \ndiff --git a/dir.c b/dir.c\nindex a14b48f..7fa372e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -682,7 +682,7 @@ static int add_excludes(const char *fname, const char *base, int baselen,\n \t\tif (sha1_stat) {\n \t\t\tint pos;\n \t\t\tif (sha1_stat->valid &&\n-\t\t\t    !match_stat_data(&sha1_stat->stat, &st))\n+\t\t\t    !match_stat_data_racy(&the_index, &sha1_stat->stat, &st))\n \t\t\t\t; /* no content change, ss->sha1 still good */\n \t\t\telse if (check_index &&\n \t\t\t\t (pos = cache_name_pos(fname, strlen(fname))) >= 0 &&\n@@ -1538,7 +1538,7 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\treturn 0;\n \t}\n \tif (!untracked->valid ||\n-\t    match_stat_data(&untracked->stat_data, &st)) {\n+\t    match_stat_data_racy(&the_index, &untracked->stat_data, &st)) {\n \t\tif (untracked->valid)\n \t\t\tinvalidate_directory(dir->untracked, untracked);\n \t\tfill_stat_data(&untracked->stat_data, &st);\ndiff --git a/read-cache.c b/read-cache.c\nindex f12a185..0ecba05 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -292,6 +292,14 @@ static int is_racy_timestamp(const struct index_state *istate,\n \t\tis_racy_stat(istate, &ce->ce_stat_data));\n }\n \n+int match_stat_data_racy(const struct index_state *istate,\n+\t\t\t const struct stat_data *sd, struct stat *st)\n+{\n+\tif (is_racy_stat(istate, sd))\n+\t\treturn MTIME_CHANGED;\n+\treturn match_stat_data(sd, st);\n+}\n+\n int ie_match_stat(const struct index_state *istate,\n \t\t  const struct cache_entry *ce, struct stat *st,\n \t\t  unsigned int options)\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253325","messageId":"1418047507-22892-17-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 15/23] untracked cache: print stats with $GIT_TRACE_UNTRACKED_STATS","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:04:59Z","receivedAt":"2014-12-08T14:04:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This could be used to verify correct behavior in tests\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 7fa372e..14dbd7a 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1922,6 +1922,18 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \tfree_simplify(simplify);\n \tqsort(dir->entries, dir->nr, sizeof(struct dir_entry *), cmp_name);\n \tqsort(dir->ignored, dir->ignored_nr, sizeof(struct dir_entry *), cmp_name);\n+\tif (dir->untracked) {\n+\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n+\t\ttrace_printf_key(&trace_untracked_stats,\n+\t\t\t\t \"node creation: %u\\n\"\n+\t\t\t\t \"gitignore invalidation: %u\\n\"\n+\t\t\t\t \"directory invalidation: %u\\n\"\n+\t\t\t\t \"opendir: %u\\n\",\n+\t\t\t\t dir->untracked->dir_created,\n+\t\t\t\t dir->untracked->gitignore_invalidated,\n+\t\t\t\t dir->untracked->dir_invalidated,\n+\t\t\t\t dir->untracked->dir_opened);\n+\t}\n \treturn dir->nr;\n }\n \n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253326","messageId":"1418047507-22892-18-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 16/23] untracked cache: mark index dirty if untracked cache is updated","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:00Z","receivedAt":"2014-12-08T14:05:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h      | 1 +\n dir.c        | 9 +++++++++\n read-cache.c | 2 +-\n 3 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex f8b3dc5..fca979b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -295,6 +295,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n #define RESOLVE_UNDO_CHANGED\t(1 << 4)\n #define CACHE_TREE_CHANGED\t(1 << 5)\n #define SPLIT_INDEX_ORDERED\t(1 << 6)\n+#define UNTRACKED_CHANGED\t(1 << 7)\n \n struct split_index;\n struct untracked_cache;\ndiff --git a/dir.c b/dir.c\nindex 14dbd7a..9b0e1a1 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1933,6 +1933,15 @@ int read_directory(struct dir_struct *dir, const char *path, int len, const stru\n \t\t\t\t dir->untracked->gitignore_invalidated,\n \t\t\t\t dir->untracked->dir_invalidated,\n \t\t\t\t dir->untracked->dir_opened);\n+\t\tif (dir->untracked == the_index.untracked &&\n+\t\t    (dir->untracked->dir_opened ||\n+\t\t     dir->untracked->gitignore_invalidated ||\n+\t\t     dir->untracked->dir_invalidated))\n+\t\t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n+\t\tif (dir->untracked != the_index.untracked) {\n+\t\t\tfree(dir->untracked);\n+\t\t\tdir->untracked = NULL;\n+\t\t}\n \t}\n \treturn dir->nr;\n }\ndiff --git a/read-cache.c b/read-cache.c\nindex 0ecba05..71d8e20 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -42,7 +42,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce,\n /* changes that can be kept in $GIT_DIR/index (basically all extensions) */\n #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \\\n \t\t CE_ENTRY_ADDED | CE_ENTRY_REMOVED | CE_ENTRY_CHANGED | \\\n-\t\t SPLIT_INDEX_ORDERED)\n+\t\t SPLIT_INDEX_ORDERED | UNTRACKED_CHANGED)\n \n struct index_state the_index;\n static const char *alternate_index_output;\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253327","messageId":"1418047507-22892-19-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 17/23] untracked-cache: temporarily disable with $GIT_DISABLE_UNTRACKED_CACHE","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:01Z","receivedAt":"2014-12-08T14:05:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This can be used to double check if results with untracked cache are\ncorrectly, compared to vanilla version. Untracked cache remains in\nindex, but not used.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9b0e1a1..916f1ed 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1800,7 +1800,7 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \tstruct untracked_cache_dir *root;\n \tint i;\n \n-\tif (!dir->untracked)\n+\tif (!dir->untracked || getenv(\"GIT_DISABLE_UNTRACKED_CACHE\"))\n \t\treturn NULL;\n \n \t/*\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253328","messageId":"1418047507-22892-20-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 18/23] status: enable untracked cache","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:02Z","receivedAt":"2014-12-08T14:05:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"update_index_if_able() is moved down so that the updated untracked\ncache could be written out.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n builtin/commit.c | 5 +++--\n wt-status.c      | 2 ++\n 2 files changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 5ed6036..bdcfa61 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1372,13 +1372,14 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \trefresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &s.pathspec, NULL, NULL);\n \n \tfd = hold_locked_index(&index_lock, 0);\n-\tif (0 <= fd)\n-\t\tupdate_index_if_able(&the_index, &index_lock);\n \n \ts.is_initial = get_sha1(s.reference, sha1) ? 1 : 0;\n \ts.ignore_submodule_arg = ignore_submodule_arg;\n \twt_status_collect(&s);\n \n+\tif (0 <= fd)\n+\t\tupdate_index_if_able(&the_index, &index_lock);\n+\n \tif (s.relative_paths)\n \t\ts.prefix = prefix;\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 27da529..8880c3b 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -585,6 +585,8 @@ static void wt_status_collect_untracked(struct wt_status *s)\n \t\t\tDIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;\n \tif (s->show_ignored_files)\n \t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n+\telse\n+\t\tdir.untracked = the_index.untracked;\n \tsetup_standard_excludes(&dir);\n \n \tfill_directory(&dir, &s->pathspec);\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253329","messageId":"1418047507-22892-21-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 19/23] update-index: manually enable or disable untracked cache","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:03Z","receivedAt":"2014-12-08T14:05:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Overall time saving on \"git status\" is about 40% in the best case\nscenario, removing ..collect_untracked() as the most time consuming\nfunction. read and refresh index operations are now at the top (which\nshould drop when index-helper and/or watchman support is added). More\nnumbers and analysis below.\n\nwebkit.git\n==========\n\n169k files. 6k dirs. Lots of test data (i.e. not touched most of the\ntime)\n\nBase status\n-----------\n\nIndex version 4 in split index mode and cache-tree populated. No\nuntracked cache. It shows how time is consumed by \"git status\". The\nsame settings are used for other repos below.\n\n18:28:10.199679 builtin/commit.c:1394   performance: 0.000000451 s: cmd_status:setup\n18:28:10.474847 read-cache.c:1407       performance: 0.274873831 s: read_index\n18:28:10.475295 read-cache.c:1407       performance: 0.000000656 s: read_index\n18:28:10.728443 preload-index.c:131     performance: 0.253147487 s: read_index_preload\n18:28:10.741422 read-cache.c:1254       performance: 0.012868340 s: refresh_index\n18:28:10.752300 wt-status.c:623         performance: 0.010421357 s: wt_status_collect_changes_worktree\n18:28:10.762069 wt-status.c:629         performance: 0.009644748 s: wt_status_collect_changes_index\n18:28:11.601019 wt-status.c:632         performance: 0.838859547 s: wt_status_collect_untracked\n18:28:11.605939 builtin/commit.c:1421   performance: 0.004835004 s: cmd_status:update_index\n18:28:11.606580 trace.c:415             performance: 1.407878388 s: git command: 'git' 'status'\n\nPopulating status\n-----------------\n\nThis is after enabling untracked cache and the cache is still empty.\nWe see a slight increase in .._collect_untracked() and update_index\n(because new cache has to be written to $GIT_DIR/index).\n\n18:28:18.915213 builtin/commit.c:1394   performance: 0.000000326 s: cmd_status:setup\n18:28:19.197364 read-cache.c:1407       performance: 0.281901416 s: read_index\n18:28:19.197754 read-cache.c:1407       performance: 0.000000546 s: read_index\n18:28:19.451355 preload-index.c:131     performance: 0.253599607 s: read_index_preload\n18:28:19.464400 read-cache.c:1254       performance: 0.012935336 s: refresh_index\n18:28:19.475115 wt-status.c:623         performance: 0.010236920 s: wt_status_collect_changes_worktree\n18:28:19.486022 wt-status.c:629         performance: 0.010801685 s: wt_status_collect_changes_index\n18:28:20.362660 wt-status.c:632         performance: 0.876551366 s: wt_status_collect_untracked\n18:28:20.396199 builtin/commit.c:1421   performance: 0.033447969 s: cmd_status:update_index\n18:28:20.396939 trace.c:415             performance: 1.482695902 s: git command: 'git' 'status'\n\nPopulated status\n----------------\n\nAfter the cache is populated, wt_status_collect_untracked() drops 82%\nfrom 0.838s to 0.144s. Overall time drops 45%. Top offenders are now\nread_index() and read_index_preload().\n\n18:28:20.408605 builtin/commit.c:1394   performance: 0.000000457 s: cmd_status:setup\n18:28:20.692864 read-cache.c:1407       performance: 0.283980458 s: read_index\n18:28:20.693273 read-cache.c:1407       performance: 0.000000661 s: read_index\n18:28:20.958814 preload-index.c:131     performance: 0.265540254 s: read_index_preload\n18:28:20.972375 read-cache.c:1254       performance: 0.013437429 s: refresh_index\n18:28:20.983959 wt-status.c:623         performance: 0.011146646 s: wt_status_collect_changes_worktree\n18:28:20.993948 wt-status.c:629         performance: 0.009879094 s: wt_status_collect_changes_index\n18:28:21.138125 wt-status.c:632         performance: 0.144084737 s: wt_status_collect_untracked\n18:28:21.173678 builtin/commit.c:1421   performance: 0.035463949 s: cmd_status:update_index\n18:28:21.174251 trace.c:415             performance: 0.766707355 s: git command: 'git' 'status'\n\ngentoo-x86.git\n==============\n\nThis repository is a strange one with a balanced, wide and shallow\nworktree (about 100k files and 23k dirs) and no .gitignore in\nworktree. .._collect_untracked() time drops 88%, total time drops 56%.\n\nBase status\n-----------\n18:20:40.828642 builtin/commit.c:1394   performance: 0.000000496 s: cmd_status:setup\n18:20:41.027233 read-cache.c:1407       performance: 0.198130532 s: read_index\n18:20:41.027670 read-cache.c:1407       performance: 0.000000581 s: read_index\n18:20:41.171716 preload-index.c:131     performance: 0.144045594 s: read_index_preload\n18:20:41.179171 read-cache.c:1254       performance: 0.007320424 s: refresh_index\n18:20:41.185785 wt-status.c:623         performance: 0.006144638 s: wt_status_collect_changes_worktree\n18:20:41.192701 wt-status.c:629         performance: 0.006780184 s: wt_status_collect_changes_index\n18:20:41.991723 wt-status.c:632         performance: 0.798927029 s: wt_status_collect_untracked\n18:20:41.994664 builtin/commit.c:1421   performance: 0.002852772 s: cmd_status:update_index\n18:20:41.995458 trace.c:415             performance: 1.168427502 s: git command: 'git' 'status'\nPopulating status\n-----------------\n18:20:48.968848 builtin/commit.c:1394   performance: 0.000000380 s: cmd_status:setup\n18:20:49.172918 read-cache.c:1407       performance: 0.203734214 s: read_index\n18:20:49.173341 read-cache.c:1407       performance: 0.000000562 s: read_index\n18:20:49.320013 preload-index.c:131     performance: 0.146671391 s: read_index_preload\n18:20:49.328039 read-cache.c:1254       performance: 0.007921957 s: refresh_index\n18:20:49.334680 wt-status.c:623         performance: 0.006172020 s: wt_status_collect_changes_worktree\n18:20:49.342526 wt-status.c:629         performance: 0.007731746 s: wt_status_collect_changes_index\n18:20:50.257510 wt-status.c:632         performance: 0.914864222 s: wt_status_collect_untracked\n18:20:50.338371 builtin/commit.c:1421   performance: 0.080776477 s: cmd_status:update_index\n18:20:50.338900 trace.c:415             performance: 1.371462446 s: git command: 'git' 'status'\nPopulated status\n----------------\n18:20:50.351160 builtin/commit.c:1394   performance: 0.000000571 s: cmd_status:setup\n18:20:50.577358 read-cache.c:1407       performance: 0.225917338 s: read_index\n18:20:50.577794 read-cache.c:1407       performance: 0.000000617 s: read_index\n18:20:50.734140 preload-index.c:131     performance: 0.156345564 s: read_index_preload\n18:20:50.745717 read-cache.c:1254       performance: 0.011463075 s: refresh_index\n18:20:50.755176 wt-status.c:623         performance: 0.008877929 s: wt_status_collect_changes_worktree\n18:20:50.763768 wt-status.c:629         performance: 0.008471633 s: wt_status_collect_changes_index\n18:20:50.854885 wt-status.c:632         performance: 0.090988721 s: wt_status_collect_untracked\n18:20:50.857765 builtin/commit.c:1421   performance: 0.002789097 s: cmd_status:update_index\n18:20:50.858411 trace.c:415             performance: 0.508647673 s: git command: 'git' 'status'\n\nlinux-2.6\n=========\n\nReference repo. Not too big. .._collect_status() drops 84%. Total time\ndrops 42%.\n\nBase status\n-----------\n18:34:09.870122 builtin/commit.c:1394   performance: 0.000000385 s: cmd_status:setup\n18:34:09.943218 read-cache.c:1407       performance: 0.072871177 s: read_index\n18:34:09.943614 read-cache.c:1407       performance: 0.000000491 s: read_index\n18:34:10.004364 preload-index.c:131     performance: 0.060748102 s: read_index_preload\n18:34:10.008190 read-cache.c:1254       performance: 0.003714285 s: refresh_index\n18:34:10.012087 wt-status.c:623         performance: 0.002775446 s: wt_status_collect_changes_worktree\n18:34:10.016054 wt-status.c:629         performance: 0.003862140 s: wt_status_collect_changes_index\n18:34:10.214747 wt-status.c:632         performance: 0.198604837 s: wt_status_collect_untracked\n18:34:10.216102 builtin/commit.c:1421   performance: 0.001244166 s: cmd_status:update_index\n18:34:10.216817 trace.c:415             performance: 0.347670735 s: git command: 'git' 'status'\nPopulating status\n-----------------\n18:34:16.595102 builtin/commit.c:1394   performance: 0.000000456 s: cmd_status:setup\n18:34:16.666600 read-cache.c:1407       performance: 0.070992413 s: read_index\n18:34:16.667012 read-cache.c:1407       performance: 0.000000606 s: read_index\n18:34:16.729375 preload-index.c:131     performance: 0.062362492 s: read_index_preload\n18:34:16.732565 read-cache.c:1254       performance: 0.003075517 s: refresh_index\n18:34:16.736148 wt-status.c:623         performance: 0.002422201 s: wt_status_collect_changes_worktree\n18:34:16.739990 wt-status.c:629         performance: 0.003746618 s: wt_status_collect_changes_index\n18:34:16.948505 wt-status.c:632         performance: 0.208426710 s: wt_status_collect_untracked\n18:34:16.961744 builtin/commit.c:1421   performance: 0.013151887 s: cmd_status:update_index\n18:34:16.962233 trace.c:415             performance: 0.368537535 s: git command: 'git' 'status'\nPopulated status\n----------------\n18:34:16.970026 builtin/commit.c:1394   performance: 0.000000631 s: cmd_status:setup\n18:34:17.046235 read-cache.c:1407       performance: 0.075904673 s: read_index\n18:34:17.046644 read-cache.c:1407       performance: 0.000000681 s: read_index\n18:34:17.113564 preload-index.c:131     performance: 0.066920253 s: read_index_preload\n18:34:17.117281 read-cache.c:1254       performance: 0.003604055 s: refresh_index\n18:34:17.121115 wt-status.c:623         performance: 0.002508345 s: wt_status_collect_changes_worktree\n18:34:17.125089 wt-status.c:629         performance: 0.003871636 s: wt_status_collect_changes_index\n18:34:17.156089 wt-status.c:632         performance: 0.030895703 s: wt_status_collect_untracked\n18:34:17.169861 builtin/commit.c:1421   performance: 0.013686404 s: cmd_status:update_index\n18:34:17.170391 trace.c:415             performance: 0.201474531 s: git command: 'git' 'status'\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-update-index.txt |  8 ++++++++\n builtin/update-index.c             | 16 ++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex dfc09d9..f9a35cd 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -172,6 +172,14 @@ may not support it yet.\n \tthe shared index file. This mode is designed for very large\n \tindexes that take a signficant amount of time to read or write.\n \n+--untracked-cache::\n+--no-untracked-cache::\n+\tEnable or disable untracked cache extension. This could speed\n+\tup for commands that involve determining untracked files such\n+\tas `git status`. The underlying operating system and file\n+\tsystem must change `st_mtime` field of a directory if files\n+\tare added or deleted in that directory.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e8c7fd4..3d2dedd 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -740,6 +740,7 @@ static int reupdate_callback(struct parse_opt_ctx_t *ctx,\n int cmd_update_index(int argc, const char **argv, const char *prefix)\n {\n \tint newfd, entries, has_errors = 0, line_termination = '\\n';\n+\tint untracked_cache = -1;\n \tint read_from_stdin = 0;\n \tint prefix_length = prefix ? strlen(prefix) : 0;\n \tint preferred_index_format = 0;\n@@ -831,6 +832,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"write index in this format\")),\n \t\tOPT_BOOL(0, \"split-index\", &split_index,\n \t\t\tN_(\"enable or disable split index\")),\n+\t\tOPT_BOOL(0, \"untracked-cache\", &untracked_cache,\n+\t\t\tN_(\"enable/disable untracked cache\")),\n \t\tOPT_END()\n \t};\n \n@@ -937,6 +940,19 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tthe_index.split_index = NULL;\n \t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n \t}\n+\tif (untracked_cache > 0 && !the_index.untracked) {\n+\t\tstruct untracked_cache *uc;\n+\n+\t\tuc = xcalloc(1, sizeof(*uc));\n+\t\tuc->exclude_per_dir = \".gitignore\";\n+\t\t/* should be the same flags used by git-status */\n+\t\tuc->dir_flags = DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;\n+\t\tthe_index.untracked = uc;\n+\t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n+\t} else if (!untracked_cache && the_index.untracked) {\n+\t\tthe_index.untracked = NULL;\n+\t\tthe_index.cache_changed |= UNTRACKED_CHANGED;\n+\t}\n \n \tif (active_cache_changed) {\n \t\tif (newfd < 0) {\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253330","messageId":"1418047507-22892-22-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 20/23] update-index: test the system before enabling untracked cache","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:04Z","receivedAt":"2014-12-08T14:05:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Helped-by: Eric Sunshine <sunshine@sunshineco.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-update-index.txt |   6 ++\n builtin/update-index.c             | 148 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 154 insertions(+)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex f9a35cd..ed32bae 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -180,6 +180,12 @@ may not support it yet.\n \tsystem must change `st_mtime` field of a directory if files\n \tare added or deleted in that directory.\n \n+--force-untracked-cache::\n+\tFor safety, `--untracked-cache` performs tests on the working\n+\tdirectory to make sure untracked cache can be used. These\n+\ttests can take a few seconds. `--force-untracked-cache` can be\n+\tused to skip the tests.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 3d2dedd..f23ec83 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -47,6 +47,147 @@ static void report(const char *fmt, ...)\n \tva_end(vp);\n }\n \n+static void remove_test_directory(void)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstrbuf_addstr(&sb, \"dir-mtime-test\");\n+\tremove_dir_recursively(&sb, 0);\n+\tstrbuf_release(&sb);\n+}\n+\n+static void xmkdir(const char *path)\n+{\n+\tif (mkdir(path, 0700))\n+\t\tdie_errno(_(\"failed to create directory %s\"), path);\n+}\n+\n+static int xstat(const char *path, struct stat *st)\n+{\n+\tif (stat(path, st))\n+\t\tdie_errno(_(\"failed to stat %s\"), path);\n+\treturn 0;\n+}\n+\n+static int create_file(const char *path)\n+{\n+\tint fd = open(path, O_CREAT | O_RDWR, 0644);\n+\tif (fd < 0)\n+\t\tdie_errno(_(\"failed to create file %s\"), path);\n+\treturn fd;\n+}\n+\n+static void xunlink(const char *path)\n+{\n+\tif (unlink(path))\n+\t\tdie_errno(_(\"failed to delete file %s\"), path);\n+}\n+\n+static void xrmdir(const char *path)\n+{\n+\tif (rmdir(path))\n+\t\tdie_errno(_(\"failed to delete directory %s\"), path);\n+}\n+\n+static void avoid_racy(void)\n+{\n+\t/*\n+\t * not use if we could usleep(10) if USE_NSEC is defined. The\n+\t * field nsec could be there, but the OS could choose to\n+\t * ignore it?\n+\t */\n+\tsleep(1);\n+}\n+\n+static int test_if_untracked_cache_is_supported(void)\n+{\n+\tstruct stat st;\n+\tstruct stat_data base;\n+\tint fd;\n+\n+\tfprintf(stderr, _(\"Testing \"));\n+\txmkdir(\"dir-mtime-test\");\n+\tatexit(remove_test_directory);\n+\txstat(\"dir-mtime-test\", &st);\n+\tfill_stat_data(&base, &st);\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\tfd = create_file(\"dir-mtime-test/newfile\");\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (!match_stat_data(&base, &st)) {\n+\t\tclose(fd);\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr,_(\"directory stat info does not \"\n+\t\t\t\t    \"change after adding a new file\"));\n+\t\treturn 0;\n+\t}\n+\tfill_stat_data(&base, &st);\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\txmkdir(\"dir-mtime-test/new-dir\");\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (!match_stat_data(&base, &st)) {\n+\t\tclose(fd);\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr, _(\"directory stat info does not change \"\n+\t\t\t\t     \"after adding a new directory\"));\n+\t\treturn 0;\n+\t}\n+\tfill_stat_data(&base, &st);\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\twrite_or_die(fd, \"data\", 4);\n+\tclose(fd);\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (match_stat_data(&base, &st)) {\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr, _(\"directory stat info changes \"\n+\t\t\t\t     \"after updating a file\"));\n+\t\treturn 0;\n+\t}\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\tclose(create_file(\"dir-mtime-test/new-dir/new\"));\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (match_stat_data(&base, &st)) {\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr, _(\"directory stat info changes after \"\n+\t\t\t\t     \"adding a file inside subdirectory\"));\n+\t\treturn 0;\n+\t}\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\txunlink(\"dir-mtime-test/newfile\");\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (!match_stat_data(&base, &st)) {\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr, _(\"directory stat info does not \"\n+\t\t\t\t     \"change after deleting a file\"));\n+\t\treturn 0;\n+\t}\n+\tfill_stat_data(&base, &st);\n+\tfputc('.', stderr);\n+\n+\tavoid_racy();\n+\txunlink(\"dir-mtime-test/new-dir/new\");\n+\txrmdir(\"dir-mtime-test/new-dir\");\n+\txstat(\"dir-mtime-test\", &st);\n+\tif (!match_stat_data(&base, &st)) {\n+\t\tfputc('\\n', stderr);\n+\t\tfprintf_ln(stderr, _(\"directory stat info does not \"\n+\t\t\t\t     \"change after deleting a directory\"));\n+\t\treturn 0;\n+\t}\n+\n+\txrmdir(\"dir-mtime-test\");\n+\tfprintf_ln(stderr, _(\" OK\"));\n+\treturn 1;\n+}\n+\n static int mark_ce_flags(const char *path, int flag, int mark)\n {\n \tint namelen = strlen(path);\n@@ -834,6 +975,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"enable or disable split index\")),\n \t\tOPT_BOOL(0, \"untracked-cache\", &untracked_cache,\n \t\t\tN_(\"enable/disable untracked cache\")),\n+\t\tOPT_SET_INT(0, \"force-untracked-cache\", &untracked_cache,\n+\t\t\t    N_(\"enable untracked cache without testing the filesystem\"), 2),\n \t\tOPT_END()\n \t};\n \n@@ -943,6 +1086,11 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tif (untracked_cache > 0 && !the_index.untracked) {\n \t\tstruct untracked_cache *uc;\n \n+\t\tif (untracked_cache < 2) {\n+\t\t\tsetup_work_tree();\n+\t\t\tif (!test_if_untracked_cache_is_supported())\n+\t\t\t\treturn 1;\n+\t\t}\n \t\tuc = xcalloc(1, sizeof(*uc));\n \t\tuc->exclude_per_dir = \".gitignore\";\n \t\t/* should be the same flags used by git-status */\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253331","messageId":"1418047507-22892-23-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 21/23] t7063: tests for untracked cache","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:05Z","receivedAt":"2014-12-08T14:05:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n .gitignore                                 |   1 +\n Makefile                                   |   1 +\n t/t7063-status-untracked-cache.sh (new +x) | 353 +++++++++++++++++++++++++++++\n test-dump-untracked-cache.c (new)          |  61 +++++\n 4 files changed, 416 insertions(+)\n create mode 100755 t/t7063-status-untracked-cache.sh\n create mode 100644 test-dump-untracked-cache.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 81e12c0..e2bb375 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -182,6 +182,7 @@\n /test-delta\n /test-dump-cache-tree\n /test-dump-split-index\n+/test-dump-untracked-cache\n /test-scrap-cache-tree\n /test-genrandom\n /test-hashmap\ndiff --git a/Makefile b/Makefile\nindex 9f984a9..fa58a53 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -555,6 +555,7 @@ TEST_PROGRAMS_NEED_X += test-date\n TEST_PROGRAMS_NEED_X += test-delta\n TEST_PROGRAMS_NEED_X += test-dump-cache-tree\n TEST_PROGRAMS_NEED_X += test-dump-split-index\n+TEST_PROGRAMS_NEED_X += test-dump-untracked-cache\n TEST_PROGRAMS_NEED_X += test-genrandom\n TEST_PROGRAMS_NEED_X += test-hashmap\n TEST_PROGRAMS_NEED_X += test-index-version\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nnew file mode 100755\nindex 0000000..2b2ffd7\n--- /dev/null\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -0,0 +1,353 @@\n+#!/bin/sh\n+\n+test_description='test untracked cache'\n+\n+. ./test-lib.sh\n+\n+avoid_racy() {\n+\tsleep 1\n+}\n+\n+git update-index --untracked-cache\n+# It's fine if git update-index returns an error code other than one,\n+# it'll be caught in the first test.\n+if test $? -eq 1; then\n+\tskip_all='This system does not support untracked cache'\n+\ttest_done\n+fi\n+\n+test_expect_success 'setup' '\n+\tgit init worktree &&\n+\tcd worktree &&\n+\tmkdir done dtwo dthree &&\n+\ttouch one two three done/one dtwo/two dthree/three &&\n+\tgit add one two done/one &&\n+\t: >.git/info/exclude &&\n+\tgit update-index --untracked-cache\n+'\n+\n+test_expect_success 'untracked cache is empty' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 0000000000000000000000000000000000000000\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+cat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+A  two\n+?? dthree/\n+?? dtwo/\n+?? three\n+EOF\n+\n+cat >../dump.expect <<EOF &&\n+info/exclude e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ 0000000000000000000000000000000000000000 recurse valid\n+dthree/\n+dtwo/\n+three\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+three\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\n+test_expect_success 'status first time (empty cache)' '\n+\tavoid_racy &&\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 3\n+gitignore invalidation: 1\n+directory invalidation: 0\n+opendir: 4\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+'\n+\n+test_expect_success 'untracked cache after first status' '\n+\ttest-dump-untracked-cache >../actual &&\n+\ttest_cmp ../dump.expect ../actual\n+'\n+\n+test_expect_success 'status second time (fully populated cache)' '\n+\tavoid_racy &&\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 0\n+directory invalidation: 0\n+opendir: 0\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+'\n+\n+test_expect_success 'untracked cache after second status' '\n+\ttest-dump-untracked-cache >../actual &&\n+\ttest_cmp ../dump.expect ../actual\n+'\n+\n+test_expect_success 'modify in root directory, one dir invalidation' '\n+\tavoid_racy &&\n+\t: >four &&\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\tcat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+A  two\n+?? dthree/\n+?? dtwo/\n+?? four\n+?? three\n+EOF\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 0\n+directory invalidation: 1\n+opendir: 1\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+\n+'\n+\n+test_expect_success 'verify untracked cache dump' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ 0000000000000000000000000000000000000000 recurse valid\n+dthree/\n+dtwo/\n+four\n+three\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+three\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'new .gitignore invalidates recursively' '\n+\tavoid_racy &&\n+\techo four >.gitignore &&\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\tcat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+A  two\n+?? .gitignore\n+?? dthree/\n+?? dtwo/\n+?? three\n+EOF\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 1\n+directory invalidation: 1\n+opendir: 4\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+\n+'\n+\n+test_expect_success 'verify untracked cache dump' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n+.gitignore\n+dthree/\n+dtwo/\n+three\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+three\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'new info/exclude invalidates everything' '\n+\tavoid_racy &&\n+\techo three >>.git/info/exclude &&\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\tcat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+A  two\n+?? .gitignore\n+?? dtwo/\n+EOF\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 1\n+directory invalidation: 0\n+opendir: 4\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+'\n+\n+test_expect_success 'verify untracked cache dump' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n+.gitignore\n+dtwo/\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'move two from tracked to untracked' '\n+\tgit rm --cached two &&\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'status after the move' '\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\tcat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+?? .gitignore\n+?? dtwo/\n+?? two\n+EOF\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 0\n+directory invalidation: 0\n+opendir: 1\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+'\n+\n+test_expect_success 'verify untracked cache dump' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n+.gitignore\n+dtwo/\n+two\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'move two from untracked to tracked' '\n+\tgit add two &&\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_expect_success 'status after the move' '\n+\t: >../trace &&\n+\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tgit status --porcelain >../actual &&\n+\tcat >../status.expect <<EOF &&\n+A  done/one\n+A  one\n+A  two\n+?? .gitignore\n+?? dtwo/\n+EOF\n+\ttest_cmp ../status.expect ../actual &&\n+\tcat >../trace.expect <<EOF &&\n+node creation: 0\n+gitignore invalidation: 0\n+directory invalidation: 0\n+opendir: 1\n+EOF\n+\ttest_cmp ../trace.expect ../trace\n+'\n+\n+test_expect_success 'verify untracked cache dump' '\n+\ttest-dump-untracked-cache >../actual &&\n+\tcat >../expect <<EOF &&\n+info/exclude 13263c0978fb9fad16b2d580fb800b6d811c3ff0\n+core.excludesfile 0000000000000000000000000000000000000000\n+exclude_per_dir .gitignore\n+flags 00000006\n+/ e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n+.gitignore\n+dtwo/\n+/done/ 0000000000000000000000000000000000000000 recurse valid\n+/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n+/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n+two\n+EOF\n+\ttest_cmp ../expect ../actual\n+'\n+\n+test_done\ndiff --git a/test-dump-untracked-cache.c b/test-dump-untracked-cache.c\nnew file mode 100644\nindex 0000000..710441e\n--- /dev/null\n+++ b/test-dump-untracked-cache.c\n@@ -0,0 +1,61 @@\n+#include \"cache.h\"\n+#include \"dir.h\"\n+\n+static int compare_untracked(const void *a_, const void *b_)\n+{\n+\tconst char *const *a = a_;\n+\tconst char *const *b = b_;\n+\treturn strcmp(*a, *b);\n+}\n+\n+static int compare_dir(const void *a_, const void *b_)\n+{\n+\tconst struct untracked_cache_dir *const *a = a_;\n+\tconst struct untracked_cache_dir *const *b = b_;\n+\treturn strcmp((*a)->name, (*b)->name);\n+}\n+\n+static void dump(struct untracked_cache_dir *ucd, struct strbuf *base)\n+{\n+\tint i, len;\n+\tqsort(ucd->untracked, ucd->untracked_nr, sizeof(*ucd->untracked),\n+\t      compare_untracked);\n+\tqsort(ucd->dirs, ucd->dirs_nr, sizeof(*ucd->dirs),\n+\t      compare_dir);\n+\tlen = base->len;\n+\tstrbuf_addf(base, \"%s/\", ucd->name);\n+\tprintf(\"%s %s\", base->buf,\n+\t       sha1_to_hex(ucd->exclude_sha1));\n+\tif (ucd->recurse)\n+\t\tfputs(\" recurse\", stdout);\n+\tif (ucd->check_only)\n+\t\tfputs(\" check_only\", stdout);\n+\tif (ucd->valid)\n+\t\tfputs(\" valid\", stdout);\n+\tprintf(\"\\n\");\n+\tfor (i = 0; i < ucd->untracked_nr; i++)\n+\t\tprintf(\"%s\\n\", ucd->untracked[i]);\n+\tfor (i = 0; i < ucd->dirs_nr; i++)\n+\t\tdump(ucd->dirs[i], base);\n+\tstrbuf_setlen(base, len);\n+}\n+\n+int main(int ac, char **av)\n+{\n+\tstruct untracked_cache *uc;\n+\tstruct strbuf base = STRBUF_INIT;\n+\tif (read_cache() < 0)\n+\t\tdie(\"unable to read index file\");\n+\tuc = the_index.untracked;\n+\tif (!uc) {\n+\t\tprintf(\"no untracked cache\\n\");\n+\t\treturn 0;\n+\t}\n+\tprintf(\"info/exclude %s\\n\", sha1_to_hex(uc->ss_info_exclude.sha1));\n+\tprintf(\"core.excludesfile %s\\n\", sha1_to_hex(uc->ss_excludes_file.sha1));\n+\tprintf(\"exclude_per_dir %s\\n\", uc->exclude_per_dir);\n+\tprintf(\"flags %08x\\n\", uc->dir_flags);\n+\tif (uc->root)\n+\t\tdump(uc->root, &base);\n+\treturn 0;\n+}\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253332","messageId":"1418047507-22892-24-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 22/23] mingw32: add uname()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:06Z","receivedAt":"2014-12-08T14:05:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Helped-by: Eric Sunshine <sunshine@sunshineco.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n compat/mingw.c | 11 +++++++++++\n compat/mingw.h |  9 +++++++++\n 2 files changed, 20 insertions(+)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex c5c37e5..88140e4 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2128,3 +2128,14 @@ void mingw_startup()\n \t/* initialize Unicode console */\n \twinansi_init();\n }\n+\n+int uname(struct utsname *buf)\n+{\n+\tDWORD v = GetVersion();\n+\tmemset(buf, 0, sizeof(*buf));\n+\tstrcpy(buf->sysname, \"Windows\");\n+\tsprintf(buf->release, \"%u.%u\", v & 0xff, (v >> 8) & 0xff);\n+\t/* assuming NT variants only.. */\n+\tsprintf(buf->version, \"%u\", (v >> 16) & 0x7fff);\n+\treturn 0;\n+}\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex df0e320..d00ba7a 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -77,6 +77,14 @@ struct itimerval {\n };\n #define ITIMER_REAL 0\n \n+struct utsname {\n+\tchar sysname[16];\n+\tchar nodename[1];\n+\tchar release[16];\n+\tchar version[16];\n+\tchar machine[1];\n+};\n+\n /*\n  * sanitize preprocessor namespace polluted by Windows headers defining\n  * macros which collide with git local versions\n@@ -166,6 +174,7 @@ struct passwd *getpwuid(uid_t uid);\n int setitimer(int type, struct itimerval *in, struct itimerval *out);\n int sigaction(int sig, struct sigaction *in, struct sigaction *out);\n int link(const char *oldpath, const char *newpath);\n+int uname(struct utsname *buf);\n \n /*\n  * replacements of existing functions\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253333","messageId":"1418047507-22892-25-git-send-email-pclouds@gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-08T14:05:07Z","receivedAt":"2014-12-08T14:05:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If the user enables untracked cache, then\n\n - move worktree to an unsupported filesystem\n - or simply upgrade OS\n - or move the whole (portable) disk from one machine to another\n - or access a shared fs from another machine\n\nthere's no guarantee that untracked cache can still function properly.\nRecord the worktree location and OS footprint in the cache. If it\nchanges, err on the safe side and disable the cache. The user can\n'update-index --untracked-cache' again to make sure all conditions are\nmet.\n\nThis adds a new requirement that setup_git_directory* must be called\nbefore read_cache() because we need worktree location by then, or the\ncache is dropped.\n\nThis change does not cover all bases, you can fool it if you try\nhard. The point is to stop accidents.\n\nHelped-by: Eric Sunshine <sunshine@sunshineco.com>\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/technical/index-format.txt |  3 +++\n dir.c                                    | 28 ++++++++++++++++++++++++++++\n git-compat-util.h                        |  1 +\n test-dump-untracked-cache.c              |  1 +\n 4 files changed, 33 insertions(+)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex b97ac8d..5dc2bee 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -242,6 +242,9 @@ Git index format\n \n   The extension starts with\n \n+  - A NUL-terminated string describing the environment when the cache\n+    is created.\n+\n   - Stat data of $GIT_DIR/info/exclude. See \"Index entry\" section from\n     ctime field until \"file size\".\n \ndiff --git a/dir.c b/dir.c\nindex 916f1ed..ef58547 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2246,10 +2246,21 @@ static void write_one_dir(struct untracked_cache_dir *untracked,\n \t\t\twrite_one_dir(untracked->dirs[i], wd);\n }\n \n+static void get_ident_string(struct strbuf *sb)\n+{\n+\tstruct utsname uts;\n+\n+\tif (uname(&uts))\n+\t\tdie_errno(_(\"failed to get kernel name and information\"));\n+\tstrbuf_addf(sb, \"Location %s, system %s %s %s\", get_git_work_tree(),\n+\t\t    uts.sysname, uts.release, uts.version);\n+}\n+\n void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked)\n {\n \tstruct ondisk_untracked_cache *ouc;\n \tstruct write_data wd;\n+\tstruct strbuf sb = STRBUF_INIT;\n \tunsigned char varbuf[16];\n \tint len = 0, varint_len;\n \tif (untracked->exclude_per_dir)\n@@ -2261,6 +2272,11 @@ void write_untracked_extension(struct strbuf *out, struct untracked_cache *untra\n \thashcpy(ouc->excludes_file_sha1, untracked->ss_excludes_file.sha1);\n \touc->dir_flags = htonl(untracked->dir_flags);\n \tmemcpy(ouc->exclude_per_dir, untracked->exclude_per_dir, len + 1);\n+\n+\tget_ident_string(&sb);\n+\tstrbuf_add(out, sb.buf, sb.len + 1);\n+\tstrbuf_release(&sb);\n+\n \tstrbuf_add(out, ouc, sizeof(*ouc) + len);\n \tif (!untracked->root) {\n \t\tvarint_len = encode_varint(0, varbuf);\n@@ -2444,12 +2460,24 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long\n \tstruct untracked_cache *uc;\n \tstruct read_data rd;\n \tconst unsigned char *next = data, *end = data + sz;\n+\tstruct strbuf sb = STRBUF_INIT;\n \tint len;\n \n \tif (sz <= 1 || end[-1] != '\\0')\n \t\treturn NULL;\n \tend--;\n \n+\tget_ident_string(&sb);\n+\tif (strcmp(sb.buf, (const char *)next)) {\n+\t\twarning(_(\"system identification does not match, untracked cache disabled.\\n\"\n+\t\t\t  \"Stored: %s\\nCurrent: %s\\n\"),\n+\t\t\tnext, sb.buf);\n+\t\tstrbuf_release(&sb);\n+\t\treturn NULL;\n+\t}\n+\tnext += sb.len + 1;\n+\tstrbuf_release(&sb);\n+\n \touc = (const struct ondisk_untracked_cache *)next;\n \tif (next + sizeof(*ouc) > end)\n \t\treturn NULL;\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f587749..e9502a1 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -132,6 +132,7 @@\n #elif defined(_MSC_VER)\n #include \"compat/msvc.h\"\n #else\n+#include <sys/utsname.h>\n #include <sys/wait.h>\n #include <sys/resource.h>\n #include <sys/socket.h>\ndiff --git a/test-dump-untracked-cache.c b/test-dump-untracked-cache.c\nindex 710441e..25d855d 100644\n--- a/test-dump-untracked-cache.c\n+++ b/test-dump-untracked-cache.c\n@@ -44,6 +44,7 @@ int main(int ac, char **av)\n {\n \tstruct untracked_cache *uc;\n \tstruct strbuf base = STRBUF_INIT;\n+\tsetup_git_directory();\n \tif (read_cache() < 0)\n \t\tdie(\"unable to read index file\");\n \tuc = the_index.untracked;\n-- \n2.2.0.60.gb7b3c64\n"},{"id":"253408","messageId":"20141209100430.GC76457@vauxhall.crustytoothpaste.net","threadId":"38142","inReplyTo":"1418047507-22892-25-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-09T10:04:30Z","receivedAt":"2014-12-09T10:04:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Dec 08, 2014 at 09:05:07PM +0700, Nguyễn Thái Ngọc Duy wrote:\n> If the user enables untracked cache, then\n> \n>  - move worktree to an unsupported filesystem\n>  - or simply upgrade OS\n>  - or move the whole (portable) disk from one machine to another\n>  - or access a shared fs from another machine\n> \n> there's no guarantee that untracked cache can still function properly.\n> Record the worktree location and OS footprint in the cache. If it\n> changes, err on the safe side and disable the cache. The user can\n> 'update-index --untracked-cache' again to make sure all conditions are\n> met.\n\nMy use case for untracked cache is where I have one machine with a\nrepository and which is also mounted via sshfs on another machine.  It\nlooks like this will disable the cache every time I change the machine I\naccess it on.  Would you be willing to accept a patch for a configuration\noption to disable this?\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187\n"},{"id":"253461","messageId":"CACsJy8A5-smRXN0k5TK8uPg4-j-Z83KWLNSO_w-eFivTNB_www@mail.gmail.com","threadId":"38142","inReplyTo":"20141209100430.GC76457@vauxhall.crustytoothpaste.net","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-09T22:53:09Z","receivedAt":"2014-12-09T22:53:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Dec 9, 2014 at 5:04 PM, brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n> On Mon, Dec 08, 2014 at 09:05:07PM +0700, Nguyễn Thái Ngọc Duy wrote:\n>> If the user enables untracked cache, then\n>>\n>>  - move worktree to an unsupported filesystem\n>>  - or simply upgrade OS\n>>  - or move the whole (portable) disk from one machine to another\n>>  - or access a shared fs from another machine\n>>\n>> there's no guarantee that untracked cache can still function properly.\n>> Record the worktree location and OS footprint in the cache. If it\n>> changes, err on the safe side and disable the cache. The user can\n>> 'update-index --untracked-cache' again to make sure all conditions are\n>> met.\n>\n> My use case for untracked cache is where I have one machine with a\n> repository and which is also mounted via sshfs on another machine.  It\n> looks like this will disable the cache every time I change the machine I\n> access it on.  Would you be willing to accept a patch for a configuration\n> option to disable this?\n\nTorsten also does not like this patch. Maybe I'm just too paranoid.\nMaybe we can drop this patch.\n-- \nDuy\n"},{"id":"253466","messageId":"CAPig+cTB6=aFR8HFOtdbGi1h9EXcOdfNB=N=THz-DgzytQ2ceA@mail.gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-12-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 10/23] untracked cache: save to an index extension","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-12-09T23:28:30Z","receivedAt":"2014-12-09T23:28:30Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Dec 8, 2014 at 9:04 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n> diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\n> index fe6f316..b97ac8d 100644\n> --- a/Documentation/technical/index-format.txt\n> +++ b/Documentation/technical/index-format.txt\n> @@ -233,3 +233,61 @@ Git index format\n>    The remaining index entries after replaced ones will be added to the\n>    final index. These added entries are also sorted by entry namme then\n>    stage.\n> +\n> +== Untracked cache\n> +\n> +  Untracked cache saves the untracked file list and necessary data to\n> +  verify the cache. The signature for this extension is { 'U', 'N',\n> +  'T', 'R' }.\n> +\n> +  The extension starts with\n> +\n> +  - Stat data of $GIT_DIR/info/exclude. See \"Index entry\" section from\n> +    ctime field until \"file size\".\n> +\n> +  - Stat data of core.excludesfile\n> +\n> +  - 32-bit dir_flags (see struct dir_struct)\n> +\n> +  - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n> +    does not exist.\n> +\n> +  - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n> +    not exist.\n> +\n> +  - NUL-terminated string of per-dir exclude file name. This usually\n> +    is \".gitignore\".\n> +\n> +  - The number of following directory blocks, variable width\n> +    encoding. If this number is zero, the extension ends here with a\n> +    following NUL.\n> +\n> +  - A number of directory blocks in depth-first-search order, each\n> +    consists of\n> +\n> +    - The number of untracked entries, variable witdh encoding.\n\ns/witdh/width/\n\n> +    - The number of sub-directory blocks, variable with encoding.\n\ns/with/width/\n\n> +    - The directory name terminated by NUL.\n> +\n> +    - A number of untrached file/dir names terminated by NUL.\n> +\n> +The remaining data of each directory block is grouped by type:\n> +\n> +  - An ewah bitmap, the n-th bit marks whether the n-th directory has\n> +    valid untracked cache entries.\n> +\n> +  - An ewah bitmap, the n-th bit records \"check-only\" bit of\n> +    read_directory_recursive() for the n-th directory.\n> +\n> +  - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n> +    is valid for the n-th directory and exists in the next data.\n> +\n> +  - An array of stat data. The n-th data corresponds with the n-th\n> +    \"one\" bit in the previous ewah bitmap.\n> +\n> +  - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n> +    in the previous ewah bitmap.\n> +\n> +  - One NUL.\n"},{"id":"253470","messageId":"CACsJy8BGfRPAMK9rFBiHs7w8L_3JwbRsZViDwPPVw6AVjUXg8g@mail.gmail.com","threadId":"38142","inReplyTo":"1418047507-22892-12-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 10/23] untracked cache: save to an index extension","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-10T00:21:07Z","receivedAt":"2014-12-10T00:21:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Dec 8, 2014 at 9:04 PM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:\n> +void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked)\n> +{\n> +       struct ondisk_untracked_cache *ouc;\n> +       struct write_data wd;\n> +       unsigned char varbuf[16];\n> +       int len = 0, varint_len;\n> +       if (untracked->exclude_per_dir)\n> +               len = strlen(untracked->exclude_per_dir);\n> +       ouc = xmalloc(sizeof(*ouc) + len);\n> +       stat_data_to_disk(&ouc->info_exclude_stat, &untracked->ss_info_exclude.stat);\n> +       stat_data_to_disk(&ouc->excludes_file_stat, &untracked->ss_excludes_file.stat);\n> +       hashcpy(ouc->info_exclude_sha1, untracked->ss_info_exclude.sha1);\n> +       hashcpy(ouc->excludes_file_sha1, untracked->ss_excludes_file.sha1);\n> +       ouc->dir_flags = htonl(untracked->dir_flags);\n> +       memcpy(ouc->exclude_per_dir, untracked->exclude_per_dir, len + 1);\n> +       strbuf_add(out, ouc, sizeof(*ouc) + len);\n\njust fyi, this sizeof(*ouc) here is wrong because we write down an\nunknown number of padding by the compiler after the last field \"char\nexclude_per_dir[1]\". I will use offsetof(struct..., exclude_per_dir)\nnext time. Spotted by valgrind.\n-- \nDuy\n"},{"id":"253476","messageId":"5487D543.7060801@web.de","threadId":"38142","inReplyTo":"CACsJy8A5-smRXN0k5TK8uPg4-j-Z83KWLNSO_w-eFivTNB_www@mail.gmail.com","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-12-10T05:08:19Z","receivedAt":"2014-12-10T05:08:19Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 12/09/2014 11:53 PM, Duy Nguyen wrote:\n> On Tue, Dec 9, 2014 at 5:04 PM, brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n>> On Mon, Dec 08, 2014 at 09:05:07PM +0700, Nguyễn Thái Ngọc Duy wrote:\n>>> If the user enables untracked cache, then\n>>>\n>>>   - move worktree to an unsupported filesystem\n>>>   - or simply upgrade OS\n>>>   - or move the whole (portable) disk from one machine to another\n>>>   - or access a shared fs from another machine\n>>>\n>>> there's no guarantee that untracked cache can still function properly.\n>>> Record the worktree location and OS footprint in the cache. If it\n>>> changes, err on the safe side and disable the cache. The user can\n>>> 'update-index --untracked-cache' again to make sure all conditions are\n>>> met.\n>> My use case for untracked cache is where I have one machine with a\n>> repository and which is also mounted via sshfs on another machine.  It\n>> looks like this will disable the cache every time I change the machine I\n>> access it on.  Would you be willing to accept a patch for a configuration\n>> option to disable this?\n> Torsten also does not like this patch. Maybe I'm just too paranoid.\n> Maybe we can drop this patch.\nThat opens another question:\nHow flexible/extensible/self-describing is the format of the UNTR \nextension ?\nIf we drop the OS name & root dir check because it disallows network use,\nbut later add a better method to verify that the underlying chain\nlocal OS - network - remote OS-remote FS is OK,\ndo we need to change the file format of UNTR ?\nIf yes, can old clients read the new format and vice versa?\nDo we need a version information of some kind, or does the\nold client skip unknown entries like we do with extensions in the index ?\n"},{"id":"253498","messageId":"CACsJy8BPzcAPJZG4=+mt=LmhhheJjXkfD2+znMjBbjbB9mODDQ@mail.gmail.com","threadId":"38142","inReplyTo":"5487D543.7060801@web.de","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-10T12:22:14Z","receivedAt":"2014-12-10T12:22:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Dec 10, 2014 at 12:08 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n> That opens another question:\n> How flexible/extensible/self-describing is the format of the UNTR extension\n> ?\n> If we drop the OS name & root dir check because it disallows network use,\n> but later add a better method to verify that the underlying chain\n> local OS - network - remote OS-remote FS is OK,\n> do we need to change the file format of UNTR ?\n> If yes, can old clients read the new format and vice versa?\n> Do we need a version information of some kind, or does the\n> old client skip unknown entries like we do with extensions in the index ?\n\nThe way index extensions are done so far, there's no actual versioning\ninside an extension.Once an extension is out, its format is set in\nstone. If you change your mind, you make a new extension (with a\ndifferent signature), so signatures are sort of \"version\". Code is\nshared mostly so it should not be a problem. Old clients don't\nrecognize new extensions, so they drop them. New clients either stick\nto old extensions or convert them to new ones. This is all local\nmatters, so I don't think we need to worry too much.\n-- \nDuy\n"},{"id":"253578","messageId":"548A0175.6000008@web.de","threadId":"38142","inReplyTo":"CACsJy8BPzcAPJZG4=+mt=LmhhheJjXkfD2+znMjBbjbB9mODDQ@mail.gmail.com","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2014-12-11T20:41:25Z","receivedAt":"2014-12-11T20:41:25Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 10.12.14 13:22, Duy Nguyen wrote:\n> On Wed, Dec 10, 2014 at 12:08 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n>> That opens another question:\n>> How flexible/extensible/self-describing is the format of the UNTR extension\n>> ?\n>> If we drop the OS name & root dir check because it disallows network use,\n>> but later add a better method to verify that the underlying chain\n>> local OS - network - remote OS-remote FS is OK,\n>> do we need to change the file format of UNTR ?\n>> If yes, can old clients read the new format and vice versa?\n>> Do we need a version information of some kind, or does the\n>> old client skip unknown entries like we do with extensions in the index ?\n> The way index extensions are done so far, there's no actual versioning\n> inside an extension.Once an extension is out, its format is set in\n> stone. If you change your mind, you make a new extension (with a\n> different signature), so signatures are sort of \"version\". Code is\n> shared mostly so it should not be a problem. Old clients don't\n> recognize new extensions, so they drop them. New clients either stick\n> to old extensions or convert them to new ones. This is all local\n> matters, so I don't think we need to worry too much.\nThanks for the info.\nEven if I share the the concerns that the cache may work on one system,\nbut not on the other, there should be better ways to protect from that.\n\nUsing the uname does not really help, if you move one repo from NTFS to VFAT,\nwe will not detect it (assuming we use Windows).\n(And how much do we need to support the move of a repo ?)\n\nThere is a concern that this may not work, when different clients are accessing\nthe repo, using the UNTR extension.\n\nSome kind of sanity check would be good to have, what can be done ?\nThe most important things are the timestamps.\nI can think of 2 sanity checks:\n- If the modified time stamp of a directory is older then the create time of any file,\n  the UNTR cache can not be used.\n- If the timestamp of a file changes, but the sha1 sum is the same, what does this mean?\n  The file (or the whole repo) has been copied, or the time stamping does not work.\n\nA simple verification of the FS could be to stat() .git/, create a temp file, delete it and\nstat() again. If mtime does not change, the FS is unusable for UNTR.\n\nThen we could extend the uname idea:\nCreate a string in UNTR which is a collection of lines like this:\n\nWorking-For: Linux;/mnt/nfs/projects/project1\nNot-OK-For: WIndows:/a:/project1\n(Of course the strings can be made nicer, and '\\n' is URL-encoded.)\n\nEach system that is not listed needs to probe the repo, add another line\nand re-write the index.\n\nWe can even add a \"best-for\" line, and invalidate the UNTR every 12 hours or so.\n\nShould we think about having an ASCII area for additional information, which is part\nof the stone, but the content is flexible ?\n\nThe patch-series really speeds up \"git status\" on a network, thanks for working on it.\nThe next days^H^H^H^H weeks I will do some more tests, using different combinations\nof OS and network protocols.\n"},{"id":"253689","messageId":"CACsJy8DLus+M65FCw=rrBStLXnWOmCJRm2NidWth9Roj8zAodw@mail.gmail.com","threadId":"38142","inReplyTo":"548A0175.6000008@web.de","subject":"Re: [PATCH v3 23/23] untracked cache: guard and disable on system changes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-12-15T03:02:22Z","receivedAt":"2014-12-15T03:02:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Dec 12, 2014 at 3:41 AM, Torsten Bögershausen <tboegi@web.de> wrote:\n> Even if I share the the concerns that the cache may work on one system,\n> but not on the other, there should be better ways to protect from that.\n>\n> Using the uname does not really help, if you move one repo from NTFS to VFAT,\n> we will not detect it (assuming we use Windows).\n> (And how much do we need to support the move of a repo ?)\n>\n> There is a concern that this may not work, when different clients are accessing\n> the repo, using the UNTR extension.\n>\n> Some kind of sanity check would be good to have, what can be done ?\n> The most important things are the timestamps.\n> I can think of 2 sanity checks:\n> - If the modified time stamp of a directory is older then the create time of any file,\n>   the UNTR cache can not be used.\n> - If the timestamp of a file changes, but the sha1 sum is the same, what does this mean?\n>   The file (or the whole repo) has been copied, or the time stamping does not work.\n>\n> A simple verification of the FS could be to stat() .git/, create a temp file, delete it and\n> stat() again. If mtime does not change, the FS is unusable for UNTR.\n\nThis is a slow test. Some filesytem only supports second resolution\ntimestamps. If you create and delete the file so fast, mtime may\nremain in the same second even if the fs is supported. We need to wait\na second between those operations (this is why \"update-index\n--untracked-cache\" takes several seconds). So it cannot be done often\n(i.e. at startup of every command)\n\n> Then we could extend the uname idea:\n> Create a string in UNTR which is a collection of lines like this:\n>\n> Working-For: Linux;/mnt/nfs/projects/project1\n> Not-OK-For: WIndows:/a:/project1\n> (Of course the strings can be made nicer, and '\\n' is URL-encoded.)\n>\n> Each system that is not listed needs to probe the repo, add another line\n> and re-write the index.\n>\n> We can even add a \"best-for\" line, and invalidate the UNTR every 12 hours or so.\n\nIt starts to look complicated. How about letting the user deal with\nit? UNTR will support storing multiple \"location\" lines. Whenever UNTR\nis needed at a new location, it's disabled. The user has to use\n'update-index' to test the new location and add it to UNTR. The user\ncan choose to keep current locations (network access), or replace them\nall with the new one (repo move), or just mark the new location\nunusable so the extension is kept for use in other places, but warning\nat this place is suppressed. THe \"localtion\" consists of worktree\npath, system name and host name.\n\n> Should we think about having an ASCII area for additional information, which is part\n> of the stone, but the content is flexible ?\n\nThese lines are already in free form. If the running system generates\nthe same string as stored in UNTR, it's allowed to use the extension.\nWe need code to understand this content, so flexibility must be within\nlimit..\n-- \nDuy\n"},{"id":"253696","messageId":"xmqq7fxsemeu.fsf@gitster.dls.corp.google.com","threadId":"38142","inReplyTo":"1418047507-22892-13-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 11/23] untracked cache: load from UNTR index extension","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-12-15T19:35:37Z","receivedAt":"2014-12-15T19:35:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:\n\n> +struct untracked_cache *read_untracked_extension(const void *data, unsigned long sz)\n> +{\n> +\tconst struct ondisk_untracked_cache *ouc;\n> +\tstruct untracked_cache *uc;\n> +\tstruct read_data rd;\n> +\tconst unsigned char *next = data, *end = data + sz;\n\ndata + sz is arithmetic on void * which is a no-no.\n"}]}