{"thread":{"id":"25846","subject":"[PATCH 0/5] Sparse checkout fixes","startedAt":"2010-11-26T18:17:42Z","lastAt":"2010-11-27T06:24:04Z","messageCount":9,"participants":["Nguyễn Thái Ngọc Duy","Thiago Farina"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"156680","messageId":"1290795467-7570-1-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":null,"subject":"[PATCH 0/5] Sparse checkout fixes","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:42Z","receivedAt":"2010-11-26T18:17:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"[on top of nd/maint-fix-add-typo-detection]\n\nThis fixes a sparse pattern sometimes won't match a directory if\n - it does not end with a slash\n - it has wildcards\n\nThe series also improves performance a bit, thanks to 3/5, skip-worktree\nbits are precomputed.\n\nImprovements in the previous round [1] are left out. I will need to\npolish them a bit more.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/161477\n\nNguyễn Thái Ngọc Duy (5):\n  cache.h: remove surrounding brackes and realign CE_* constants\n  dir.c: add free_excludes()\n  unpack-trees: move all skip-worktree checks back to unpack_trees()\n  unpack-trees: fix sparse checkout's \"unable to match directories\"\n  Revert \"excluded_1(): support exclude files in index\"\n\n Documentation/git-read-tree.txt      |    7 -\n cache.h                              |   26 ++--\n dir.c                                |   19 ++-\n dir.h                                |    1 +\n t/t1011-read-tree-sparse-checkout.sh |   14 ++-\n unpack-trees.c                       |  240 ++++++++++++++++++++++++++++++----\n 6 files changed, 254 insertions(+), 53 deletions(-)\n\n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156681","messageId":"1290795467-7570-2-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/5] cache.h: remove surrounding brackes and realign CE_* constants","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:43Z","receivedAt":"2010-11-26T18:17:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h |   25 ++++++++++++-------------\n 1 files changed, 12 insertions(+), 13 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 33decd9..4819cf5 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -170,26 +170,25 @@ struct cache_entry {\n  *\n  * In-memory only flags\n  */\n-#define CE_UPDATE    (0x10000)\n-#define CE_REMOVE    (0x20000)\n-#define CE_UPTODATE  (0x40000)\n-#define CE_ADDED     (0x80000)\n+#define CE_UPDATE              0x10000\n+#define CE_REMOVE              0x20000\n+#define CE_UPTODATE            0x40000\n+#define CE_ADDED               0x80000\n \n-#define CE_HASHED    (0x100000)\n-#define CE_UNHASHED  (0x200000)\n-#define CE_CONFLICTED (0x800000)\n+#define CE_HASHED             0x100000\n+#define CE_UNHASHED           0x200000\n+#define CE_WT_REMOVE          0x400000 /* remove in work directory */\n+#define CE_CONFLICTED         0x800000\n \n-#define CE_WT_REMOVE (0x400000) /* remove in work directory */\n-\n-#define CE_UNPACKED  (0x1000000)\n+#define CE_UNPACKED          0x1000000\n \n /*\n  * Extended on-disk flags\n  */\n-#define CE_INTENT_TO_ADD 0x20000000\n-#define CE_SKIP_WORKTREE 0x40000000\n+#define CE_INTENT_TO_ADD    0x20000000\n+#define CE_SKIP_WORKTREE    0x40000000\n /* CE_EXTENDED2 is for future extension */\n-#define CE_EXTENDED2 0x80000000\n+#define CE_EXTENDED2        0x80000000\n \n #define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)\n \n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156683","messageId":"1290795467-7570-3-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/5] dir.c: add free_excludes()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:44Z","receivedAt":"2010-11-26T18:17:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c          |   12 ++++++++++++\n dir.h          |    1 +\n unpack-trees.c |    6 +-----\n 3 files changed, 14 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b2dfb69..39eb7a3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -223,6 +223,18 @@ static void *read_skip_worktree_file_from_index(const char *path, size_t *size)\n \treturn data;\n }\n \n+void free_excludes(struct exclude_list *el)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < el->nr; i++)\n+\t\tfree(el->excludes[i]);\n+\tfree(el->excludes);\n+\n+\tel->nr = 0;\n+\tel->excludes = NULL;\n+}\n+\n int add_excludes_from_file_to_list(const char *fname,\n \t\t\t\t   const char *base,\n \t\t\t\t   int baselen,\ndiff --git a/dir.h b/dir.h\nindex 278d84c..ce55008 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -78,6 +78,7 @@ extern int add_excludes_from_file_to_list(const char *fname, const char *base, i\n extern void add_excludes_from_file(struct dir_struct *, const char *fname);\n extern void add_exclude(const char *string, const char *base,\n \t\t\tint baselen, struct exclude_list *which);\n+extern void free_excludes(struct exclude_list *el);\n extern int file_exists(const char *);\n \n extern char *get_relative_cwd(char *buffer, int size, const char *dir);\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 803445a..4d92412 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -945,11 +945,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t*o->dst_index = o->result;\n \n done:\n-\tfor (i = 0;i < el.nr;i++)\n-\t\tfree(el.excludes[i]);\n-\tif (el.excludes)\n-\t\tfree(el.excludes);\n-\n+\tfree_excludes(&el);\n \treturn ret;\n \n return_failed:\n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156682","messageId":"1290795467-7570-4-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] unpack-trees: move all skip-worktree checks back to unpack_trees()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:45Z","receivedAt":"2010-11-26T18:17:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Earlier, the will_have_skip_worktree() checks are done in various\nplaces, which makes it hard to traverse the index tree-alike, required\nby excluded_from_list(). This patch moves all the checks into two\nloops in unpack_trees().\n\nEntries in index in this operation can be classified into two\ngroups: ones already in index before unpack_trees() is called and ones\nadded to index after traverse_trees() is called.\n\nIn both groups, before checking file status on worktree, the future\nskip-worktree bit must be checked, so that if an entry will be outside\nworktree, worktree should not be checked.\n\nFor the first group, the future skip-worktree bit is precomputed and\nstored as CE_NEW_SKIP_WORKTREE in the first loop before\ntraverse_trees() is called so that *way_merge() function does not need\nto compute it again.\n\nFor the second group, because we don't know what entries will be in\nthis group until traverse_trees() finishes, operations that need\nfuture skip-worktree check is delayed until CE_NEW_SKIP_WORKTREE is\ncomputed in the second loop. CE_ADDED is used to mark entries in the\nsecond group.\n\nCE_ADDED and CE_NEW_SKIP_WORKTREE are temporary flags used in\nunpack_trees().  CE_ADDED is only used by add_to_index(), which should\nnot be called while unpack_trees() is running.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h        |    1 +\n unpack-trees.c |   82 +++++++++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 74 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4819cf5..52b998b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -181,6 +181,7 @@ struct cache_entry {\n #define CE_CONFLICTED         0x800000\n \n #define CE_UNPACKED          0x1000000\n+#define CE_NEW_SKIP_WORKTREE 0x2000000\n \n /*\n  * Extended on-disk flags\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 4d92412..a6518db 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -258,7 +258,7 @@ static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_opt\n {\n \tint was_skip_worktree = ce_skip_worktree(ce);\n \n-\tif (!ce_stage(ce) && will_have_skip_worktree(ce, o))\n+\tif (ce->ce_flags & CE_NEW_SKIP_WORKTREE)\n \t\tce->ce_flags |= CE_SKIP_WORKTREE;\n \telse\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n@@ -333,7 +333,7 @@ static void mark_all_ce_unused(struct index_state *index)\n {\n \tint i;\n \tfor (i = 0; i < index->cache_nr; i++)\n-\t\tindex->cache[i]->ce_flags &= ~CE_UNPACKED;\n+\t\tindex->cache[i]->ce_flags &= ~(CE_UNPACKED | CE_ADDED | CE_NEW_SKIP_WORKTREE);\n }\n \n static int locate_in_src_index(struct cache_entry *ce,\n@@ -835,8 +835,33 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n }\n \n /*\n+ * Set/Clear CE_NEW_SKIP_WORKTREE according to $GIT_DIR/info/sparse-checkout\n+ */\n+static void mark_new_skip_worktree(struct exclude_list *el,\n+\t\t\t\t   struct index_state *the_index,\n+\t\t\t\t   int select_flag, int skip_wt_flag)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < the_index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = the_index->cache[i];\n+\n+\t\tif (select_flag && !(ce->ce_flags & select_flag))\n+\t\t\tcontinue;\n+\n+\t\tif (!ce_stage(ce) && will_have_skip_worktree(ce, o))\n+\t\t\tce->ce_flags |= skip_wt_flag;\n+\t\telse\n+\t\t\tce->ce_flags &= ~skip_wt_flag;\n+\t}\n+}\n+\n+static int verify_absent(struct cache_entry *, enum unpack_trees_error_types, struct unpack_trees_options *);\n+/*\n  * N-way merge \"len\" trees.  Returns 0 on success, -1 on failure to manipulate the\n  * resulting index, -2 on failure to reflect the changes to the work tree.\n+ *\n+ * CE_ADDED, CE_UNPACKED and CE_NEW_SKIP_WORKTREE are used internally\n  */\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n@@ -869,6 +894,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->merge_size = len;\n \tmark_all_ce_unused(o->src_index);\n \n+\t/*\n+\t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n+\t */\n+\tif (!o->skip_sparse_checkout)\n+\t\tmark_new_skip_worktree(o->el, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n \to->df_conflict_entry = dfc;\n@@ -922,9 +953,29 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \tif (!o->skip_sparse_checkout) {\n \t\tint empty_worktree = 1;\n-\t\tfor (i = 0;i < o->result.cache_nr;i++) {\n+\n+\t\t/*\n+\t\t * Sparse checkout loop #2: set NEW_SKIP_WORKTREE on entries not in loop #1\n+\t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n+\t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n+\t\t */\n+\t\tmark_new_skip_worktree(o->el, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\n+\t\tfor (i = 0; i < o->result.cache_nr; i++) {\n \t\t\tstruct cache_entry *ce = o->result.cache[i];\n \n+\t\t\t/*\n+\t\t\t * Entries marked with CE_ADDED in merged_entry() do not have\n+\t\t\t * verify_absent() check (the check is effectively disabled\n+\t\t\t * because CE_NEW_SKIP_WORKTREE is set unconditionally).\n+\t\t\t *\n+\t\t\t * Do the real check now because we have had\n+\t\t\t * correct CE_NEW_SKIP_WORKTREE\n+\t\t\t */\n+\t\t\tif (ce->ce_flags & CE_ADDED &&\n+\t\t\t    verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n+\t\t\t\t\treturn -1;\n+\n \t\t\tif (apply_sparse_checkout(ce, o)) {\n \t\t\t\tret = -1;\n \t\t\t\tgoto done;\n@@ -1013,7 +1064,7 @@ static int verify_uptodate_1(struct cache_entry *ce,\n static int verify_uptodate(struct cache_entry *ce,\n \t\t\t   struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n }\n@@ -1200,7 +1251,7 @@ static int verify_absent(struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n \t\t\t struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_absent_1(ce, error_type, o);\n }\n@@ -1222,10 +1273,23 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \tint update = CE_UPDATE;\n \n \tif (!old) {\n+\t\t/*\n+\t\t * New index entries. In sparse checkout, the following\n+\t\t * verify_absent() will be delayed until after\n+\t\t * traverse_trees() finishes in unpack_trees(), then:\n+\t\t *\n+\t\t *  - CE_NEW_SKIP_WORKTREE will be computed correctly\n+\t\t *  - verify_absent() be called again, this time with\n+\t\t *    correct CE_NEW_SKIP_WORKTREE\n+\t\t *\n+\t\t * verify_absent() call here does nothing in sparse\n+\t\t * checkout (i.e. o->skip_sparse_checkout == 0)\n+\t\t */\n+\t\tupdate |= CE_ADDED;\n+\t\tmerge->ce_flags |= CE_NEW_SKIP_WORKTREE;\n+\n \t\tif (verify_absent(merge, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n \t\t\treturn -1;\n-\t\tif (!o->skip_sparse_checkout && will_have_skip_worktree(merge, o))\n-\t\t\tupdate |= CE_SKIP_WORKTREE;\n \t\tinvalidate_ce_path(merge, o);\n \t} else if (!(old->ce_flags & CE_CONFLICTED)) {\n \t\t/*\n@@ -1241,8 +1305,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o))\n \t\t\t\treturn -1;\n-\t\t\tif (ce_skip_worktree(old))\n-\t\t\t\tupdate |= CE_SKIP_WORKTREE;\n+\t\t\t/* Migrate old flags over */\n+\t\t\tupdate |= old->ce_flags & (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n \t\t\tinvalidate_ce_path(old, o);\n \t\t}\n \t} else {\n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156684","messageId":"1290795467-7570-5-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/5] unpack-trees: fix sparse checkout's \"unable to match directories\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:46Z","receivedAt":"2010-11-26T18:17:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Matching index entries against an excludes file currently has two\nproblems.\n\nFirst, there's no function to do it.  Code paths (like sparse\ncheckout) that wanted to try it would iterate over index entries and\nfor each index entry pass that path to excluded_from_list().  But that\nis not how excluded_from_list() works; one is supposed to feed in each\nancester of a path before a given path to find out if it was excluded\nbecause of some parent or grandparent matching a\n\n  bigsubdirectory/\n\npattern despite the path not matching any .gitignore pattern directly.\n\nSecond, it's inefficient.  The excludes mechanism is supposed to let\nus block off vast swaths of the filesystem as uninteresting; separately\nchecking every index entry doesn't fit that model.\n\nIntroduce a new function to take care of both these problems.  This\ntraverses the index in depth-first order (well, that's what order the\nindex is in) to mark un-excluded entries.\n\nMaybe some day the in-core index format will be restructured to make\nthis sort of operation easier.  Or maybe we will want to try some\nbinary search based thing.  The interface is simple enough to allow\nall those things.  Example:\n\n  clear_ce_flags(the_index.cache, the_index.cache_nr,\n                 CE_CANDIDATE, CE_CLEARME, exclude_list);\n\nwould clear the CE_CLEARME flag on all index entries with\nCE_CANDIDATE flag and not matched by exclude_list.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-read-tree.txt      |    7 --\n t/t1011-read-tree-sparse-checkout.sh |   14 +++-\n unpack-trees.c                       |  154 +++++++++++++++++++++++++++++++--\n 3 files changed, 155 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 2e78da4..f6037c4 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -412,13 +412,6 @@ turn `core.sparseCheckout` on in order to have sparse checkout\n support.\n \n \n-BUGS\n-----\n-In order to match a directory with $GIT_DIR/info/sparse-checkout,\n-trailing slash must be used. The form without trailing slash, while\n-works with .gitignore, does not work with sparse checkout.\n-\n-\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 8008fa2..67d9217 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -94,12 +94,20 @@ test_expect_success 'match directories with trailing slash' '\n \ttest -f sub/added\n '\n \n-test_expect_failure 'match directories without trailing slash' '\n-\techo init.t >.git/info/sparse-checkout &&\n+test_expect_success 'match directories without trailing slash' '\n \techo sub >>.git/info/sparse-checkout &&\n \tgit read-tree -m -u HEAD &&\n \tgit ls-files -t >result &&\n-\ttest_cmp expected.swt result &&\n+\ttest_cmp expected.swt-noinit result &&\n+\ttest ! -f init.t &&\n+\ttest -f sub/added\n+'\n+\n+test_expect_success 'match directory pattern' '\n+\techo \"s?b\" >>.git/info/sparse-checkout &&\n+\tgit read-tree -m -u HEAD &&\n+\tgit ls-files -t >result &&\n+\ttest_cmp expected.swt-noinit result &&\n \ttest ! -f init.t &&\n \ttest -f sub/added\n '\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a6518db..06de723 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -245,15 +245,6 @@ static int check_updates(struct unpack_trees_options *o)\n static int verify_uptodate_sparse(struct cache_entry *ce, struct unpack_trees_options *o);\n static int verify_absent_sparse(struct cache_entry *ce, enum unpack_trees_error_types, struct unpack_trees_options *o);\n \n-static int will_have_skip_worktree(const struct cache_entry *ce, struct unpack_trees_options *o)\n-{\n-\tconst char *basename;\n-\n-\tbasename = strrchr(ce->name, '/');\n-\tbasename = basename ? basename+1 : ce->name;\n-\treturn excluded_from_list(ce->name, ce_namelen(ce), basename, NULL, o->el) <= 0;\n-}\n-\n static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tint was_skip_worktree = ce_skip_worktree(ce);\n@@ -834,6 +825,138 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \treturn mask;\n }\n \n+/* Whole directory matching */\n+static int clear_ce_flags_dir(struct cache_entry **cache, int nr,\n+\t\t\t      char *prefix, int prefix_len,\n+\t\t\t      char *basename,\n+\t\t\t      int select_mask, int clear_mask,\n+\t\t\t      struct exclude_list *el)\n+{\n+\tstruct cache_entry **cache_end = cache + nr;\n+\tint dtype = DT_DIR;\n+\tint ret = excluded_from_list(prefix, prefix_len, basename, &dtype, el);\n+\n+\tprefix[prefix_len++] = '/';\n+\n+\t/* included, no clearing for any entries under this directory */\n+\tif (!ret) {\n+\t\tfor (; cache != cache_end; cache++) {\n+\t\t\tstruct cache_entry *ce = *cache;\n+\t\t\tif (strncmp(ce->name, prefix, prefix_len))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\treturn nr - (cache_end - cache);\n+\t}\n+\n+\t/* excluded, clear all selected entries under this directory. */\n+\tif (ret == 1) {\n+\t\tfor (; cache != cache_end; cache++) {\n+\t\t\tstruct cache_entry *ce = *cache;\n+\t\t\tif (select_mask && !(ce->ce_flags & select_mask))\n+\t\t\t\tcontinue;\n+\t\t\tif (strncmp(ce->name, prefix, prefix_len))\n+\t\t\t\tbreak;\n+\t\t\tce->ce_flags &= ~clear_mask;\n+\t\t}\n+\t\treturn nr - (cache_end - cache);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Traverse the index, find every entry that matches according to\n+ * o->el. Do \"ce_flags &= ~clear_mask\" on those entries. Return the\n+ * number of traversed entries.\n+ *\n+ * If select_mask is non-zero, only entries whose ce_flags has on of\n+ * those bits enabled are traversed.\n+ *\n+ * cache\t: pointer to an index entry\n+ * prefix_len\t: an offset to its path\n+ *\n+ * The current path (\"prefix\") including the trailing '/' is\n+ *   cache[0]->name[0..(prefix_len-1)]\n+ * Top level path has prefix_len zero.\n+ */\n+static int clear_ce_flags_1(struct cache_entry **cache, int nr,\n+\t\t\t    char *prefix, int prefix_len,\n+\t\t\t    int select_mask, int clear_mask,\n+\t\t\t    struct exclude_list *el)\n+{\n+\tstruct cache_entry **cache_end = cache + nr;\n+\n+\t/*\n+\t * Process all entries that have the given prefix and meet\n+\t * select_mask condition\n+\t */\n+\twhile(cache != cache_end) {\n+\t\tstruct cache_entry *ce = *cache;\n+\t\tconst char *name, *slash;\n+\t\tint len, dtype;\n+\n+\t\tif (select_mask && !(ce->ce_flags & select_mask)) {\n+\t\t\tcache++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (prefix_len && strncmp(ce->name, prefix, prefix_len))\n+\t\t\tbreak;\n+\n+\t\tname = ce->name + prefix_len;\n+\t\tslash = strchr(name, '/');\n+\n+\t\t/* If it's a directory, try whole directory match first */\n+\t\tif (slash) {\n+\t\t\tlen = slash - name;\n+\t\t\tint processed;\n+\n+\t\t\tmemcpy(prefix + prefix_len, name, len);\n+\n+\t\t\t/*\n+\t\t\t * terminate the string (no trailing slash),\n+\t\t\t * clear_c_f_dir needs it\n+\t\t\t */\n+\t\t\tprefix[prefix_len + len] = '\\0';\n+\t\t\tprocessed = clear_ce_flags_dir(cache, cache_end - cache,\n+\t\t\t\t\t\t       prefix, prefix_len + len,\n+\t\t\t\t\t\t       prefix + prefix_len,\n+\t\t\t\t\t\t       select_mask, clear_mask,\n+\t\t\t\t\t\t       el);\n+\n+\t\t\t/* clear_c_f_dir eats a whole dir already? */\n+\t\t\tif (processed) {\n+\t\t\t\tcache += processed;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tprefix[prefix_len + len++] = '/';\n+\t\t\tcache += clear_ce_flags_1(cache, cache_end - cache,\n+\t\t\t\t\t\t  prefix, prefix_len + len,\n+\t\t\t\t\t\t  select_mask, clear_mask, el);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Non-directory */\n+\t\tdtype = ce_to_dtype(ce);\n+\t\tif (excluded_from_list(ce->name, ce_namelen(ce), name, &dtype, el) > 0)\n+\t\t\tce->ce_flags &= ~clear_mask;\n+\t\tcache++;\n+\t}\n+\treturn nr - (cache_end - cache);\n+}\n+\n+static int clear_ce_flags(struct cache_entry **cache, int nr,\n+\t\t\t    int select_mask, int clear_mask,\n+\t\t\t    struct exclude_list *el)\n+{\n+\tchar prefix[PATH_MAX];\n+\treturn clear_ce_flags_1(cache, nr,\n+\t\t\t\tprefix, 0,\n+\t\t\t\tselect_mask, clear_mask,\n+\t\t\t\tel);\n+}\n+\n /*\n  * Set/Clear CE_NEW_SKIP_WORKTREE according to $GIT_DIR/info/sparse-checkout\n  */\n@@ -843,17 +966,28 @@ static void mark_new_skip_worktree(struct exclude_list *el,\n {\n \tint i;\n \n+\t/*\n+\t * 1. Pretend the narrowest worktree: only unmerged entries\n+\t * are checked out\n+\t */\n \tfor (i = 0; i < the_index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = the_index->cache[i];\n \n \t\tif (select_flag && !(ce->ce_flags & select_flag))\n \t\t\tcontinue;\n \n-\t\tif (!ce_stage(ce) && will_have_skip_worktree(ce, o))\n+\t\tif (!ce_stage(ce))\n \t\t\tce->ce_flags |= skip_wt_flag;\n \t\telse\n \t\t\tce->ce_flags &= ~skip_wt_flag;\n \t}\n+\n+\t/*\n+\t * 2. Widen worktree according to sparse-checkout file.\n+\t * Matched entries will have skip_wt_flag cleared (i.e. \"in\")\n+\t */\n+\tclear_ce_flags(the_index->cache, the_index->cache_nr,\n+\t\t       select_flag, skip_wt_flag, el);\n }\n \n static int verify_absent(struct cache_entry *, enum unpack_trees_error_types, struct unpack_trees_options *);\n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156685","messageId":"1290795467-7570-6-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/5] Revert \"excluded_1(): support exclude files in index\"","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-26T18:17:47Z","receivedAt":"2010-11-26T18:17:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This reverts commit c84de70781674a35b9bfd20aa5bc8c47582615df.\nThe commit provided a workaround for matching directories in\nindex. But it is no longer needed.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n dir.c |    7 -------\n 1 files changed, 0 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 39eb7a3..d059aa1 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -371,13 +371,6 @@ int excluded_from_list(const char *pathname,\n \t\t\tint to_exclude = x->to_exclude;\n \n \t\t\tif (x->flags & EXC_FLAG_MUSTBEDIR) {\n-\t\t\t\tif (!dtype) {\n-\t\t\t\t\tif (!prefixcmp(pathname, exclude) &&\n-\t\t\t\t\t    pathname[x->patternlen] == '/')\n-\t\t\t\t\t\treturn to_exclude;\n-\t\t\t\t\telse\n-\t\t\t\t\t\tcontinue;\n-\t\t\t\t}\n \t\t\t\tif (*dtype == DT_UNKNOWN)\n \t\t\t\t\t*dtype = get_dtype(NULL, pathname, pathlen);\n \t\t\t\tif (*dtype != DT_DIR)\n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156691","messageId":"AANLkTinwP=tDYyeyjX8uUChtvOeVmT4QfZjxT0A67hop@mail.gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 1/5] cache.h: remove surrounding brackes and realign CE_* constants","fromName":"Thiago Farina","fromEmail":"tfransosi@gmail.com","sentAt":"2010-11-26T19:20:11Z","receivedAt":"2010-11-26T19:20:11Z","isPatch":true,"sender":{"key":"tfransosi@gmail.com","avatar":"https://avatars.githubusercontent.com/u/970071?v=4"},"body":"2010/11/26 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n> ---\n>  cache.h |   25 ++++++++++++-------------\n>  1 files changed, 12 insertions(+), 13 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index 33decd9..4819cf5 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -170,26 +170,25 @@ struct cache_entry {\n>  *\n>  * In-memory only flags\n>  */\n> -#define CE_UPDATE    (0x10000)\n> -#define CE_REMOVE    (0x20000)\n> -#define CE_UPTODATE  (0x40000)\n> -#define CE_ADDED     (0x80000)\n> +#define CE_UPDATE              0x10000\n> +#define CE_REMOVE              0x20000\n> +#define CE_UPTODATE            0x40000\n> +#define CE_ADDED               0x80000\n>\nThanks for fixing this. Maybe these hex can be rewriting as 1 << 0, 1\n<< 1 and so on?\n"},{"id":"156714","messageId":"1290838936-25601-1-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"AANLkTinwP=tDYyeyjX8uUChtvOeVmT4QfZjxT0A67hop@mail.gmail.com","subject":"[PATCH 1/5] cache.h: realign and use (1 << x) form for CE_* constants","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-27T06:22:16Z","receivedAt":"2010-11-27T06:22:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n 2010/11/27 Thiago Farina <tfransosi@gmail.com>:\n > Thanks for fixing this. Maybe these hex can be rewriting as 1 << 0, 1\n > << 1 and so on?\n >\n\n It does look better. There a few 0xXXXX constants above, but they\n are short enough.\n\n cache.h |   25 ++++++++++++-------------\n 1 files changed, 12 insertions(+), 13 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 33decd9..f9545a1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -170,26 +170,25 @@ struct cache_entry {\n  *\n  * In-memory only flags\n  */\n-#define CE_UPDATE    (0x10000)\n-#define CE_REMOVE    (0x20000)\n-#define CE_UPTODATE  (0x40000)\n-#define CE_ADDED     (0x80000)\n+#define CE_UPDATE            (1 << 16)\n+#define CE_REMOVE            (1 << 17)\n+#define CE_UPTODATE          (1 << 18)\n+#define CE_ADDED             (1 << 19)\n \n-#define CE_HASHED    (0x100000)\n-#define CE_UNHASHED  (0x200000)\n-#define CE_CONFLICTED (0x800000)\n+#define CE_HASHED            (1 << 20)\n+#define CE_UNHASHED          (1 << 21)\n+#define CE_WT_REMOVE         (1 << 22) /* remove in work directory */\n+#define CE_CONFLICTED        (1 << 23)\n \n-#define CE_WT_REMOVE (0x400000) /* remove in work directory */\n-\n-#define CE_UNPACKED  (0x1000000)\n+#define CE_UNPACKED          (1 << 24)\n \n /*\n  * Extended on-disk flags\n  */\n-#define CE_INTENT_TO_ADD 0x20000000\n-#define CE_SKIP_WORKTREE 0x40000000\n+#define CE_INTENT_TO_ADD     (1 << 29)\n+#define CE_SKIP_WORKTREE     (1 << 30)\n /* CE_EXTENDED2 is for future extension */\n-#define CE_EXTENDED2 0x80000000\n+#define CE_EXTENDED2         (1 << 31)\n \n #define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)\n \n-- \n1.7.3.2.316.gda8b3\n"},{"id":"156715","messageId":"1290839044-25775-1-git-send-email-pclouds@gmail.com","threadId":"25846","inReplyTo":"1290795467-7570-4-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] unpack-trees: move all skip-worktree checks back to unpack_trees()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-11-27T06:24:04Z","receivedAt":"2010-11-27T06:24:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Earlier, the will_have_skip_worktree() checks are done in various\nplaces, which makes it hard to traverse the index tree-alike, required\nby excluded_from_list(). This patch moves all the checks into two\nloops in unpack_trees().\n\nEntries in index in this operation can be classified into two\ngroups: ones already in index before unpack_trees() is called and ones\nadded to index after traverse_trees() is called.\n\nIn both groups, before checking file status on worktree, the future\nskip-worktree bit must be checked, so that if an entry will be outside\nworktree, worktree should not be checked.\n\nFor the first group, the future skip-worktree bit is precomputed and\nstored as CE_NEW_SKIP_WORKTREE in the first loop before\ntraverse_trees() is called so that *way_merge() function does not need\nto compute it again.\n\nFor the second group, because we don't know what entries will be in\nthis group until traverse_trees() finishes, operations that need\nfuture skip-worktree check is delayed until CE_NEW_SKIP_WORKTREE is\ncomputed in the second loop. CE_ADDED is used to mark entries in the\nsecond group.\n\nCE_ADDED and CE_NEW_SKIP_WORKTREE are temporary flags used in\nunpack_trees().  CE_ADDED is only used by add_to_index(), which should\nnot be called while unpack_trees() is running.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n CE_NEW_SKIP_WORKTREE is changed to (1 << 25) instead of 0x2000000\n\n cache.h        |    1 +\n unpack-trees.c |   82 +++++++++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 74 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f9545a1..cb69318 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -181,6 +181,7 @@ struct cache_entry {\n #define CE_CONFLICTED        (1 << 23)\n \n #define CE_UNPACKED          (1 << 24)\n+#define CE_NEW_SKIP_WORKTREE (1 << 25)\n \n /*\n  * Extended on-disk flags\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 4d92412..a6518db 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -258,7 +258,7 @@ static int apply_sparse_checkout(struct cache_entry *ce, struct unpack_trees_opt\n {\n \tint was_skip_worktree = ce_skip_worktree(ce);\n \n-\tif (!ce_stage(ce) && will_have_skip_worktree(ce, o))\n+\tif (ce->ce_flags & CE_NEW_SKIP_WORKTREE)\n \t\tce->ce_flags |= CE_SKIP_WORKTREE;\n \telse\n \t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n@@ -333,7 +333,7 @@ static void mark_all_ce_unused(struct index_state *index)\n {\n \tint i;\n \tfor (i = 0; i < index->cache_nr; i++)\n-\t\tindex->cache[i]->ce_flags &= ~CE_UNPACKED;\n+\t\tindex->cache[i]->ce_flags &= ~(CE_UNPACKED | CE_ADDED | CE_NEW_SKIP_WORKTREE);\n }\n \n static int locate_in_src_index(struct cache_entry *ce,\n@@ -835,8 +835,33 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n }\n \n /*\n+ * Set/Clear CE_NEW_SKIP_WORKTREE according to $GIT_DIR/info/sparse-checkout\n+ */\n+static void mark_new_skip_worktree(struct exclude_list *el,\n+\t\t\t\t   struct index_state *the_index,\n+\t\t\t\t   int select_flag, int skip_wt_flag)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < the_index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = the_index->cache[i];\n+\n+\t\tif (select_flag && !(ce->ce_flags & select_flag))\n+\t\t\tcontinue;\n+\n+\t\tif (!ce_stage(ce) && will_have_skip_worktree(ce, o))\n+\t\t\tce->ce_flags |= skip_wt_flag;\n+\t\telse\n+\t\t\tce->ce_flags &= ~skip_wt_flag;\n+\t}\n+}\n+\n+static int verify_absent(struct cache_entry *, enum unpack_trees_error_types, struct unpack_trees_options *);\n+/*\n  * N-way merge \"len\" trees.  Returns 0 on success, -1 on failure to manipulate the\n  * resulting index, -2 on failure to reflect the changes to the work tree.\n+ *\n+ * CE_ADDED, CE_UNPACKED and CE_NEW_SKIP_WORKTREE are used internally\n  */\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n@@ -869,6 +894,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->merge_size = len;\n \tmark_all_ce_unused(o->src_index);\n \n+\t/*\n+\t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n+\t */\n+\tif (!o->skip_sparse_checkout)\n+\t\tmark_new_skip_worktree(o->el, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n \to->df_conflict_entry = dfc;\n@@ -922,9 +953,29 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \tif (!o->skip_sparse_checkout) {\n \t\tint empty_worktree = 1;\n-\t\tfor (i = 0;i < o->result.cache_nr;i++) {\n+\n+\t\t/*\n+\t\t * Sparse checkout loop #2: set NEW_SKIP_WORKTREE on entries not in loop #1\n+\t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n+\t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n+\t\t */\n+\t\tmark_new_skip_worktree(o->el, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\n+\t\tfor (i = 0; i < o->result.cache_nr; i++) {\n \t\t\tstruct cache_entry *ce = o->result.cache[i];\n \n+\t\t\t/*\n+\t\t\t * Entries marked with CE_ADDED in merged_entry() do not have\n+\t\t\t * verify_absent() check (the check is effectively disabled\n+\t\t\t * because CE_NEW_SKIP_WORKTREE is set unconditionally).\n+\t\t\t *\n+\t\t\t * Do the real check now because we have had\n+\t\t\t * correct CE_NEW_SKIP_WORKTREE\n+\t\t\t */\n+\t\t\tif (ce->ce_flags & CE_ADDED &&\n+\t\t\t    verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n+\t\t\t\t\treturn -1;\n+\n \t\t\tif (apply_sparse_checkout(ce, o)) {\n \t\t\t\tret = -1;\n \t\t\t\tgoto done;\n@@ -1013,7 +1064,7 @@ static int verify_uptodate_1(struct cache_entry *ce,\n static int verify_uptodate(struct cache_entry *ce,\n \t\t\t   struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n }\n@@ -1200,7 +1251,7 @@ static int verify_absent(struct cache_entry *ce,\n \t\t\t enum unpack_trees_error_types error_type,\n \t\t\t struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && will_have_skip_worktree(ce, o))\n+\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_absent_1(ce, error_type, o);\n }\n@@ -1222,10 +1273,23 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \tint update = CE_UPDATE;\n \n \tif (!old) {\n+\t\t/*\n+\t\t * New index entries. In sparse checkout, the following\n+\t\t * verify_absent() will be delayed until after\n+\t\t * traverse_trees() finishes in unpack_trees(), then:\n+\t\t *\n+\t\t *  - CE_NEW_SKIP_WORKTREE will be computed correctly\n+\t\t *  - verify_absent() be called again, this time with\n+\t\t *    correct CE_NEW_SKIP_WORKTREE\n+\t\t *\n+\t\t * verify_absent() call here does nothing in sparse\n+\t\t * checkout (i.e. o->skip_sparse_checkout == 0)\n+\t\t */\n+\t\tupdate |= CE_ADDED;\n+\t\tmerge->ce_flags |= CE_NEW_SKIP_WORKTREE;\n+\n \t\tif (verify_absent(merge, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n \t\t\treturn -1;\n-\t\tif (!o->skip_sparse_checkout && will_have_skip_worktree(merge, o))\n-\t\t\tupdate |= CE_SKIP_WORKTREE;\n \t\tinvalidate_ce_path(merge, o);\n \t} else if (!(old->ce_flags & CE_CONFLICTED)) {\n \t\t/*\n@@ -1241,8 +1305,8 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \t\t} else {\n \t\t\tif (verify_uptodate(old, o))\n \t\t\t\treturn -1;\n-\t\t\tif (ce_skip_worktree(old))\n-\t\t\t\tupdate |= CE_SKIP_WORKTREE;\n+\t\t\t/* Migrate old flags over */\n+\t\t\tupdate |= old->ce_flags & (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n \t\t\tinvalidate_ce_path(old, o);\n \t\t}\n \t} else {\n-- \n1.7.3.2.316.gda8b3\n"}]}