{"thread":{"id":"20272","subject":"[RFC PATCH 0/5] Sparse checkout resurrection","startedAt":"2009-07-29T06:49:09Z","lastAt":"2009-08-04T13:20:26Z","messageCount":15,"participants":["Nguyễn Thái Ngọc Duy","Jakub Narebski","Nguyen Thai Ngoc Duy","Junio C Hamano","skillzero@gmail.com"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"119019","messageId":"1248850154-5469-1-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":null,"subject":"[RFC PATCH 0/5] Sparse checkout resurrection","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:09Z","receivedAt":"2009-07-29T06:49:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Hi,\n\nI rip most of the code from my sparse checkout series,\nleaving only basic stuff. Now there is no CE_NO_CHECKOUT bit (CE_VALID\nwill be used), no sparse pattterns. You will need to provide a script,\nnamely \"git-shape-workdir\", to specify how you want to shape your\nworkdir. That's all. No porcelain, only plumbing. Need to add some\noptions to read-tree and checkout in order to enable/disable this, but\nI haven't got that far.\n\nThe first four patches is more or less CE_VALID fixups. Real stuff is\nin the last patch. Sparse patterns may return as a separate\nscript. Currently no idea how the porcelain UI should be. The only\nthing that is probably impossible this way is \"git clone\n--sparse\". You may need to \"clone -n\" first, then apply sparse\ncheckout later.\n\nComments?\n\nNguyễn Thái Ngọc Duy (5):\n Prevent diff machinery from examining worktree outside sparse checkout\n grep: skip files outside sparse checkout area\n gitignore: read from index if .gitignore is not in worktree\n unpack_trees(): keep track of unmerged entries\n unpack_trees(): add support for sparse checkout\n\n Documentation/git-grep.txt | 4 +-\n builtin-grep.c | 7 ++-\n builtin-read-tree.c | 6 ++-\n cache.h | 4 +\n diff-lib.c | 5 +-\n diff.c | 4 +-\n dir.c | 70 ++++++++++++------\n t/t3001-ls-files-others-exclude.sh | 20 +++++\n unpack-trees.c | 140 ++++++++++++++++++++++++++++++++++--\n unpack-trees.h | 4 +\n 10 files changed, 229 insertions(+), 35 deletions(-)\n"},{"id":"119020","messageId":"1248850154-5469-2-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/5] Prevent diff machinery from examining worktree outside sparse checkout","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:10Z","receivedAt":"2009-07-29T06:49:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"---\n diff-lib.c | 5 +++--\n diff.c | 4 +++-\n 2 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex b7813af..4094f18 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -159,7 +159,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (ce_uptodate(ce))\n+\t\tif (ce_uptodate(ce) || ce->ce_flags & CE_VALID)\n \t\t\tcontinue;\n \n \t\tchanged = check_removed(ce, &st);\n@@ -337,6 +337,8 @@ static void do_oneway_diff(struct unpack_trees_options *o,\n \tstruct rev_info *revs = o->unpack_data;\n \tint match_missing, cached;\n \n+\t/* if the entry is not checked out, don't examine work tree */\n+\tcached = o->index_only || (idx && idx->ce_flags & CE_VALID);\n \t/*\n \t * Backward compatibility wart - \"diff-index -m\" does\n \t * not mean \"do not ignore merges\", but \"match_missing\".\n@@ -344,7 +346,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,\n \t * But with the revision flag parsing, that's found in\n \t * \"!revs->ignore_merges\".\n \t */\n-\tcached = o->index_only;\n \tmatch_missing = !revs->ignore_merges;\n \n \tif (cached && idx && ce_stage(idx)) {\ndiff --git a/diff.c b/diff.c\nindex 91d6ea2..de1bd87 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -1810,8 +1810,10 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int\n \n \t/*\n \t * If ce matches the file in the work tree, we can reuse it.\n+\t * For sparse checkout case, ce_uptodate() may be true although\n+\t * the file may or may not exist in the work tree.\n \t */\n-\tif (ce_uptodate(ce) ||\n+\tif ((ce_uptodate(ce) && !(ce->ce_flags & CE_VALID)) ||\n \t (!lstat(name, &st) && !ce_match_stat(ce, &st, 0)))\n \t\treturn 1;\n \n-- \n1.6.3.2.448.gdf8b6\n"},{"id":"119021","messageId":"1248850154-5469-3-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-2-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/5] grep: skip files outside sparse checkout area","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:11Z","receivedAt":"2009-07-29T06:49:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"---\n Documentation/git-grep.txt | 4 +++-\n builtin-grep.c | 7 ++++++-\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 8c70020..8d9cba7 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -27,7 +27,9 @@ SYNOPSIS\n DESCRIPTION\n -----------\n Look for specified patterns in the working tree files, blobs\n-registered in the index file, or given tree objects.\n+registered in the index file, or given tree objects. By default\n+it will search in the working tree files. When in sparse checkout\n+mode, it only searches checked-out files.\n \n \n OPTIONS\ndiff --git a/builtin-grep.c b/builtin-grep.c\nindex ad0e0a5..55078cb 100644\n--- a/builtin-grep.c\n+++ b/builtin-grep.c\n@@ -455,6 +455,8 @@ static int external_grep(struct grep_opt *opt, const char **paths, int cached)\n \t\t\tcontinue;\n \t\tif (!pathspec_matches(paths, ce->name, opt->max_depth))\n \t\t\tcontinue;\n+\t\tif (ce->ce_flags & CE_VALID)\n+\t\t\tcontinue;\n \t\tname = ce->name;\n \t\tif (name[0] == '-') {\n \t\t\tint len = ce_namelen(ce);\n@@ -522,8 +524,11 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached,\n \t\t\t\tcontinue;\n \t\t\thit |= grep_sha1(opt, ce->sha1, ce->name, 0);\n \t\t}\n-\t\telse\n+\t\telse {\n+\t\t\tif (ce->ce_flags & CE_VALID)\n+\t\t\t\tcontinue;\n \t\t\thit |= grep_file(opt, ce->name);\n+\t\t}\n \t\tif (ce_stage(ce)) {\n \t\t\tdo {\n \t\t\t\tnr++;\n-- \n1.6.3.2.448.gdf8b6\n"},{"id":"119022","messageId":"1248850154-5469-4-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-3-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/5] gitignore: read from index if .gitignore is not in worktree","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:12Z","receivedAt":"2009-07-29T06:49:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"this allows to use .gitignore in sparse mode, where .gitignore may or\nmay not be checked out (CE_VALID)\n---\n dir.c | 70 ++++++++++++++++++++++++------------\n t/t3001-ls-files-others-exclude.sh | 20 ++++++++++\n 2 files changed, 67 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex e05b850..86bc90c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -200,11 +200,36 @@ void add_exclude(const char *string, const char *base,\n \twhich->excludes[which->nr++] = x;\n }\n \n+static void *read_index_data(const char *path, size_t *size)\n+{\n+\tint pos, len;\n+\tunsigned long sz;\n+\tenum object_type type;\n+\tvoid *data;\n+\tstruct index_state *istate = &the_index;\n+\n+\tlen = strlen(path);\n+\tpos = index_name_pos(istate, path, len);\n+\tif (pos < 0)\n+\t\treturn NULL;\n+\t/* only applies to no-checkout items */\n+\tif (!(istate->cache[pos]->ce_flags & CE_VALID))\n+\t\treturn NULL;\n+\tdata = read_sha1_file(istate->cache[pos]->sha1, &type, &sz);\n+\tif (!data || type != OBJ_BLOB) {\n+\t\tfree(data);\n+\t\treturn NULL;\n+\t}\n+\t*size = xsize_t(sz);\n+\treturn data;\n+}\n+\n static int add_excludes_from_file_1(const char *fname,\n \t\t\t\t const char *base,\n \t\t\t\t int baselen,\n \t\t\t\t char **buf_p,\n-\t\t\t\t struct exclude_list *which)\n+\t\t\t\t struct exclude_list *which,\n+\t\t\t\t int check_index)\n {\n \tstruct stat st;\n \tint fd, i;\n@@ -212,27 +237,31 @@ static int add_excludes_from_file_1(const char *fname,\n \tchar *buf, *entry;\n \n \tfd = open(fname, O_RDONLY);\n-\tif (fd < 0 || fstat(fd, &st) < 0)\n-\t\tgoto err;\n-\tsize = xsize_t(st.st_size);\n-\tif (size == 0) {\n-\t\tclose(fd);\n-\t\treturn 0;\n+\tif (fd < 0 || fstat(fd, &st) < 0) {\n+\t\tif (0 <= fd)\n+\t\t\tclose(fd);\n+\t\tif (!check_index || (buf = read_index_data(fname, &size)) == NULL)\n+\t\t\treturn -1;\n \t}\n-\tbuf = xmalloc(size+1);\n-\tif (read_in_full(fd, buf, size) != size)\n-\t{\n-\t\tfree(buf);\n-\t\tgoto err;\n+\telse {\n+\t\tsize = xsize_t(st.st_size);\n+\t\tif (size == 0) {\n+\t\t\tclose(fd);\n+\t\t\treturn 0;\n+\t\t}\n+\t\tbuf = xmalloc(size);\n+\t\tif (read_in_full(fd, buf, size) != size) {\n+\t\t\tclose(fd);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tclose(fd);\n \t}\n-\tclose(fd);\n \n \tif (buf_p)\n \t\t*buf_p = buf;\n-\tbuf[size++] = '\\n';\n \tentry = buf;\n-\tfor (i = 0; i < size; i++) {\n-\t\tif (buf[i] == '\\n') {\n+\tfor (i = 0; i <= size; i++) {\n+\t\tif (i == size || buf[i] == '\\n') {\n \t\t\tif (entry != buf + i && entry[0] != '#') {\n \t\t\t\tbuf[i - (i && buf[i-1] == '\\r')] = 0;\n \t\t\t\tadd_exclude(entry, base, baselen, which);\n@@ -241,17 +270,12 @@ static int add_excludes_from_file_1(const char *fname,\n \t\t}\n \t}\n \treturn 0;\n-\n- err:\n-\tif (0 <= fd)\n-\t\tclose(fd);\n-\treturn -1;\n }\n \n void add_excludes_from_file(struct dir_struct *dir, const char *fname)\n {\n \tif (add_excludes_from_file_1(fname, \"\", 0, NULL,\n-\t\t\t\t &dir->exclude_list[EXC_FILE]) < 0)\n+\t\t\t\t &dir->exclude_list[EXC_FILE], 0) < 0)\n \t\tdie(\"cannot use %s as an exclude file\", fname);\n }\n \n@@ -302,7 +326,7 @@ static void prep_exclude(struct dir_struct *dir, const char *base, int baselen)\n \t\tstrcpy(dir->basebuf + stk->baselen, dir->exclude_per_dir);\n \t\tadd_excludes_from_file_1(dir->basebuf,\n \t\t\t\t\t dir->basebuf, stk->baselen,\n-\t\t\t\t\t &stk->filebuf, el);\n+\t\t\t\t\t &stk->filebuf, el, 1);\n \t\tdir->exclude_stack = stk;\n \t\tcurrent = stk->baselen;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex c65bca8..e8b25d3 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -85,6 +85,26 @@ test_expect_success \\\n >output &&\n test_cmp expect output'\n \n+test_expect_success 'setup sparse gitignore' '\n+\tgit add .gitignore one/.gitignore one/two/.gitignore &&\n+\tgit update-index --no-checkout .gitignore one/.gitignore one/two/.gitignore &&\n+\trm .gitignore one/.gitignore one/two/.gitignore\n+'\n+\n+test_expect_success \\\n+ 'git ls-files --others with various exclude options.' \\\n+ 'git ls-files --others \\\n+ --exclude=\\*.6 \\\n+ --exclude-per-directory=.gitignore \\\n+ --exclude-from=.git/ignore \\\n+ >output &&\n+ test_cmp expect output'\n+\n+test_expect_success 'restore gitignore' '\n+\tgit checkout .gitignore one/.gitignore one/two/.gitignore &&\n+\trm .git/index\n+'\n+\n cat > excludes-file <<\\EOF\n *.[1-8]\n e*\n-- \n1.6.3.2.448.gdf8b6\n"},{"id":"119023","messageId":"1248850154-5469-5-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-4-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/5] unpack_trees(): keep track of unmerged entries","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:13Z","receivedAt":"2009-07-29T06:49:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\"git read-tree --reset\" will currently remove all unmerged entries in\nindex before feeding the index to unpack_trees(). Because the lack of\nunmerged entries, these entries, when read from tree, will be seen as\n\"new entries\" by {one,two,three}way_merge().\n\nThis is fine for now. But for sparse checkout, it needs to know\nwhether an entry is new entry, because it will handle it different way\nthan already-in entry.\n\nSo the patch moves \"unmerged entries removal\" part into\nunpack_trees(), actually unpack_callback(). The function then can turn\non o->has_unmerged flag, which can be utilized by sparse checkout.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin-read-tree.c | 6 +++++-\n unpack-trees.c | 7 +++++++\n unpack-trees.h | 2 ++\n 3 files changed, 14 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-read-tree.c b/builtin-read-tree.c\nindex 9c2d634..17ca265 100644\n--- a/builtin-read-tree.c\n+++ b/builtin-read-tree.c\n@@ -113,7 +113,11 @@ int cmd_read_tree(int argc, const char **argv, const char *unused_prefix)\n \targc = parse_options(argc, argv, unused_prefix, read_tree_options,\n \t\t\t read_tree_usage, 0);\n \n-\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n+\tif (opts.reset) {\n+\t\topts.prune_unmerged = 1;\n+\t\tread_cache();\n+\t}\n+\telse if (read_cache_unmerged() && (opts.prefix || opts.merge))\n \t\tdie(\"You need to resolve your current index first\");\n \n \tprefix_set = opts.prefix ? 1 : 0;\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 720f7a1..c67eed8 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -289,6 +289,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\to->has_unmerged = 0;\n \t\twhile (o->pos < o->src_index->cache_nr) {\n \t\t\tstruct cache_entry *ce = o->src_index->cache[o->pos];\n \t\t\tint cmp = compare_entry(ce, info, p);\n@@ -308,6 +309,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t\t\t\tadd_entry(o, ce, 0, 0);\n \t\t\t\t\t\treturn mask;\n \t\t\t\t\t}\n+\n+\t\t\t\t\tif (o->prune_unmerged) {\n+\t\t\t\t\t\to->has_unmerged = 1;\n+\t\t\t\t\t\t/* leave src[0] as NULL and go over all other staged entries */\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\t}\n \t\t\t\t}\n \t\t\t\tsrc[0] = ce;\n \t\t\t}\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex d19df44..aa3ac17 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -26,6 +26,8 @@ struct unpack_trees_options {\n \t\t verbose_update,\n \t\t aggressive,\n \t\t skip_unmerged,\n+\t\t prune_unmerged,\n+\t\t has_unmerged,\n \t\t initial_checkout,\n \t\t diff_index_cached,\n \t\t gently;\n-- \n1.6.3.2.448.gdf8b6\n"},{"id":"119024","messageId":"1248850154-5469-6-git-send-email-pclouds@gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-5-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/5] unpack_trees(): add support for sparse checkout","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T06:49:14Z","receivedAt":"2009-07-29T06:49:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch teaches unpack_trees() to checkout/remove entries\non working directories appropriately when sparse checkout area is\nchanged. A helper \"git shape-workdir\" is needed to help determine\nwhich entry will be checked out, which will be not.\n\n\"git shape-workdir\" will receive from stdin in this format\n\nX\\tpathname\n\nwhere X is either\n - '!' current entry is already CE_VALID\n - 'N' current entry is \"new\" (it has not been in index before)\n - '-' current entry is \"normal\" entry\n\n\"git shape-workdir\" is expected to return either \"1\" or \"0\"\nimmediately. \"1\" means the entry should be in workdir. \"0\" means\nsetting CE_VALID and get rid of it from workdir.\n---\n cache.h | 4 ++\n unpack-trees.c | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++++---\n unpack-trees.h | 2 +\n 3 files changed, 133 insertions(+), 6 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1a2a3c9..eac3151 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -177,6 +177,10 @@ struct cache_entry {\n #define CE_HASHED (0x100000)\n #define CE_UNHASHED (0x200000)\n \n+/* Only remove in work directory, not index */\n+#define CE_WT_REMOVE (0x400000)\n+#define CE_NEW (0x800000)\n+\n /*\n * Extended on-disk flags\n */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex c67eed8..125a662 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -8,6 +8,7 @@\n #include \"progress.h\"\n #include \"refs.h\"\n #include \"attr.h\"\n+#include \"run-command.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -32,6 +33,12 @@ static struct unpack_trees_error_msgs unpack_plumbing_errors = {\n \n \t/* bind_overlap */\n \t\"Entry '%s' overlaps with '%s'. Cannot bind.\",\n+\n+\t/* sparse_not_uptodate_file */\n+\t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n+\n+\t/* would_lose_orphaned */\n+\t\"Orphaned working tree file '%s' would be %s by sparse checkout update.\",\n };\n \n #define ERRORMSG(o,fld) \\\n@@ -78,7 +85,7 @@ static int check_updates(struct unpack_trees_options *o)\n \tif (o->update && o->verbose_update) {\n \t\tfor (total = cnt = 0; cnt < index->cache_nr; cnt++) {\n \t\t\tstruct cache_entry *ce = index->cache[cnt];\n-\t\t\tif (ce->ce_flags & (CE_UPDATE | CE_REMOVE))\n+\t\t\tif (ce->ce_flags & (CE_UPDATE | CE_REMOVE | CE_WT_REMOVE))\n \t\t\t\ttotal++;\n \t\t}\n \n@@ -92,6 +99,13 @@ static int check_updates(struct unpack_trees_options *o)\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n+\t\tif (ce->ce_flags & CE_WT_REMOVE) {\n+\t\t\tdisplay_progress(progress, ++cnt);\n+\t\t\tif (o->update)\n+\t\t\t\tunlink_entry(ce);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (ce->ce_flags & CE_REMOVE) {\n \t\t\tdisplay_progress(progress, ++cnt);\n \t\t\tif (o->update)\n@@ -118,6 +132,81 @@ static int check_updates(struct unpack_trees_options *o)\n \treturn errs != 0;\n }\n \n+static int verify_uptodate_sparse(struct cache_entry *ce, struct unpack_trees_options *o);\n+static int verify_absent_sparse(struct cache_entry *ce, const char *action, struct unpack_trees_options *o);\n+static int apply_narrow_spec(struct unpack_trees_options *o)\n+{\n+\tstruct index_state *index = &o->result;\n+\tstruct child_process cp;\n+\tconst char *argv[] = { \"shape-workdir\", NULL };\n+\tint i, ret;\n+\n+\tmemset(&cp, 0, sizeof(cp));\n+\tcp.in = -1;\n+\tcp.out = -1;\n+\tcp.git_cmd = 1;\n+\tcp.argv = argv;\n+\n+\tret = start_command(&cp);\n+\tif (ret)\n+\t\treturn -1;\n+\n+\tfor (i = 0; i < index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = index->cache[i];\n+\t\tint was_checkout = !(ce->ce_flags & CE_VALID);\n+\t\tchar buf[PATH_MAX+3];\n+\t\tchar result[2];\n+\n+\t\tif (ce_stage(ce))\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * We only care about files getting into the checkout area\n+\t\t * If merge strategies want to remove some, go ahead\n+\t\t */\n+\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\n+\t\tsnprintf(buf, PATH_MAX+3, \"%c\\t%s\\n\",\n+\t\t\t (ce->ce_flags & CE_NEW) ? 'N' : (was_checkout ? '!' : '-'),\n+\t\t\t ce->name);\n+\t\txwrite(cp.in, buf, strlen(buf));\n+\t\txread(cp.out, result, 2);\n+\n+\t\tif (result[0] == '1')\n+\t\t\tce->ce_flags &= ~CE_VALID;\n+\t\telse\n+\t\t\tce->ce_flags |= CE_VALID;\n+\n+\t\t/* Update worktree, add/remove entries if needed */\n+\n+\t\tif (was_checkout && ce->ce_flags & CE_VALID) {\n+\t\t\t/*\n+\t\t\t * If CE_UPDATE is set, verify_uptodate() must be called already\n+\t\t\t * also stat info may have lost after merged_entry() so calling\n+\t\t\t * verify_uptodate() again may fail\n+\t\t\t */\n+\t\t\tif (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))\n+\t\t\t\treturn -1;\n+\t\t\tce->ce_flags |= CE_WT_REMOVE;\n+\t\t}\n+\t\tif (!was_checkout && !(ce->ce_flags & CE_VALID)) {\n+\t\t\tif (verify_absent_sparse(ce, \"overwritten\", o))\n+\t\t\t\treturn -1;\n+\t\t\tce->ce_flags |= CE_UPDATE;\n+\t\t}\n+\n+\t\t/* merge strategies may set CE_UPDATE outside checkout area */\n+\t\tif (ce->ce_flags & CE_VALID)\n+\t\t\tce->ce_flags &= ~CE_UPDATE;\n+\n+\t}\n+\tclose(cp.in);\n+\tfinish_command(&cp);\n+\tclose(cp.out);\n+\treturn 0;\n+}\n+\n static inline int call_unpack_fn(struct cache_entry **src, struct unpack_trees_options *o)\n {\n \tint ret = o->fn(src, o);\n@@ -423,6 +512,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tif (o->trivial_merges_only && o->nontrivial_merge)\n \t\treturn unpack_failed(o, \"Merge requires file-level merging\");\n \n+\tif (apply_narrow_spec(o))\n+\t\treturn unpack_failed(o, NULL);\n+\n \to->src_index = NULL;\n \tret = check_updates(o) ? (-2) : 0;\n \tif (o->dst_index)\n@@ -452,8 +544,9 @@ static int same(struct cache_entry *a, struct cache_entry *b)\n * When a CE gets turned into an unmerged entry, we\n * want it to be up-to-date\n */\n-static int verify_uptodate(struct cache_entry *ce,\n-\t\tstruct unpack_trees_options *o)\n+static int verify_uptodate_generic(struct cache_entry *ce,\n+\t\t\t\t struct unpack_trees_options *o,\n+\t\t\t\t const char *error_msg)\n {\n \tstruct stat st;\n \n@@ -478,7 +571,18 @@ static int verify_uptodate(struct cache_entry *ce,\n \tif (errno == ENOENT)\n \t\treturn 0;\n \treturn o->gently ? -1 :\n-\t\terror(ERRORMSG(o, not_uptodate_file), ce->name);\n+\t\terror(error_msg, ce->name);\n+}\n+\n+static int verify_uptodate(struct cache_entry *ce,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\treturn verify_uptodate_generic(ce, o, ERRORMSG(o, not_uptodate_file));\n+}\n+static int verify_uptodate_sparse(struct cache_entry *ce,\n+\t\t\t\t struct unpack_trees_options *o)\n+{\n+\treturn verify_uptodate_generic(ce, o, ERRORMSG(o, sparse_not_uptodate_file));\n }\n \n static void invalidate_ce_path(struct cache_entry *ce, struct unpack_trees_options *o)\n@@ -586,8 +690,9 @@ static int icase_exists(struct unpack_trees_options *o, struct cache_entry *dst,\n * We do not want to remove or overwrite a working tree file that\n * is not tracked, unless it is ignored.\n */\n-static int verify_absent(struct cache_entry *ce, const char *action,\n-\t\t\t struct unpack_trees_options *o)\n+static int verify_absent_generic(struct cache_entry *ce, const char *action,\n+\t\t\t\t struct unpack_trees_options *o,\n+\t\t\t\t const char *error_msg)\n {\n \tstruct stat st;\n \n@@ -667,6 +772,16 @@ static int verify_absent(struct cache_entry *ce, const char *action,\n \t}\n \treturn 0;\n }\n+static int verify_absent(struct cache_entry *ce, const char *action,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\treturn verify_absent_generic(ce, action, o, ERRORMSG(o, would_lose_untracked));\n+}\n+static int verify_absent_sparse(struct cache_entry *ce, const char *action,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\treturn verify_absent_generic(ce, action, o, ERRORMSG(o, would_lose_orphaned));\n+}\n \n static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \t\tstruct unpack_trees_options *o)\n@@ -689,10 +804,14 @@ static int merged_entry(struct cache_entry *merge, struct cache_entry *old,\n \t\t\t\treturn -1;\n \t\t\tinvalidate_ce_path(old, o);\n \t\t}\n+\t\tif (old->ce_flags & CE_VALID)\n+\t\t\tupdate |= CE_VALID;\n \t}\n \telse {\n \t\tif (verify_absent(merge, \"overwritten\", o))\n \t\t\treturn -1;\n+\t\tif (!o->has_unmerged)\n+\t\t\tupdate |= CE_NEW;\n \t\tinvalidate_ce_path(merge, o);\n \t}\n \n@@ -1017,6 +1136,8 @@ int oneway_merge(struct cache_entry **src, struct unpack_trees_options *o)\n \t\t\t ie_match_stat(o->src_index, old, &st, CE_MATCH_IGNORE_VALID))\n \t\t\t\tupdate |= CE_UPDATE;\n \t\t}\n+\t\tif (old->ce_flags & CE_VALID)\n+\t\t\tupdate |= CE_VALID;\n \t\tadd_entry(o, old, update, 0);\n \t\treturn 0;\n \t}\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex aa3ac17..8838517 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -14,6 +14,8 @@ struct unpack_trees_error_msgs {\n \tconst char *not_uptodate_dir;\n \tconst char *would_lose_untracked;\n \tconst char *bind_overlap;\n+\tconst char *sparse_not_uptodate_file;\n+\tconst char *would_lose_orphaned;\n };\n \n struct unpack_trees_options {\n-- \n1.6.3.2.448.gdf8b6\n"},{"id":"119055","messageId":"m3zlan1zhv.fsf@localhost.localdomain","threadId":"20272","inReplyTo":"1248850154-5469-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 5/5] unpack_trees(): add support for sparse checkout","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-07-29T11:48:31Z","receivedAt":"2009-07-29T11:48:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This patch teaches unpack_trees() to checkout/remove entries\n> on working directories appropriately when sparse checkout area is\n> changed. A helper \"git shape-workdir\" is needed to help determine\n> which entry will be checked out, which will be not.\n\nWouldn't \"git update-index --index-info\" (perhaps extended) be enough?\n\n> \n> \"git shape-workdir\" will receive from stdin in this format\n> \n> X\\tpathname\n> \n> where X is either\n> - '!' current entry is already CE_VALID\n> - 'N' current entry is \"new\" (it has not been in index before)\n> - '-' current entry is \"normal\" entry\n> \n> \"git shape-workdir\" is expected to return either \"1\" or \"0\"\n> immediately. \"1\" means the entry should be in workdir. \"0\" means\n> setting CE_VALID and get rid of it from workdir.\n\nJust in case it would be better IMVHO if it returned \"1\\tpathname\" or\n\"2\\tpathname\". By the way, is 'pathname' quoted if necessary, and\ndoes git-shape-workdir support -z/--null option?\n\nSignoff (also in some other patches in this series)?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"119120","messageId":"fcaeb9bf0907291632n418508ccke6ad274211690e59@mail.gmail.com","threadId":"20272","inReplyTo":"m3zlan1zhv.fsf@localhost.localdomain","subject":"Re: [PATCH 5/5] unpack_trees(): add support for sparse checkout","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-07-29T23:32:01Z","receivedAt":"2009-07-29T23:32:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2009/7/29 Jakub Narebski :\n> Nguyễn Thái Ngọc Duy     writes:\n>\n>> This patch teaches unpack_trees() to checkout/remove entries\n>> on working directories appropriately when sparse checkout area is\n>> changed. A helper \"git shape-workdir\" is needed to help determine\n>> which entry will be checked out, which will be not.\n>\n> Wouldn't \"git update-index --index-info\" (perhaps extended) be enough?\n\nIt's a bit more complicated because the in-memory index is filled with\nvarious information and can't just be dumped out to be manipulated\nwith \"git update-index\".\n\n>>\n>> \"git shape-workdir\" will receive from stdin in this format\n>>\n>> X\\tpathname\n>>\n>> where X is either\n>>  - '!' current entry is already CE_VALID\n>>  - 'N' current entry is \"new\" (it has not been in index before)\n>>  - '-' current entry is \"normal\" entry\n>>\n>> \"git shape-workdir\" is expected to return either \"1\" or \"0\"\n>> immediately. \"1\" means the entry should be in workdir. \"0\" means\n>> setting CE_VALID and get rid of it from workdir.\n>\n> Just in case it would be better IMVHO if it returned \"1\\tpathname\" or\n> \"2\\tpathname\".  By the way, is 'pathname' quoted if necessary, and\n> does git-shape-workdir support -z/--null option?\n\nIt doesn't currently. Thanks for the suggestion.\n\n> Signoff (also in some other patches in this series)?\n\nThis series is mainly for taking input on how git-shape-workdir should\nbehave, what format is good... That's why I did not sign off. Anyway\nthe series is not really well tested (and obvious lacks tests)\n-- \nDuy\n"},{"id":"119134","messageId":"200907300242.09950.jnareb@gmail.com","threadId":"20272","inReplyTo":"fcaeb9bf0907291632n418508ccke6ad274211690e59@mail.gmail.com","subject":"Re: [PATCH 5/5] unpack_trees(): add support for sparse checkout","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-07-30T00:42:09Z","receivedAt":"2009-07-30T00:42:09Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 30 July 2009, Nguyen Thai Ngoc Duy wrote:\n> 2009/7/29 Jakub Narebski :\n\n> > Signoff (also in some other patches in this series)?\n> \n> This series is mainly for taking input on how git-shape-workdir should\n> behave, what format is good... That's why I did not sign off. Anyway\n> the series is not really well tested (and obvious lacks tests)\n\nIn that case it would be IMHO a good idea to mark those patches also\nby using \"[PATCH/RFC m/n] ...\" or \"[RFC/PATCH m/n] ...\" in subject.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"119289","messageId":"7vprbgstfr.fsf@alter.siamese.dyndns.org","threadId":"20272","inReplyTo":"1248850154-5469-2-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 1/5] Prevent diff machinery from examining worktree outside sparse checkout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-31T16:34:00Z","receivedAt":"2009-07-31T16:34:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> ---\n\nSign off? Oh, this is an RFC... Ok...\n\n> diff --git a/diff-lib.c b/diff-lib.c\n> index b7813af..4094f18 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -337,6 +337,8 @@ static void do_oneway_diff(struct unpack_trees_options *o,\n> \tstruct rev_info *revs = o->unpack_data;\n> \tint match_missing, cached;\n> \n> +\t/* if the entry is not checked out, don't examine work tree */\n> +\tcached = o->index_only || (idx && idx->ce_flags & CE_VALID);\n> \t/*\n> \t * Backward compatibility wart - \"diff-index -m\" does\n> \t * not mean \"do not ignore merges\", but \"match_missing\".\n> @@ -344,7 +346,6 @@ static void do_oneway_diff(struct unpack_trees_options *o,\n> \t * But with the revision flag parsing, that's found in\n> \t * \"!revs->ignore_merges\".\n> \t */\n> -\tcached = o->index_only;\n> \tmatch_missing = !revs->ignore_merges;\n> \n> \tif (cached && idx && ce_stage(idx)) {\n\nThe big comment is about match_missing and not about cached, so the code\nmovement is good. When we pass NULL idx to this function, it means that\nthe tree has it but the index doesn't, so it cannot possibly be marked as\n\"exists in the index but is not checked out\". So the hunk makes sense.\n\n> diff --git a/diff.c b/diff.c\n> index 91d6ea2..de1bd87 100644\n> --- a/diff.c\n> +++ b/diff.c\n> @@ -1810,8 +1810,10 @@ static int reuse_worktree_file(const char *name, const unsigned char *sha1, int\n> \n> \t/*\n> \t * If ce matches the file in the work tree, we can reuse it.\n> +\t * For sparse checkout case, ce_uptodate() may be true although\n> +\t * the file may or may not exist in the work tree.\n> \t */\n> -\tif (ce_uptodate(ce) ||\n> +\tif ((ce_uptodate(ce) && !(ce->ce_flags & CE_VALID)) ||\n> \t (!lstat(name, &st) && !ce_match_stat(ce, &st, 0)))\n> \t\treturn 1;\n\nWhy is this hunk necessary? A few lines above this we return 0 for any\nCE_VALID cache entry.\n"},{"id":"119290","messageId":"7vljm4st0w.fsf@alter.siamese.dyndns.org","threadId":"20272","inReplyTo":"1248850154-5469-3-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 2/5] grep: skip files outside sparse checkout area","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-31T16:42:55Z","receivedAt":"2009-07-31T16:42:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> ---\n> Documentation/git-grep.txt | 4 +++-\n> builtin-grep.c | 7 ++++++-\n> 2 files changed, 9 insertions(+), 2 deletions(-)\n\nI do not think you can call this \"CE_VALID fixup\". This is a possibly\nuseful but nevertheless a new, different behaviour.\n\nWe probably want opt->ignoreassumeunchanged in struct grep_opt, and\ncontrol it via grep.ignoreassumeunchanged configuration, that perhaps\ndefaults to true (i.e. \"ignore\"), with a command line override.\n\nOh, and perhaps a shorter-and-sweeter name than \"ignoreAssumeUnchanged\".\n\n> diff --git a/builtin-grep.c b/builtin-grep.c\n> index ad0e0a5..55078cb 100644\n> --- a/builtin-grep.c\n> +++ b/builtin-grep.c\n> @@ -455,6 +455,8 @@ static int external_grep(struct grep_opt *opt, const char **paths, int cached)\n> \t\t\tcontinue;\n> \t\tif (!pathspec_matches(paths, ce->name, opt->max_depth))\n> \t\t\tcontinue;\n> +\t\tif (ce->ce_flags & CE_VALID)\n> +\t\t\tcontinue;\n\nThe test probably should come before pathspec_matches().\n\n> @@ -522,8 +524,11 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached,\n> \t\t\t\tcontinue;\n> \t\t\thit |= grep_sha1(opt, ce->sha1, ce->name, 0);\n> \t\t}\n> -\t\telse\n> +\t\telse {\n> +\t\t\tif (ce->ce_flags & CE_VALID)\n> +\t\t\t\tcontinue;\n> \t\t\thit |= grep_file(opt, ce->name);\n> +\t\t}\n> \t\tif (ce_stage(ce)) {\n> \t\t\tdo {\n> \t\t\t\tnr++;\n\nI do not think the new code in this hunk will be reached as the other arm\nof the \"else\" greps in the index when CE_VALID is set.\n\nHas this series been tested or self-reviewed yet?\n"},{"id":"119291","messageId":"7vhbwsssg1.fsf@alter.siamese.dyndns.org","threadId":"20272","inReplyTo":"1248850154-5469-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 3/5] gitignore: read from index if .gitignore is not in worktree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-07-31T16:55:26Z","receivedAt":"2009-07-31T16:55:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> this allows to use .gitignore in sparse mode, where .gitignore may or\n> may not be checked out (CE_VALID)\n\nHmm, do we have a guarantee that we have read the index when .gitignore is\nused in all the codepath in the current codebase?\n\n\"git add\" used to do the fill_directory() traversal before reading the\nindex in some cases back in v1.6.0 days for a reason. \"git add\" does not\ndo so anymore, but I am wondering if there are others.\n"},{"id":"119309","messageId":"2729632a0907311418m6e60b1b6g9475f7668dec0192@mail.gmail.com","threadId":"20272","inReplyTo":"1248850154-5469-1-git-send-email-pclouds@gmail.com","subject":"Re: [RFC PATCH 0/5] Sparse checkout resurrection","fromName":"","fromEmail":"skillzero@gmail.com","sentAt":"2009-07-31T21:18:26Z","receivedAt":"2009-07-31T21:18:26Z","isPatch":true,"sender":{"key":"skillzero@gmail.com","avatar":null},"body":"2009/7/28 Nguyễn Thái Ngọc Duy :\n> Hi,\n>\n> I rip most of the code from my sparse checkout series,\n> leaving only basic stuff. Now there is no CE_NO_CHECKOUT bit (CE_VALID\n> will be used), no sparse pattterns. You will need to provide a script,\n> namely \"git-shape-workdir\", to specify how you want to shape your\n> workdir. That's all. No porcelain, only plumbing. Need to add some\n> options to read-tree and checkout in order to enable/disable this, but\n> I haven't got that far.\n>\n> The first four patches is more or less CE_VALID fixups. Real stuff is\n> in the last patch. Sparse patterns may return as a separate\n> script. Currently no idea how the porcelain UI should be. The only\n> thing that is probably impossible this way is \"git clone\n> --sparse\". You may need to \"clone -n\" first, then apply sparse\n> checkout later.\n>\n> Comments?\n\nI don't know the git code enough to really comment on the details yet,\nbut I just wanted to say that I'd love to see sparse checkout support\nin git. I tried your other patch series using a config variable\ninstead of the \"git-shape-workdir\" script and I liked it (although I\ndidn't thoroughly test it). It shouldn't be hard to write a\ngit-shape-workdir script to parse a modules file to decide which files\nto include/exclude (I'm hoping to do that soon).\n\nJudging by the surprisingly large number of people commenting in the\nrecent git survey that they want some form of sparse/partial checkout\nsupport (myself included), I hope something like this patch series\ngets some traction.\n\nBTW...Is there a repository I can clone from that has these changes\nand possibly future enhancements? I checked\n, but I didn't see them.\n"},{"id":"119495","messageId":"fcaeb9bf0908040614r6c1ba65dw98c63a48b82df0aa@mail.gmail.com","threadId":"20272","inReplyTo":"7vljm4st0w.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/5] grep: skip files outside sparse checkout area","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-08-04T13:14:08Z","receivedAt":"2009-08-04T13:14:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2009/7/31 Junio C Hamano :\n>> @@ -522,8 +524,11 @@ static int grep_cache(struct grep_opt *opt, const char **paths, int cached,\n>>                               continue;\n>>                       hit |= grep_sha1(opt, ce->sha1, ce->name, 0);\n>>               }\n>> -             else\n>> +             else {\n>> +                     if (ce->ce_flags & CE_VALID)\n>> +                             continue;\n>>                       hit |= grep_file(opt, ce->name);\n>> +             }\n>>               if (ce_stage(ce)) {\n>>                       do {\n>>                               nr++;\n>\n> I do not think the new code in this hunk will be reached as the other arm\n> of the \"else\" greps in the index when CE_VALID is set.\n>\n> Has this series been tested or self-reviewed yet?\n\nYes and no. It had been tested and self-reviewed back when it was part\nof original series. There the new code was CE_NO_CHECKOUT check, not\nCE_VALID. I grepped through and replaced with CE_VALID to form a new\ndraft in order to get comments about this direction, this time it was\nnot well tested because my focus was in the last patch.\n-- \nDuy\n"},{"id":"119496","messageId":"fcaeb9bf0908040620n6a8292br5e62427bc60b0256@mail.gmail.com","threadId":"20272","inReplyTo":"2729632a0907311418m6e60b1b6g9475f7668dec0192@mail.gmail.com","subject":"Re: [RFC PATCH 0/5] Sparse checkout resurrection","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-08-04T13:20:26Z","receivedAt":"2009-08-04T13:20:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2009/8/1 :\n> BTW...Is there a repository I can clone from that has these changes\n> and possibly future enhancements? I checked\n> , but I didn't see them.\n\nNo. This is far from usable. I think I'll make \"git-shape-workdir\" a\nhook, put an example hook that should cover basic cases and repost.\n-- \nDuy\n"}]}