{"thread":{"id":"55489","subject":"[PATCH 02/10] unpack-trees: make sparse aware","startedAt":"2021-04-13T14:01:50Z","lastAt":"2021-07-15T02:41:51Z","messageCount":215,"participants":["Derrick Stolee via GitGitGadget","Matheus Tavares Bernardino","Derrick Stolee","Elijah Newren","Junio C Hamano","Bagas Sanjaya"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"421893","messageId":"0a3892d2ec9e4acd4cba1c1d0390acc60dc6e50f.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 02/10] unpack-trees: make sparse aware","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:29Z","receivedAt":"2021-04-13T14:01:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs a first step to integrate 'git status' and 'git add' with the sparse\nindex, we must start integrating unpack_trees() with sparse directory\nentries. These changes are currently impossible to trigger because\nunpack_trees() calls ensure_full_index() if command_requires_full_index\nis true. This is the case for all commands at the moment. As we expand\nmore commands to be sparse-aware, we might find that more changes are\nrequired to unpack_trees(). The current changes will suffice for\n'status' and 'add'.\n\nunpack_trees() calls the traverse_trees() API using unpack_callback()\nto decide if we should recurse into a subtree. We must add new abilities\nto skip a subtree if it corresponds to a sparse directory entry.\n\nIt is important to be careful about the trailing directory separator\nthat exists in the sparse directory entries but not in the subtree\npaths.\n\nSigned-off-by: Derrick Stolee \n---\n dir.h | 2 +-\n preload-index.c | 2 ++\n read-cache.c | 3 +++\n unpack-trees.c | 24 ++++++++++++++++++++++--\n 4 files changed, 28 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.h b/dir.h\nindex 51cb0e217247..9d6666f520f3 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -503,7 +503,7 @@ static inline int ce_path_match(struct index_state *istate,\n \t\t\t\tchar *seen)\n {\n \treturn match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen,\n-\t\t\t S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n+\t\t\t S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n }\n \n static inline int dir_path_match(struct index_state *istate,\ndiff --git a/preload-index.c b/preload-index.c\nindex e5529a586366..35e67057ca9b 100644\n--- a/preload-index.c\n+++ b/preload-index.c\n@@ -55,6 +55,8 @@ static void *preload_thread(void *_data)\n \t\t\tcontinue;\n \t\tif (S_ISGITLINK(ce->ce_mode))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n \t\tif (ce_uptodate(ce))\n \t\t\tcontinue;\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/read-cache.c b/read-cache.c\nindex 29ffa9ac5db9..6308234b4838 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\tif (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dddf106d5bd4..9a62e823928a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n@@ -984,6 +991,9 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n+\t/* remove directory separator if a sparse directory entry */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\tce_len--;\n \treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n }\n \n@@ -993,6 +1003,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* If ce is a sparse directory, then allow equality here. */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n@@ -1243,6 +1257,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n \tstruct unpack_trees_options *o = info->data;\n \tconst struct name_entry *p = names;\n+\tunsigned recurse = 1;\n \n \t/* Find first entry with a real name (we could use \"mask\" too) */\n \twhile (!p->mode)\n@@ -1284,12 +1299,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t\t\t}\n \t\t\t\t}\n \t\t\t\tsrc[0] = ce;\n+\n+\t\t\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\trecurse = 0;\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (recurse &&\n+\t unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n@@ -1319,7 +1338,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\tif (recurse &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n \t\t\t\t\t names, info) < 0)\n \t\t\treturn -1;\n \t\treturn mask;\n-- \ngitgitgadget\n\n"},{"id":"421894","messageId":"pull.932.git.1618322497.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":null,"subject":"[PATCH 00/10] Sparse-index: integrate with status and add","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:27Z","receivedAt":"2021-04-13T14:01:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' and 'git add' very fast when a sparse-index is enabled on a\nrepository with cone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * git add -A\n * git add . (and other pathspecs)\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\nDerrick Stolee (10):\n t1092: add tests for status/add and sparse files\n unpack-trees: make sparse aware\n dir.c: accept a directory as part of cone-mode patterns\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n dir: use expand_to_path() for sparse directories\n add: allow operating on a sparse-only index\n pathspec: stop calling ensure_full_index\n t7519: add sparse directories to FS monitor tests\n fsmonitor: test with sparse index\n\n builtin/add.c | 3 +\n builtin/commit.c | 3 +\n dir.c | 5 ++\n dir.h | 2 +-\n pathspec.c | 2 -\n preload-index.c | 2 +\n read-cache.c | 5 +-\n t/t1092-sparse-checkout-compatibility.sh | 73 +++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 65 +++++++++++++++++++++\n unpack-trees.c | 24 +++++++-\n wt-status.c | 14 ++++-\n wt-status.h | 1 +\n 12 files changed, 186 insertions(+), 13 deletions(-)\n\n\nbase-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n-- \ngitgitgadget\n"},{"id":"421895","messageId":"b2cb5401eff83c43ca805a36bf41a28a6ffc3630.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 01/10] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:28Z","receivedAt":"2021-04-13T14:02:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file\n'folder1/a' is shown as modified, but adding it fails. This is new\nbehavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\nentries, 2021-04-08). Before that change, these adds would be silently\nignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 36 ++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 12e6c453024f..6598c12a2069 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -232,6 +232,42 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_all ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" is completely ignored\n+\t# by the sparse-checkout repos. It causes the\n+\t# full repo to have a different staged environment.\n+\ttest_must_fail git -C sparse-checkout add folder1/a &&\n+\ttest_must_fail git -C sparse-index add folder1/a &&\n+\tgit -C full-checkout checkout HEAD -- folder1/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"421896","messageId":"d7d4cad8be0b2a27a332a2796ba0dce92783355f.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 05/10] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:32Z","receivedAt":"2021-04-13T14:02:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nBy running the debugger for 'git status -uno' after that change, we find\ntwo instances of ensure_full_index() that were added for extra safety,\nbut can be removed without issue.\n\nIn refresh_index(), we loop through the index entries. The\nrefresh_cache_ent() method copies the sparse directories into the\nrefreshed index without issue.\n\nThe loop within run_diff_files() skips things that are in stage 0 and\nhave skip-worktree enabled, so seems safe to disable ensure_full_index()\nhere.\n\nThis allows some cases of 'git status' to no longer expand a sparse\nindex to a full one, giving the following performance improvements for\np2000-sparse-checkout-operations.sh:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.38(0.36+0.07) 0.37(0.31+0.10) -2.6%\n2000.3: git status (full-index-v4) 0.38(0.29+0.12) 0.37(0.30+0.11) -2.6%\n2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.37s to\n0.05s improvement of 82%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 2 --\n t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++----\n 3 files changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex cf0c36d1dcb2..e529da7beadd 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 6308234b4838..83e6bdef7604 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1578,8 +1578,6 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e488ef9bd941..380a085f8ec4 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -449,12 +449,16 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status -uno &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"421897","messageId":"e86f874dd41291da66848068e7725a172dee231e.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 04/10] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:31Z","receivedAt":"2021-04-13T14:02:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 6598c12a2069..e488ef9bd941 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -196,6 +196,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 0c8287a023e4..0425169c1895 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e1..ab9cc9d8f032 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"421898","messageId":"28ca717e6526f0b66df696f3237d72b9bee2ffc3.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 03/10] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:30Z","receivedAt":"2021-04-13T14:02:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 166238e79f52..57e22e605cec 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1378,6 +1378,11 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/* Directory requests should be added as if they are a file */\n+\tif (parent_pathname.len > 1 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"421899","messageId":"9d1183ddd280b1e9a580dcbfe9761ceec8d0bc80.1618322498.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 10/10] fsmonitor: test with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:37Z","receivedAt":"2021-04-13T14:02:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the effort to protect uses of the index to operate on a full\nindex, we did not modify fsmonitor.c. This is because it already works\neffectively with only the change to index_name_stage_pos(). The only\nthing left to do is to test that it works correctly.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 23879d967297..306157d48abf 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -78,6 +78,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -400,4 +401,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+test_expect_success 'status succeeds with sparse index' '\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"421900","messageId":"f1a9ce4ef0e56a9be7adbdd81b4eeb6404906eb3.1618322497.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 07/10] add: allow operating on a sparse-only index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:34Z","receivedAt":"2021-04-13T14:02:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDisable command_requires_full_index for 'git add'. This does not require\nany additional removals of ensure_full_index(). The main reason is that\n'git add' discovers changes based on the pathspec and the worktree\nitself. These are then inserted into the index directly, and calls to\nindex_name_pos() or index_file_exists() already call expand_to_path() at\nthe appropriate time to support a sparse-index.\n\nAdd a test to check that 'git add -A' and 'git add ' does not\nexpand the index at all, as long as is not within a sparse\ndirectory. This does not help the global 'git add .' case.\n\nWe can measure the improvement using p2000-sparse-operations.sh with\nthese results:\n\nTest HEAD~1 HEAD\n------------------------------------------------------------------------------\n2000.6: git add -A (full-index-v3) 1.35(1.00+0.20) 1.33(0.98+0.19) -1.5%\n2000.7: git add -A (full-index-v4) 1.25(0.97+0.17) 1.23(0.96+0.16) -1.6%\n2000.8: git add -A (sparse-index-v3) 2.38(2.28+0.13) 0.06(0.04+0.08) -97.5%\n2000.9: git add -A (sparse-index-v4) 2.39(2.25+0.18) 0.06(0.04+0.07) -97.5%\n\nWhile the 97% improvement seems impressive, it's important to recognize\nthat previously we had significant overhead for expanding the\nsparse-index. Comparing to the full index case, 'git add -A' goes from\n1.33s to 0.06s, which is \"only\" a 95% improvement.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 3 +++\n t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 58ee3f954ef7..0572d0344065 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -526,6 +526,9 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \tadd_new_files = !take_worktree_changes && !refresh_only && !add_renormalize;\n \trequire_pathspec = !(take_worktree_changes || (0 < addremove_explicit));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n \n \t/*\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b937d7096afd..c210dba78067 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,6 +459,18 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\n+\trm trace2.txt &&\n+\techo >>sparse-index/README.md &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index add -A &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\n+\trm trace2.txt &&\n+\techo >>sparse-index/extra.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index add extra.txt &&\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"421901","messageId":"6d7f30f2b90ac2151645a29c6a12639074c8f2d3.1618322498.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 08/10] pathspec: stop calling ensure_full_index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:35Z","receivedAt":"2021-04-13T14:02:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe add_pathspec_matches_against_index() focuses on matching a pathspec\nto file entries in the index. This already works correctly for its only\nuse: checking if untracked files exist in the index.\n\nThe compatibility checks in t1092 already test that 'git add '\nworks for a directory outside of the sparse cone. That provides coverage\nfor removing this guard.\n\nThis finalizes our ability to run 'git add .' without expanding a sparse\nindex to a full one. This is evidenced by an update to t1092 and by\nthese performance numbers for p2000-sparse-operations.sh:\n\nTest HEAD~1 HEAD\n--------------------------------------------------------------------------------\n2000.10: git add . (full-index-v3) 1.37(1.02+0.18) 1.38(1.01+0.20) +0.7%\n2000.11: git add . (full-index-v4) 1.26(1.00+0.15) 1.27(0.99+0.17) +0.8%\n2000.12: git add . (sparse-index-v3) 2.39(2.29+0.14) 0.06(0.05+0.07) -97.5%\n2000.13: git add . (sparse-index-v4) 2.42(2.32+0.14) 0.06(0.05+0.06) -97.5%\n\nWhile the 97% improvement is shown by the test results, it is worth\nnoting that expanding the sparse index was adding overhead in previous\ncommits. Comparing to the full index case, we see the performance go\nfrom 1.27s to 0.06s, a 95% improvement.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 2 --\n t/t1092-sparse-checkout-compatibility.sh | 6 ++++++\n 2 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex 54813c0c4e8e..b51b48471fe6 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\t\tnum_unmatched++;\n \tif (!num_unmatched)\n \t\treturn;\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c210dba78067..738013b00191 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -471,6 +471,12 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/extra.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index add extra.txt &&\n+\ttest_region ! index ensure_full_index trace2.txt &&\n+\n+\trm trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index add . &&\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"421902","messageId":"75199bbe8ca179b7d2e2b09831af0d479bd28e6e.1618322498.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH 09/10] t7519: add sparse directories to FS monitor tests","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-13T14:01:36Z","receivedAt":"2021-04-13T14:02:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe File System Monitor (FS Monitor) tests in t7519 demonstrate some\nimportant interactions with the index and the response from the FS\nMonitor hook. Later changes will integrate the FS Monitor extension in\nthe index with the existence of sparse directory entries in a sparse\nindex. To do so, we need to include directories outside of the sparse\ncheckout definition.\n\nAdd a new directory, dir1a, between dir1 and dir2 in the test repo used\nby this script. By inserting it in the middle, we are more likely to\ntrigger incorrect behavior when the fsmonitor_dirty bitmap is involved\nwith sparse directories changing the position of cache entries.\n\nI could have modified the test to create two repos, one sparse and one\nnot, but that causes confusion in the expected output. Further, it makes\nthe test take twice as long. With this approach, we can validate that FS\nMonitor works with the sparse index feature using the\nGIT_TEST_SPARSE_INDEX=1 environment variable. The test currently fails\nwith that environment variable because FS Monitor is disabled when a\nsparse index exists. The following changes will update this behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 45d025f96010..23879d967297 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -62,11 +62,16 @@ test_expect_success 'setup' '\n \tmkdir dir1 &&\n \t: >dir1/tracked &&\n \t: >dir1/modified &&\n+\tmkdir dir1a &&\n+\t: >dir1a/a &&\n+\t: >dir1a/b &&\n \tmkdir dir2 &&\n \t: >dir2/tracked &&\n \t: >dir2/modified &&\n \tgit -c core.fsmonitor= add . &&\n \tgit -c core.fsmonitor= commit -m initial &&\n+\tgit sparse-checkout init --cone --no-sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n \tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n \tcat >.gitignore <<-\\EOF\n \t.gitignore\n@@ -99,6 +104,8 @@ test_expect_success 'update-index --no-fsmonitor\" removes the fsmonitor extensio\n cat >expect <expect <expect <expect <expect <expect <\n\nThe recently-implemented expand_to_path() method can supply position\nqueries a faster response if they are specifically asking for a path\nwithin the sparse cone. Since this is the most-common scenario, this\nprovides a significant speedup.\n\nUpdate t1092-sparse-checkout-compatibility.sh to fully ensure that 'git\nstatus' does not expand a sparse index to a full one, even when there\nexist untracked files.\n\nThe performance test script p2000-sparse-operations.sh demonstrates\nthat this is the final hole to fill to allow 'git status' to speed up\nwhen using a sparse index:\n\nTest HEAD~1 HEAD\n------------------------------------------------------------------------------\n2000.4: git status (sparse-index-v3) 1.50(1.43+0.10) 0.04(0.04+0.03) -97.3%\n2000.5: git status (sparse-index-v4) 1.50(1.43+0.10) 0.04(0.03+0.04) -97.3%\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 380a085f8ec4..b937d7096afd 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -456,8 +456,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tinit_repos &&\n \n \trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index status -uno &&\n+\t\tgit -C sparse-index status &&\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"421930","messageId":"CAHd-oW7vCKC-XRM=rX37+jQn_XDzjtar9nNHKQ-4OHSZ=2=KFA@mail.gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/10] Sparse-index: integrate with status and add","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-13T20:45:58Z","receivedAt":"2021-04-13T20:46:19Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Stolee\n\nOn Tue, Apr 13, 2021 at 11:02 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' and 'git add' very fast when a sparse-index is enabled on a\n> repository with cone-mode sparse-checkout (and a small populated set).\n>\n> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n\nI just noticed that our ds/sparse-index-protections and\nmt/add-rm-sparse-checkout had a small semantic conflict. It didn't\nappear before, but it does now with this new series.\n\nds/sparse-index-protections added `ensure_full_index()` guards before\nthe loops that traverse over all cache entries. At the same time,\nmt/add-rm-sparse-checkout added yet another one of these loops, at\n`pathspec.c::find_pathspecs_matching_skip_worktree()`. Although the\nnew place didn't get the `ensure_full_index()` guard, all of its\ncallers (in `add` and `rm`) did call `ensure_full_index()` before\ncalling it, so it was fine.\n\nHowever, patches 7 and 8 remove some of these protections in `add`s\ncode. And, as a result, if \"dir\" is a sparse directory entry, `git add\n[--refresh] dir/file` no longer emits the warning added at\nmt/add-rm-sparse-checkout.\n\nAdding `ensure_full_index()` at\n`find_pathspecs_matching_skip_worktree()` fixes the problem. We have\nto consider the performance implications, but they _might_ be\nacceptable as we only call this function when a pathspec given to\n`add` or `rm` does not match any non-ignored file inside the sparse\ncheckout.\n\nAdditionally, the tests I added at t3705 won't catch this problem,\neven when running with GIT_TEST_SPARSE_INDEX=true :( That's because\nthey don't set core.sparseCheckout and core.sparseCheckoutCone, they\nonly set individual index entries with the SKIP_WORKTREE bit. And\ntherefore, the index is always written fully. Perhaps, should I reroll\nmy series using cone mode for these tests?\n\n(And a semi-related question: do you plan on adding\nGIT_TEST_SPARSE_INDEX=true to one of the CI jobs? )\n"},{"id":"421971","messageId":"ddb08fd9-995e-dcb5-9e2c-60e3d761490b@gmail.com","threadId":"55489","inReplyTo":"CAHd-oW7vCKC-XRM=rX37+jQn_XDzjtar9nNHKQ-4OHSZ=2=KFA@mail.gmail.com","subject":"Re: [PATCH 00/10] Sparse-index: integrate with status and add","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-14T16:31:41Z","receivedAt":"2021-04-14T16:31:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/13/2021 4:45 PM, Matheus Tavares Bernardino wrote:\n> Hi, Stolee\n> \n> On Tue, Apr 13, 2021 at 11:02 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n>> status' and 'git add' very fast when a sparse-index is enabled on a\n>> repository with cone-mode sparse-checkout (and a small populated set).\n>>\n>> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n> \n> I just noticed that our ds/sparse-index-protections and\n> mt/add-rm-sparse-checkout had a small semantic conflict. It didn't\n> appear before, but it does now with this new series.\n\nThank you for taking a close look.\n \n> ds/sparse-index-protections added `ensure_full_index()` guards before\n> the loops that traverse over all cache entries. At the same time,\n> mt/add-rm-sparse-checkout added yet another one of these loops, at\n> `pathspec.c::find_pathspecs_matching_skip_worktree()`. Although the\n> new place didn't get the `ensure_full_index()` guard, all of its\n> callers (in `add` and `rm`) did call `ensure_full_index()` before\n> calling it, so it was fine.\n>\n> However, patches 7 and 8 remove some of these protections in `add`s\n> code. And, as a result, if \"dir\" is a sparse directory entry, `git add\n> [--refresh] dir/file` no longer emits the warning added at\n> mt/add-rm-sparse-checkout.\n\nYou are right, it does not emit the warning. I will add a test that\nensures that behavior is the same across the two sparse repos in\nt1092 as part of my v2 in this series.\n \n> Adding `ensure_full_index()` at\n> `find_pathspecs_matching_skip_worktree()` fixes the problem. We have\n> to consider the performance implications, but they _might_ be\n> acceptable as we only call this function when a pathspec given to\n> `add` or `rm` does not match any non-ignored file inside the sparse\n> checkout.\n\nI'll want to do the right thing here to make the warning work, so\nI'll take a look soon.\n\n> Additionally, the tests I added at t3705 won't catch this problem,\n> even when running with GIT_TEST_SPARSE_INDEX=true :( That's because\n> they don't set core.sparseCheckout and core.sparseCheckoutCone, they\n> only set individual index entries with the SKIP_WORKTREE bit. And\n> therefore, the index is always written fully. Perhaps, should I reroll\n> my series using cone mode for these tests?\n\nYour series should not be re-rolled for this. Instead, this is valuable\nfeedback for this series: there is behavior in 'git add' that I am not\nchecking stays the same when the sparse-index is enabled. That's my\nresponsibility and I'll get it fixed.\n \n> (And a semi-related question: do you plan on adding\n> GIT_TEST_SPARSE_INDEX=true to one of the CI jobs? )\n\nI do plan to add that, after things calm down. It won't do much right\nnow because it requires core.sparseCheckout[Cone] to be enabled. Not\nmany tests provide that, so they don't add much coverage. I thought at\none point to adjust the initial repo creation to include a\nsparse-checkout in cone mode, but that would change too many tests.\nI still haven't found the right way to expand the test coverage to\ntake advantage of our deep test suite for this feature.\n\nThanks,\n-Stolee\n"},{"id":"422482","messageId":"CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com","threadId":"55489","inReplyTo":"b2cb5401eff83c43ca805a36bf41a28a6ffc3630.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/10] t1092: add tests for status/add and sparse files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-20T21:52:04Z","receivedAt":"2021-04-20T21:52:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before moving to update 'git status' and 'git add' to work with sparse\n> indexes, add an explicit test that ensures the sparse-index works the\n> same as a normal sparse-checkout when the worktree contains directories\n> and files outside of the sparse cone.\n>\n> Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is\n> not in the sparse cone. When 'folder1/a' is modified, the file\n> 'folder1/a' is shown as modified, but adding it fails. This is new\n> behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\n> entries, 2021-04-08). Before that change, these adds would be silently\n> ignored.\n>\n> Untracked files are fine: adding new files both with 'git add .' and\n> 'git add folder1/' works just as in a full checkout. This may not be\n> entirely desirable, but we are not intending to change behavior at the\n> moment, only document it.\n\nPersonally, I'd say not desirable and we should throw an error just\nlike we do with skip-worktree entries that the user happens to try to\ngit add. I've had reports from users that got confused by what\nhappens after this. I've been meaning to create some patches to fix\nit up, but wanted to avoid getting in the way of the sparse-index work\nand have been a bit tied up on other projects to boot.\n\nI'll note in particular that it's easy for users after running \"git\nadd\" to run other things such as \"git sparse-checkout reapply\" or \"git\nswitch $otherbranch\" and suddenly the file disappears from the working\ntree. From the sparse-checkout machinery that makes sense; this path\ndoesn't match the .git/info/sparse-checkout list of paths, so it\nshould be removed from the working tree. But it's very disorienting\nto users. Especially if some of those commands are side-effects of\nother commands (e.g. our build system invokes \"git sparse-checkout\nreapply\" in various cases, most common of which is that even a simple\n\"git pull\" can bring down code with dependency changes and thus a need\nfor new sparsity rules and whatnot), but it definitely can just happen\nin ways users don't expect with their own commands (e.g. the git\nswitch/checkout example).\n\nThe patch looks good, but it'd be nice if while documenting it we also\nadd a comment that we believe we want to change the behavior (for\nsparse-checkout both with and without sparse-index). It's one of\nthose many paper-cuts we still have.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 36 ++++++++++++++++++++++++\n> 1 file changed, 36 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 12e6c453024f..6598c12a2069 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -232,6 +232,42 @@ test_expect_success 'add, commit, checkout' '\n> test_all_match git checkout -\n> '\n>\n> +test_expect_success 'status/add: outside sparse cone' '\n> + init_repos &&\n> +\n> + # folder1 is at HEAD, but outside the sparse cone\n> + run_on_sparse mkdir folder1 &&\n> + cp initial-repo/folder1/a sparse-checkout/folder1/a &&\n> + cp initial-repo/folder1/a sparse-index/folder1/a &&\n> +\n> + test_sparse_match git status &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> + run_on_all ../edit-contents folder1/a &&\n> + run_on_all ../edit-contents folder1/new &&\n> +\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + # This \"git add folder1/a\" is completely ignored\n> + # by the sparse-checkout repos. It causes the\n> + # full repo to have a different staged environment.\n> + test_must_fail git -C sparse-checkout add folder1/a &&\n> + test_must_fail git -C sparse-index add folder1/a &&\n> + git -C full-checkout checkout HEAD -- folder1/a &&\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/new &&\n> +\n> + run_on_all ../edit-contents folder1/newer &&\n> + test_all_match git add folder1/ &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/newer\n> +'\n> +\n> test_expect_success 'checkout and reset --hard' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n>\n"},{"id":"422492","messageId":"CABPp-BHcdhO_kEdqR23sDdGAgoSu2R-HBWv-RmzQvJ0ysWtzxg@mail.gmail.com","threadId":"55489","inReplyTo":"0a3892d2ec9e4acd4cba1c1d0390acc60dc6e50f.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-20T23:00:35Z","receivedAt":"2021-04-20T23:00:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> As a first step to integrate 'git status' and 'git add' with the sparse\n> index, we must start integrating unpack_trees() with sparse directory\n> entries. These changes are currently impossible to trigger because\n> unpack_trees() calls ensure_full_index() if command_requires_full_index\n> is true. This is the case for all commands at the moment. As we expand\n> more commands to be sparse-aware, we might find that more changes are\n> required to unpack_trees(). The current changes will suffice for\n> 'status' and 'add'.\n>\n> unpack_trees() calls the traverse_trees() API using unpack_callback()\n> to decide if we should recurse into a subtree. We must add new abilities\n> to skip a subtree if it corresponds to a sparse directory entry.\n>\n> It is important to be careful about the trailing directory separator\n> that exists in the sparse directory entries but not in the subtree\n> paths.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.h | 2 +-\n> preload-index.c | 2 ++\n> read-cache.c | 3 +++\n> unpack-trees.c | 24 ++++++++++++++++++++++--\n> 4 files changed, 28 insertions(+), 3 deletions(-)\n>\n> diff --git a/dir.h b/dir.h\n> index 51cb0e217247..9d6666f520f3 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -503,7 +503,7 @@ static inline int ce_path_match(struct index_state *istate,\n> char *seen)\n> {\n> return match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen,\n> - S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n> + S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n\nI'm confused why this change would be needed, or why it'd semantically\nbe meaningful here either. Doesn't S_ISSPARSEDIR() being true imply\nS_ISDIR() is true (and perhaps even vice versa?).\n\nBy chance, was this a leftover from your early RFC changes from a few\nseries ago when you had an entirely different mode for sparse\ndirectory entries?\n\n> }\n>\n> static inline int dir_path_match(struct index_state *istate,\n> diff --git a/preload-index.c b/preload-index.c\n> index e5529a586366..35e67057ca9b 100644\n> --- a/preload-index.c\n> +++ b/preload-index.c\n> @@ -55,6 +55,8 @@ static void *preload_thread(void *_data)\n> continue;\n> if (S_ISGITLINK(ce->ce_mode))\n> continue;\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + continue;\n> if (ce_uptodate(ce))\n> continue;\n> if (ce_skip_worktree(ce))\n\nDon't we have S_ISSPARSEDIR(ce->ce_mode) implies ce_skip_worktree(ce)?\n Is this a duplicate check? If so, is it still desirable for\nfuture-proofing or code clarity, or is it strictly redundant?\n\n> diff --git a/read-cache.c b/read-cache.c\n> index 29ffa9ac5db9..6308234b4838 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> continue;\n>\n> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> + continue;\n> +\n\nI'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n!ignore_skip_worktree and why it'd be desirable to refresh\nskip-worktree entries. However, this is tangential to your patch and\nhas apparently been around since 2009 (in particular, from 56cac48c35\n(\"ie_match_stat(): do not ignore skip-worktree bit with\nCE_MATCH_IGNORE_VALID\", 2009-12-14)).\n\n> if (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n> filtered = 1;\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index dddf106d5bd4..9a62e823928a 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n> {\n> ce->ce_flags |= CE_UNPACKED;\n>\n> + /*\n> + * If this is a sparse directory, don't advance cache_bottom.\n> + * That will be advanced later using the cache-tree data.\n> + */\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + return;\n> +\n\nI don't understand cache_bottom stuff; we might want to get Junio to\nlook over it. Or maybe I just need to dig a bit further and attempt\nto understand it.\n\n> if (o->cache_bottom < o->src_index->cache_nr &&\n> o->src_index->cache[o->cache_bottom] == ce) {\n> int bottom = o->cache_bottom;\n> @@ -984,6 +991,9 @@ static int do_compare_entry(const struct cache_entry *ce,\n> ce_len -= pathlen;\n> ce_name = ce->name + pathlen;\n>\n> + /* remove directory separator if a sparse directory entry */\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + ce_len--;\n> return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n\nShouldn't we be passing ce->ce_mode instead of S_IFREG here as well?\n\nNote the following sort order:\n foo\n foo.txt\n foo/\n foo/bar\n\nYou've trimmed off the '/', so 'foo/' would be ordered where 'foo' is,\nbut df_name_compare() exists to make \"foo\" sort exactly where \"foo/\"\nwould when \"foo\" is a directory. Will your df_name_compare() call\nhere result in foo.txt being placed after all the \"foo/\"\nentries in the index and perhaps cause other problems down the line?\n(Are there issues, e.g. with cache-trees getting wrong ordering from\nthis, or even writing out indexes or tree objects with the wrong\nordering? I've written out trees to disk with wrong ordering before\nand git usually survives but gets really confused with diffs.)\n\nSince at least one caller of compare_entry() takes the return result\nand does a \"if (cmp < 0)\", this order is going to matter in some\ncases. Perhaps we need some testcases where there is a sparse\ndirectory entry named \"foo/\" and a file recorded in some relevant tree\nwith the name \"foo.txt\" to be able to trigger these lines of code?\n\n> }\n>\n> @@ -993,6 +1003,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n> if (cmp)\n> return cmp;\n>\n> + /* If ce is a sparse directory, then allow equality here. */\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + return 0;\n> +\n\nUm...so a sparse directory compares equal to _anything_ at all? I'm\nreally confused why this would be desirable. Am I missing something\nhere?\n\n> /*\n> * Even if the beginning compared identically, the ce should\n> * compare as bigger than a directory leading up to it!\n> @@ -1243,6 +1257,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> struct unpack_trees_options *o = info->data;\n> const struct name_entry *p = names;\n> + unsigned recurse = 1;\n\n\"recurse\" sent my mind off into questions about safety checks, base\ncases, etc., instead of just the simple \"we don't want to read in\ndirectories corresponding to sparse entries\". I think this would be\nclearer either if the variable had the sparsity concept embedded in\nits name somewhere (e.g. \"unsigned sparse_entry = 0\", and check for\n(!sparse_entry) instead of (recurse) below), or with a comment about\nwhy there are cases where you want to avoid recursion.\n\n>\n> /* Find first entry with a real name (we could use \"mask\" too) */\n> while (!p->mode)\n> @@ -1284,12 +1299,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n> src[0] = ce;\n> +\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + recurse = 0;\n\nAh, the context here doesn't show it but this is in the \"if (!cmp)\"\nblock, i.e. if we found a match for the sparse directory. This makes\nsense, to me, _if_ we ignore the above question about sparse\ndirectories matching equal to anything and everything.\n\n> }\n> break;\n> }\n> }\n>\n> - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> + if (recurse &&\n> + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> return -1;\n>\n> if (o->merge && src[0]) {\n> @@ -1319,7 +1338,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> + if (recurse &&\n> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> names, info) < 0)\n> return -1;\n> return mask;\n\nNice. :-)\n\n\nI think your patch was mostly about the recurse stuff, which other\nthan the name or a comment about it look good to me. However, all the\nother preparatory small tweaks brought up a lot of questions or\nconfusion for me. I'm worried there might be a bug or two, though I\nmay have just misunderstood some of the code bits.\n"},{"id":"422497","messageId":"CABPp-BH8LHA_gvBF2pB2+S_5ttYoesDw_qScNrRiyYBVzPHdGw@mail.gmail.com","threadId":"55489","inReplyTo":"28ca717e6526f0b66df696f3237d72b9bee2ffc3.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 03/10] dir.c: accept a directory as part of cone-mode patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-20T23:21:13Z","receivedAt":"2021-04-20T23:21:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When we have sparse directory entries in the index, we want to compare\n> that directory against sparse-checkout patterns. Those pattern matching\n> algorithms are built expecting a file path, not a directory path. This\n> is especially important in the \"cone mode\" patterns which will match\n> files that exist within the \"parent directories\" as well as the\n> recursive directory matches.\n>\n> If path_matches_pattern_list() is given a directory, we can add a fake\n> filename (\"-\") to the directory and get the same results as before,\n> assuming we are in cone mode. Since sparse index requires cone mode\n> patterns, this is an acceptable assumption.\n\nMakes sense; thanks for the good description.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 5 +++++\n> 1 file changed, 5 insertions(+)\n>\n> diff --git a/dir.c b/dir.c\n> index 166238e79f52..57e22e605cec 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1378,6 +1378,11 @@ enum pattern_match_result path_matches_pattern_list(\n> strbuf_addch(&parent_pathname, '/');\n> strbuf_add(&parent_pathname, pathname, pathlen);\n>\n> + /* Directory requests should be added as if they are a file */\n\n\"added\" or \"matched\"? Also, the description seems a bit brief and\nlikely to surprise; I'd at least want to expand \"file\" to \"file within\ntheir given directory\" but it might be nice to get some summarized\nversion of the commit message or at least state that \"-\" is just a\nrandom simple name within the given directory.\n\n> + if (parent_pathname.len > 1 &&\n\nIs this line...\n\n> + parent_pathname.buf[parent_pathname.len - 1] == '/')\n\nto prevent an out-of-bounds indexing? If so, shouldn't it be \"> 0\" or\n\">= 1\" rather than \"> 1\"? And if so, doesn't the strbuf_addch() call\nabove ensure the condition is always met?\n\nOr are we trying to avoid adding the \"-\" when we parent_pathname is\njust a plain \"/\"?\n\n> + strbuf_add(&parent_pathname, \"-\", 1);\n> +\n\nSorry for all the questions on such a tiny change. It makes sense to\nme, I'm just curious whether it'll confuse future code readers.\n\n\n> if (hashmap_contains_path(&pl->recursive_hashmap,\n> &parent_pathname)) {\n> result = MATCHED_RECURSIVE;\n> --\n"},{"id":"422498","messageId":"CABPp-BHVSPLsXT36=pyrXRR_d9oWCs2aFhA0jh6_u4DzVcTMDg@mail.gmail.com","threadId":"55489","inReplyTo":"e86f874dd41291da66848068e7725a172dee231e.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/10] status: skip sparse-checkout percentage with sparse-index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-20T23:26:28Z","receivedAt":"2021-04-20T23:26:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> 'git status' began reporting a percentage of populated paths when\n> sparse-checkout is enabled in 051df3cf (wt-status: show sparse\n> checkout status as well, 2020-07-18). This percentage is incorrect when\n> the index has sparse directories. It would also be expensive to\n> calculate as we would need to parse trees to count the total number of\n> possible paths.\n>\n> Avoid the expensive computation by simplifying the output to only report\n> that a sparse checkout exists, without the percentage.\n\nMakes sense. The percentage wasn't critical, it was just a nice UI\nbonus. The critical part is notifying about being in a sparse\ncheckout.\n\nIt makes me wonder slightly if we'd want to remove the percentage for\nboth modes just to keep them more similar. I'll ask some folks for\ntheir thoughts/opinions. Of course, that could always be tweaked\nlater and doesn't necessarily need to go into your series.\n\n> This change is the reason we use 'git status --porcelain=v2' in\n> t1092-sparse-checkout-compatibility.sh. We don't want to ensure that\n> this message is equal across both modes, but instead just the important\n> information about staged, modified, and untracked files are compared.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n> wt-status.c | 14 +++++++++++---\n> wt-status.h | 1 +\n> 3 files changed, 20 insertions(+), 3 deletions(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 6598c12a2069..e488ef9bd941 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -196,6 +196,14 @@ test_expect_success 'status with options' '\n> test_all_match git status --porcelain=v2 -uno\n> '\n>\n> +test_expect_success 'status reports sparse-checkout' '\n> + init_repos &&\n> + git -C sparse-checkout status >full &&\n> + git -C sparse-index status >sparse &&\n> + test_i18ngrep \"You are in a sparse checkout with \" full &&\n> + test_i18ngrep \"You are in a sparse checkout.\" sparse\n> +'\n> +\n> test_expect_success 'add, commit, checkout' '\n> init_repos &&\n>\n> diff --git a/wt-status.c b/wt-status.c\n> index 0c8287a023e4..0425169c1895 100644\n> --- a/wt-status.c\n> +++ b/wt-status.c\n> @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n> if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n> return;\n>\n> - status_printf_ln(s, color,\n> - _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n> - s->state.sparse_checkout_percentage);\n> + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n> + status_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n> + else\n> + status_printf_ln(s, color,\n> + _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n> + s->state.sparse_checkout_percentage);\n> wt_longstatus_print_trailer(s);\n> }\n>\n> @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n> return;\n> }\n>\n> + if (r->index->sparse_index) {\n> + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n> + return;\n> + }\n> +\n> for (i = 0; i < r->index->cache_nr; i++) {\n> struct cache_entry *ce = r->index->cache[i];\n> if (ce_skip_worktree(ce))\n> diff --git a/wt-status.h b/wt-status.h\n> index 0d32799b28e1..ab9cc9d8f032 100644\n> --- a/wt-status.h\n> +++ b/wt-status.h\n> @@ -78,6 +78,7 @@ enum wt_status_format {\n> };\n>\n> #define SPARSE_CHECKOUT_DISABLED -1\n> +#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n>\n> struct wt_status_state {\n> int merge_in_progress;\n> --\n> gitgitgadget\n\nLooks good.\n"},{"id":"422510","messageId":"CABPp-BHzfcCNqhHOnK0gsC1r6XX9KAqMYVHhsAzdLpJj5E5TEw@mail.gmail.com","threadId":"55489","inReplyTo":"d7d4cad8be0b2a27a332a2796ba0dce92783355f.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/10] status: use sparse-index throughout","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T00:44:48Z","receivedAt":"2021-04-21T00:45:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> By testing 'git -c core.fsmonitor= status -uno', we can check for the\n> simplest index operations that can be made sparse-aware. The necessary\n> implementation details are already integrated with sparse-checkout, so\n> modify command_requires_full_index to be zero for cmd_status().\n>\n> By running the debugger for 'git status -uno' after that change, we find\n> two instances of ensure_full_index() that were added for extra safety,\n> but can be removed without issue.\n>\n> In refresh_index(), we loop through the index entries. The\n> refresh_cache_ent() method copies the sparse directories into the\n> refreshed index without issue.\n\nI do see the removal of a call to ensure_full_index() in\nrefresh_index() that you mention in this paragraph in the patch below.\n\nI'm confused, though; I would have thought we wanted to avoid a\nrefresh_cache_ent() call. Also, one of your previous patches added a\n\n if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n continue;\n\ncheck before the code ever gets to the refresh_cache_ent() call, so as\nfar as I can tell, that function won't be called from refresh_entry()\nfor sparse entries. Maybe your commit message here is out-of-date?\nOr am I confused somehow?\n\n> The loop within run_diff_files() skips things that are in stage 0 and\n> have skip-worktree enabled, so seems safe to disable ensure_full_index()\n> here.\n\nUnlike the above, I don't see a removal of a ensure_full_index() call\nin run_diff_files() as claimed by this paragraph. Has the commit\nmessage gotten out of date with refactorings you did while developing\nthis series?\n\n> This allows some cases of 'git status' to no longer expand a sparse\n> index to a full one, giving the following performance improvements for\n> p2000-sparse-checkout-operations.sh:\n>\n> Test HEAD~1 HEAD\n> -----------------------------------------------------------------------------\n> 2000.2: git status (full-index-v3) 0.38(0.36+0.07) 0.37(0.31+0.10) -2.6%\n> 2000.3: git status (full-index-v4) 0.38(0.29+0.12) 0.37(0.30+0.11) -2.6%\n> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n>\n> Note that since HEAD~1 was expanding the sparse index by parsing trees,\n> it was artificially slower than the full index case. Thus, the 98%\n> improvement is misleading, and instead we should celebrate the 0.37s to\n> 0.05s improvement of 82%. This is more indicative of the peformance\n> gains we are expecting by using a sparse index.\n\n82%, very nice. Was this with git.git as the test repository, or some\nother repo? If it's git.git, then we'd actually expect a much bigger\nspeedup for other repositories, as git.git is pretty small.\n\n\n> Note: we are dropping the assignment of core.fsmonitor here. This is not\n> necessary for the test script as we are not altering the config any\n> other way. Correct integration with FS Monitor will be validated in\n> later changes.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/commit.c | 3 +++\n> read-cache.c | 2 --\n> t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++----\n> 3 files changed, 11 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/commit.c b/builtin/commit.c\n> index cf0c36d1dcb2..e529da7beadd 100644\n> --- a/builtin/commit.c\n> +++ b/builtin/commit.c\n> @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n> if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> usage_with_options(builtin_status_usage, builtin_status_options);\n>\n> + prepare_repo_settings(the_repository);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> status_init_config(&s, git_status_config);\n> argc = parse_options(argc, argv, prefix,\n> builtin_status_options,\n> diff --git a/read-cache.c b/read-cache.c\n> index 6308234b4838..83e6bdef7604 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1578,8 +1578,6 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> */\n> preload_index(istate, pathspec, 0);\n> trace2_region_enter(\"index\", \"refresh\", NULL);\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> struct cache_entry *ce, *new_entry;\n> int cache_errno = 0;\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index e488ef9bd941..380a085f8ec4 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -449,12 +449,16 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> git -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n> test_region index convert_to_sparse trace2.txt &&\n> - test_region index ensure_full_index trace2.txt &&\n> + test_region index ensure_full_index trace2.txt\n> +'\n>\n> - rm trace2.txt &&\n> +test_expect_success 'sparse-index is not expanded' '\n> + init_repos &&\n> +\n> + rm -f trace2.txt &&\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> - git -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n> - test_region index ensure_full_index trace2.txt\n> + git -C sparse-index status -uno &&\n> + test_region ! index ensure_full_index trace2.txt\n> '\n>\n> test_done\n> --\n> gitgitgadget\n\nOther than what looks like a couple issues in the commit message, the\nchange looks good to me.\n"},{"id":"422513","messageId":"CABPp-BE2qzvFwp0Gg_o+Sd5oC2itpkLNuhvEv1qWTvbCGUKWhQ@mail.gmail.com","threadId":"55489","inReplyTo":"434306541613cbd0b9bb4ebb3102d97e3df9eb94.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/10] dir: use expand_to_path() for sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T00:52:35Z","receivedAt":"2021-04-21T00:52:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The recently-implemented expand_to_path() method can supply position\n> queries a faster response if they are specifically asking for a path\n> within the sparse cone. Since this is the most-common scenario, this\n> provides a significant speedup.\n>\n> Update t1092-sparse-checkout-compatibility.sh to fully ensure that 'git\n> status' does not expand a sparse index to a full one, even when there\n> exist untracked files.\n>\n> The performance test script p2000-sparse-operations.sh demonstrates\n> that this is the final hole to fill to allow 'git status' to speed up\n> when using a sparse index:\n>\n> Test HEAD~1 HEAD\n> ------------------------------------------------------------------------------\n> 2000.4: git status (sparse-index-v3) 1.50(1.43+0.10) 0.04(0.04+0.03) -97.3%\n> 2000.5: git status (sparse-index-v4) 1.50(1.43+0.10) 0.04(0.03+0.04) -97.3%\n\nUm, I'm confused. In the previous patch you claimed the following speedups:\n\n2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n\nI don't understand why the \"Before\" for this patch claims 1.50 as the\ninitial speed, if the \"After\" for the last patch was 0.04. Should the\nprevious commit message have instead claimed:\n\n2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 1.50(1.43+0.10) -38.3%\n2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 1.50(1.43+0.10) -38.5%\n\n?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 380a085f8ec4..b937d7096afd 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -456,8 +456,9 @@ test_expect_success 'sparse-index is not expanded' '\n> init_repos &&\n>\n> rm -f trace2.txt &&\n> + echo >>sparse-index/untracked.txt &&\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> - git -C sparse-index status -uno &&\n> + git -C sparse-index status &&\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> --\n> gitgitgadget\n\nOh! So, the previous patch was testing without enumerating untracked\nfiles (because it did those slowly), whereas this one enumerates\nuntracked files and is still able to achieve the same performance?\nThis wasn't very clear from the commit message. Maybe I'm just bad at\nreading, but perhaps the commit message could be tweaked slightly to\nmake this more clear?\n"},{"id":"422514","messageId":"CABPp-BHLTKWW_j53P9=mwGpmAiuBUcUvRp2Oaim8T+SB46+h+g@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BE2qzvFwp0Gg_o+Sd5oC2itpkLNuhvEv1qWTvbCGUKWhQ@mail.gmail.com","subject":"Re: [PATCH 06/10] dir: use expand_to_path() for sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T00:53:45Z","receivedAt":"2021-04-21T00:53:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"One more thing:\n\nOn Tue, Apr 20, 2021 at 5:52 PM Elijah Newren wrote:\n>\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > From: Derrick Stolee \n> >\n> > The recently-implemented expand_to_path() method can supply position\n> > queries a faster response if they are specifically asking for a path\n> > within the sparse cone. Since this is the most-common scenario, this\n> > provides a significant speedup.\n> >\n> > Update t1092-sparse-checkout-compatibility.sh to fully ensure that 'git\n> > status' does not expand a sparse index to a full one, even when there\n> > exist untracked files.\n> >\n> > The performance test script p2000-sparse-operations.sh demonstrates\n> > that this is the final hole to fill to allow 'git status' to speed up\n> > when using a sparse index:\n> >\n> > Test HEAD~1 HEAD\n> > ------------------------------------------------------------------------------\n> > 2000.4: git status (sparse-index-v3) 1.50(1.43+0.10) 0.04(0.04+0.03) -97.3%\n> > 2000.5: git status (sparse-index-v4) 1.50(1.43+0.10) 0.04(0.03+0.04) -97.3%\n>\n> Um, I'm confused. In the previous patch you claimed the following speedups:\n>\n> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n>\n> I don't understand why the \"Before\" for this patch claims 1.50 as the\n> initial speed, if the \"After\" for the last patch was 0.04. Should the\n> previous commit message have instead claimed:\n>\n> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 1.50(1.43+0.10) -38.3%\n> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 1.50(1.43+0.10) -38.5%\n>\n> ?\n>\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > t/t1092-sparse-checkout-compatibility.sh | 3 ++-\n> > 1 file changed, 2 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> > index 380a085f8ec4..b937d7096afd 100755\n> > --- a/t/t1092-sparse-checkout-compatibility.sh\n> > +++ b/t/t1092-sparse-checkout-compatibility.sh\n> > @@ -456,8 +456,9 @@ test_expect_success 'sparse-index is not expanded' '\n> > init_repos &&\n> >\n> > rm -f trace2.txt &&\n> > + echo >>sparse-index/untracked.txt &&\n> > GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> > - git -C sparse-index status -uno &&\n> > + git -C sparse-index status &&\n> > test_region ! index ensure_full_index trace2.txt\n> > '\n> >\n> > --\n> > gitgitgadget\n>\n> Oh! So, the previous patch was testing without enumerating untracked\n> files (because it did those slowly), whereas this one enumerates\n> untracked files and is still able to achieve the same performance?\n> This wasn't very clear from the commit message. Maybe I'm just bad at\n> reading, but perhaps the commit message could be tweaked slightly to\n> make this more clear?\n\nWhy is the subject of this commit \"dir: use expand_to_path() ...\" if\nit only touches t1092-sparse-checkout-compatibility.sh?\n"},{"id":"422515","messageId":"CABPp-BHf-Tq7kmoWFGu5jA7j8-MiqhysG-vjVvGfUwKMe9Z2-g@mail.gmail.com","threadId":"55489","inReplyTo":"6d7f30f2b90ac2151645a29c6a12639074c8f2d3.1618322498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/10] pathspec: stop calling ensure_full_index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T00:57:53Z","receivedAt":"2021-04-21T00:58:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The add_pathspec_matches_against_index() focuses on matching a pathspec\n> to file entries in the index. This already works correctly for its only\n> use: checking if untracked files exist in the index.\n>\n> The compatibility checks in t1092 already test that 'git add '\n> works for a directory outside of the sparse cone. That provides coverage\n> for removing this guard.\n>\n> This finalizes our ability to run 'git add .' without expanding a sparse\n> index to a full one. This is evidenced by an update to t1092 and by\n> these performance numbers for p2000-sparse-operations.sh:\n>\n> Test HEAD~1 HEAD\n> --------------------------------------------------------------------------------\n> 2000.10: git add . (full-index-v3) 1.37(1.02+0.18) 1.38(1.01+0.20) +0.7%\n> 2000.11: git add . (full-index-v4) 1.26(1.00+0.15) 1.27(0.99+0.17) +0.8%\n> 2000.12: git add . (sparse-index-v3) 2.39(2.29+0.14) 0.06(0.05+0.07) -97.5%\n> 2000.13: git add . (sparse-index-v4) 2.42(2.32+0.14) 0.06(0.05+0.06) -97.5%\n>\n> While the 97% improvement is shown by the test results, it is worth\n> noting that expanding the sparse index was adding overhead in previous\n> commits. Comparing to the full index case, we see the performance go\n> from 1.27s to 0.06s, a 95% improvement.\n\nThis is awesome. :-)\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> pathspec.c | 2 --\n> t/t1092-sparse-checkout-compatibility.sh | 6 ++++++\n> 2 files changed, 6 insertions(+), 2 deletions(-)\n>\n> diff --git a/pathspec.c b/pathspec.c\n> index 54813c0c4e8e..b51b48471fe6 100644\n> --- a/pathspec.c\n> +++ b/pathspec.c\n> @@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n> num_unmatched++;\n> if (!num_unmatched)\n> return;\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> const struct cache_entry *ce = istate->cache[i];\n> if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index c210dba78067..738013b00191 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -471,6 +471,12 @@ test_expect_success 'sparse-index is not expanded' '\n> echo >>sparse-index/extra.txt &&\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> git -C sparse-index add extra.txt &&\n> + test_region ! index ensure_full_index trace2.txt &&\n> +\n> + rm trace2.txt &&\n> + echo >>sparse-index/untracked.txt &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git -C sparse-index add . &&\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> --\n> gitgitgadget\n>\n"},{"id":"422536","messageId":"CABPp-BGLsmNPjtdhb+SMPcXkHCPNaKcknO7ghUYdBoyrTiymTQ@mail.gmail.com","threadId":"55489","inReplyTo":"9d1183ddd280b1e9a580dcbfe9761ceec8d0bc80.1618322498.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/10] fsmonitor: test with sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T07:00:15Z","receivedAt":"2021-04-21T07:00:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> During the effort to protect uses of the index to operate on a full\n> index, we did not modify fsmonitor.c. This is because it already works\n> effectively with only the change to index_name_stage_pos(). The only\n> thing left to do is to test that it works correctly.\n>\n> These tests are added to demonstrate that the behavior is the same\n> across a full index and a sparse index, but also that file modifications\n> to a tracked directory outside of the sparse cone will trigger\n> ensure_full_index().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n> 1 file changed, 48 insertions(+)\n>\n> diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\n> index 23879d967297..306157d48abf 100755\n> --- a/t/t7519-status-fsmonitor.sh\n> +++ b/t/t7519-status-fsmonitor.sh\n> @@ -78,6 +78,7 @@ test_expect_success 'setup' '\n> expect*\n> actual*\n> marker*\n> + trace2*\n> EOF\n> '\n>\n> @@ -400,4 +401,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n> )\n> '\n>\n> +test_expect_success 'status succeeds with sparse index' '\n> + test_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n> + git status --porcelain=v2 >expect &&\n> + git sparse-checkout init --cone --sparse-index &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git status --porcelain=v2 >actual &&\n> + test_region ! index ensure_full_index trace2.txt &&\n> + test_cmp expect actual &&\n> + rm trace2.txt &&\n> +\n> + write_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> + printf \"last_update_token\\0\"\n> + EOF\n> + git config core.fsmonitor .git/hooks/fsmonitor-test &&\n> + git status --porcelain=v2 >expect &&\n> + git sparse-checkout init --cone --sparse-index &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git status --porcelain=v2 >actual &&\n> + test_region ! index ensure_full_index trace2.txt &&\n> + test_cmp expect actual &&\n> + rm trace2.txt &&\n> +\n> + write_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> + printf \"last_update_token\\0\"\n> + printf \"dir1/modified\\0\"\n> + EOF\n> + git config core.fsmonitor .git/hooks/fsmonitor-test &&\n> + git status --porcelain=v2 >expect &&\n> + git sparse-checkout init --cone --sparse-index &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git status --porcelain=v2 >actual &&\n> + test_region ! index ensure_full_index trace2.txt &&\n> + test_cmp expect actual &&\n> +\n> + write_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> + printf \"last_update_token\\0\"\n> + printf \"dir1a/modified\\0\"\n> + EOF\n> + git config core.fsmonitor .git/hooks/fsmonitor-test &&\n> + git status --porcelain=v2 >expect &&\n> + git sparse-checkout init --cone --sparse-index &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git status --porcelain=v2 >actual &&\n> + test_region index ensure_full_index trace2.txt &&\n> + test_cmp expect actual\n\nThere's a lot of duplicated lines here; would it make sense to have a\nhelper function you call, making it easier to see the differences\nbetween the four subsections of this test? Also, do you want to use\ntest_config instead of git config, so that it automatically gets unset\nat the end of the test?\n"},{"id":"422580","messageId":"184087ce-b560-11e5-1c5d-e25b6bd58127@gmail.com","threadId":"55489","inReplyTo":"CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com","subject":"Re: [PATCH 01/10] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:21:11Z","receivedAt":"2021-04-21T13:24:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 5:52 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n> I'll note in particular that it's easy for users after running \"git\n> add\" to run other things such as \"git sparse-checkout reapply\" or \"git\n> switch $otherbranch\" and suddenly the file disappears from the working\n> tree. From the sparse-checkout machinery that makes sense; this path\n> doesn't match the .git/info/sparse-checkout list of paths, so it\n> should be removed from the working tree. But it's very disorienting\n> to users. Especially if some of those commands are side-effects of\n> other commands (e.g. our build system invokes \"git sparse-checkout\n> reapply\" in various cases, most common of which is that even a simple\n> \"git pull\" can bring down code with dependency changes and thus a need\n> for new sparsity rules and whatnot), but it definitely can just happen\n> in ways users don't expect with their own commands (e.g. the git\n> switch/checkout example).\n> \n> The patch looks good, but it'd be nice if while documenting it we also\n> add a comment that we believe we want to change the behavior (for\n> sparse-checkout both with and without sparse-index). It's one of\n> those many paper-cuts we still have.\n\nI can try to comment on these corner case tests that the behavior is\nnot intended to be permanent, especially when already needing to comment\nhow strange it is acting.\n\nThanks,\n-Stolee\n"},{"id":"422581","messageId":"a777331a-aff6-766f-30da-eef052ad83ee@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHcdhO_kEdqR23sDdGAgoSu2R-HBWv-RmzQvJ0ysWtzxg@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:41:19Z","receivedAt":"2021-04-21T13:41:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> diff --git a/dir.h b/dir.h\n>> index 51cb0e217247..9d6666f520f3 100644\n>> --- a/dir.h\n>> +++ b/dir.h\n>> @@ -503,7 +503,7 @@ static inline int ce_path_match(struct index_state *istate,\n>> char *seen)\n>> {\n>> return match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen,\n>> - S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n>> + S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n> \n> I'm confused why this change would be needed, or why it'd semantically\n> be meaningful here either. Doesn't S_ISSPARSEDIR() being true imply\n> S_ISDIR() is true (and perhaps even vice versa?).\n> \n> By chance, was this a leftover from your early RFC changes from a few\n> series ago when you had an entirely different mode for sparse\n> directory entries?\n\nI will double-check on this with additional testing and debugging.\nYour comments below make it clear that this patch would benefit from\nsome additional splitting.\n\n>> }\n>>\n>> static inline int dir_path_match(struct index_state *istate,\n>> diff --git a/preload-index.c b/preload-index.c\n>> index e5529a586366..35e67057ca9b 100644\n>> --- a/preload-index.c\n>> +++ b/preload-index.c\n>> @@ -55,6 +55,8 @@ static void *preload_thread(void *_data)\n>> continue;\n>> if (S_ISGITLINK(ce->ce_mode))\n>> continue;\n>> + if (S_ISSPARSEDIR(ce->ce_mode))\n>> + continue;\n>> if (ce_uptodate(ce))\n>> continue;\n>> if (ce_skip_worktree(ce))\n> \n> Don't we have S_ISSPARSEDIR(ce->ce_mode) implies ce_skip_worktree(ce)?\n> Is this a duplicate check? If so, is it still desirable for\n> future-proofing or code clarity, or is it strictly redundant?\n\nYou're right, we could skip this one because the ce_skip_worktree(ce)\nis enough to cover this case. I think I created this one because I was\nauditing uses of S_ISGITLINK().\n\n>> diff --git a/read-cache.c b/read-cache.c\n>> index 29ffa9ac5db9..6308234b4838 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n>> if (ignore_skip_worktree && ce_skip_worktree(ce))\n>> continue;\n>>\n>> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n>> + continue;\n>> +\n> \n> I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> !ignore_skip_worktree and why it'd be desirable to refresh\n> skip-worktree entries. However, this is tangential to your patch and\n> has apparently been around since 2009 (in particular, from 56cac48c35\n> (\"ie_match_stat(): do not ignore skip-worktree bit with\n> CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n\nThis is probably better served with a statement like this earlier in\nthe method:\n\n\tif (ignore_skip_worktree)\n\t\tensure_full_index(istate);\n\nIt seems like ignoring the skip worktree bits is a rare occasion and\nit will be worth expanding the index for that case.\n\n>> if (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n>> filtered = 1;\n>>\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index dddf106d5bd4..9a62e823928a 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n>> {\n>> ce->ce_flags |= CE_UNPACKED;\n>>\n>> + /*\n>> + * If this is a sparse directory, don't advance cache_bottom.\n>> + * That will be advanced later using the cache-tree data.\n>> + */\n>> + if (S_ISSPARSEDIR(ce->ce_mode))\n>> + return;\n>> +\n> \n> I don't understand cache_bottom stuff; we might want to get Junio to\n> look over it. Or maybe I just need to dig a bit further and attempt\n> to understand it.\n\nI remember looking very careful at this when I created this (and found\nit worth a comment) but I don't recall enough off the top of my head.\nThis is worth splitting out with a careful message, which will force me\nto reexamine the cache_bottom member.\n\n>> if (o->cache_bottom < o->src_index->cache_nr &&\n>> o->src_index->cache[o->cache_bottom] == ce) {\n>> int bottom = o->cache_bottom;\n>> @@ -984,6 +991,9 @@ static int do_compare_entry(const struct cache_entry *ce,\n>> ce_len -= pathlen;\n>> ce_name = ce->name + pathlen;\n>>\n>> + /* remove directory separator if a sparse directory entry */\n>> + if (S_ISSPARSEDIR(ce->ce_mode))\n>> + ce_len--;\n>> return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n> \n> Shouldn't we be passing ce->ce_mode instead of S_IFREG here as well?\n> \n> Note the following sort order:\n> foo\n> foo.txt\n> foo/\n> foo/bar\n> \n> You've trimmed off the '/', so 'foo/' would be ordered where 'foo' is,\n> but df_name_compare() exists to make \"foo\" sort exactly where \"foo/\"\n> would when \"foo\" is a directory. Will your df_name_compare() call\n> here result in foo.txt being placed after all the \"foo/\"\n> entries in the index and perhaps cause other problems down the line?\n> (Are there issues, e.g. with cache-trees getting wrong ordering from\n> this, or even writing out indexes or tree objects with the wrong\n> ordering? I've written out trees to disk with wrong ordering before\n> and git usually survives but gets really confused with diffs.)\n> \n> Since at least one caller of compare_entry() takes the return result\n> and does a \"if (cmp < 0)\", this order is going to matter in some\n> cases. Perhaps we need some testcases where there is a sparse\n> directory entry named \"foo/\" and a file recorded in some relevant tree\n> with the name \"foo.txt\" to be able to trigger these lines of code?\n\nI will do some testing to find out why removing the separator here was\nnecessary or valuable.\n\n>> }\n>>\n>> @@ -993,6 +1003,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n>> if (cmp)\n>> return cmp;\n>>\n>> + /* If ce is a sparse directory, then allow equality here. */\n>> + if (S_ISSPARSEDIR(ce->ce_mode))\n>> + return 0;\n>> +\n> \n> Um...so a sparse directory compares equal to _anything_ at all? I'm\n> really confused why this would be desirable. Am I missing something\n> here?\n\nThe context is that is removed from the patch is that \"cmp\" is the\nresponse from do_compare_entry(), which does a length-limited comparison.\nIf cmp is non-zero, then we've already returned the difference.\n\nThe rest of the method is checking if the 'info' input is actually a\nparent directory of the _path_ given at this cache entry.\n\n>> /*\n>> * Even if the beginning compared identically, the ce should\n>> * compare as bigger than a directory leading up to it!\n\nThe line after this is:\n\n\treturn ce_namelen(ce) > traverse_path_len(info, tree_entry_len(n));\n\nThis comparison is saying \"these paths match up to the directory specified\nby info and n, but we need 'ce' to be a file within that directory.\" But\nin the case of a sparse directory entry, we can skip this comparison.\n\n>> @@ -1243,6 +1257,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n>> struct unpack_trees_options *o = info->data;\n>> const struct name_entry *p = names;\n>> + unsigned recurse = 1;\n> \n> \"recurse\" sent my mind off into questions about safety checks, base\n> cases, etc., instead of just the simple \"we don't want to read in\n> directories corresponding to sparse entries\". I think this would be\n> clearer either if the variable had the sparsity concept embedded in\n> its name somewhere (e.g. \"unsigned sparse_entry = 0\", and check for\n> (!sparse_entry) instead of (recurse) below), or with a comment about\n> why there are cases where you want to avoid recursion.\n\nI can understand that. This callback is confusing because it _does_\nrecurse, but through a sequence of methods instead of actually calling\nitself.\n\nIt would be better to say something like \"unpack_subdirectories = 1\"\nand disabling it when we are in a sparse directory.\n\n>>\n>> /* Find first entry with a real name (we could use \"mask\" too) */\n>> while (!p->mode)\n>> @@ -1284,12 +1299,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>> }\n>> }\n>> src[0] = ce;\n>> +\n>> + if (S_ISSPARSEDIR(ce->ce_mode))\n>> + recurse = 0;\n> \n> Ah, the context here doesn't show it but this is in the \"if (!cmp)\"\n> block, i.e. if we found a match for the sparse directory. This makes\n> sense, to me, _if_ we ignore the above question about sparse\n> directories matching equal to anything and everything.\n\nI believe that \"anything and everything\" concern has been resolved.\n\n>> @@ -1319,7 +1338,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>> }\n>> }\n>>\n>> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n>> + if (recurse &&\n>> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n>> names, info) < 0)\n>> return -1;\n>> return mask;\n> \n> Nice. :-)\n> \n> \n> I think your patch was mostly about the recurse stuff, which other\n> than the name or a comment about it look good to me. However, all the\n> other preparatory small tweaks brought up a lot of questions or\n> confusion for me. I'm worried there might be a bug or two, though I\n> may have just misunderstood some of the code bits.\n \nThis patch could probably be split up a little to make these things\nclearer. Thanks for bringing up the tricky bits.\n\n-Stolee\n"},{"id":"422582","messageId":"252df328-a615-b2fd-ec52-f37ee4397bc1@gmail.com","threadId":"55489","inReplyTo":"CABPp-BH8LHA_gvBF2pB2+S_5ttYoesDw_qScNrRiyYBVzPHdGw@mail.gmail.com","subject":"Re: [PATCH 03/10] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:47:41Z","receivedAt":"2021-04-21T13:47:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 7:21 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When we have sparse directory entries in the index, we want to compare\n>> that directory against sparse-checkout patterns. Those pattern matching\n>> algorithms are built expecting a file path, not a directory path. This\n>> is especially important in the \"cone mode\" patterns which will match\n>> files that exist within the \"parent directories\" as well as the\n>> recursive directory matches.\n>>\n>> If path_matches_pattern_list() is given a directory, we can add a fake\n>> filename (\"-\") to the directory and get the same results as before,\n>> assuming we are in cone mode. Since sparse index requires cone mode\n>> patterns, this is an acceptable assumption.\n> \n> Makes sense; thanks for the good description.\n> \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> dir.c | 5 +++++\n>> 1 file changed, 5 insertions(+)\n>>\n>> diff --git a/dir.c b/dir.c\n>> index 166238e79f52..57e22e605cec 100644\n>> --- a/dir.c\n>> +++ b/dir.c\n>> @@ -1378,6 +1378,11 @@ enum pattern_match_result path_matches_pattern_list(\n>> strbuf_addch(&parent_pathname, '/');\n>> strbuf_add(&parent_pathname, pathname, pathlen);\n>>\n>> + /* Directory requests should be added as if they are a file */\n> \n> \"added\" or \"matched\"? Also, the description seems a bit brief and\n> likely to surprise; I'd at least want to expand \"file\" to \"file within\n> their given directory\" but it might be nice to get some summarized\n> version of the commit message or at least state that \"-\" is just a\n> random simple name within the given directory.\n\nI can improve this comment.\n\n>> + if (parent_pathname.len > 1 &&\n> \n> Is this line...\n> \n>> + parent_pathname.buf[parent_pathname.len - 1] == '/')\n> \n> to prevent an out-of-bounds indexing? If so, shouldn't it be \"> 0\" or\n> \">= 1\" rather than \"> 1\"? And if so, doesn't the strbuf_addch() call\n> above ensure the condition is always met?\n> \n> Or are we trying to avoid adding the \"-\" when we parent_pathname is\n> just a plain \"/\"?\n\nI believe plain \"/\" is impossible. There needs to be a valid tree entry\nbefore that first slash (\"a/\", for example). But that isn't super\nimportant to the logic here and just adds confusion.\n\n> \n>> + strbuf_add(&parent_pathname, \"-\", 1);\n>> +\n> \n> Sorry for all the questions on such a tiny change. It makes sense to\n> me, I'm just curious whether it'll confuse future code readers.\n\nYes, let's avoid confusion by doing the simple thing and use \"> 0\".\n\nThanks,\n-Stolee\n"},{"id":"422583","messageId":"e5374162-fc30-3e17-deec-4e6988958763@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHVSPLsXT36=pyrXRR_d9oWCs2aFhA0jh6_u4DzVcTMDg@mail.gmail.com","subject":"Re: [PATCH 04/10] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:51:31Z","receivedAt":"2021-04-21T13:51:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 7:26 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> Avoid the expensive computation by simplifying the output to only report\n>> that a sparse checkout exists, without the percentage.\n> \n> Makes sense. The percentage wasn't critical, it was just a nice UI\n> bonus. The critical part is notifying about being in a sparse\n> checkout.\n> \n> It makes me wonder slightly if we'd want to remove the percentage for\n> both modes just to keep them more similar. I'll ask some folks for\n> their thoughts/opinions. Of course, that could always be tweaked\n> later and doesn't necessarily need to go into your series.\n\nI find the percentage helpful for users who are exploring the\nsparse-checkout feature in their repositories. It's nice to know how\nmuch time it is saving, because \"percentage of files\" frequently\ntranslates to \"percentage of time it takes to update the worktree\".\n\nI was sad to lose it here, but I don't see any way to keep it.\n\nThanks,\n-Stolee\n"},{"id":"422584","messageId":"17feba15-a7a9-1948-3e0b-a395df69010e@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHzfcCNqhHOnK0gsC1r6XX9KAqMYVHhsAzdLpJj5E5TEw@mail.gmail.com","subject":"Re: [PATCH 05/10] status: use sparse-index throughout","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T13:55:59Z","receivedAt":"2021-04-21T13:56:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 8:44 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> By testing 'git -c core.fsmonitor= status -uno', we can check for the\n>> simplest index operations that can be made sparse-aware. The necessary\n>> implementation details are already integrated with sparse-checkout, so\n>> modify command_requires_full_index to be zero for cmd_status().\n>>\n>> By running the debugger for 'git status -uno' after that change, we find\n>> two instances of ensure_full_index() that were added for extra safety,\n>> but can be removed without issue.\n>>\n>> In refresh_index(), we loop through the index entries. The\n>> refresh_cache_ent() method copies the sparse directories into the\n>> refreshed index without issue.\n> \n> I do see the removal of a call to ensure_full_index() in\n> refresh_index() that you mention in this paragraph in the patch below.\n> \n> I'm confused, though; I would have thought we wanted to avoid a\n> refresh_cache_ent() call. Also, one of your previous patches added a\n> \n> if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> continue;\n> \n> check before the code ever gets to the refresh_cache_ent() call, so as\n> far as I can tell, that function won't be called from refresh_entry()\n> for sparse entries. Maybe your commit message here is out-of-date?\n> Or am I confused somehow?\n> \n>> The loop within run_diff_files() skips things that are in stage 0 and\n>> have skip-worktree enabled, so seems safe to disable ensure_full_index()\n>> here.\n> \n> Unlike the above, I don't see a removal of a ensure_full_index() call\n> in run_diff_files() as claimed by this paragraph. Has the commit\n> message gotten out of date with refactorings you did while developing\n> this series?\n\nI greatly reduced the number of ensure_full_index() calls in the\nprevious topic (ds/sparse-index-protections) since first writing this\npatch, so it is very likely to be out-of-date. Thanks for calling it out.\n\n>> This allows some cases of 'git status' to no longer expand a sparse\n>> index to a full one, giving the following performance improvements for\n>> p2000-sparse-checkout-operations.sh:\n>>\n>> Test HEAD~1 HEAD\n>> -----------------------------------------------------------------------------\n>> 2000.2: git status (full-index-v3) 0.38(0.36+0.07) 0.37(0.31+0.10) -2.6%\n>> 2000.3: git status (full-index-v4) 0.38(0.29+0.12) 0.37(0.30+0.11) -2.6%\n>> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n>> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n>>\n>> Note that since HEAD~1 was expanding the sparse index by parsing trees,\n>> it was artificially slower than the full index case. Thus, the 98%\n>> improvement is misleading, and instead we should celebrate the 0.37s to\n>> 0.05s improvement of 82%. This is more indicative of the peformance\n>> gains we are expecting by using a sparse index.\n> \n> 82%, very nice. Was this with git.git as the test repository, or some\n> other repo? If it's git.git, then we'd actually expect a much bigger\n> speedup for other repositories, as git.git is pretty small.\nThis test script takes the input repository (git.git in this case) and\ncreates a tree that contains that repository many times over, but only\nfour copies remain in the sparse-checkout definition. This creates the\nbig speedup, because of the enormous difference in index size.\n\nAs I am exploring commands such as 'merge' and 'rebase' I am finding\nthat this test setup is too expensive to cover those commands. I will\nneed to reduce the size of the test repository (by a factor of 4) and\nthat will reduce how impressive these results are while making the more\ncomplicated commands testable in a reasonable amount of time.\n\nThanks,\n-Stolee\n"},{"id":"422585","messageId":"c120e61f-7eed-78f5-c3ac-ae4c1d9f7c14@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHLTKWW_j53P9=mwGpmAiuBUcUvRp2Oaim8T+SB46+h+g@mail.gmail.com","subject":"Re: [PATCH 06/10] dir: use expand_to_path() for sparse directories","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T14:03:58Z","receivedAt":"2021-04-21T14:04:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 8:53 PM, Elijah Newren wrote:\n> One more thing:\n> \n> On Tue, Apr 20, 2021 at 5:52 PM Elijah Newren wrote:\n>>\n>> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n>> wrote:\n>>> Test HEAD~1 HEAD\n>>> ------------------------------------------------------------------------------\n>>> 2000.4: git status (sparse-index-v3) 1.50(1.43+0.10) 0.04(0.04+0.03) -97.3%\n>>> 2000.5: git status (sparse-index-v4) 1.50(1.43+0.10) 0.04(0.03+0.04) -97.3%\n>>\n>> Um, I'm confused. In the previous patch you claimed the following speedups:\n>>\n>> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n>> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n>>\n>> I don't understand why the \"Before\" for this patch claims 1.50 as the\n>> initial speed, if the \"After\" for the last patch was 0.04. Should the\n>> previous commit message have instead claimed:\n>>\n>> 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 1.50(1.43+0.10) -38.3%\n>> 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 1.50(1.43+0.10) -38.5%\n...\n>> Oh! So, the previous patch was testing without enumerating untracked\n>> files (because it did those slowly), whereas this one enumerates\n>> untracked files and is still able to achieve the same performance?\n>> This wasn't very clear from the commit message. Maybe I'm just bad at\n>> reading, but perhaps the commit message could be tweaked slightly to\n>> make this more clear?\n> \n> Why is the subject of this commit \"dir: use expand_to_path() ...\" if\n> it only touches t1092-sparse-checkout-compatibility.sh?\n \nYou are right to be confused. This is another patch that simplified due\nto refactors in the protections branch. This should just be squashed into\nthe previous.\n\nFor context: an earlier version inserted ensure_full_index() before\nevery call to index_name_pos() and then this patch swapped that for\na call to expand_to_path(). The change in the protections branch was\nto have index_name_pos() call expand_to_path() itself, preventing the\nneed for these ensure_full_index() calls.\n\nThanks,\n-Stolee\n"},{"id":"422596","messageId":"CAHd-oW4wq6T1kmBGXUfEO4R8h0dfUAGudKvO0xLgCgm2bQt=qA@mail.gmail.com","threadId":"55489","inReplyTo":"b2cb5401eff83c43ca805a36bf41a28a6ffc3630.1618322497.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/10] t1092: add tests for status/add and sparse files","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-21T15:14:58Z","receivedAt":"2021-04-21T15:15:14Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Stolee\n\nYou already said you will make changes in this test to make sure\ngit-add's sparse warning is kept on a sparse index (BTW thanks for\nthat :), but I just wanted to give a couple suggestions that came to\nmy mind while reading the patch.\n\nOn Tue, Apr 13, 2021 at 11:02 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 12e6c453024f..6598c12a2069 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -232,6 +232,42 @@ test_expect_success 'add, commit, checkout' '\n> test_all_match git checkout -\n> '\n>\n> +test_expect_success 'status/add: outside sparse cone' '\n> + init_repos &&\n> +\n> + # folder1 is at HEAD, but outside the sparse cone\n> + run_on_sparse mkdir folder1 &&\n> + cp initial-repo/folder1/a sparse-checkout/folder1/a &&\n> + cp initial-repo/folder1/a sparse-index/folder1/a &&\n> +\n> + test_sparse_match git status &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> + run_on_all ../edit-contents folder1/a &&\n> + run_on_all ../edit-contents folder1/new &&\n> +\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + # This \"git add folder1/a\" is completely ignored\n> + # by the sparse-checkout repos. It causes the\n> + # full repo to have a different staged environment.\n> +\n> + test_must_fail git -C sparse-checkout add folder1/a &&\n> + test_must_fail git -C sparse-index add folder1/a &&\n\nTo make sure the output is the same, could we collapse these two lines into:\n\ntest_sparse_match test_must_fail git add folder1/a ?\n\nAnd additionally, I think we could repeat this check with `add\n--refresh` and also after removing `folder1/a`. The reason I'm saying\nthis is because the check currently succeeds when `folder1/a` is in\nthe working tree (maybe because `fill_directory()` ends up expanding\nthe sparse index in this case?), but not under the two other\ncircumstances I mentioned (as we've discussed in [1]).\n\n[1]: https://lore.kernel.org/git/CAHd-oW7vCKC-XRM=rX37+jQn_XDzjtar9nNHKQ-4OHSZ=2=KFA@mail.gmail.com/\n\n> + git -C full-checkout checkout HEAD -- folder1/a &&\n> + test_sparse_match git status --porcelain=v2 &&\n\nHmm, shouldn't this be `test_all_match`? IIUC, we've resetted\n`folder1/a` on the full repo to make sure the status report is the\nsame across all repos, right?\n\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/new &&\n> +\n> + run_on_all ../edit-contents folder1/newer &&\n> + test_all_match git add folder1/ &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/newer\n> +'\n> +\n> test_expect_success 'checkout and reset --hard' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n>\n"},{"id":"422597","messageId":"CABPp-BEdXOrDecGZBxmwdMvLBse80s3J+O34Bwz5dRRvFPgnUA@mail.gmail.com","threadId":"55489","inReplyTo":"a777331a-aff6-766f-30da-eef052ad83ee@gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T16:11:37Z","receivedAt":"2021-04-21T16:11:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"// Adding Matheus to cc due to the ignore_skip_worktree bit, given his\nexperience and expertise with the checkout and unpack-trees code.\n\nOn Wed, Apr 21, 2021 at 6:41 AM Derrick Stolee wrote:\n>\n> On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >> diff --git a/dir.h b/dir.h\n> >> index 51cb0e217247..9d6666f520f3 100644\n> >> --- a/dir.h\n> >> +++ b/dir.h\n> >> @@ -503,7 +503,7 @@ static inline int ce_path_match(struct index_state *istate,\n> >> char *seen)\n> >> {\n> >> return match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen,\n> >> - S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n> >> + S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n> >\n> > I'm confused why this change would be needed, or why it'd semantically\n> > be meaningful here either. Doesn't S_ISSPARSEDIR() being true imply\n> > S_ISDIR() is true (and perhaps even vice versa?).\n> >\n> > By chance, was this a leftover from your early RFC changes from a few\n> > series ago when you had an entirely different mode for sparse\n> > directory entries?\n>\n> I will double-check on this with additional testing and debugging.\n> Your comments below make it clear that this patch would benefit from\n> some additional splitting.\n>\n> >> }\n> >>\n> >> static inline int dir_path_match(struct index_state *istate,\n> >> diff --git a/preload-index.c b/preload-index.c\n> >> index e5529a586366..35e67057ca9b 100644\n> >> --- a/preload-index.c\n> >> +++ b/preload-index.c\n> >> @@ -55,6 +55,8 @@ static void *preload_thread(void *_data)\n> >> continue;\n> >> if (S_ISGITLINK(ce->ce_mode))\n> >> continue;\n> >> + if (S_ISSPARSEDIR(ce->ce_mode))\n> >> + continue;\n> >> if (ce_uptodate(ce))\n> >> continue;\n> >> if (ce_skip_worktree(ce))\n> >\n> > Don't we have S_ISSPARSEDIR(ce->ce_mode) implies ce_skip_worktree(ce)?\n> > Is this a duplicate check? If so, is it still desirable for\n> > future-proofing or code clarity, or is it strictly redundant?\n>\n> You're right, we could skip this one because the ce_skip_worktree(ce)\n> is enough to cover this case. I think I created this one because I was\n> auditing uses of S_ISGITLINK().\n>\n> >> diff --git a/read-cache.c b/read-cache.c\n> >> index 29ffa9ac5db9..6308234b4838 100644\n> >> --- a/read-cache.c\n> >> +++ b/read-cache.c\n> >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> >> continue;\n> >>\n> >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> >> + continue;\n> >> +\n> >\n> > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > !ignore_skip_worktree and why it'd be desirable to refresh\n> > skip-worktree entries. However, this is tangential to your patch and\n> > has apparently been around since 2009 (in particular, from 56cac48c35\n> > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n>\n> This is probably better served with a statement like this earlier in\n> the method:\n>\n> if (ignore_skip_worktree)\n> ensure_full_index(istate);\n>\n> It seems like ignoring the skip worktree bits is a rare occasion and\n> it will be worth expanding the index for that case.\n\nMaybe...I read the commit message that introduced the behavior and\nit's not very convincing to me that SKIP_WORKTREE should be ignored\n(it's also not that clear to me what the conditions are; is it just\nupdate-index --really-refresh?); it may be worth double checking on\nthat assumption first, especially given how many other bugs existed\nwith skip_worktree stuff for years. If it's necessary, then I agree\nthat your extra if-check makes sense.\n\nIn particular, I think it'd be really dumb for \"update-index\n--really-refresh\" to read in and populate a huge subdirectory just to\nstat files that don't exist because they are in directories that don't\nexist. And I think there's a pretty good argument to not update stat\ninformation for skip_worktree entries in non-sparse-index cases even\nin the presence of that flag, especially given Matheus' other recent\nchanges in this area (the emails just before we got to the point of\ndiscussing SKIP_WORKTREE and racy clean entries...speaking of which,\nit might be worthwhile pinging Matheus' for opinions on this issue\ntoo.)\n\n> >> if (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n> >> filtered = 1;\n> >>\n> >> diff --git a/unpack-trees.c b/unpack-trees.c\n> >> index dddf106d5bd4..9a62e823928a 100644\n> >> --- a/unpack-trees.c\n> >> +++ b/unpack-trees.c\n> >> @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n> >> {\n> >> ce->ce_flags |= CE_UNPACKED;\n> >>\n> >> + /*\n> >> + * If this is a sparse directory, don't advance cache_bottom.\n> >> + * That will be advanced later using the cache-tree data.\n> >> + */\n> >> + if (S_ISSPARSEDIR(ce->ce_mode))\n> >> + return;\n> >> +\n> >\n> > I don't understand cache_bottom stuff; we might want to get Junio to\n> > look over it. Or maybe I just need to dig a bit further and attempt\n> > to understand it.\n>\n> I remember looking very careful at this when I created this (and found\n> it worth a comment) but I don't recall enough off the top of my head.\n> This is worth splitting out with a careful message, which will force me\n> to reexamine the cache_bottom member.\n>\n> >> if (o->cache_bottom < o->src_index->cache_nr &&\n> >> o->src_index->cache[o->cache_bottom] == ce) {\n> >> int bottom = o->cache_bottom;\n> >> @@ -984,6 +991,9 @@ static int do_compare_entry(const struct cache_entry *ce,\n> >> ce_len -= pathlen;\n> >> ce_name = ce->name + pathlen;\n> >>\n> >> + /* remove directory separator if a sparse directory entry */\n> >> + if (S_ISSPARSEDIR(ce->ce_mode))\n> >> + ce_len--;\n> >> return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n> >\n> > Shouldn't we be passing ce->ce_mode instead of S_IFREG here as well?\n> >\n> > Note the following sort order:\n> > foo\n> > foo.txt\n> > foo/\n> > foo/bar\n> >\n> > You've trimmed off the '/', so 'foo/' would be ordered where 'foo' is,\n> > but df_name_compare() exists to make \"foo\" sort exactly where \"foo/\"\n> > would when \"foo\" is a directory. Will your df_name_compare() call\n> > here result in foo.txt being placed after all the \"foo/\"\n> > entries in the index and perhaps cause other problems down the line?\n> > (Are there issues, e.g. with cache-trees getting wrong ordering from\n> > this, or even writing out indexes or tree objects with the wrong\n> > ordering? I've written out trees to disk with wrong ordering before\n> > and git usually survives but gets really confused with diffs.)\n> >\n> > Since at least one caller of compare_entry() takes the return result\n> > and does a \"if (cmp < 0)\", this order is going to matter in some\n> > cases. Perhaps we need some testcases where there is a sparse\n> > directory entry named \"foo/\" and a file recorded in some relevant tree\n> > with the name \"foo.txt\" to be able to trigger these lines of code?\n>\n> I will do some testing to find out why removing the separator here was\n> necessary or valuable.\n\nI think you removed the separator because df_name_compare() assumes it\ngets a regular filename (i.e. no trailing '/') and manually adds one\nbased on mode for directories. You were probably worried about what\namounts to a non-sensical double '/', but df_name_compare() wouldn't\nactually get to that point unless someone somehow recorded a path\nwithin a git tree object that ended with a trailing '/'. I'd rather\nnot have to worry about the double '/' and explain why it isn't\npossible (or wonder about whether git trees with trailing '/'\ncharacters could be recorded on some OS), so I think the trimming of\nthe separator as you did makes sense.\n\nWhat doesn't make sense to me is that the code just below had a\nhardcoded S_IFREG that it passed to df_name_compare, based on \"this is\na cache entry, and index entries are _always_ regular files\". You\ndidn't change that, even though it's now a false assumption.\nsymlinks, and regular files should be passed as S_IFREG there, I'm not\nsure what should be passed for submodules (though the fact that it's\nbeen using S_IFREG for years suggests maybe that is the mode we want\nfor it, so we can't use ce->ce_mode), and I'm pretty sure sparse\ndirectory entries should be passed as S_IFDIR in order to get the\nsorting right unless you stop stripping the trailing '/' character.\nI'm not exactly sure where the sorting for do_compare_entry() affects\nthe code later, but I tried to trace it out a little in my comments\nabove in order to guide some testing.\n\n> >> }\n> >>\n> >> @@ -993,6 +1003,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n> >> if (cmp)\n> >> return cmp;\n> >>\n> >> + /* If ce is a sparse directory, then allow equality here. */\n> >> + if (S_ISSPARSEDIR(ce->ce_mode))\n> >> + return 0;\n> >> +\n> >\n> > Um...so a sparse directory compares equal to _anything_ at all? I'm\n> > really confused why this would be desirable. Am I missing something\n> > here?\n>\n> The context is that is removed from the patch is that \"cmp\" is the\n> response from do_compare_entry(), which does a length-limited comparison.\n> If cmp is non-zero, then we've already returned the difference.\n>\n> The rest of the method is checking if the 'info' input is actually a\n> parent directory of the _path_ given at this cache entry.\n\nAh, thanks for the explanation. So the only way we get here with\ncmp==0 when we're dealing with a sparse directory entry is if we found\na directory by the same name....\n\n> >> /*\n> >> * Even if the beginning compared identically, the ce should\n> >> * compare as bigger than a directory leading up to it!\n>\n> The line after this is:\n>\n> return ce_namelen(ce) > traverse_path_len(info, tree_entry_len(n));\n>\n> This comparison is saying \"these paths match up to the directory specified\n> by info and n, but we need 'ce' to be a file within that directory.\" But\n> in the case of a sparse directory entry, we can skip this comparison.\n\nIsn't this \"must skip\" rather than \"can skip\"? If we're considering\nthe ce path \"foo/bar/\", then the traverse_path would be \"foo/bar\" and\nwe'd have:\n ce_namelen(ce) == 1 + traverse_path_len(info, tree_entry_len(n))\nso this would return 1 for the comparison making them be treated as\nnon-equal even though they are what we consider equal entries.\n\nIn any event, it seems like this new check could use a better comment\nthan \"then allow equality here\".\n\n> >> @@ -1243,6 +1257,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> >> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> >> struct unpack_trees_options *o = info->data;\n> >> const struct name_entry *p = names;\n> >> + unsigned recurse = 1;\n> >\n> > \"recurse\" sent my mind off into questions about safety checks, base\n> > cases, etc., instead of just the simple \"we don't want to read in\n> > directories corresponding to sparse entries\". I think this would be\n> > clearer either if the variable had the sparsity concept embedded in\n> > its name somewhere (e.g. \"unsigned sparse_entry = 0\", and check for\n> > (!sparse_entry) instead of (recurse) below), or with a comment about\n> > why there are cases where you want to avoid recursion.\n>\n> I can understand that. This callback is confusing because it _does_\n> recurse, but through a sequence of methods instead of actually calling\n> itself.\n>\n> It would be better to say something like \"unpack_subdirectories = 1\"\n> and disabling it when we are in a sparse directory.\n\nI like that name.\n\n>\n> >>\n> >> /* Find first entry with a real name (we could use \"mask\" too) */\n> >> while (!p->mode)\n> >> @@ -1284,12 +1299,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> >> }\n> >> }\n> >> src[0] = ce;\n> >> +\n> >> + if (S_ISSPARSEDIR(ce->ce_mode))\n> >> + recurse = 0;\n> >\n> > Ah, the context here doesn't show it but this is in the \"if (!cmp)\"\n> > block, i.e. if we found a match for the sparse directory. This makes\n> > sense, to me, _if_ we ignore the above question about sparse\n> > directories matching equal to anything and everything.\n>\n> I believe that \"anything and everything\" concern has been resolved.\n\nYes, if we just improve the \"then allow equality here\" comment.\n\n> >> @@ -1319,7 +1338,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> >> }\n> >> }\n> >>\n> >> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> >> + if (recurse &&\n> >> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> >> names, info) < 0)\n> >> return -1;\n> >> return mask;\n> >\n> > Nice. :-)\n> >\n> >\n> > I think your patch was mostly about the recurse stuff, which other\n> > than the name or a comment about it look good to me. However, all the\n> > other preparatory small tweaks brought up a lot of questions or\n> > confusion for me. I'm worried there might be a bug or two, though I\n> > may have just misunderstood some of the code bits.\n>\n> This patch could probably be split up a little to make these things\n> clearer. Thanks for bringing up the tricky bits.\n>\n> -Stolee\n"},{"id":"422606","messageId":"34972851-7d3b-c759-ba87-7022702f74c0@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHcdhO_kEdqR23sDdGAgoSu2R-HBWv-RmzQvJ0ysWtzxg@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-21T17:27:03Z","receivedAt":"2021-04-21T17:27:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> wrote:\n\n>> diff --git a/read-cache.c b/read-cache.c\n>> index 29ffa9ac5db9..6308234b4838 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n>> if (ignore_skip_worktree && ce_skip_worktree(ce))\n>> continue;\n>>\n>> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n>> + continue;\n>> +\n> \n> I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> !ignore_skip_worktree and why it'd be desirable to refresh\n> skip-worktree entries. However, this is tangential to your patch and\n> has apparently been around since 2009 (in particular, from 56cac48c35\n> (\"ie_match_stat(): do not ignore skip-worktree bit with\n> CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n\nI did some more digging on this part here. There has been movement in\nthis space!\n\nThe thing that triggers this ignore_skip_worktree variable inside\nrefresh_index() is now the REFRESH_IGNORE_SKIP_WORKTREE flag which was\nintroduced recently and is set only by builtin/add.c:refresh(), by\nMatheus: a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08).\n\nThis means that we can (for now) keep the behavior the same by adding\n\n\tif (ignore_skip_worktree)\n\t\tensure_full_index(istate);\n\nbefore the loop. This prevents the expansion during 'git status', but\nrequires modification before we are ready for 'git add' to work\ncorrectly. Specifically, 'git add' currently warns only when adding\nsomething that exactly matches a tracked file with SKIP_WORKTREE. It\ndoes _not_ warn when adding something that is untracked but would have\nthe SKIP_WORKTREE bit if it was tracked. We will need to add that\nextra warning if we want to avoid expanding during 'git add'.\n\nAlternatively, we can decide to change the behavior here and send an\nerror() and return failure if they try to add something that would\nlive within a sparse-directory entry. I will think more on this and\nhave a good answer before v2 is ready.\n\nThanks,\n-Stolee\n"},{"id":"422620","messageId":"CAHd-oW76_8D_wu7dixX+2i7-_M3Je=JGk3JonL18yeA5PRrsRg@mail.gmail.com","threadId":"55489","inReplyTo":"34972851-7d3b-c759-ba87-7022702f74c0@gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-21T18:55:33Z","receivedAt":"2021-04-21T18:55:49Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Apr 21, 2021 at 2:27 PM Derrick Stolee wrote:\n>\n> On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > wrote:\n>\n> >> diff --git a/read-cache.c b/read-cache.c\n> >> index 29ffa9ac5db9..6308234b4838 100644\n> >> --- a/read-cache.c\n> >> +++ b/read-cache.c\n> >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> >> continue;\n> >>\n> >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> >> + continue;\n> >> +\n> >\n> > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > !ignore_skip_worktree and why it'd be desirable to refresh\n> > skip-worktree entries. However, this is tangential to your patch and\n> > has apparently been around since 2009 (in particular, from 56cac48c35\n> > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n>\n> I did some more digging on this part here. There has been movement in\n> this space!\n>\n> The thing that triggers this ignore_skip_worktree variable inside\n> refresh_index() is now the REFRESH_IGNORE_SKIP_WORKTREE flag which was\n> introduced recently and is set only by builtin/add.c:refresh(), by\n> Matheus: a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n> 2021-04-08).\n>\n> This means that we can (for now) keep the behavior the same by adding\n>\n> if (ignore_skip_worktree)\n> ensure_full_index(istate);\n>\n> before the loop.\n\nHmm, I don't think we need to expand the index here.\nignore_skip_worktree makes the loop below ignore entries with the\nskip_worktree bit set. Since sparse dirs also have this bit set, we\nwill already get the behavior we want :)\n\nHowever, I think we will need to expand the index at\n`find_pathspecs_matching_against_index()` in order to find and warn\nabout the pathspecs that have matches among skip_worktree entries...\n\n> This prevents the expansion during 'git status', but\n> requires modification before we are ready for 'git add' to work\n> correctly. Specifically, 'git add' currently warns only when adding\n> something that exactly matches a tracked file with SKIP_WORKTREE. It\n> does _not_ warn when adding something that is untracked but would have\n> the SKIP_WORKTREE bit if it was tracked. We will need to add that\n> extra warning if we want to avoid expanding during 'git add'.\n\nHmm, I see :( I was trying to think if it would be possible to do the\npathspec matching (for the warning) without having to expand the\nindex, but then there are the untracked files... If the user gives\n\"a/*/c\" and we have \"a/b/\" as a sparse dir, we don't know if \"a/b/c\"\nis a skip_worktree entry or an untracked file without expanding the\nindex...\n\n> Alternatively, we can decide to change the behavior here and send an\n> error() and return failure if they try to add something that would\n> live within a sparse-directory entry.\n\nI think this behavior would be tricky to replicate on non-sparse-index\nsparse-checkouts, if we were to do that. We would have to pathspec\nmatch each untracked file against the sparsity patterns, perhaps?\n"},{"id":"422622","messageId":"CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com","threadId":"55489","inReplyTo":"34972851-7d3b-c759-ba87-7022702f74c0@gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T18:56:24Z","receivedAt":"2021-04-21T18:56:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Apr 21, 2021 at 10:27 AM Derrick Stolee wrote:\n>\n> On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > wrote:\n>\n> >> diff --git a/read-cache.c b/read-cache.c\n> >> index 29ffa9ac5db9..6308234b4838 100644\n> >> --- a/read-cache.c\n> >> +++ b/read-cache.c\n> >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> >> continue;\n> >>\n> >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> >> + continue;\n> >> +\n> >\n> > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > !ignore_skip_worktree and why it'd be desirable to refresh\n> > skip-worktree entries. However, this is tangential to your patch and\n> > has apparently been around since 2009 (in particular, from 56cac48c35\n> > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n>\n> I did some more digging on this part here. There has been movement in\n> this space!\n>\n> The thing that triggers this ignore_skip_worktree variable inside\n> refresh_index() is now the REFRESH_IGNORE_SKIP_WORKTREE flag which was\n> introduced recently and is set only by builtin/add.c:refresh(), by\n> Matheus: a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n> 2021-04-08).\n>\n> This means that we can (for now) keep the behavior the same by adding\n>\n> if (ignore_skip_worktree)\n> ensure_full_index(istate);\n>\n> before the loop. This prevents the expansion during 'git status', but\n> requires modification before we are ready for 'git add' to work\n> correctly. Specifically, 'git add' currently warns only when adding\n> something that exactly matches a tracked file with SKIP_WORKTREE. It\n> does _not_ warn when adding something that is untracked but would have\n> the SKIP_WORKTREE bit if it was tracked. We will need to add that\n> extra warning if we want to avoid expanding during 'git add'.\n>\n> Alternatively, we can decide to change the behavior here and send an\n> error() and return failure if they try to add something that would\n> live within a sparse-directory entry. I will think more on this and\n> have a good answer before v2 is ready.\n\nSee my comments on 01/10; users are already getting surprised by \"git\nadd\" today and has been going on for months (though not super\nfrequently). When they try to \"git add\" an untracked path that would\nnot match any path specifications in $GIT_DIR/info/sparse-checkout,\nthe fact that \"git add\" doesn't error out (or at the very least give a\nwarning) causes _subsequent_ commands to surprise the user with their\nbehavior; the fact that it is some later command that does weird stuff\n(removing the file from the working tree) makes it harder for them to\ntry to understand and make sense of. So, I'd say we do want to change\nthe behavior here...and not just for sparse-indexes but\nsparse-checkouts in general.\n\nAs for how this affects the code, I think I'm behind both you and\nMatheus on understanding here, but I'm starting to think it was a good\nidea for me to spout my offhand comment on what looked like a funny\ncode smell that I thought was unrelated to your patch. Sounds like it\nis causing some good digging...I'll try to read up more on the results\nwhen you send v2. :-)\n"},{"id":"422624","messageId":"CABPp-BG5uSUHy6bQ7xEWSQ8VAy0GCqX0P670HfcopZPUwAfQcw@mail.gmail.com","threadId":"55489","inReplyTo":"CAHd-oW76_8D_wu7dixX+2i7-_M3Je=JGk3JonL18yeA5PRrsRg@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-21T19:10:58Z","receivedAt":"2021-04-21T19:11:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Apr 21, 2021 at 11:55 AM Matheus Tavares Bernardino\n wrote:\n>\n> On Wed, Apr 21, 2021 at 2:27 PM Derrick Stolee wrote:\n> >\n> > On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > > wrote:\n> >\n> > >> diff --git a/read-cache.c b/read-cache.c\n> > >> index 29ffa9ac5db9..6308234b4838 100644\n> > >> --- a/read-cache.c\n> > >> +++ b/read-cache.c\n> > >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> > >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> > >> continue;\n> > >>\n> > >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> > >> + continue;\n> > >> +\n> > >\n> > > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > > !ignore_skip_worktree and why it'd be desirable to refresh\n> > > skip-worktree entries. However, this is tangential to your patch and\n> > > has apparently been around since 2009 (in particular, from 56cac48c35\n> > > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n> >\n> > I did some more digging on this part here. There has been movement in\n> > this space!\n> >\n> > The thing that triggers this ignore_skip_worktree variable inside\n> > refresh_index() is now the REFRESH_IGNORE_SKIP_WORKTREE flag which was\n> > introduced recently and is set only by builtin/add.c:refresh(), by\n> > Matheus: a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n> > 2021-04-08).\n> >\n> > This means that we can (for now) keep the behavior the same by adding\n> >\n> > if (ignore_skip_worktree)\n> > ensure_full_index(istate);\n> >\n> > before the loop.\n>\n> Hmm, I don't think we need to expand the index here.\n> ignore_skip_worktree makes the loop below ignore entries with the\n> skip_worktree bit set. Since sparse dirs also have this bit set, we\n> will already get the behavior we want :)\n>\n> However, I think we will need to expand the index at\n> `find_pathspecs_matching_against_index()` in order to find and warn\n> about the pathspecs that have matches among skip_worktree entries...\n>\n> > This prevents the expansion during 'git status', but\n> > requires modification before we are ready for 'git add' to work\n> > correctly. Specifically, 'git add' currently warns only when adding\n> > something that exactly matches a tracked file with SKIP_WORKTREE. It\n> > does _not_ warn when adding something that is untracked but would have\n> > the SKIP_WORKTREE bit if it was tracked. We will need to add that\n> > extra warning if we want to avoid expanding during 'git add'.\n>\n> Hmm, I see :( I was trying to think if it would be possible to do the\n> pathspec matching (for the warning) without having to expand the\n> index, but then there are the untracked files... If the user gives\n> \"a/*/c\" and we have \"a/b/\" as a sparse dir, we don't know if \"a/b/c\"\n> is a skip_worktree entry or an untracked file without expanding the\n> index...\n\nI thought Stolee's series added something that could allow us to check\nthat e.g. \"a/b/c\" corresponded to an entry under the sparse directory\n\"a/b/\" and thus is a would-be-sparse entry. Can we use that?\n\n> > Alternatively, we can decide to change the behavior here and send an\n> > error() and return failure if they try to add something that would\n> > live within a sparse-directory entry.\n>\n> I think this behavior would be tricky to replicate on non-sparse-index\n> sparse-checkouts, if we were to do that. We would have to pathspec\n> match each untracked file against the sparsity patterns, perhaps?\n\nBy way of analogy, don't we have to pay the cost of pathspec matching\neach tree entry against the sparsity patterns when doing a checkout\nbefore putting those entries into the index? Since \"git add\" is\ntrying to put new entries into the index, doesn't it make sense for it\nto pay the same cost for the untracked paths it is about to place\nthere?\n\nSure, that can be expensive for non-cone mode, but that's the price\nusers pay for using sparse-checkouts and not using cone mode, and they\npay it every time they try to update the index with some new checkout.\nI think \"git add\" should be treated similarly as another way to update\nthe index -- especially since users will get confused (and have gotten\nconfused) by subsequent commands if we don't do those checks.\n"},{"id":"422630","messageId":"CAHd-oW4xm_Uis6yCNtCJZ9qm7a6Oy4ty-05E1VPYGZvvXqaRwg@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BG5uSUHy6bQ7xEWSQ8VAy0GCqX0P670HfcopZPUwAfQcw@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-21T19:51:45Z","receivedAt":"2021-04-21T19:52:00Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Apr 21, 2021 at 4:11 PM Elijah Newren wrote:\n>\n> On Wed, Apr 21, 2021 at 11:55 AM Matheus Tavares Bernardino\n> wrote:\n> >\n> > On Wed, Apr 21, 2021 at 2:27 PM Derrick Stolee wrote:\n> > >\n> > > On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > > > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > > > wrote:\n> > >\n> > > >> diff --git a/read-cache.c b/read-cache.c\n> > > >> index 29ffa9ac5db9..6308234b4838 100644\n> > > >> --- a/read-cache.c\n> > > >> +++ b/read-cache.c\n> > > >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> > > >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> > > >> continue;\n> > > >>\n> > > >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> > > >> + continue;\n> > > >> +\n> > > >\n> > > > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > > > !ignore_skip_worktree and why it'd be desirable to refresh\n> > > > skip-worktree entries. However, this is tangential to your patch and\n> > > > has apparently been around since 2009 (in particular, from 56cac48c35\n> > > > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > > > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n> > >\n> > > I did some more digging on this part here. There has been movement in\n> > > this space!\n> > >\n> > > The thing that triggers this ignore_skip_worktree variable inside\n> > > refresh_index() is now the REFRESH_IGNORE_SKIP_WORKTREE flag which was\n> > > introduced recently and is set only by builtin/add.c:refresh(), by\n> > > Matheus: a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n> > > 2021-04-08).\n> > >\n> > > This means that we can (for now) keep the behavior the same by adding\n> > >\n> > > if (ignore_skip_worktree)\n> > > ensure_full_index(istate);\n> > >\n> > > before the loop.\n> >\n> > Hmm, I don't think we need to expand the index here.\n> > ignore_skip_worktree makes the loop below ignore entries with the\n> > skip_worktree bit set. Since sparse dirs also have this bit set, we\n> > will already get the behavior we want :)\n> >\n> > However, I think we will need to expand the index at\n> > `find_pathspecs_matching_against_index()` in order to find and warn\n> > about the pathspecs that have matches among skip_worktree entries...\n> >\n> > > This prevents the expansion during 'git status', but\n> > > requires modification before we are ready for 'git add' to work\n> > > correctly. Specifically, 'git add' currently warns only when adding\n> > > something that exactly matches a tracked file with SKIP_WORKTREE. It\n> > > does _not_ warn when adding something that is untracked but would have\n> > > the SKIP_WORKTREE bit if it was tracked. We will need to add that\n> > > extra warning if we want to avoid expanding during 'git add'.\n> >\n> > Hmm, I see :( I was trying to think if it would be possible to do the\n> > pathspec matching (for the warning) without having to expand the\n> > index, but then there are the untracked files... If the user gives\n> > \"a/*/c\" and we have \"a/b/\" as a sparse dir, we don't know if \"a/b/c\"\n> > is a skip_worktree entry or an untracked file without expanding the\n> > index...\n>\n> I thought Stolee's series added something that could allow us to check\n> that e.g. \"a/b/c\" corresponded to an entry under the sparse directory\n> \"a/b/\" and thus is a would-be-sparse entry. Can we use that?\n\nYes, you mean for the warning on untracked paths that would become\nsparse entries, right? The problem I was considering there was the\nwarning on tracked entries only, in which case I'm not sure if it\nwould help.\n\n> > > Alternatively, we can decide to change the behavior here and send an\n> > > error() and return failure if they try to add something that would\n> > > live within a sparse-directory entry.\n> >\n> > I think this behavior would be tricky to replicate on non-sparse-index\n> > sparse-checkouts, if we were to do that. We would have to pathspec\n> > match each untracked file against the sparsity patterns, perhaps?\n>\n> By way of analogy, don't we have to pay the cost of pathspec matching\n> each tree entry against the sparsity patterns when doing a checkout\n> before putting those entries into the index? Since \"git add\" is\n> trying to put new entries into the index, doesn't it make sense for it\n> to pay the same cost for the untracked paths it is about to place\n> there?\n>\n> Sure, that can be expensive for non-cone mode, but that's the price\n> users pay for using sparse-checkouts and not using cone mode, and they\n> pay it every time they try to update the index with some new checkout.\n> I think \"git add\" should be treated similarly as another way to update\n> the index -- especially since users will get confused (and have gotten\n> confused) by subsequent commands if we don't do those checks.\n\nGood point. Yeah, that all makes sense :)\n"},{"id":"422658","messageId":"CAHd-oW4Ei3p9rG5z2DRJMVx2dc5B7LeimnoRvBOsStM42v+9HQ@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BEdXOrDecGZBxmwdMvLBse80s3J+O34Bwz5dRRvFPgnUA@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-04-22T02:24:27Z","receivedAt":"2021-04-22T02:24:44Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Apr 21, 2021 at 1:11 PM Elijah Newren wrote:\n>\n> // Adding Matheus to cc due to the ignore_skip_worktree bit, given his\n> experience and expertise with the checkout and unpack-trees code.\n>\n> On Wed, Apr 21, 2021 at 6:41 AM Derrick Stolee wrote:\n> >\n> > On 4/20/2021 7:00 PM, Elijah Newren wrote:\n> > > On Tue, Apr 13, 2021 at 7:01 AM Derrick Stolee via GitGitGadget\n> > > wrote:\n> > >>\n> > >> diff --git a/read-cache.c b/read-cache.c\n> > >> index 29ffa9ac5db9..6308234b4838 100644\n> > >> --- a/read-cache.c\n> > >> +++ b/read-cache.c\n> > >> @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n> > >> if (ignore_skip_worktree && ce_skip_worktree(ce))\n> > >> continue;\n> > >>\n> > >> + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n> > >> + continue;\n> > >> +\n> > >\n> > > I'm a bit confused about what could trigger ce_skip_worktree(ce) &&\n> > > !ignore_skip_worktree and why it'd be desirable to refresh\n> > > skip-worktree entries.\n\nThe skip-worktree entries are not really refreshed in refresh_index(),\neven when !ignore_skip_worktree (which is the default case; i.e.\nwithout the REFRESH_IGNORE_SKIP_WORKTREE flag).\n\nThis flag (which is currently only used by `git add --refresh`s code\nat `builtin/add.c:refresh()`), just makes refresh_index() skip the\nfollowing operations on skip-worktree entries: pathspec matching,\nmarking the matches on `seen`, checking/warning if unmerged, and\nmarking the entry as up-to-date (i.e. with the in-memory CE_UPTODATE\nbit).\n\nI added this flag in mt/add-rm-in-sparse-checkout and changed\n`builtin/add.c:refresh()` to use it mainly because we needed a `seen`\narray with only matches from non-skip-worktree entries so that we\ncould later decide when to emit the warning. (In fact, the original\nimplementation of the flag only controlled whether sparse matches\nwould be marked on `seen` or not [1])\n\n[1]: https://lore.kernel.org/git/d65b214dd1d83a2e8710a9bbf98477c1929f0d5e.1614138107.git.matheus.bernardino@usp.br/\n\nPerhaps we could alternatively make refresh_index() skip the\npreviously mentioned operations on all skip-worktrees entries\n*unconditionally*. I.e. having, early in the loop:\n\nif (ce_skip_worktree(ce))\n continue;\n\nBut I'm not familiar enough with CE_UPTODATE and how it's used in\ndifferent parts of the code base, so I didn't want to risk introducing\nany bugs at refresh_index() callers that might want/expect the\nfunction to set the CE_UPTODATE bit on the skip-worktree entries. The\ncase of `git add --refresh` was much narrower and easier to analyze,\nand that's what we were interested in for the warning. That's why I\nonly changed the behavior there :)\n\n> > > However, this is tangential to your patch and\n> > > has apparently been around since 2009 (in particular, from 56cac48c35\n> > > (\"ie_match_stat(): do not ignore skip-worktree bit with\n> > > CE_MATCH_IGNORE_VALID\", 2009-12-14)).\n\nNote that the `CE_MATCH_IGNORE_SKIP_WORKTREE` added in this patch does\ncontrol if refresh_cache_ent() will refresh skip-worktree entries, but\nrefresh_index() allways calls this function *without* this flag.\n"},{"id":"422809","messageId":"94a175eb-8b1a-638e-df85-2d33713ab4fb@gmail.com","threadId":"55489","inReplyTo":"CAHd-oW4wq6T1kmBGXUfEO4R8h0dfUAGudKvO0xLgCgm2bQt=qA@mail.gmail.com","subject":"Re: [PATCH 01/10] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-23T20:12:00Z","receivedAt":"2021-04-23T20:12:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/21/2021 11:14 AM, Matheus Tavares Bernardino wrote:\n> Hi, Stolee\n> \n> You already said you will make changes in this test to make sure\n> git-add's sparse warning is kept on a sparse index (BTW thanks for\n> that :), but I just wanted to give a couple suggestions that came to\n> my mind while reading the patch.\n\nI appreciate the suggestions! More tests always help me from\nmaking mistakes, and you are definitely more of a 'git add'\nexpert than me.\n \n>> + test_must_fail git -C sparse-checkout add folder1/a &&\n>> + test_must_fail git -C sparse-index add folder1/a &&\n> \n> To make sure the output is the same, could we collapse these two lines into:\n> \n> test_sparse_match test_must_fail git add folder1/a ?\n\nThis is elegant. I'm sad I didn't think of it earlier.\n\n> And additionally, I think we could repeat this check with `add\n> --refresh` and also after removing `folder1/a`. The reason I'm saying\n> this is because the check currently succeeds when `folder1/a` is in\n> the working tree (maybe because `fill_directory()` ends up expanding\n> the sparse index in this case?), but not under the two other\n> circumstances I mentioned (as we've discussed in [1]).\n> \n> [1]: https://lore.kernel.org/git/CAHd-oW7vCKC-XRM=rX37+jQn_XDzjtar9nNHKQ-4OHSZ=2=KFA@mail.gmail.com/\n\nCan do!\n\n>> + git -C full-checkout checkout HEAD -- folder1/a &&\n>> + test_sparse_match git status --porcelain=v2 &&\n> \n> Hmm, shouldn't this be `test_all_match`? IIUC, we've resetted\n> `folder1/a` on the full repo to make sure the status report is the\n> same across all repos, right?\n\nYes!\n\nThanks,\n-Stolee\n"},{"id":"422811","messageId":"74f21db2-d8ba-3aae-99f6-2712d581f2cb@gmail.com","threadId":"55489","inReplyTo":"CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com","subject":"Re: [PATCH 02/10] unpack-trees: make sparse aware","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-23T20:16:05Z","receivedAt":"2021-04-23T20:16:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/21/2021 2:56 PM, Elijah Newren wrote:\n> On Wed, Apr 21, 2021 at 10:27 AM Derrick Stolee wrote:\n>> Alternatively, we can decide to change the behavior here and send an\n>> error() and return failure if they try to add something that would\n>> live within a sparse-directory entry. I will think more on this and\n>> have a good answer before v2 is ready.\n> \n> See my comments on 01/10; users are already getting surprised by \"git\n> add\" today and has been going on for months (though not super\n> frequently). When they try to \"git add\" an untracked path that would\n> not match any path specifications in $GIT_DIR/info/sparse-checkout,\n> the fact that \"git add\" doesn't error out (or at the very least give a\n> warning) causes _subsequent_ commands to surprise the user with their\n> behavior; the fact that it is some later command that does weird stuff\n> (removing the file from the working tree) makes it harder for them to\n> try to understand and make sense of. So, I'd say we do want to change\n> the behavior here...and not just for sparse-indexes but\n> sparse-checkouts in general.\n> \n> As for how this affects the code, I think I'm behind both you and\n> Matheus on understanding here, but I'm starting to think it was a good\n> idea for me to spout my offhand comment on what looked like a funny\n> code smell that I thought was unrelated to your patch. Sounds like it\n> is causing some good digging...I'll try to read up more on the results\n> when you send v2. :-)\n\nI think there are enough strange thing happening with 'git add' that I\nwant to take some time to figure out the right approach here. In v2, I\nwill delete the changes to builtin/add.c and instead focus on making\n'git status' faster with a sparse-index. The 'git add' improvements\nwill follow in another series after I take enough time to understand\nall of these special modes.\n\nI think this split is especially important if we decide that changing\nthe behavior is the best thing to do here.\n\nThanks,\n-Stolee\n"},{"id":"422821","messageId":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.git.1618322497.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:17Z","receivedAt":"2021-04-23T21:34:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdates in V2\n=============\n\n * Based on the feedback, it is clear that 'git add' will require much more\n careful testing and thought. I'm splitting it out of this series and it\n will return with a follow-up.\n * Test cases are improved, both in coverage and organization.\n * The previous \"unpack-trees: make sparse aware\" patch is split into three\n now.\n * Stale messages based on an old implementation of the \"protections\" topic\n are now fixed.\n * Performance tests were re-run.\n\nDerrick Stolee (8):\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: stop recursing into sparse directories\n dir.c: accept a directory as part of cone-mode patterns\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n fsmonitor: test with sparse index\n\n builtin/commit.c | 3 ++\n dir.c | 11 +++++\n read-cache.c | 10 +++-\n t/t1092-sparse-checkout-compatibility.sh | 61 ++++++++++++++++++++++--\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++\n unpack-trees.c | 25 ++++++++--\n wt-status.c | 14 ++++--\n wt-status.h | 1 +\n 8 files changed, 161 insertions(+), 12 deletions(-)\n\n\nbase-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v1:\n\n 1: b2cb5401eff8 ! 1: 3bac9edae7d8 t1092: add tests for status/add and sparse files\n @@ Commit message\n Untracked files are fine: adding new files both with 'git add .' and\n 'git add folder1/' works just as in a full checkout. This may not be\n entirely desirable, but we are not intending to change behavior at the\n - moment, only document it.\n + moment, only document it. A future change could alter the behavior to\n + be more sensible, and this test could be modified to satisfy the new\n + expected behavior.\n \n Signed-off-by: Derrick Stolee \n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n +\t# This \"git add folder1/a\" is completely ignored\n +\t# by the sparse-checkout repos. It causes the\n +\t# full repo to have a different staged environment.\n -+\ttest_must_fail git -C sparse-checkout add folder1/a &&\n -+\ttest_must_fail git -C sparse-index add folder1/a &&\n ++\t#\n ++\t# This is not a desirable behavior, but this test\n ++\t# ensures that the sparse-index is not the cause\n ++\t# of a behavior change.\n ++\ttest_sparse_match test_must_fail git add folder1/a &&\n ++\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n +\tgit -C full-checkout checkout HEAD -- folder1/a &&\n -+\ttest_sparse_match git status --porcelain=v2 &&\n ++\ttest_all_match git status --porcelain=v2 &&\n +\n +\ttest_all_match git add . &&\n +\ttest_all_match git status --porcelain=v2 &&\n -: ------------ > 2: 19344394379d unpack-trees: preserve cache_bottom\n -: ------------ > 3: 24e71d8c0622 unpack-trees: compare sparse directories correctly\n 2: 0a3892d2ec9e ! 4: d3c8948d0a33 unpack-trees: make sparse aware\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - unpack-trees: make sparse aware\n + unpack-trees: stop recursing into sparse directories\n \n - As a first step to integrate 'git status' and 'git add' with the sparse\n - index, we must start integrating unpack_trees() with sparse directory\n - entries. These changes are currently impossible to trigger because\n - unpack_trees() calls ensure_full_index() if command_requires_full_index\n - is true. This is the case for all commands at the moment. As we expand\n - more commands to be sparse-aware, we might find that more changes are\n - required to unpack_trees(). The current changes will suffice for\n - 'status' and 'add'.\n + When walking trees using traverse_trees_recursive() and\n + unpack_callback(), we must not attempt to walk into a sparse directory\n + entry. There are no index entries within that directory to compare to\n + the tree object at that position, so skip over the entries of that tree.\n \n - unpack_trees() calls the traverse_trees() API using unpack_callback()\n - to decide if we should recurse into a subtree. We must add new abilities\n - to skip a subtree if it corresponds to a sparse directory entry.\n -\n - It is important to be careful about the trailing directory separator\n - that exists in the sparse directory entries but not in the subtree\n - paths.\n + This code is used in many places, so the only way to test it is to start\n + removing the command_requres_full_index option from one builtin at a\n + time and carefully test that its use of unpack_trees() behaves correctly\n + with a sparse-index. Such tests will be added by later changes.\n \n Signed-off-by: Derrick Stolee \n \n - ## dir.h ##\n -@@ dir.h: static inline int ce_path_match(struct index_state *istate,\n - \t\t\t\tchar *seen)\n - {\n - \treturn match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen,\n --\t\t\t S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n -+\t\t\t S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n - }\n - \n - static inline int dir_path_match(struct index_state *istate,\n -\n - ## preload-index.c ##\n -@@ preload-index.c: static void *preload_thread(void *_data)\n - \t\t\tcontinue;\n - \t\tif (S_ISGITLINK(ce->ce_mode))\n - \t\t\tcontinue;\n -+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\t\tcontinue;\n - \t\tif (ce_uptodate(ce))\n - \t\t\tcontinue;\n - \t\tif (ce_skip_worktree(ce))\n -\n - ## read-cache.c ##\n -@@ read-cache.c: int refresh_index(struct index_state *istate, unsigned int flags,\n - \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n - \t\t\tcontinue;\n - \n -+\t\tif (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode))\n -+\t\t\tcontinue;\n -+\n - \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n - \t\t\tfiltered = 1;\n - \n -\n ## unpack-trees.c ##\n -@@ unpack-trees.c: static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n - {\n - \tce->ce_flags |= CE_UNPACKED;\n - \n -+\t/*\n -+\t * If this is a sparse directory, don't advance cache_bottom.\n -+\t * That will be advanced later using the cache-tree data.\n -+\t */\n -+\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\treturn;\n -+\n - \tif (o->cache_bottom < o->src_index->cache_nr &&\n - \t o->src_index->cache[o->cache_bottom] == ce) {\n - \t\tint bottom = o->cache_bottom;\n -@@ unpack-trees.c: static int do_compare_entry(const struct cache_entry *ce,\n - \tce_len -= pathlen;\n - \tce_name = ce->name + pathlen;\n - \n -+\t/* remove directory separator if a sparse directory entry */\n -+\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\tce_len--;\n - \treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n - }\n - \n -@@ unpack-trees.c: static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n - \tif (cmp)\n - \t\treturn cmp;\n - \n -+\t/* If ce is a sparse directory, then allow equality here. */\n -+\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\treturn 0;\n -+\n - \t/*\n - \t * Even if the beginning compared identically, the ce should\n - \t * compare as bigger than a directory leading up to it!\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n \tstruct unpack_trees_options *o = info->data;\n \tconst struct name_entry *p = names;\n -+\tunsigned recurse = 1;\n ++\tunsigned unpack_tree = 1;\n \n \t/* Find first entry with a real name (we could use \"mask\" too) */\n \twhile (!p->mode)\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned l\n \t\t\t\tsrc[0] = ce;\n +\n +\t\t\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\t\t\t\trecurse = 0;\n ++\t\t\t\t\tunpack_tree = 0;\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n \t}\n \n -\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n -+\tif (recurse &&\n ++\tif (unpack_tree &&\n +\t unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned l\n \t\t}\n \n -\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n -+\t\tif (recurse &&\n ++\t\tif (unpack_tree &&\n +\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n \t\t\t\t\t names, info) < 0)\n \t\t\treturn -1;\n 3: 28ca717e6526 ! 5: fd96b71968b6 dir.c: accept a directory as part of cone-mode patterns\n @@ dir.c: enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n -+\t/* Directory requests should be added as if they are a file */\n -+\tif (parent_pathname.len > 1 &&\n ++\t/*\n ++\t * Directory entries are matched if and only if a file\n ++\t * contained immediately within them is matched. For the\n ++\t * case of a directory entry, modify the path to create\n ++\t * a fake filename within this directory, allowing us to\n ++\t * use the file-base matching logic in an equivalent way.\n ++\t */\n ++\tif (parent_pathname.len > 0 &&\n +\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n +\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n +\n 4: e86f874dd412 = 6: 1f4ba56e7416 status: skip sparse-checkout percentage with sparse-index\n 5: d7d4cad8be0b ! 7: 3d09368c0541 status: use sparse-index throughout\n @@ Commit message\n implementation details are already integrated with sparse-checkout, so\n modify command_requires_full_index to be zero for cmd_status().\n \n - By running the debugger for 'git status -uno' after that change, we find\n - two instances of ensure_full_index() that were added for extra safety,\n - but can be removed without issue.\n + In refresh_index(), we loop through the index entries to refresh their\n + stat() information. However, sparse directories have no stat()\n + information to populate. Ignore these entries.\n \n - In refresh_index(), we loop through the index entries. The\n - refresh_cache_ent() method copies the sparse directories into the\n - refreshed index without issue.\n + This allows 'git status' to no longer expand a sparse index to a full\n + one. This is further tested by dropping the \"-uno\" option and adding an\n + untracked file into the worktree.\n \n - The loop within run_diff_files() skips things that are in stage 0 and\n - have skip-worktree enabled, so seems safe to disable ensure_full_index()\n - here.\n -\n - This allows some cases of 'git status' to no longer expand a sparse\n - index to a full one, giving the following performance improvements for\n - p2000-sparse-checkout-operations.sh:\n + The performance test p2000-sparse-checkout-operations.sh demonstrates\n + these improvements:\n \n Test HEAD~1 HEAD\n -----------------------------------------------------------------------------\n - 2000.2: git status (full-index-v3) 0.38(0.36+0.07) 0.37(0.31+0.10) -2.6%\n - 2000.3: git status (full-index-v4) 0.38(0.29+0.12) 0.37(0.30+0.11) -2.6%\n - 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4%\n - 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0%\n + 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n + 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n + 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n + 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n \n Note that since HEAD~1 was expanding the sparse index by parsing trees,\n it was artificially slower than the full index case. Thus, the 98%\n - improvement is misleading, and instead we should celebrate the 0.37s to\n - 0.05s improvement of 82%. This is more indicative of the peformance\n + improvement is misleading, and instead we should celebrate the 0.34s to\n + 0.05s improvement of 85%. This is more indicative of the peformance\n gains we are expecting by using a sparse index.\n \n Note: we are dropping the assignment of core.fsmonitor here. This is not\n @@ read-cache.c: int refresh_index(struct index_state *istate, unsigned int flags,\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n -\t/* TODO: audit for interaction with sparse-index. */\n -\tensure_full_index(istate);\n ++\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n +@@ read-cache.c: int refresh_index(struct index_state *istate, unsigned int flags,\n + \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n + \t\t\tcontinue;\n + \n ++\t\t/*\n ++\t\t * If this entry is a sparse directory, then there isn't\n ++\t\t * any stat() information to update. Ignore the entry.\n ++\t\t */\n ++\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n ++\t\t\tcontinue;\n ++\n + \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n + \t\t\tfiltered = 1;\n + \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is expanded and converted back' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is e\n +\tinit_repos &&\n +\n +\trm -f trace2.txt &&\n ++\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n -\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n -\ttest_region index ensure_full_index trace2.txt\n -+\t\tgit -C sparse-index status -uno &&\n ++\t\tgit -C sparse-index status &&\n +\ttest_region ! index ensure_full_index trace2.txt\n '\n \n 6: 434306541613 < -: ------------ dir: use expand_to_path() for sparse directories\n 7: f1a9ce4ef0e5 < -: ------------ add: allow operating on a sparse-only index\n 8: 6d7f30f2b90a < -: ------------ pathspec: stop calling ensure_full_index\n 9: 75199bbe8ca1 < -: ------------ t7519: add sparse directories to FS monitor tests\n 10: 9d1183ddd280 ! 8: 1fd033a6ebb2 fsmonitor: test with sparse index\n @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/\n \t)\n '\n \n -+test_expect_success 'status succeeds with sparse index' '\n -+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n ++# Usage:\n ++# check_sparse_index_behavior [!]\n ++# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n ++# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n ++check_sparse_index_behavior () {\n +\tgit status --porcelain=v2 >expect &&\n +\tgit sparse-checkout init --cone --sparse-index &&\n ++\tgit sparse-checkout set dir1 dir2 &&\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n +\t\tgit status --porcelain=v2 >actual &&\n -+\ttest_region ! index ensure_full_index trace2.txt &&\n ++\ttest_region $1 index ensure_full_index trace2.txt &&\n +\ttest_cmp expect actual &&\n +\trm trace2.txt &&\n ++\tgit sparse-checkout disable\n ++}\n ++\n ++test_expect_success 'status succeeds with sparse index' '\n ++\tgit reset --hard &&\n ++\n ++\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n ++\tcheck_sparse_index_behavior ! &&\n +\n +\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n +\t\tprintf \"last_update_token\\0\"\n +\tEOF\n +\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n -+\tgit status --porcelain=v2 >expect &&\n -+\tgit sparse-checkout init --cone --sparse-index &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n -+\t\tgit status --porcelain=v2 >actual &&\n -+\ttest_region ! index ensure_full_index trace2.txt &&\n -+\ttest_cmp expect actual &&\n -+\trm trace2.txt &&\n ++\tcheck_sparse_index_behavior ! &&\n +\n +\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n +\t\tprintf \"last_update_token\\0\"\n +\t\tprintf \"dir1/modified\\0\"\n +\tEOF\n -+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n -+\tgit status --porcelain=v2 >expect &&\n -+\tgit sparse-checkout init --cone --sparse-index &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n -+\t\tgit status --porcelain=v2 >actual &&\n -+\ttest_region ! index ensure_full_index trace2.txt &&\n -+\ttest_cmp expect actual &&\n ++\tcheck_sparse_index_behavior ! &&\n +\n ++\tcp -r dir1 dir1a &&\n ++\tgit add dir1a &&\n ++\tgit commit -m \"add dir1a\" &&\n ++\n ++\t# This one modifies outside the sparse-checkout definition\n ++\t# and hence we expect to expand the sparse-index.\n +\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n +\t\tprintf \"last_update_token\\0\"\n +\t\tprintf \"dir1a/modified\\0\"\n +\tEOF\n -+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n -+\tgit status --porcelain=v2 >expect &&\n -+\tgit sparse-checkout init --cone --sparse-index &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n -+\t\tgit status --porcelain=v2 >actual &&\n -+\ttest_region index ensure_full_index trace2.txt &&\n -+\ttest_cmp expect actual\n ++\tcheck_sparse_index_behavior\n +'\n +\n test_done\n\n-- \ngitgitgadget\n"},{"id":"422822","messageId":"3bac9edae7d82ef9fdabbe2f3959e574f79f1dd0.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:18Z","receivedAt":"2021-04-23T21:34:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file\n'folder1/a' is shown as modified, but adding it fails. This is new\nbehavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\nentries, 2021-04-08). Before that change, these adds would be silently\nignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 40 ++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 12e6c453024f..0ec487acd283 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -232,6 +232,46 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_all ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" is completely ignored\n+\t# by the sparse-checkout repos. It causes the\n+\t# full repo to have a different staged environment.\n+\t#\n+\t# This is not a desirable behavior, but this test\n+\t# ensures that the sparse-index is not the cause\n+\t# of a behavior change.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\tgit -C full-checkout checkout HEAD -- folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"422823","messageId":"19344394379dd1ac84f21ae5d72747af5e554668.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:19Z","receivedAt":"2021-04-23T21:34:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dddf106d5bd4..1067db19c9d2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"422824","messageId":"24e71d8c062239be9b995bcc31dd12edf84106e3.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:20Z","receivedAt":"2021-04-23T21:34:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1067db19c9d2..3af797093095 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1000,6 +1002,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/* If ce is a sparse directory, then allow an exact match. */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"422825","messageId":"1f4ba56e741632be5d55effc4ed8b6d40e3863d2.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:23Z","receivedAt":"2021-04-23T21:34:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0ec487acd283..0dc551b25f67 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -196,6 +196,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 0c8287a023e4..0425169c1895 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e1..ab9cc9d8f032 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"422826","messageId":"d3c8948d0a33d81e0d426130c89783df25b06703.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] unpack-trees: stop recursing into sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:21Z","receivedAt":"2021-04-23T21:34:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen walking trees using traverse_trees_recursive() and\nunpack_callback(), we must not attempt to walk into a sparse directory\nentry. There are no index entries within that directory to compare to\nthe tree object at that position, so skip over the entries of that tree.\n\nThis code is used in many places, so the only way to test it is to start\nremoving the command_requres_full_index option from one builtin at a\ntime and carefully test that its use of unpack_trees() behaves correctly\nwith a sparse-index. Such tests will be added by later changes.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3af797093095..67777570f829 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1256,6 +1256,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n \tstruct unpack_trees_options *o = info->data;\n \tconst struct name_entry *p = names;\n+\tunsigned unpack_tree = 1;\n \n \t/* Find first entry with a real name (we could use \"mask\" too) */\n \twhile (!p->mode)\n@@ -1297,12 +1298,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t\t\t}\n \t\t\t\t}\n \t\t\t\tsrc[0] = ce;\n+\n+\t\t\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\tunpack_tree = 0;\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_tree &&\n+\t unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n@@ -1332,7 +1337,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\tif (unpack_tree &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n \t\t\t\t\t names, info) < 0)\n \t\t\treturn -1;\n \t\treturn mask;\n-- \ngitgitgadget\n\n"},{"id":"422827","messageId":"fd96b71968b69a357d2a427b990150bc090e3073.1619213665.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:22Z","receivedAt":"2021-04-23T21:34:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 166238e79f52..ab76ef286495 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"422828","messageId":"3d09368c0541523d92ba272f3401b350f3ba3c0b.1619213666.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:24Z","receivedAt":"2021-04-23T21:34:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex cf0c36d1dcb2..e529da7beadd 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 29ffa9ac5db9..f80e26831b36 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0dc551b25f67..5a8fe88dc894 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -453,12 +453,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"422829","messageId":"1fd033a6ebb2eebc85667920e93a4eeeab1b4fe0.1619213666.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] fsmonitor: test with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-23T21:34:25Z","receivedAt":"2021-04-23T21:34:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the effort to protect uses of the index to operate on a full\nindex, we did not modify fsmonitor.c. This is because it already works\neffectively with only the change to index_name_stage_pos(). The only\nthing left to do is to test that it works correctly.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 48 insertions(+)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 45d025f96010..f70fe961902e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"424409","messageId":"CABPp-BHoy9U21tdPtq==MrjcT3m2w0FpaG46pnnzW+VwcoFUcQ@mail.gmail.com","threadId":"55489","inReplyTo":"24e71d8c062239be9b995bcc31dd12edf84106e3.1619213665.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/8] unpack-trees: compare sparse directories correctly","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-13T03:26:10Z","receivedAt":"2021-05-13T03:26:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Apr 23, 2021 at 2:34 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> As we further integrate the sparse-index into unpack-trees, we need to\n> ensure that we compare sparse directory entries correctly with other\n> entries. This affects searching for an exact path as well as sorting\n> index entries.\n>\n> Sparse directory entries contain the trailing directory separator. This\n> is important for the sorting, in particular. Thus, within\n> do_compare_entry() we stop using S_IFREG in all cases, since sparse\n> directories should use S_IFDIR to indicate that the comparison should\n> treat the entry name as a dirctory.\n>\n> Within compare_entry(), it first calls do_compare_entry() to check the\n> leading portion of the name. When the input path is a directory name, we\n> could match exactly already. Thus, we should return 0 if we have an\n> exact string match on a sparse directory entry.\n\nThanks for splitting up patch 2 from the original series; it's much\neasier to understand these separate patches.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 1067db19c9d2..3af797093095 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n> int pathlen, ce_len;\n> const char *ce_name;\n> int cmp;\n> + unsigned ce_mode;\n>\n> /*\n> * If we have not precomputed the traverse path, it is quicker\n> @@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n> ce_len -= pathlen;\n> ce_name = ce->name + pathlen;\n>\n> - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n> + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n\nAh, so here the fact that S_ISSPARSEDIR is defined as\n #define S_ISSPARSEDIR(m) ((m) == S_IFDIR)\nwhereas S_ISDIR is defined as\n #define S_ISDIR(m) (((m) & S_IFMT) == S_IFDIR)\nturns out to be critically important, because if you used S_ISDIR()\nhere, then we'd get ce_mode = S_IFDIR for submodules and break the\nsorting. S_ISSPARSEDIR() gives us the correct value.\n\n> + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n> }\n>\n> static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n> @@ -1000,6 +1002,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n> if (cmp)\n> return cmp;\n>\n> + /* If ce is a sparse directory, then allow an exact match. */\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + return 0;\n\nI think the comment from the commit message belongs in the code; the\ncomment in the code is too jarring without the more detailed\nexplanation.\n\n> +\n> /*\n> * Even if the beginning compared identically, the ce should\n> * compare as bigger than a directory leading up to it!\n> --\n> gitgitgadget\n"},{"id":"424410","messageId":"CABPp-BFkqK6ccVeV8N8WozsJC0nx05PTjNueqXgZzuURYR3e=A@mail.gmail.com","threadId":"55489","inReplyTo":"d3c8948d0a33d81e0d426130c89783df25b06703.1619213665.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/8] unpack-trees: stop recursing into sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-13T03:31:18Z","receivedAt":"2021-05-13T03:31:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Apr 23, 2021 at 2:34 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When walking trees using traverse_trees_recursive() and\n> unpack_callback(), we must not attempt to walk into a sparse directory\n> entry. There are no index entries within that directory to compare to\n> the tree object at that position, so skip over the entries of that tree.\n>\n> This code is used in many places, so the only way to test it is to start\n> removing the command_requres_full_index option from one builtin at a\n> time and carefully test that its use of unpack_trees() behaves correctly\n> with a sparse-index. Such tests will be added by later changes.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 3af797093095..67777570f829 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1256,6 +1256,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> struct unpack_trees_options *o = info->data;\n> const struct name_entry *p = names;\n> + unsigned unpack_tree = 1;\n>\n> /* Find first entry with a real name (we could use \"mask\" too) */\n> while (!p->mode)\n> @@ -1297,12 +1298,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n> src[0] = ce;\n> +\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + unpack_tree = 0;\n> }\n> break;\n> }\n> }\n>\n> - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> + if (unpack_tree &&\n> + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> return -1;\n>\n> if (o->merge && src[0]) {\n> @@ -1332,7 +1337,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> + if (unpack_tree &&\n> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> names, info) < 0)\n> return -1;\n> return mask;\n> --\n> gitgitgadget\n\nThe splitting of the previous patch looks really good here too, and\nthe variable rename makes it flow nicely. Looking good.\n"},{"id":"424415","messageId":"CABPp-BF0EGo-WLTN45qOu02O4sEzmSCUXT3XeoKQdbTHToNQOw@mail.gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/8] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-13T04:12:34Z","receivedAt":"2021-05-13T04:12:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Apr 23, 2021 at 2:34 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' very fast when a sparse-index is enabled on a repository with\n> cone-mode sparse-checkout (and a small populated set).\n>\n> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n> The latter branch is needed because it changes the behavior of 'git add'\n> around sparse entries, which changes the expectations of a test added in\n> patch 1.\n>\n> The approach here is to audit the places where ensure_full_index() pops up\n> while doing normal commands with pathspecs within the sparse-checkout\n> definition. Each of these are checked and tested. In the end, the\n> sparse-index is integrated with these features:\n>\n> * git status\n> * FS Monitor index extension.\n>\n> The performance tests in p2000-sparse-operations.sh improve by 95% or more,\n> even when compared with the full-index cases, not just the sparse-index\n> cases that previously had extra overhead.\n>\n> Hopefully this is the first example of how ds/sparse-index-protections has\n> done the basic work to do these conversions safely, making them look easier\n> than they seemed when starting this adventure.\n>\n> Thanks, -Stolee\n>\n>\n> Updates in V2\n> =============\n>\n> * Based on the feedback, it is clear that 'git add' will require much more\n> careful testing and thought. I'm splitting it out of this series and it\n> will return with a follow-up.\n> * Test cases are improved, both in coverage and organization.\n> * The previous \"unpack-trees: make sparse aware\" patch is split into three\n> now.\n> * Stale messages based on an old implementation of the \"protections\" topic\n> are now fixed.\n> * Performance tests were re-run.\n\nI read through the topic, both my old comments, the range-diff, and\nthe new patches where the range-diff wasn't enough. I tried to spot\nissues, and was hoping to find problems you alluded to in your recent\ncomments at https://lore.kernel.org/git/05932ebc-04ac-b3c5-a460-5d37d8604fd9@gmail.com/,\nbut I failed to spot them. I hope it has to do with the cache bottom\nstuff that I just don't understand, because otherwise I just missed\nthe problems in my review. I can say that in v2 you fixed the issues\nI did spot in my review of v1.\n\nI'll look forward to v3 to see what it was I missed. If I somehow\ndon't respond soon (in a week at the latest), do feel free to ping me;\nsorry for somehow having this one slip through the cracks.\n"},{"id":"424465","messageId":"CAHd-oW7X07EU+bX7vZQ+6Jr6ghEQ8kQMuFFCFew6T3mYArBTKg@mail.gmail.com","threadId":"55489","inReplyTo":"3bac9edae7d82ef9fdabbe2f3959e574f79f1dd0.1619213665.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] t1092: add tests for status/add and sparse files","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-05-13T12:40:09Z","receivedAt":"2021-05-13T12:40:27Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Apr 23, 2021 at 6:34 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before moving to update 'git status' and 'git add' to work with sparse\n> indexes, add an explicit test that ensures the sparse-index works the\n> same as a normal sparse-checkout when the worktree contains directories\n> and files outside of the sparse cone.\n>\n> Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is\n> not in the sparse cone. When 'folder1/a' is modified, the file\n> 'folder1/a' is shown as modified, but adding it fails.\n\nHmm, I might be doing something wrong, but I think `folder1/a` is not\nshown as modified.\n\n$ git init test\n$ mkdir test/folder1\n$ echo original >test/folder1/a\n$ echo original >test/b\n$ git -C test add . && git -C test commit -m files\n$ git -C test sparse-checkout init --cone --sparse-index\n$ ls test\nb\n$ mkdir test/folder1 && echo modified >test/folder1/a\n$ git -C test status\nOn branch master\nYou are in a sparse checkout with 50% of tracked files present.\nnothing to commit, working tree clean\n\n> This is new\n> behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\n> entries, 2021-04-08). Before that change, these adds would be silently\n> ignored.\n>\n> Untracked files are fine: adding new files both with 'git add .' and\n> 'git add folder1/' works just as in a full checkout. This may not be\n> entirely desirable, but we are not intending to change behavior at the\n> moment, only document it. A future change could alter the behavior to\n> be more sensible, and this test could be modified to satisfy the new\n> expected behavior.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 40 ++++++++++++++++++++++++\n> 1 file changed, 40 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 12e6c453024f..0ec487acd283 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -232,6 +232,46 @@ test_expect_success 'add, commit, checkout' '\n> test_all_match git checkout -\n> '\n>\n> +test_expect_success 'status/add: outside sparse cone' '\n> + init_repos &&\n\nA minor suggestion: before recreating folder1/a, we could also test\nthat `git add folder1/a` will not remove the sparse entry from the\nindex and will properly warn about it on both sparse repos. I.e.\nadding a:\n\n test_sparse_match test_must_fail git add folder1/a\n\n> + # folder1 is at HEAD, but outside the sparse cone\n> + run_on_sparse mkdir folder1 &&\n> + cp initial-repo/folder1/a sparse-checkout/folder1/a &&\n> + cp initial-repo/folder1/a sparse-index/folder1/a &&\n> +\n> + test_sparse_match git status &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> + run_on_all ../edit-contents folder1/a &&\n\nHmm, we modify `folder1/a` in all repos, but we only try adding it on\nthe sparse repos, and then we immediately restore it on the full repo.\nAs we won't use the modified version on the full repo, could this\nperhaps be `run_on_sparse` instead? If so, we could also save the\nlater `git -C full-checkout checkout HEAD -- folder1/a`.\n\n> + run_on_all ../edit-contents folder1/new &&\n> +\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + # This \"git add folder1/a\" is completely ignored\n> + # by the sparse-checkout repos. It causes the\n> + # full repo to have a different staged environment.\n> + #\n> + # This is not a desirable behavior, but this test\n> + # ensures that the sparse-index is not the cause\n> + # of a behavior change.\n\nI'm not sure I understand what the undesirable behavior is in this\nsentence. Is it \"git add folder1/a\" erroring out and not updating\n`folder1/a`? Or the full repo having a different staged environment?\n\n> + test_sparse_match test_must_fail git add folder1/a &&\n> + test_sparse_match test_must_fail git add --refresh folder1/a &&\n> + git -C full-checkout checkout HEAD -- folder1/a &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/new &&\n> +\n> + run_on_all ../edit-contents folder1/newer &&\n> + test_all_match git add folder1/ &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git commit -m folder1/newer\n> +'\n> +\n> test_expect_success 'checkout and reset --hard' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n>\n"},{"id":"424578","messageId":"289809a1-2956-f412-256e-ca92308ac3f5@gmail.com","threadId":"55489","inReplyTo":"CAHd-oW7X07EU+bX7vZQ+6Jr6ghEQ8kQMuFFCFew6T3mYArBTKg@mail.gmail.com","subject":"Re: [PATCH v2 1/8] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-14T12:27:30Z","receivedAt":"2021-05-14T12:27:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/13/2021 8:40 AM, Matheus Tavares Bernardino wrote:\n> On Fri, Apr 23, 2021 at 6:34 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> Before moving to update 'git status' and 'git add' to work with sparse\n>> indexes, add an explicit test that ensures the sparse-index works the\n>> same as a normal sparse-checkout when the worktree contains directories\n>> and files outside of the sparse cone.\n>>\n>> Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is\n>> not in the sparse cone. When 'folder1/a' is modified, the file\n>> 'folder1/a' is shown as modified, but adding it fails.\n> \n> Hmm, I might be doing something wrong, but I think `folder1/a` is not\n> shown as modified.\n> \n> $ git init test\n> $ mkdir test/folder1\n> $ echo original >test/folder1/a\n> $ echo original >test/b\n> $ git -C test add . && git -C test commit -m files\n> $ git -C test sparse-checkout init --cone --sparse-index\n> $ ls test\n> b\n> $ mkdir test/folder1 && echo modified >test/folder1/a\n> $ git -C test status\n> On branch master\n> You are in a sparse checkout with 50% of tracked files present.\n> nothing to commit, working tree clean\n\nYou are correct. This happens in both the sparse-index case and the\nregular full-index case. The modifications outside of the sparse-checkout\ndefinition are ignored, as long as they matched a tracked file.\n\nI checked my latest code against this example and see that the sparse\nindex is not expanded to a full one. It _will_ be if we add an untracked\nfile outside of the sparse cone.\n\n>> This is new\n>> behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\n>> entries, 2021-04-08). Before that change, these adds would be silently\n>> ignored.\n>>\n>> Untracked files are fine: adding new files both with 'git add .' and\n>> 'git add folder1/' works just as in a full checkout. This may not be\n>> entirely desirable, but we are not intending to change behavior at the\n>> moment, only document it. A future change could alter the behavior to\n>> be more sensible, and this test could be modified to satisfy the new\n>> expected behavior.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t1092-sparse-checkout-compatibility.sh | 40 ++++++++++++++++++++++++\n>> 1 file changed, 40 insertions(+)\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 12e6c453024f..0ec487acd283 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -232,6 +232,46 @@ test_expect_success 'add, commit, checkout' '\n>> test_all_match git checkout -\n>> '\n>>\n>> +test_expect_success 'status/add: outside sparse cone' '\n>> + init_repos &&\n> \n> A minor suggestion: before recreating folder1/a, we could also test\n> that `git add folder1/a` will not remove the sparse entry from the\n> index and will properly warn about it on both sparse repos. I.e.\n> adding a:\n> \n> test_sparse_match test_must_fail git add folder1/a\n\nWill do.\n\n>> + # folder1 is at HEAD, but outside the sparse cone\n>> + run_on_sparse mkdir folder1 &&\n>> + cp initial-repo/folder1/a sparse-checkout/folder1/a &&\n>> + cp initial-repo/folder1/a sparse-index/folder1/a &&\n>> +\n>> + test_sparse_match git status &&\n>> +\n>> + write_script edit-contents <<-\\EOF &&\n>> + echo text >>$1\n>> + EOF\n>> + run_on_all ../edit-contents folder1/a &&\n> \n> Hmm, we modify `folder1/a` in all repos, but we only try adding it on\n> the sparse repos, and then we immediately restore it on the full repo.\n> As we won't use the modified version on the full repo, could this\n> perhaps be `run_on_sparse` instead? If so, we could also save the\n> later `git -C full-checkout checkout HEAD -- folder1/a`.\n\nGood idea.\n\n>> + run_on_all ../edit-contents folder1/new &&\n>> +\n>> + test_sparse_match git status --porcelain=v2 &&\n>> +\n>> + # This \"git add folder1/a\" is completely ignored\n>> + # by the sparse-checkout repos. It causes the\n>> + # full repo to have a different staged environment.\n>> + #\n>> + # This is not a desirable behavior, but this test\n>> + # ensures that the sparse-index is not the cause\n>> + # of a behavior change.\n> \n> I'm not sure I understand what the undesirable behavior is in this\n> sentence. Is it \"git add folder1/a\" erroring out and not updating\n> `folder1/a`? Or the full repo having a different staged environment?\n\nPerhaps this isn't actually undesirable, now that we are actually\nreturning an error. It's no longer silent, so maybe my comment is\nstale from an earlier version.\n\nThanks,\n-Stolee\n"},{"id":"424597","messageId":"9204f7f6-85d5-f974-a77f-4e8e412cdc55@gmail.com","threadId":"55489","inReplyTo":"CABPp-BF0EGo-WLTN45qOu02O4sEzmSCUXT3XeoKQdbTHToNQOw@mail.gmail.com","subject":"Re: [PATCH v2 0/8] Sparse-index: integrate with status","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-14T18:28:54Z","receivedAt":"2021-05-14T18:28:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/13/2021 12:12 AM, Elijah Newren wrote:\n> On Fri, Apr 23, 2021 at 2:34 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n>> status' very fast when a sparse-index is enabled on a repository with\n>> cone-mode sparse-checkout (and a small populated set).\n> \n> I read through the topic, both my old comments, the range-diff, and\n> the new patches where the range-diff wasn't enough. I tried to spot\n> issues, and was hoping to find problems you alluded to in your recent\n> comments at https://lore.kernel.org/git/05932ebc-04ac-b3c5-a460-5d37d8604fd9@gmail.com/,\n> but I failed to spot them. I hope it has to do with the cache bottom\n> stuff that I just don't understand, because otherwise I just missed\n> the problems in my review. I can say that in v2 you fixed the issues\n> I did spot in my review of v1.\n> \n> I'll look forward to v3 to see what it was I missed. If I somehow\n> don't respond soon (in a week at the latest), do feel free to ping me;\n> sorry for somehow having this one slip through the cracks.\n\nv3 is on the way. The changes related to issues I found in my\ndeeper testing are more about what wasn't previously tested in\nmy test script as opposed to things actually being wrong in\nthe patch series. (There is one case where some new code was\nincorrect, but it wasn't being tested because of the test repo's\ndata shape.)\n\nThanks,\n-Stolee\n"},{"id":"424598","messageId":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v2.git.1619213665.gitgitgadget@gmail.com","subject":"[PATCH v3 00/12] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:30:59Z","receivedAt":"2021-05-14T18:31:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdates in V3\n=============\n\nSorry that this was a long time coming. I got a little side-tracked on other\nprojects, but I also worked to get the sparse-index feature working against\nthe Scalar functional tests, which contain many special cases around the\nsparse-checkout feature as they were inherited from special cases that arose\nin the virtualized environment of VFS for Git. This version contains my\nfixes based on that investigation. Most of these were easy to identify and\nfix, but I was blocked for a long time struggling with a bug when combining\nthe sparse-index with the builtin FS Monitor feature, but I've reported my\nfindings already [1].\n\n[1]\nhttps://lore.kernel.org/git/0b9e54ba-ac27-e537-7bef-1b4448f92352@gmail.com/\n\n * Updated comments and tests based on the v2 feedback.\n * Expanded the test repository data shape based on the special cases found\n during my investigation.\n * Added several commits that either fix errors in the status code, or fix\n errors in the previous sparse-index series, specifically:\n * When in a conflict state, the cache-tree fails to update. For now, skip\n writing a sparse-index until this can be resolved more carefully.\n * When expanding a sparse-directory entry, we set the CE_SKIP_WORKTREE\n bit but forgot the CE_EXTENDED bit.\n * git status had failures if there was a sparse-directory entry as the\n first entry within a directory.\n * When expanding a directory to report its status, such as when a\n sparse-directory is staged but doesn't exist at HEAD (such as in an\n orphaned commit) we did not previously recurse correctly into\n subdirectories.\n * Be extra careful with the FS Monitor data when expanding or contracting\n an index. This version now abandons all FS Monitor data at these\n conversion points with the expectation that in the future these\n conversions will be rare so the FS Monitor feature can work\n efficiently. Updates in V2\n\n----------------------------------------------------------------------------\n\n * Based on the feedback, it is clear that 'git add' will require much more\n careful testing and thought. I'm splitting it out of this series and it\n will return with a follow-up.\n * Test cases are improved, both in coverage and organization.\n * The previous \"unpack-trees: make sparse aware\" patch is split into three\n now.\n * Stale messages based on an old implementation of the \"protections\" topic\n are now fixed.\n * Performance tests were re-run.\n\nDerrick Stolee (12):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: stop recursing into sparse directories\n dir.c: accept a directory as part of cone-mode patterns\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n diff-lib.c | 6 ++\n dir.c | 11 +++\n read-cache.c | 10 +-\n sparse-index.c | 27 +++++-\n t/t1092-sparse-checkout-compatibility.sh | 117 ++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 48 ++++++++++\n unpack-trees.c | 27 +++++-\n wt-status.c | 64 ++++++++++++-\n wt-status.h | 1 +\n 10 files changed, 300 insertions(+), 14 deletions(-)\n\n\nbase-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v2:\n\n -: ------------ > 1: 5a2ed3d1d701 sparse-index: skip indexes with unmerged entries\n -: ------------ > 2: 8aa41e749471 sparse-index: include EXTENDED flag when expanding\n -: ------------ > 3: 70971b1f9261 t1092: expand repository data shape\n 1: 3bac9edae7d8 ! 4: a80b5a41153f t1092: add tests for status/add and sparse files\n @@ Commit message\n and files outside of the sparse cone.\n \n Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is\n - not in the sparse cone. When 'folder1/a' is modified, the file\n - 'folder1/a' is shown as modified, but adding it fails. This is new\n - behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE\n - entries, 2021-04-08). Before that change, these adds would be silently\n - ignored.\n + not in the sparse cone. When 'folder1/a' is modified, the file is not\n + shown as modified and adding it will fail. This is new behavior as of\n + a20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n + 2021-04-08). Before that change, these adds would be silently ignored.\n \n Untracked files are fine: adding new files both with 'git add .' and\n 'git add folder1/' works just as in a full checkout. This may not be\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n +test_expect_success 'status/add: outside sparse cone' '\n +\tinit_repos &&\n +\n ++\t# adding a \"missing\" file outside the cone should fail\n ++\ttest_sparse_match test_must_fail git add folder1/a &&\n ++\n +\t# folder1 is at HEAD, but outside the sparse cone\n +\trun_on_sparse mkdir folder1 &&\n +\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n +\twrite_script edit-contents <<-\\EOF &&\n +\techo text >>$1\n +\tEOF\n -+\trun_on_all ../edit-contents folder1/a &&\n ++\trun_on_sparse ../edit-contents folder1/a &&\n +\trun_on_all ../edit-contents folder1/new &&\n +\n +\ttest_sparse_match git status --porcelain=v2 &&\n +\n -+\t# This \"git add folder1/a\" is completely ignored\n -+\t# by the sparse-checkout repos. It causes the\n -+\t# full repo to have a different staged environment.\n -+\t#\n -+\t# This is not a desirable behavior, but this test\n -+\t# ensures that the sparse-index is not the cause\n -+\t# of a behavior change.\n ++\t# This \"git add folder1/a\" fails with a warning\n ++\t# in the sparse repos, differing from the full\n ++\t# repo. This is intentional.\n +\ttest_sparse_match test_must_fail git add folder1/a &&\n +\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n -+\tgit -C full-checkout checkout HEAD -- folder1/a &&\n +\ttest_all_match git status --porcelain=v2 &&\n +\n +\ttest_all_match git add . &&\n 2: 19344394379d = 5: 07a45b661c4a unpack-trees: preserve cache_bottom\n 3: 24e71d8c0622 ! 6: cc4a526e7947 unpack-trees: compare sparse directories correctly\n @@ unpack-trees.c: static int compare_entry(const struct cache_entry *ce, const str\n \tif (cmp)\n \t\treturn cmp;\n \n -+\t/* If ce is a sparse directory, then allow an exact match. */\n ++\t/*\n ++\t * At this point, we know that we have a prefix match. If ce\n ++\t * is a sparse directory, then allow an exact match. This only\n ++\t * works when the input name is a directory, since ce->name\n ++\t * ends in a directory separator.\n ++\t */\n +\tif (S_ISSPARSEDIR(ce->ce_mode))\n +\t\treturn 0;\n +\n 4: d3c8948d0a33 ! 7: 598375d3531f unpack-trees: stop recursing into sparse directories\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n + ## diff-lib.c ##\n +@@ diff-lib.c: static void show_new_file(struct rev_info *revs,\n + \tunsigned int mode;\n + \tunsigned dirty_submodule = 0;\n + \n ++\tif (S_ISSPARSEDIR(new_file->ce_mode))\n ++\t\treturn;\n ++\n + \t/*\n + \t * New file in the index: it might actually be different in\n + \t * the working tree.\n +@@ diff-lib.c: static int show_modified(struct rev_info *revs,\n + \tconst struct object_id *oid;\n + \tunsigned dirty_submodule = 0;\n + \n ++\tif (S_ISSPARSEDIR(new_entry->ce_mode))\n ++\t\treturn 0;\n ++\n + \tif (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n + \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n + \t\tif (report_missing)\n +\n ## unpack-trees.c ##\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned l\n \t/* Find first entry with a real name (we could use \"mask\" too) */\n \twhile (!p->mode)\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n - \t\t\t\t\t}\n - \t\t\t\t}\n - \t\t\t\tsrc[0] = ce;\n -+\n -+\t\t\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n -+\t\t\t\t\tunpack_tree = 0;\n - \t\t\t}\n - \t\t\tbreak;\n \t\t}\n \t}\n \n 5: fd96b71968b6 = 8: 47da2b317237 dir.c: accept a directory as part of cone-mode patterns\n 6: 1f4ba56e7416 = 9: bc1512981493 status: skip sparse-checkout percentage with sparse-index\n 7: 3d09368c0541 = 10: 5b1ae369a7cd status: use sparse-index throughout\n -: ------------ > 11: 3b42783d4a86 wt-status: expand added sparse directory entries\n 8: 1fd033a6ebb2 ! 12: b72507f514d1 fsmonitor: test with sparse index\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - fsmonitor: test with sparse index\n + fsmonitor: integrate with sparse index\n \n - During the effort to protect uses of the index to operate on a full\n - index, we did not modify fsmonitor.c. This is because it already works\n - effectively with only the change to index_name_stage_pos(). The only\n - thing left to do is to test that it works correctly.\n + If we need to expand a sparse-index into a full one, then the FS Monitor\n + bitmap is going to be incorrect. Ensure that we start fresh at such an\n + event.\n +\n + While this is currently a performance drawback, the eventual hope of the\n + sparse-index feature is that these expansions will be rare and hence we\n + will be able to keep the FS Monitor data accurate across multiple Git\n + commands.\n \n These tests are added to demonstrate that the behavior is the same\n across a full index and a sparse index, but also that file modifications\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n + ## sparse-index.c ##\n +@@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n + \tcache_tree_free(&istate->cache_tree);\n + \tcache_tree_update(istate, 0);\n + \n ++\tistate->fsmonitor_has_run_once = 0;\n ++\tFREE_AND_NULL(istate->fsmonitor_dirty);\n ++\tFREE_AND_NULL(istate->fsmonitor_last_update);\n ++\n + \tistate->sparse_index = 1;\n + \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n + \treturn 0;\n +@@ sparse-index.c: void ensure_full_index(struct index_state *istate)\n + \tistate->cache = full->cache;\n + \tistate->cache_nr = full->cache_nr;\n + \tistate->cache_alloc = full->cache_alloc;\n ++\tistate->fsmonitor_has_run_once = 0;\n ++\tFREE_AND_NULL(istate->fsmonitor_dirty);\n ++\tFREE_AND_NULL(istate->fsmonitor_last_update);\n + \n + \tstrbuf_release(&base);\n + \tfree(full);\n +\n ## t/t7519-status-fsmonitor.sh ##\n @@ t/t7519-status-fsmonitor.sh: test_expect_success 'setup' '\n \texpect*\n\n-- \ngitgitgadget\n"},{"id":"424599","messageId":"8aa41e749471df3bd9d593b8f55db6506eafea12.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:01Z","receivedAt":"2021-05-14T18:31:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but they must also be\nmarked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\ncorrectly written to disk in the case that the index is not converted\nback down to a sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 1b49898d0cb7..b2b3fbd75050 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"424600","messageId":"5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 01/12] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:00Z","receivedAt":"2021-05-14T18:31:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 6f21397e2ee0..1b49898d0cb7 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 12e6c453024f..4f2f09b53a32 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"424601","messageId":"70971b1f9261f2623ad6d6393be1a8e191003eb4.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 03/12] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:02Z","receivedAt":"2021-05-14T18:31:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4f2f09b53a32..98257695979a 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,16 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n-- \ngitgitgadget\n\n"},{"id":"424602","messageId":"a80b5a41153f07a58167be9c9759f3af3048cc5d.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 04/12] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:03Z","receivedAt":"2021-05-14T18:31:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 98257695979a..fba98d5484ae 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -238,6 +238,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"424603","messageId":"07a45b661c4a6ecd63ede19c9e058a8879069742.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 05/12] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:04Z","receivedAt":"2021-05-14T18:31:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dddf106d5bd4..1067db19c9d2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"424604","messageId":"598375d3531fabe8582cb6d93838df09e1bd06c3.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 07/12] unpack-trees: stop recursing into sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:06Z","receivedAt":"2021-05-14T18:31:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen walking trees using traverse_trees_recursive() and\nunpack_callback(), we must not attempt to walk into a sparse directory\nentry. There are no index entries within that directory to compare to\nthe tree object at that position, so skip over the entries of that tree.\n\nThis code is used in many places, so the only way to test it is to start\nremoving the command_requres_full_index option from one builtin at a\ntime and carefully test that its use of unpack_trees() behaves correctly\nwith a sparse-index. Such tests will be added by later changes.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 6 ++++++\n unpack-trees.c | 7 +++++--\n 2 files changed, 11 insertions(+), 2 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex b73cc1859a49..d5e7e01132ee 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -322,6 +322,9 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned int mode;\n \tunsigned dirty_submodule = 0;\n \n+\tif (S_ISSPARSEDIR(new_file->ce_mode))\n+\t\treturn;\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -343,6 +346,9 @@ static int show_modified(struct rev_info *revs,\n \tconst struct object_id *oid;\n \tunsigned dirty_submodule = 0;\n \n+\tif (S_ISSPARSEDIR(new_entry->ce_mode))\n+\t\treturn 0;\n+\n \tif (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex ef6a2b1c951c..703b0bdc9dfd 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1261,6 +1261,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n \tstruct unpack_trees_options *o = info->data;\n \tconst struct name_entry *p = names;\n+\tunsigned unpack_tree = 1;\n \n \t/* Find first entry with a real name (we could use \"mask\" too) */\n \twhile (!p->mode)\n@@ -1307,7 +1308,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_tree &&\n+\t unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n@@ -1337,7 +1339,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\tif (unpack_tree &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n \t\t\t\t\t names, info) < 0)\n \t\t\treturn -1;\n \t\treturn mask;\n-- \ngitgitgadget\n\n"},{"id":"424605","messageId":"cc4a526e7947f40a0ef65396f2ec8d21d2445edf.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 06/12] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:05Z","receivedAt":"2021-05-14T18:31:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1067db19c9d2..ef6a2b1c951c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"424606","messageId":"bc151298149331e75b0a5aacb66c7bfcbe9822b8.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 09/12] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:08Z","receivedAt":"2021-05-14T18:31:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex fba98d5484ae..34dae7fbcadd 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -202,6 +202,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 0c8287a023e4..0425169c1895 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e1..ab9cc9d8f032 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"424607","messageId":"5b1ae369a7cd2700288a6719c496c149b4240d82.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 10/12] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:09Z","receivedAt":"2021-05-14T18:31:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex cf0c36d1dcb2..e529da7beadd 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 29ffa9ac5db9..f80e26831b36 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 34dae7fbcadd..59faf7381093 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -479,12 +479,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"424608","messageId":"3b42783d4a86473420480b2789d61d8103e6e7d4.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 11/12] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:10Z","receivedAt":"2021-05-14T18:31:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++\n wt-status.c | 50 ++++++++++++++++++++++++\n 2 files changed, 78 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 59faf7381093..cd3669d36b53 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\ttest_all_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 0425169c1895..90db8bd659fa 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trun_diff_index(&rev, 1);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tchar *full_name;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n+\tit = string_list_insert(&s->change, full_name);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps;\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tmemset(&ps, 0, sizeof(ps));\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"424609","messageId":"b72507f514d130afc4aa651116a96a007c92e242.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 12/12] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:11Z","receivedAt":"2021-05-14T18:31:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b2b3fbd75050..32ba0d17ef7c 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 45d025f96010..f70fe961902e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"424610","messageId":"47da2b317237c4ce8c95ce847e1c65c2d82fcfb7.1621017072.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v3 08/12] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-14T18:31:07Z","receivedAt":"2021-05-14T18:31:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 166238e79f52..ab76ef286495 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"424802","messageId":"CABPp-BFx9=7j5LrLnQwzxba+i7_cAEXngzyt9rfOJfBuqOPBsw@mail.gmail.com","threadId":"55489","inReplyTo":"8aa41e749471df3bd9d593b8f55db6506eafea12.1621017072.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T01:33:12Z","receivedAt":"2021-05-18T01:33:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When creating a full index from a sparse one, we create cache entries\n> for every blob within a given sparse directory entry. These are\n> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n> correctly written to disk in the case that the index is not converted\n> back down to a sparse-index.\n\nThis seems odd to me. When sparse-index is not involved and we are\njust doing simple sparse checkouts, do we mark CE_SKIP_WORKTREE\nentries with CE_EXTENDED? I can't find any code that does so.\n\nIs it possible that the setting of CE_EXTENDED is just a workaround\nthat happens to force the index to be written in cases where the logic\nis otherwise thinking it can get away without one? Or is there\nsomething I'm missing about why the CE_EXTENDED flag is actually\nneeded here?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> sparse-index.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index 1b49898d0cb7..b2b3fbd75050 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid,\n> strbuf_addstr(base, path);\n>\n> ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n> - ce->ce_flags |= CE_SKIP_WORKTREE;\n> + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n> set_index_entry(istate, istate->cache_nr++, ce);\n>\n> strbuf_setlen(base, len);\n> --\n> gitgitgadget\n>\n"},{"id":"424803","messageId":"CABPp-BG=5huMt2FOrzscNrg9rCpo40we7ms8K=508+Kur0yGnQ@mail.gmail.com","threadId":"55489","inReplyTo":"70971b1f9261f2623ad6d6393be1a8e191003eb4.1621017072.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/12] t1092: expand repository data shape","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T01:49:27Z","receivedAt":"2021-05-18T01:49:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> As more features integrate with the sparse-index feature, more and more\n> special cases arise that require different data shapes within the tree\n> structure of the repository in order to demonstrate those cases.\n>\n> Add several interesting special cases all at once instead of sprinkling\n> them across several commits. The interesting cases being added here are:\n>\n> * Add sparse-directory entries on both sides of directories within the\n> sparse-checkout definition.\n>\n> * Add directories outside the sparse-checkout definition who have only\n> one entry and are the first entry of a directory with multiple\n> entries.\n>\n> Later tests will take advantage of these shapes, but they also deepen\n> the tests that already exist.\n\nMakes sense. Do we also want to add ones of the form\n\n foo/bar\n foo.txt\n\n?\n\nHere we'd be particularly looking that if foo is a sparse directory,\nwe want to avoid messing up its order. ('foo' sorts before 'foo.txt',\nbut 'foo/' sorts after, and thus 'foo' the directory should be after\n'foo.txt')\n\n\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 8 +++++++-\n> 1 file changed, 7 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 4f2f09b53a32..98257695979a 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -17,7 +17,7 @@ test_expect_success 'setup' '\n> echo \"after folder1\" >g &&\n> echo \"after x\" >z &&\n> mkdir folder1 folder2 deep x &&\n> - mkdir deep/deeper1 deep/deeper2 &&\n> + mkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n> mkdir deep/deeper1/deepest &&\n> echo \"after deeper1\" >deep/e &&\n> echo \"after deepest\" >deep/deeper1/e &&\n> @@ -25,10 +25,16 @@ test_expect_success 'setup' '\n> cp a folder2 &&\n> cp a x &&\n> cp a deep &&\n> + cp a deep/before &&\n> cp a deep/deeper1 &&\n> cp a deep/deeper2 &&\n> + cp a deep/later &&\n> cp a deep/deeper1/deepest &&\n> cp -r deep/deeper1/deepest deep/deeper2 &&\n> + mkdir deep/deeper1/0 &&\n> + mkdir deep/deeper1/0/0 &&\n> + touch deep/deeper1/0/1 &&\n> + touch deep/deeper1/0/0/0 &&\n> git add . &&\n> git commit -m \"initial commit\" &&\n> git checkout -b base &&\n> --\n> gitgitgadget\n\nLooks good.\n"},{"id":"424804","messageId":"CABPp-BGc_Cduk3aFJqk4MjJ2k6KMDWSnwZfJM+E-1oK8txsf7Q@mail.gmail.com","threadId":"55489","inReplyTo":"598375d3531fabe8582cb6d93838df09e1bd06c3.1621017072.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 07/12] unpack-trees: stop recursing into sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T02:03:40Z","receivedAt":"2021-05-18T02:03:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When walking trees using traverse_trees_recursive() and\n> unpack_callback(), we must not attempt to walk into a sparse directory\n> entry. There are no index entries within that directory to compare to\n> the tree object at that position, so skip over the entries of that tree.\n>\n> This code is used in many places, so the only way to test it is to start\n> removing the command_requres_full_index option from one builtin at a\n> time and carefully test that its use of unpack_trees() behaves correctly\n> with a sparse-index. Such tests will be added by later changes.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff-lib.c | 6 ++++++\n> unpack-trees.c | 7 +++++--\n> 2 files changed, 11 insertions(+), 2 deletions(-)\n>\n> diff --git a/diff-lib.c b/diff-lib.c\n> index b73cc1859a49..d5e7e01132ee 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -322,6 +322,9 @@ static void show_new_file(struct rev_info *revs,\n> unsigned int mode;\n> unsigned dirty_submodule = 0;\n>\n> + if (S_ISSPARSEDIR(new_file->ce_mode))\n> + return;\n> +\n\nMakes sense, but is this related to the unpack-trees.c changes and the\ncommit message, or should it be in a separate commit?\n\n> /*\n> * New file in the index: it might actually be different in\n> * the working tree.\n> @@ -343,6 +346,9 @@ static int show_modified(struct rev_info *revs,\n> const struct object_id *oid;\n> unsigned dirty_submodule = 0;\n>\n> + if (S_ISSPARSEDIR(new_entry->ce_mode))\n> + return 0;\n> +\n\nSame question as above. And a few more questions...\n\nWhat if the old commit/tree had a file at this path, and the new\ncommit/tree has a (sparse) directory at this path? Shouldn't\n_something_ be shown for the file deletion? Or does such a case not\nrun through this code path?\n\nAlso, wouldn't we expect it to be an error for show_modified() to be\ncalled on a sparse directory? If two sparse directories differed, we\nshould have inflated the trees to find the differences in the path\nunderneath them, right? And if they didn't differ, then\nshow_modified() should not have been invoked?\n\nI can see cases where we wouldn't want to bother looking at the\ndifferences between to sparse directories, e.g. a\n--restrict-to-sparsity-paths option to diff/log/etc, but I don't see\nyou setting this behind an option here.\n\n> if (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n> &dirty_submodule, &revs->diffopt) < 0) {\n> if (report_missing)\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index ef6a2b1c951c..703b0bdc9dfd 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1261,6 +1261,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> struct unpack_trees_options *o = info->data;\n> const struct name_entry *p = names;\n> + unsigned unpack_tree = 1;\n>\n> /* Find first entry with a real name (we could use \"mask\" too) */\n> while (!p->mode)\n> @@ -1307,7 +1308,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> + if (unpack_tree &&\n> + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> return -1;\n>\n> if (o->merge && src[0]) {\n> @@ -1337,7 +1339,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> + if (unpack_tree &&\n> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> names, info) < 0)\n> return -1;\n> return mask;\n> --\n> gitgitgadget\n\nThe unpack-trees.c changes make sense to me still.\n"},{"id":"424805","messageId":"CABPp-BEdWMnzUay9R_GnbiWzuddrFMu87DLmCz4_s8HaBsrYQQ@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BGc_Cduk3aFJqk4MjJ2k6KMDWSnwZfJM+E-1oK8txsf7Q@mail.gmail.com","subject":"Re: [PATCH v3 07/12] unpack-trees: stop recursing into sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T02:06:38Z","receivedAt":"2021-05-18T02:06:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry, I spoke too soon...\n\nOn Mon, May 17, 2021 at 7:03 PM Elijah Newren wrote:\n>\n> > diff --git a/unpack-trees.c b/unpack-trees.c\n> > index ef6a2b1c951c..703b0bdc9dfd 100644\n> > --- a/unpack-trees.c\n> > +++ b/unpack-trees.c\n> > @@ -1261,6 +1261,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> > struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> > struct unpack_trees_options *o = info->data;\n> > const struct name_entry *p = names;\n> > + unsigned unpack_tree = 1;\n\nHere, you set unpack_tree to 1.\n\n> >\n> > /* Find first entry with a real name (we could use \"mask\" too) */\n> > while (!p->mode)\n> > @@ -1307,7 +1308,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> > }\n> > }\n> >\n> > - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> > + if (unpack_tree &&\n\nYou check it's value here...\n\n> > + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> > return -1;\n> >\n> > if (o->merge && src[0]) {\n> > @@ -1337,7 +1339,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> > }\n> > }\n> >\n> > - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> > + if (unpack_tree &&\n...and here....\n\n> > + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> > names, info) < 0)\n> > return -1;\n> > return mask;\n\nbut you never set unpack_tree to 0, so this is wasted effort and you\nalways recurse. The previous iteration had a case where it'd set\nunpack_tree to 0 in a certain case, but you deleted that code in this\nversion. Why?\n"},{"id":"424808","messageId":"CABPp-BGww6VPMmp9dDyRJ2vgNCOdj16-LjU3aM3MM1NQ+T-pmg@mail.gmail.com","threadId":"55489","inReplyTo":"3b42783d4a86473420480b2789d61d8103e6e7d4.1621017072.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 11/12] wt-status: expand added sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T02:27:17Z","receivedAt":"2021-05-18T02:27:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> It is difficult, but possible, to get into a state where we intend to\n> add a directory that is outside of the sparse-checkout definition. Add a\n\nThen we need to fix that; allowing things to be added outside the\nsparse-checkout definition is a bug[1][2]. That's an invariant I\nbelieve we should maintain everywhere; things get really confusing to\nusers somewhere later down the road if we don't. Matheus worked to\nfix that with 'git add'; if there are other commands that need fixing\ntoo, then we should also fix them.\n\n[1] https://lore.kernel.org/git/CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com/\n[2] https://lore.kernel.org/git/CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com/\n\n> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nI think `git checkout --orphan` should just throw an error if\nsparse-checkout is in use. Allowing adding paths outside the\nsparse-checkout set causes too much collateral and deferred confusion\nfor users.\n\n> This test failed before because the output of 'git status\n> --porcelain=v2' would not match on the lines for folder1/:\n>\n> * The sparse-checkout repo (with a full index) would output each path\n> name that is intended to be added.\n>\n> * The sparse-index repo would only output that \"folder1/\" is staged for\n> addition.\n>\n> The status should report the full list of files to be added, and so this\n> sparse-directory entry should be expanded to a full list when reaching\n> it inside the wt_status_collect_changes_initial() method. Use\n> read_tree_at() to assist.\n\nHaving a sparse directory entry whose object_id in the index does not\nmatch HEAD should be an error. Have a CE_SKIP_WORKTREE non-directory\nwhose object_id in the index does not match HEAD should also be an\nerror. I don't think we should complicate the code to try to handle\nviolations of those assumptions. I do think we should add checks to\nenforce that constraint (or BUG() if it's violated).\n\nAnd yeah, that also means 'git sparse-checkout add/set' would need to\nerror out if paths are requested to be sparsified despite being\ndifferent from HEAD.\n\n> Somehow, this loop over the cache entries was not guarded by\n> ensure_full_index() as intended.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++\n> wt-status.c | 50 ++++++++++++++++++++++++\n> 2 files changed, 78 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 59faf7381093..cd3669d36b53 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' '\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> +test_expect_success 'reset mixed and checkout orphan' '\n> + init_repos &&\n> +\n> + test_all_match git checkout rename-out-to-in &&\n> + test_all_match git reset --mixed HEAD~1 &&\n> + test_sparse_match test-tool read-cache --table --expand &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + # At this point, sparse-checkouts behave differently\n> + # from the full-checkout.\n> + test_sparse_match git checkout --orphan new-branch &&\n> + test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_sparse_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'add everything with deep new file' '\n> + init_repos &&\n> +\n> + run_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n> +\n> + run_on_all touch deep/deeper1/x &&\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> test_done\n> diff --git a/wt-status.c b/wt-status.c\n> index 0425169c1895..90db8bd659fa 100644\n> --- a/wt-status.c\n> +++ b/wt-status.c\n> @@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n> run_diff_index(&rev, 1);\n> }\n>\n> +static int add_file_to_list(const struct object_id *oid,\n> + struct strbuf *base, const char *path,\n> + unsigned int mode, void *context)\n> +{\n> + struct string_list_item *it;\n> + struct wt_status_change_data *d;\n> + struct wt_status *s = context;\n> + char *full_name;\n> +\n> + if (S_ISDIR(mode))\n> + return READ_TREE_RECURSIVE;\n> +\n> + full_name = xstrfmt(\"%s%s\", base->buf, path);\n> + it = string_list_insert(&s->change, full_name);\n> + d = it->util;\n> + if (!d) {\n> + CALLOC_ARRAY(d, 1);\n> + it->util = d;\n> + }\n> +\n> + d->index_status = DIFF_STATUS_ADDED;\n> + /* Leave {mode,oid}_head zero for adds. */\n> + d->mode_index = mode;\n> + oidcpy(&d->oid_index, oid);\n> + s->committable = 1;\n> + return 0;\n> +}\n> +\n> static void wt_status_collect_changes_initial(struct wt_status *s)\n> {\n> struct index_state *istate = s->repo->index;\n> @@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n> continue;\n> if (ce_intent_to_add(ce))\n> continue;\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + /*\n> + * This is a sparse directory entry, so we want to collect all\n> + * of the added files within the tree. This requires recursively\n> + * expanding the trees to find the elements that are new in this\n> + * tree and marking them with DIFF_STATUS_ADDED.\n> + */\n> + struct strbuf base = STRBUF_INIT;\n> + struct pathspec ps;\n> + struct tree *tree = lookup_tree(istate->repo, &ce->oid);\n> +\n> + memset(&ps, 0, sizeof(ps));\n> + ps.recursive = 1;\n> + ps.has_wildcard = 1;\n> + ps.max_depth = -1;\n> +\n> + strbuf_add(&base, ce->name, ce->ce_namelen);\n> + read_tree_at(istate->repo, tree, &base, &ps,\n> + add_file_to_list, s);\n> + continue;\n> + }\n> +\n> it = string_list_insert(&s->change, ce->name);\n> d = it->util;\n> if (!d) {\n> --\n> gitgitgadget\n\nThis was a really nice catch that you got this particular testcase.\nWhile I disagree with the fix, I do have to say nice work on the catch\nand the implementation otherwise.\n"},{"id":"424883","messageId":"ddeda8ec-bc21-1f62-5d0e-839be74f1ec8@gmail.com","threadId":"55489","inReplyTo":"CABPp-BFx9=7j5LrLnQwzxba+i7_cAEXngzyt9rfOJfBuqOPBsw@mail.gmail.com","subject":"Re: [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T14:57:17Z","receivedAt":"2021-05-18T14:57:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/17/2021 9:33 PM, Elijah Newren wrote:\n> On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When creating a full index from a sparse one, we create cache entries\n>> for every blob within a given sparse directory entry. These are\n>> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n>> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n>> correctly written to disk in the case that the index is not converted\n>> back down to a sparse-index.\n> \n> This seems odd to me. When sparse-index is not involved and we are\n> just doing simple sparse checkouts, do we mark CE_SKIP_WORKTREE\n> entries with CE_EXTENDED? I can't find any code that does so.\n> \n> Is it possible that the setting of CE_EXTENDED is just a workaround\n> that happens to force the index to be written in cases where the logic\n> is otherwise thinking it can get away without one? Or is there\n> something I'm missing about why the CE_EXTENDED flag is actually\n> needed here?\n\nThis is happening within the context of ensure_full_index(), so we\nare creating new cache entries and want to mimic what they would\nlook like on-disk. Something within do_write_index() discovers that\nsince CE_SKIP_WORKTREE is set, then also CE_EXTENDED should be set\nin order to ensure that the on-disk representation has enough room\nfor the CE_SKIP_WORKTREE bit.\n\nI suppose this might not have a meaningful purpose other than when\nI compare a full index against an expanded sparse-index and check\nif their flags match.\n\nThanks,\n-Stolee\n"},{"id":"424884","messageId":"48a361fd-33d5-29ba-556f-10075139de33@gmail.com","threadId":"55489","inReplyTo":"CABPp-BG=5huMt2FOrzscNrg9rCpo40we7ms8K=508+Kur0yGnQ@mail.gmail.com","subject":"Re: [PATCH v3 03/12] t1092: expand repository data shape","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T14:59:59Z","receivedAt":"2021-05-18T15:00:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/17/2021 9:49 PM, Elijah Newren wrote:\n> On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> Later tests will take advantage of these shapes, but they also deepen\n>> the tests that already exist.\n> \n> Makes sense. Do we also want to add ones of the form\n> \n> foo/bar\n> foo.txt\n> \n> ?\n> \n> Here we'd be particularly looking that if foo is a sparse directory,\n> we want to avoid messing up its order. ('foo' sorts before 'foo.txt',\n> but 'foo/' sorts after, and thus 'foo' the directory should be after\n> 'foo.txt')\n\nGood idea!\n\nThanks,\n-Stolee\n"},{"id":"424890","messageId":"CABPp-BHXNLTcQC=XmPoaVoy8xLKpru3yuJU1DfJVpOf_tJy8Yg@mail.gmail.com","threadId":"55489","inReplyTo":"ddeda8ec-bc21-1f62-5d0e-839be74f1ec8@gmail.com","subject":"Re: [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T17:48:06Z","receivedAt":"2021-05-18T17:48:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 18, 2021 at 7:57 AM Derrick Stolee wrote:\n>\n> On 5/17/2021 9:33 PM, Elijah Newren wrote:\n> > On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n> >> From: Derrick Stolee \n> >>\n> >> When creating a full index from a sparse one, we create cache entries\n> >> for every blob within a given sparse directory entry. These are\n> >> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n> >> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n> >> correctly written to disk in the case that the index is not converted\n> >> back down to a sparse-index.\n> >\n> > This seems odd to me. When sparse-index is not involved and we are\n> > just doing simple sparse checkouts, do we mark CE_SKIP_WORKTREE\n> > entries with CE_EXTENDED? I can't find any code that does so.\n> >\n> > Is it possible that the setting of CE_EXTENDED is just a workaround\n> > that happens to force the index to be written in cases where the logic\n> > is otherwise thinking it can get away without one? Or is there\n> > something I'm missing about why the CE_EXTENDED flag is actually\n> > needed here?\n>\n> This is happening within the context of ensure_full_index(), so we\n> are creating new cache entries and want to mimic what they would\n> look like on-disk. Something within do_write_index() discovers that\n> since CE_SKIP_WORKTREE is set, then also CE_EXTENDED should be set\n> in order to ensure that the on-disk representation has enough room\n> for the CE_SKIP_WORKTREE bit.\n\nYeah, I think it's this part:\n\n /* reduce extended entries if possible */\n cache[i]->ce_flags &= ~CE_EXTENDED;\n if (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n extended++;\n cache[i]->ce_flags |= CE_EXTENDED;\n }\n\n>\n> I suppose this might not have a meaningful purpose other than when\n> I compare a full index against an expanded sparse-index and check\n> if their flags match.\n\nAh, you're just setting this flag in advance of do_write_index() being\ncalled so that you can compare in memory values and check they match\nwithout doing a write-to-disk-and-read-back cycle. Makes sense, but\nit'd be nice to see this in the commit message.\n"},{"id":"424891","messageId":"cb9161ca-dc6e-b77b-1a41-385ed8920bb2@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHXNLTcQC=XmPoaVoy8xLKpru3yuJU1DfJVpOf_tJy8Yg@mail.gmail.com","subject":"Re: [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T18:16:46Z","receivedAt":"2021-05-18T18:16:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/18/2021 1:48 PM, Elijah Newren wrote:\n> On Tue, May 18, 2021 at 7:57 AM Derrick Stolee wrote:\n>>\n>> On 5/17/2021 9:33 PM, Elijah Newren wrote:\n>>> Is it possible that the setting of CE_EXTENDED is just a workaround\n>>> that happens to force the index to be written in cases where the logic\n>>> is otherwise thinking it can get away without one? Or is there\n>>> something I'm missing about why the CE_EXTENDED flag is actually\n>>> needed here?\n>>\n>> This is happening within the context of ensure_full_index(), so we\n>> are creating new cache entries and want to mimic what they would\n>> look like on-disk. Something within do_write_index() discovers that\n>> since CE_SKIP_WORKTREE is set, then also CE_EXTENDED should be set\n>> in order to ensure that the on-disk representation has enough room\n>> for the CE_SKIP_WORKTREE bit.\n> \n> Yeah, I think it's this part:\n> \n> /* reduce extended entries if possible */\n> cache[i]->ce_flags &= ~CE_EXTENDED;\n> if (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n> extended++;\n> cache[i]->ce_flags |= CE_EXTENDED;\n> }\n> \n>>\n>> I suppose this might not have a meaningful purpose other than when\n>> I compare a full index against an expanded sparse-index and check\n>> if their flags match.\n> \n> Ah, you're just setting this flag in advance of do_write_index() being\n> called so that you can compare in memory values and check they match\n> without doing a write-to-disk-and-read-back cycle. Makes sense, but\n> it'd be nice to see this in the commit message.\n\nWill do. Thanks,\n\n-Stolee\n"},{"id":"424892","messageId":"a96b5081-e913-22a8-45eb-8d5de432dccb@gmail.com","threadId":"55489","inReplyTo":"CABPp-BGww6VPMmp9dDyRJ2vgNCOdj16-LjU3aM3MM1NQ+T-pmg@mail.gmail.com","subject":"Re: [PATCH v3 11/12] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T18:26:28Z","receivedAt":"2021-05-18T18:26:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/17/2021 10:27 PM, Elijah Newren wrote:\n> On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> It is difficult, but possible, to get into a state where we intend to\n>> add a directory that is outside of the sparse-checkout definition. Add a\n> \n> Then we need to fix that; allowing things to be added outside the\n> sparse-checkout definition is a bug[1][2]. That's an invariant I\n> believe we should maintain everywhere; things get really confusing to\n> users somewhere later down the road if we don't. Matheus worked to\n> fix that with 'git add'; if there are other commands that need fixing\n> too, then we should also fix them.\n> \n> [1] https://lore.kernel.org/git/CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com/\n> [2] https://lore.kernel.org/git/CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com/\n> \n>> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n>> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n> \n> I think `git checkout --orphan` should just throw an error if\n> sparse-checkout is in use. Allowing adding paths outside the\n> sparse-checkout set causes too much collateral and deferred confusion\n> for users.\n\nI've been trying to strike an interesting balance of creating\nperformance improvements without changing behavior, trying to\ndefer those behavior changes to an isolated instance. I think\nthat approach is unavoidable with the 'git add' work that I\npulled out of this series and will return to soon.\n\nHowever, here I think it would be too much to start throwing\nan error in this case. I think that change is a bit too much.\n\nThe thing I can try to do, instead of the current approach, is\nto not allow sparse directory entries to differ between the\nindex and HEAD. That will satisfy this case, but also a lot of\nother painful cases.\n\nI have no idea how to actually accomplish that, but I'll start\ndigging.\n\n>> This test failed before because the output of 'git status\n>> --porcelain=v2' would not match on the lines for folder1/:\n>>\n>> * The sparse-checkout repo (with a full index) would output each path\n>> name that is intended to be added.\n>>\n>> * The sparse-index repo would only output that \"folder1/\" is staged for\n>> addition.\n>>\n>> The status should report the full list of files to be added, and so this\n>> sparse-directory entry should be expanded to a full list when reaching\n>> it inside the wt_status_collect_changes_initial() method. Use\n>> read_tree_at() to assist.\n> \n> Having a sparse directory entry whose object_id in the index does not\n> match HEAD should be an error.\n\nI can get behind this understanding.\n\n> Have a CE_SKIP_WORKTREE non-directory\n> whose object_id in the index does not match HEAD should also be an\n> error.\n\nI'm less convinced here. At minimum, I'm not willing to stake\na firm claim and change the behavior around this statement in\nthe current series.\n\n> I don't think we should complicate the code to try to handle\n> violations of those assumptions. I do think we should add checks to\n> enforce that constraint (or BUG() if it's violated).\n\nA BUG() is likely too strict, because existing Git clients can\nget users into this state, and then they upgrade and are suddenly\nin a BUG() state. We should perhaps do our best effort to avoid\nthis case and handle it as appropriately as possible.\n\n> And yeah, that also means 'git sparse-checkout add/set' would need to\n> error out if paths are requested to be sparsified despite being\n> different from HEAD.\n\nThis would be a reasonable thing, assuming the established\nbehavior is changed.\n\n>> Somehow, this loop over the cache entries was not guarded by\n>> ensure_full_index() as intended.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++\n>> wt-status.c | 50 ++++++++++++++++++++++++\n>> 2 files changed, 78 insertions(+)\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 59faf7381093..cd3669d36b53 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' '\n>> test_region ! index ensure_full_index trace2.txt\n>> '\n>>\n>> +test_expect_success 'reset mixed and checkout orphan' '\n>> + init_repos &&\n>> +\n>> + test_all_match git checkout rename-out-to-in &&\n>> + test_all_match git reset --mixed HEAD~1 &&\n>> + test_sparse_match test-tool read-cache --table --expand &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_all_match git status --porcelain=v2 &&\n>> +\n>> + # At this point, sparse-checkouts behave differently\n>> + # from the full-checkout.\n>> + test_sparse_match git checkout --orphan new-branch &&\n>> + test_sparse_match test-tool read-cache --table --expand &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_sparse_match git status --porcelain=v2\n>> +'\n>> +\n>> +test_expect_success 'add everything with deep new file' '\n>> + init_repos &&\n>> +\n>> + run_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n>> +\n>> + run_on_all touch deep/deeper1/x &&\n>> + test_all_match git add . &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_all_match git status --porcelain=v2\n>> +'>\n> This was a really nice catch that you got this particular testcase.\n> While I disagree with the fix, I do have to say nice work on the catch\n> and the implementation otherwise.\n\nThis test exists almost verbatim in the Scalar and VFS For Git\nfunctional tests. I have no idea what context caused it to be\nnecessary.\n\nI can understand your aversion to the solution I presented here.\nPreventing sparse directory entries that differ from the tree at\nHEAD for that path should be more robust to future integrations.\n\nThanks,\n-Stolee\n"},{"id":"424899","messageId":"f592fa8e-b607-1665-5f2e-3a3ad834b30b@gmail.com","threadId":"55489","inReplyTo":"a96b5081-e913-22a8-45eb-8d5de432dccb@gmail.com","subject":"Re: [PATCH v3 11/12] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T19:04:58Z","receivedAt":"2021-05-18T19:05:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/18/2021 2:26 PM, Derrick Stolee wrote:\n> On 5/17/2021 10:27 PM, Elijah Newren wrote:\n>> On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n>> wrote:\n>>>\n>>> From: Derrick Stolee \n>>>\n>>> It is difficult, but possible, to get into a state where we intend to\n>>> add a directory that is outside of the sparse-checkout definition. Add a\n>>\n>> Then we need to fix that; allowing things to be added outside the\n>> sparse-checkout definition is a bug[1][2]. That's an invariant I\n>> believe we should maintain everywhere; things get really confusing to\n>> users somewhere later down the road if we don't. Matheus worked to\n>> fix that with 'git add'; if there are other commands that need fixing\n>> too, then we should also fix them.\n>>\n>> [1] https://lore.kernel.org/git/CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com/\n>> [2] https://lore.kernel.org/git/CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com/\n>>\n>>> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n>>> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n>>\n>> I think `git checkout --orphan` should just throw an error if\n>> sparse-checkout is in use. Allowing adding paths outside the\n>> sparse-checkout set causes too much collateral and deferred confusion\n>> for users.\n> \n> I've been trying to strike an interesting balance of creating\n> performance improvements without changing behavior, trying to\n> defer those behavior changes to an isolated instance. I think\n> that approach is unavoidable with the 'git add' work that I\n> pulled out of this series and will return to soon.\n> \n> However, here I think it would be too much to start throwing\n> an error in this case. I think that change is a bit too much.\n> \n> The thing I can try to do, instead of the current approach, is\n> to not allow sparse directory entries to differ between the\n> index and HEAD. That will satisfy this case, but also a lot of\n> other painful cases.\n> \n> I have no idea how to actually accomplish that, but I'll start\n> digging.\n\nIt didn't take much digging to discover that this is likely\nimpossible, or rather it would be a drastic change to make this\nhappen.\n\nThe immediate issue is trying to prevent sparse directory entries\nfrom existing when the contained paths don't match what exists at\nHEAD. However, in the 'git checkout --orphan' case, we are using\na full index for the unpack_trees() that updates the in-memory\nindex according to the paths at HEAD, then updates HEAD to point\nto a non-existing ref. The sparse directories are only created as\npart of convert_to_sparse() within do_write_index(). At that\npoint, there is no HEAD provided. Trying to load it from scratch\nviolates the fact that HEAD is being staged to change _after_ the\nindex updates in a command like 'git checkout'.\n\nSo, the drastic change to make this work would be to update the\nindex API to require a root tree to be provided whenever writing\nthe index. However, that doesn't make sense, either! What do we\ndo when in a conflicted state?\n\nWhat if a user modifies HEAD manually to point to a new ref?\n\nSuch a change would couple the index to the concept of HEAD in\nan unproductive way, I think. The index data structure exists\nas a separate entity that is frequently _compared_ to HEAD, and\nthe solution presented in this patch presents a way to keep the\ncomparison of a sparse-index and HEAD to be the same as if we\nhad a full index.\n\nSo, after looking into it, I'm back in favor of this change and\nforever allowing sparse cache entries to differ from HEAD,\nbecause there is no way to avoid it.\n\nThanks,\n-Stolee\n"},{"id":"424900","messageId":"e583fe80-4872-0e2b-622f-8288d2b18bd0@gmail.com","threadId":"55489","inReplyTo":"CABPp-BEdWMnzUay9R_GnbiWzuddrFMu87DLmCz4_s8HaBsrYQQ@mail.gmail.com","subject":"Re: [PATCH v3 07/12] unpack-trees: stop recursing into sparse directories","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-18T19:20:47Z","receivedAt":"2021-05-18T19:20:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/17/2021 10:06 PM, Elijah Newren wrote:\n> Sorry, I spoke too soon...\n> \n> On Mon, May 17, 2021 at 7:03 PM Elijah Newren wrote:\n>>\n>>> diff --git a/unpack-trees.c b/unpack-trees.c\n>>> index ef6a2b1c951c..703b0bdc9dfd 100644\n>>> --- a/unpack-trees.c\n>>> +++ b/unpack-trees.c\n>>> @@ -1261,6 +1261,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>>> struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n>>> struct unpack_trees_options *o = info->data;\n>>> const struct name_entry *p = names;\n>>> + unsigned unpack_tree = 1;\n> \n> Here, you set unpack_tree to 1.\n> \n>>>\n>>> /* Find first entry with a real name (we could use \"mask\" too) */\n>>> while (!p->mode)\n>>> @@ -1307,7 +1308,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>>> }\n>>> }\n>>>\n>>> - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n>>> + if (unpack_tree &&\n> \n> You check it's value here...\n> \n>>> + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n>>> return -1;\n>>>\n>>> if (o->merge && src[0]) {\n>>> @@ -1337,7 +1339,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n>>> }\n>>> }\n>>>\n>>> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n>>> + if (unpack_tree &&\n> ...and here....\n> \n>>> + traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n>>> names, info) < 0)\n>>> return -1;\n>>> return mask;\n> \n> but you never set unpack_tree to 0, so this is wasted effort and you\n> always recurse. The previous iteration had a case where it'd set\n> unpack_tree to 0 in a certain case, but you deleted that code in this\n> version. Why?\n\nIt appears that the changes to unpack-trees.c are no longer relevant,\nand instead the changes to diff-lib.c (which were already out of place)\nshould instead be the focus. In fact, those changes to diff-lib.c can\nbe simplified and moved to path 10, so I will do that.\n\nThanks,\n-Stolee\n"},{"id":"424923","messageId":"CABPp-BEnE47iH4zgf4dU-_nby9NCs0s=rOKDOHEqSKiL01A6pQ@mail.gmail.com","threadId":"55489","inReplyTo":"f592fa8e-b607-1665-5f2e-3a3ad834b30b@gmail.com","subject":"Re: [PATCH v3 11/12] wt-status: expand added sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-19T08:38:20Z","receivedAt":"2021-05-19T08:38:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 18, 2021 at 12:05 PM Derrick Stolee wrote:\n>\n> On 5/18/2021 2:26 PM, Derrick Stolee wrote:\n> > On 5/17/2021 10:27 PM, Elijah Newren wrote:\n> >> On Fri, May 14, 2021 at 11:31 AM Derrick Stolee via GitGitGadget\n> >> wrote:\n> >>>\n> >>> From: Derrick Stolee \n> >>>\n> >>> It is difficult, but possible, to get into a state where we intend to\n> >>> add a directory that is outside of the sparse-checkout definition. Add a\n> >>\n> >> Then we need to fix that; allowing things to be added outside the\n> >> sparse-checkout definition is a bug[1][2]. That's an invariant I\n> >> believe we should maintain everywhere; things get really confusing to\n> >> users somewhere later down the road if we don't. Matheus worked to\n> >> fix that with 'git add'; if there are other commands that need fixing\n> >> too, then we should also fix them.\n> >>\n> >> [1] https://lore.kernel.org/git/CABPp-BFhyFiKSXdLM5q5t=ZKzr6V0pY7dbheierRaOHFbMEdkg@mail.gmail.com/\n> >> [2] https://lore.kernel.org/git/CABPp-BF0ZhbSs42R3Bw_r-hbhQ71qtbXSBqXdq0djyaan=8p=A@mail.gmail.com/\n> >>\n> >>> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n> >>> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n> >>\n> >> I think `git checkout --orphan` should just throw an error if\n> >> sparse-checkout is in use. Allowing adding paths outside the\n> >> sparse-checkout set causes too much collateral and deferred confusion\n> >> for users.\n> >\n> > I've been trying to strike an interesting balance of creating\n> > performance improvements without changing behavior, trying to\n> > defer those behavior changes to an isolated instance. I think\n> > that approach is unavoidable with the 'git add' work that I\n> > pulled out of this series and will return to soon.\n> >\n> > However, here I think it would be too much to start throwing\n> > an error in this case. I think that change is a bit too much.\n> >\n> > The thing I can try to do, instead of the current approach, is\n> > to not allow sparse directory entries to differ between the\n> > index and HEAD. That will satisfy this case, but also a lot of\n> > other painful cases.\n> >\n> > I have no idea how to actually accomplish that, but I'll start\n> > digging.\n>\n> It didn't take much digging to discover that this is likely\n> impossible, or rather it would be a drastic change to make this\n> happen.\n>\n> The immediate issue is trying to prevent sparse directory entries\n> from existing when the contained paths don't match what exists at\n> HEAD. However, in the 'git checkout --orphan' case, we are using\n> a full index for the unpack_trees() that updates the in-memory\n> index according to the paths at HEAD, then updates HEAD to point\n> to a non-existing ref. The sparse directories are only created as\n> part of convert_to_sparse() within do_write_index(). At that\n> point, there is no HEAD provided. Trying to load it from scratch\n> violates the fact that HEAD is being staged to change _after_ the\n> index updates in a command like 'git checkout'.\n>\n> So, the drastic change to make this work would be to update the\n> index API to require a root tree to be provided whenever writing\n> the index. However, that doesn't make sense, either! What do we\n> do when in a conflicted state?\n>\n> What if a user modifies HEAD manually to point to a new ref?\n>\n> Such a change would couple the index to the concept of HEAD in\n> an unproductive way, I think. The index data structure exists\n> as a separate entity that is frequently _compared_ to HEAD, and\n> the solution presented in this patch presents a way to keep the\n> comparison of a sparse-index and HEAD to be the same as if we\n> had a full index.\n>\n> So, after looking into it, I'm back in favor of this change and\n> forever allowing sparse cache entries to differ from HEAD,\n> because there is no way to avoid it.\n\nDoh, thanks for digging in and entertaining the idea. I'm worried\nwe'll get lots of confused users over the years from not being able to\ndo this, but you do make some good points.\n\nI still think `git checkout --orphan` should be an error when in a\nsparse checkout -- the point of a sparse checkout is that you only\ncare about a subset of files, whereas checkout --orphan fundamentally\nsays you are throwing away history but care about each and every file\nsince you are staging \"changes\" from all of them to include in some\nnew commit soon. They just seem in strong opposition to me, and it\nseems likely to result in surprises for some of the users when despite\nthe --orphan request and them fixing up the working directory how they\nlike, they get some new commit that contains files that aren't in\ntheir working tree. (In contrast, `git switch --orphan` would\nprobably be fine in a sparse checkout, precisely because it really\ndoes empty everything). However, I do agree with you that such a\nchange belongs in a separate series. So, yes, your patch is good, and\nI'll raise the behavioral change later.\n\n(Sorry for being slow to respond and still not getting to all your\ngood reviews of my series; I'm a bit limited in my time for git right\nnow...)\n"},{"id":"425203","messageId":"5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 01/12] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:30Z","receivedAt":"2021-05-21T11:59:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 6f21397e2ee0..1b49898d0cb7 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 12e6c453024f..4f2f09b53a32 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"425204","messageId":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v3.git.1621017072.gitgitgadget@gmail.com","subject":"[PATCH v4 00/12] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:29Z","receivedAt":"2021-05-21T11:59:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdates in V4\n=============\n\n * The previous patch \"unpack-trees: stop recursing into sparse directories\"\n was confusing, and actually a bit sloppy.\n * It has been replaced with \"unpack-trees: be careful around sparse\n directory entries\" which takes the sparse-directory checks and raises\n them higher up into unpack_trees.c instead of in diff-lib.c.\n\n\nUpdates in V3\n=============\n\nSorry that this was a long time coming. I got a little side-tracked on other\nprojects, but I also worked to get the sparse-index feature working against\nthe Scalar functional tests, which contain many special cases around the\nsparse-checkout feature as they were inherited from special cases that arose\nin the virtualized environment of VFS for Git. This version contains my\nfixes based on that investigation. Most of these were easy to identify and\nfix, but I was blocked for a long time struggling with a bug when combining\nthe sparse-index with the builtin FS Monitor feature, but I've reported my\nfindings already [1].\n\n[1]\nhttps://lore.kernel.org/git/0b9e54ba-ac27-e537-7bef-1b4448f92352@gmail.com/\n\n * Updated comments and tests based on the v2 feedback.\n * Expanded the test repository data shape based on the special cases found\n during my investigation.\n * Added several commits that either fix errors in the status code, or fix\n errors in the previous sparse-index series, specifically:\n * When in a conflict state, the cache-tree fails to update. For now, skip\n writing a sparse-index until this can be resolved more carefully.\n * When expanding a sparse-directory entry, we set the CE_SKIP_WORKTREE\n bit but forgot the CE_EXTENDED bit.\n * git status had failures if there was a sparse-directory entry as the\n first entry within a directory.\n * When expanding a directory to report its status, such as when a\n sparse-directory is staged but doesn't exist at HEAD (such as in an\n orphaned commit) we did not previously recurse correctly into\n subdirectories.\n * Be extra careful with the FS Monitor data when expanding or contracting\n an index. This version now abandons all FS Monitor data at these\n conversion points with the expectation that in the future these\n conversions will be rare so the FS Monitor feature can work\n efficiently. Updates in V2\n\n----------------------------------------------------------------------------\n\n * Based on the feedback, it is clear that 'git add' will require much more\n careful testing and thought. I'm splitting it out of this series and it\n will return with a follow-up.\n * Test cases are improved, both in coverage and organization.\n * The previous \"unpack-trees: make sparse aware\" patch is split into three\n now.\n * Stale messages based on an old implementation of the \"protections\" topic\n are now fixed.\n * Performance tests were re-run.\n\nDerrick Stolee (12):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: be careful around sparse directory entries\n dir.c: accept a directory as part of cone-mode patterns\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n dir.c | 11 +++\n read-cache.c | 10 +-\n sparse-index.c | 27 +++++-\n t/t1092-sparse-checkout-compatibility.sh | 117 ++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 48 ++++++++++\n unpack-trees.c | 26 ++++-\n wt-status.c | 64 ++++++++++++-\n wt-status.h | 1 +\n 9 files changed, 295 insertions(+), 12 deletions(-)\n\n\nbase-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v3:\n\n 1: 5a2ed3d1d701 = 1: 5a2ed3d1d701 sparse-index: skip indexes with unmerged entries\n 2: 8aa41e749471 = 2: 8aa41e749471 sparse-index: include EXTENDED flag when expanding\n 3: 70971b1f9261 = 3: 70971b1f9261 t1092: expand repository data shape\n 4: a80b5a41153f = 4: a80b5a41153f t1092: add tests for status/add and sparse files\n 5: 07a45b661c4a = 5: 07a45b661c4a unpack-trees: preserve cache_bottom\n 6: cc4a526e7947 = 6: cc4a526e7947 unpack-trees: compare sparse directories correctly\n 7: 598375d3531f < -: ------------ unpack-trees: stop recursing into sparse directories\n -: ------------ > 7: e28df7f9395d unpack-trees: be careful around sparse directory entries\n 8: 47da2b317237 = 8: 2cc3a93d4434 dir.c: accept a directory as part of cone-mode patterns\n 9: bc1512981493 = 9: 5011feb1aa04 status: skip sparse-checkout percentage with sparse-index\n 10: 5b1ae369a7cd = 10: 9f2ce5301dc9 status: use sparse-index throughout\n 11: 3b42783d4a86 = 11: 24417e095243 wt-status: expand added sparse directory entries\n 12: b72507f514d1 = 12: 584d4b559a91 fsmonitor: integrate with sparse index\n\n-- \ngitgitgadget\n"},{"id":"425205","messageId":"70971b1f9261f2623ad6d6393be1a8e191003eb4.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 03/12] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:32Z","receivedAt":"2021-05-21T12:00:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4f2f09b53a32..98257695979a 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,16 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n-- \ngitgitgadget\n\n"},{"id":"425206","messageId":"8aa41e749471df3bd9d593b8f55db6506eafea12.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 02/12] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:31Z","receivedAt":"2021-05-21T12:00:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but they must also be\nmarked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\ncorrectly written to disk in the case that the index is not converted\nback down to a sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 1b49898d0cb7..b2b3fbd75050 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"425207","messageId":"a80b5a41153f07a58167be9c9759f3af3048cc5d.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 04/12] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:33Z","receivedAt":"2021-05-21T12:00:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 98257695979a..fba98d5484ae 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -238,6 +238,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"425208","messageId":"07a45b661c4a6ecd63ede19c9e058a8879069742.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 05/12] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:34Z","receivedAt":"2021-05-21T12:00:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dddf106d5bd4..1067db19c9d2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"425209","messageId":"cc4a526e7947f40a0ef65396f2ec8d21d2445edf.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 06/12] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:35Z","receivedAt":"2021-05-21T12:00:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1067db19c9d2..ef6a2b1c951c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"425210","messageId":"e28df7f9395da33f24d6b75fa30081074ac6b801.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 07/12] unpack-trees: be careful around sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:36Z","receivedAt":"2021-05-21T12:00:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe methods traverse_by_cache_tree() and unpack_nondirectories() have\nsimilar behavior in trying to demonstrate the difference between and\nindex and a tree, with some differences about how they walk the index.\n\nEach of these is expecting every cache entry to correspond to a file\npath. We need to skip over the sparse directory entries in the case of a\nsparse-index. Those entries are discovered in the portion that looks for\nsubtrees among the cache entries by scanning the paths for slashes.\n\nSkipping these sparse directory entries will have a measurable effect\nwhen we relax 'git status' to work with sparse-indexes: without this\nchange these methods would call call_unpack_fn() which in turn calls\noneway_diff() and then shows these sparse directory entries as added or\nmodified files.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex ef6a2b1c951c..22634d98e72b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -802,6 +802,9 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,\n \n \t\tsrc[0] = o->src_index->cache[pos + i];\n \n+\t\tif (S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tlen = ce_namelen(src[0]);\n \t\tnew_ce_len = cache_entry_size(len);\n \n@@ -1074,6 +1077,9 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\tif (src[0] && S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n-- \ngitgitgadget\n\n"},{"id":"425211","messageId":"2cc3a93d4434d546cf385fa2c902064748f1b2b6.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 08/12] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:37Z","receivedAt":"2021-05-21T12:00:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 166238e79f52..ab76ef286495 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"425212","messageId":"5011feb1aa04885f6bb272305913257db5c304eb.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 09/12] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:38Z","receivedAt":"2021-05-21T12:00:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex fba98d5484ae..34dae7fbcadd 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -202,6 +202,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 0c8287a023e4..0425169c1895 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e1..ab9cc9d8f032 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"425214","messageId":"9f2ce5301dc944d54f748390655dcc7ca3f0f9f9.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 10/12] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:39Z","receivedAt":"2021-05-21T12:00:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex cf0c36d1dcb2..e529da7beadd 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 29ffa9ac5db9..f80e26831b36 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 34dae7fbcadd..59faf7381093 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -479,12 +479,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"425213","messageId":"24417e095243979b13acde785a5bbd79145b050d.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 11/12] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:40Z","receivedAt":"2021-05-21T12:00:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++\n wt-status.c | 50 ++++++++++++++++++++++++\n 2 files changed, 78 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 59faf7381093..cd3669d36b53 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\ttest_all_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 0425169c1895..90db8bd659fa 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trun_diff_index(&rev, 1);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tchar *full_name;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n+\tit = string_list_insert(&s->change, full_name);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps;\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tmemset(&ps, 0, sizeof(ps));\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"425215","messageId":"584d4b559a91213287221b9731281431fc557627.1621598382.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v4 12/12] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-21T11:59:41Z","receivedAt":"2021-05-21T12:00:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b2b3fbd75050..32ba0d17ef7c 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 45d025f96010..f70fe961902e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"425764","messageId":"73d09910-6d1e-014a-ff0a-931cdb71b31b@gmail.com","threadId":"55489","inReplyTo":"e28df7f9395da33f24d6b75fa30081074ac6b801.1621598382.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 07/12] unpack-trees: be careful around sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-28T11:36:32Z","receivedAt":"2021-05-28T11:36:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/21/2021 7:59 AM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The methods traverse_by_cache_tree() and unpack_nondirectories() have\n> similar behavior in trying to demonstrate the difference between and\n> index and a tree, with some differences about how they walk the index.\n\nAs I have been working on further sparse-index integrations,\nspecifically with 'git checkout', I have found an issue with this\npatch that doesn't show itself in the current t1092 test script,\nbut appears later as more complicated scenarios appear.\n\nI am pursuing the correct fix (that will also make 'git checkout'\nwork better) but it might be a week or two before I can send a v5\nwith that fix.\n\nThanks,\n-Stolee\n"},{"id":"426582","messageId":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v4.git.1621598382.gitgitgadget@gmail.com","subject":"[PATCH v5 00/14] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:33:58Z","receivedAt":"2021-06-07T12:34:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdates in V5\n=============\n\nI replaced one patch with a few that are more complicated. The reason is\nthat I started integrating with git checkout and realized that some of the\nchanges I was making in unpack-trees.c were incorrect for that situation, so\nI might as well do them right here. The tests can't demonstrate the bugs\nwith the previous case until we integrate with git checkout, which will\nfollow in another series after this one is submitted.\n\nFor testing, I've integrated this series along with extensions that work for\ngit commit and git checkout into the Scalar functional tests, which test\nmany scenarios with cone mode sparse-checkout and hence provides good\nevidence that this is working correctly.\n\n\nUpdates in V4\n=============\n\n * The previous patch \"unpack-trees: stop recursing into sparse directories\"\n was confusing, and actually a bit sloppy.\n * It has been replaced with \"unpack-trees: be careful around sparse\n directory entries\" which takes the sparse-directory checks and raises\n them higher up into unpack_trees.c instead of in diff-lib.c.\n\n\nUpdates in V3\n=============\n\nSorry that this was a long time coming. I got a little side-tracked on other\nprojects, but I also worked to get the sparse-index feature working against\nthe Scalar functional tests, which contain many special cases around the\nsparse-checkout feature as they were inherited from special cases that arose\nin the virtualized environment of VFS for Git. This version contains my\nfixes based on that investigation. Most of these were easy to identify and\nfix, but I was blocked for a long time struggling with a bug when combining\nthe sparse-index with the builtin FS Monitor feature, but I've reported my\nfindings already [1].\n\n[1]\nhttps://lore.kernel.org/git/0b9e54ba-ac27-e537-7bef-1b4448f92352@gmail.com/\n\n * Updated comments and tests based on the v2 feedback.\n * Expanded the test repository data shape based on the special cases found\n during my investigation.\n * Added several commits that either fix errors in the status code, or fix\n errors in the previous sparse-index series, specifically:\n * When in a conflict state, the cache-tree fails to update. For now, skip\n writing a sparse-index until this can be resolved more carefully.\n * When expanding a sparse-directory entry, we set the CE_SKIP_WORKTREE\n bit but forgot the CE_EXTENDED bit.\n * git status had failures if there was a sparse-directory entry as the\n first entry within a directory.\n * When expanding a directory to report its status, such as when a\n sparse-directory is staged but doesn't exist at HEAD (such as in an\n orphaned commit) we did not previously recurse correctly into\n subdirectories.\n * Be extra careful with the FS Monitor data when expanding or contracting\n an index. This version now abandons all FS Monitor data at these\n conversion points with the expectation that in the future these\n conversions will be rare so the FS Monitor feature can work\n efficiently. Updates in V2\n\n----------------------------------------------------------------------------\n\n * Based on the feedback, it is clear that 'git add' will require much more\n careful testing and thought. I'm splitting it out of this series and it\n will return with a follow-up.\n * Test cases are improved, both in coverage and organization.\n * The previous \"unpack-trees: make sparse aware\" patch is split into three\n now.\n * Stale messages based on an old implementation of the \"protections\" topic\n are now fixed.\n * Performance tests were re-run.\n\nDerrick Stolee (14):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: replace incorrect 'echo' with 'cat'\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: unpack sparse directory entries\n dir.c: accept a directory as part of cone-mode patterns\n diff-lib: handle index diffs with sparse dirs\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n diff-lib.c | 188 +++++++++++++++++++++++\n dir.c | 11 ++\n read-cache.c | 10 +-\n sparse-index.c | 27 +++-\n t/t1092-sparse-checkout-compatibility.sh | 158 ++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 48 ++++++\n unpack-trees.c | 121 +++++++++++++--\n wt-status.c | 64 +++++++-\n wt-status.h | 1 +\n 10 files changed, 607 insertions(+), 24 deletions(-)\n\n\nbase-commit: f723f370c89ad61f4f40aabfd3540b1ce19c00e5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v4:\n\n 1: 5a2ed3d1d701 = 1: 5a2ed3d1d701 sparse-index: skip indexes with unmerged entries\n 2: 8aa41e749471 = 2: 8aa41e749471 sparse-index: include EXTENDED flag when expanding\n -: ------------ > 3: b99371c7dd61 t1092: replace incorrect 'echo' with 'cat'\n 3: 70971b1f9261 ! 4: f4dddac1859e t1092: expand repository data shape\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'setup' '\n +\t\tmkdir deep/deeper1/0/0 &&\n +\t\ttouch deep/deeper1/0/1 &&\n +\t\ttouch deep/deeper1/0/0/0 &&\n ++\t\tcp -r deep/deeper1/0 folder1 &&\n ++\t\tcp -r deep/deeper1/0 folder2 &&\n ++\t\techo >>folder1/0/0/0 &&\n ++\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'setup' '\n + \t\tmv folder1/a folder2/b &&\n + \t\tmv folder1/larger-content folder2/edited-content &&\n + \t\techo >>folder2/edited-content &&\n ++\t\techo >>folder2/0/1 &&\n ++\t\techo stuff >>deep/deeper1/a &&\n + \t\tgit add . &&\n + \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n + \n + \t\tgit checkout -b rename-out-to-in rename-base &&\n + \t\tmv folder1/a deep/deeper1/b &&\n ++\t\techo more stuff >>deep/deeper1/a &&\n ++\t\trm folder2/0/1 &&\n ++\t\tmkdir folder2/0/1 &&\n ++\t\techo >>folder2/0/1/1 &&\n + \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n + \t\techo >>deep/deeper1/edited-content &&\n + \t\tgit add . &&\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'setup' '\n + \n + \t\tgit checkout -b rename-in-to-out rename-base &&\n + \t\tmv deep/deeper1/a folder1/b &&\n ++\t\techo >>folder2/0/1 &&\n ++\t\trm -rf folder1/0/0 &&\n ++\t\techo >>folder1/0/0 &&\n + \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n + \t\techo >>folder1/edited-content &&\n + \t\tgit add . &&\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'diff --staged' '\n + \ttest_all_match git diff --staged\n + '\n + \n +-test_expect_success 'diff with renames' '\n ++test_expect_success 'diff with renames and conflicts' '\n + \tinit_repos &&\n + \n + \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n + \tdo\n + \t\ttest_all_match git checkout rename-base &&\n + \t\ttest_all_match git checkout $branch -- .&&\n ++\t\ttest_all_match git status --porcelain=v2 &&\n ++\t\ttest_all_match git diff --staged --no-renames &&\n ++\t\ttest_all_match git diff --staged --find-renames || return 1\n ++\tdone\n ++'\n ++\n ++test_expect_success 'diff with directory/file conflicts' '\n ++\tinit_repos &&\n ++\n ++\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n ++\tdo\n ++\t\tgit -C full-checkout reset --hard &&\n ++\t\ttest_sparse_match git reset --hard &&\n ++\t\ttest_all_match git checkout $branch &&\n ++\t\ttest_all_match git checkout rename-base -- . &&\n ++\t\ttest_all_match git status --porcelain=v2 &&\n + \t\ttest_all_match git diff --staged --no-renames &&\n + \t\ttest_all_match git diff --staged --find-renames || return 1\n + \tdone\n 4: a80b5a41153f = 5: 856346b72f79 t1092: add tests for status/add and sparse files\n 5: 07a45b661c4a = 6: f3f6223e955f unpack-trees: preserve cache_bottom\n 6: cc4a526e7947 = 7: 45ae96adf285 unpack-trees: compare sparse directories correctly\n 7: e28df7f9395d ! 8: 724194eef9f6 unpack-trees: be careful around sparse directory entries\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - unpack-trees: be careful around sparse directory entries\n + unpack-trees: unpack sparse directory entries\n \n - The methods traverse_by_cache_tree() and unpack_nondirectories() have\n - similar behavior in trying to demonstrate the difference between and\n - index and a tree, with some differences about how they walk the index.\n + During unpack_callback(), index entries are compared against tree\n + entries. These are matched according to names and types. One goal is to\n + decide if we should recurse into subtrees or simply operate on one index\n + entry.\n \n - Each of these is expecting every cache entry to correspond to a file\n - path. We need to skip over the sparse directory entries in the case of a\n - sparse-index. Those entries are discovered in the portion that looks for\n - subtrees among the cache entries by scanning the paths for slashes.\n + In the case of a sparse-directory entry, we do not want to recurse into\n + that subtree and instead simply compare the trees. In some cases, we\n + might want to perform a merge operation on the entry, such as during\n + 'git checkout ' which wants to replace a sparse tree entry with\n + the tree for that path at the target commit. We extend the logic within\n + unpack_nondirectories() to create a sparse-directory entry in this case,\n + and then that is sent to call_unpack_fn().\n \n - Skipping these sparse directory entries will have a measurable effect\n - when we relax 'git status' to work with sparse-indexes: without this\n - change these methods would call call_unpack_fn() which in turn calls\n - oneway_diff() and then shows these sparse directory entries as added or\n - modified files.\n + There are some subtleties in this process. For instance, we need to\n + update find_cache_entry() to allow finding a sparse-directory entry that\n + exactly matches a given path.\n \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n -@@ unpack-trees.c: static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,\n +@@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n + \tconst struct name_entry *n,\n + \tint stage,\n + \tstruct index_state *istate,\n +-\tint is_transient)\n ++\tint is_transient,\n ++\tint is_sparse_directory)\n + {\n + \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n ++\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n + \tstruct cache_entry *ce =\n + \t\tis_transient ?\n +-\t\tmake_empty_transient_cache_entry(len) :\n +-\t\tmake_empty_cache_entry(istate, len);\n ++\t\tmake_empty_transient_cache_entry(alloc_len) :\n ++\t\tmake_empty_cache_entry(istate, alloc_len);\n \n - \t\tsrc[0] = o->src_index->cache[pos + i];\n + \tce->ce_mode = create_ce_mode(n->mode);\n + \tce->ce_flags = create_ce_flags(stage);\n +@@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n + \t/* len+1 because the cache_entry allocates space for NUL */\n + \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n -+\t\tif (S_ISSPARSEDIR(src[0]->ce_mode))\n -+\t\t\tcontinue;\n ++\tif (is_sparse_directory) {\n ++\t\tce->name[len] = '/';\n ++\t\tce->name[len + 1] = 0;\n ++\t\tce->ce_namelen++;\n ++\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n ++\t}\n +\n - \t\tlen = ce_namelen(src[0]);\n - \t\tnew_ce_len = cache_entry_size(len);\n + \treturn ce;\n + }\n \n @@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n + \t\t\t\t unsigned long dirmask,\n + \t\t\t\t struct cache_entry **src,\n + \t\t\t\t const struct name_entry *names,\n +-\t\t\t\t const struct traverse_info *info)\n ++\t\t\t\t const struct traverse_info *info,\n ++\t\t\t\t int sparse_directory)\n + {\n + \tint i;\n + \tstruct unpack_trees_options *o = info->data;\n + \tunsigned long conflicts = info->df_conflicts | dirmask;\n + \n +-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n -+\tif (src[0] && S_ISSPARSEDIR(src[0]->ce_mode))\n ++\t/* no-op if our cache entry doesn't match the expectations. */\n ++\tif (sparse_directory) {\n ++\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n ++\t\t\tBUG(\"expected sparse directory entry\");\n ++\t} else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n +\t\treturn 0;\n ++\t}\n +\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n +@@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n + \t\t * not stored in the index. otherwise construct the\n + \t\t * cache entry from the index aware logic.\n + \t\t */\n +-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n ++\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n ++\t\t\t\t\t\t &o->result, o->merge,\n ++\t\t\t\t\t\t sparse_directory);\n + \t}\n + \n + \tif (o->merge) {\n +@@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n + static struct cache_entry *find_cache_entry(struct traverse_info *info,\n + \t\t\t\t\t const struct name_entry *p)\n + {\n ++\tstruct cache_entry *ce;\n + \tint pos = find_cache_pos(info, p->path, p->pathlen);\n + \tstruct unpack_trees_options *o = info->data;\n + \n + \tif (0 <= pos)\n + \t\treturn o->src_index->cache[pos];\n +-\telse\n ++\n ++\t/*\n ++\t * Check for a sparse-directory entry named \"path/\".\n ++\t * Due to the input p->path not having a trailing\n ++\t * slash, the negative 'pos' value overshoots the\n ++\t * expected position by one, hence \"-2\" here.\n ++\t */\n ++\tpos = -pos - 2;\n ++\n ++\tif (pos < 0 || pos >= o->src_index->cache_nr)\n ++\t\treturn NULL;\n ++\n ++\tce = o->src_index->cache[pos];\n ++\n ++\tif (!S_ISSPARSEDIR(ce->ce_mode))\n + \t\treturn NULL;\n ++\n ++\t/*\n ++\t * Compare ce->name to info->name + '/' + p->path + '/'\n ++\t * if info->name is non-empty. Compare ce->name to\n ++\t * p-.path + '/' otherwise.\n ++\t */\n ++\tif (info->namelen) {\n ++\t\tif (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n ++\t\t ce->name[info->namelen] == '/' &&\n ++\t\t !strncmp(ce->name, info->name, info->namelen) &&\n ++\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n ++\t\t\treturn ce;\n ++\t} else if (ce->ce_namelen == p->pathlen + 1 &&\n ++\t\t !strncmp(ce->name, p->path, p->pathlen))\n ++\t\treturn ce;\n ++\treturn NULL;\n + }\n + \n + static void debug_path(struct traverse_info *info)\n +@@ unpack-trees.c: static void debug_unpack_callback(int n,\n + \t\tdebug_name_entry(i, names + i);\n + }\n + \n ++/*\n ++ * Returns true if and only if the given cache_entry is a\n ++ * sparse-directory entry that matches the given name_entry\n ++ * from the tree walk at the given traverse_info.\n ++ */\n ++static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n ++{\n ++\tsize_t expected_len, name_start;\n ++\n ++\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n ++\t\treturn 0;\n ++\n ++\tif (info->namelen)\n ++\t\tname_start = info->namelen + 1;\n ++\telse\n ++\t\tname_start = 0;\n ++\texpected_len = name->pathlen + 1 + name_start;\n ++\n ++\tif (ce->ce_namelen != expected_len ||\n ++\t strncmp(ce->name, info->name, info->namelen) ||\n ++\t strncmp(ce->name + name_start, name->path, name->pathlen))\n ++\t\treturn 0;\n ++\n ++\treturn 1;\n ++}\n ++\n + /*\n + * Note that traverse_by_cache_tree() duplicates some logic in this function\n + * without actually calling it. If you change the logic here you may need to\n +@@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n + \t\t}\n + \t}\n + \n +-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n ++\tif (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0)\n + \t\treturn -1;\n + \n + \tif (o->merge && src[0]) {\n +@@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n + \t\t\t}\n + \t\t}\n + \n +-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n +-\t\t\t\t\t names, info) < 0)\n ++\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n ++\t\t\tif (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n ++\t\t\t\treturn -1;\n ++\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n ++\t\t\t\t\t\t names, info) < 0) {\n + \t\t\treturn -1;\n ++\t\t}\n ++\n + \t\treturn mask;\n + \t}\n + \n 8: 2cc3a93d4434 = 9: b8ff179f43e3 dir.c: accept a directory as part of cone-mode patterns\n -: ------------ > 10: b9b97e011293 diff-lib: handle index diffs with sparse dirs\n 9: 5011feb1aa04 = 11: 611b9f61fb2c status: skip sparse-checkout percentage with sparse-index\n 10: 9f2ce5301dc9 = 12: 0c0a765dde80 status: use sparse-index throughout\n 11: 24417e095243 ! 13: 02f2c7b63982 wt-status: expand added sparse directory entries\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n +\tinit_repos &&\n +\n +\ttest_all_match git checkout rename-out-to-in &&\n -+\ttest_all_match git reset --mixed HEAD~1 &&\n ++\n ++\t# Sparse checkouts do not agree with full checkouts about\n ++\t# how to report a directory/file conflict during a reset.\n ++\t# This command would fail with test_all_match because the\n ++\t# full checkout reports \"T folder1/0/1\" while a sparse\n ++\t# checkout reports \"D folder1/0/1\". This matches because\n ++\t# the sparse checkouts skip \"adding\" the other side of\n ++\t# the conflict.\n ++\ttest_sparse_match git reset --mixed HEAD~1 &&\n +\ttest_sparse_match test-tool read-cache --table --expand &&\n -+\ttest_all_match git status --porcelain=v2 &&\n -+\ttest_all_match git status --porcelain=v2 &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n +\n +\t# At this point, sparse-checkouts behave differently\n +\t# from the full-checkout.\n 12: 584d4b559a91 = 14: 46ca150c3548 fsmonitor: integrate with sparse index\n\n-- \ngitgitgadget\n"},{"id":"426583","messageId":"b8ff179f43e3f9713079f9b39d56ae3e3587faae.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 09/14] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:07Z","receivedAt":"2021-06-07T12:34:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 166238e79f52..ab76ef286495 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"426584","messageId":"5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 01/14] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:33:59Z","receivedAt":"2021-06-07T12:34:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 6f21397e2ee0..1b49898d0cb7 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 12e6c453024f..4f2f09b53a32 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"426585","messageId":"45ae96adf28546c586cbf4e67f3ec59090ef2377.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 07/14] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:05Z","receivedAt":"2021-06-07T12:34:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1067db19c9d2..ef6a2b1c951c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"426586","messageId":"724194eef9f6d5514bcd7761daf0eb2bb87c0619.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 08/14] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:06Z","receivedAt":"2021-06-07T12:34:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring unpack_callback(), index entries are compared against tree\nentries. These are matched according to names and types. One goal is to\ndecide if we should recurse into subtrees or simply operate on one index\nentry.\n\nIn the case of a sparse-directory entry, we do not want to recurse into\nthat subtree and instead simply compare the trees. In some cases, we\nmight want to perform a merge operation on the entry, such as during\n'git checkout ' which wants to replace a sparse tree entry with\nthe tree for that path at the target commit. We extend the logic within\nunpack_nondirectories() to create a sparse-directory entry in this case,\nand then that is sent to call_unpack_fn().\n\nThere are some subtleties in this process. For instance, we need to\nupdate find_cache_entry() to allow finding a sparse-directory entry that\nexactly matches a given path.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 101 ++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 91 insertions(+), 10 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex ef6a2b1c951c..ff448ee8424e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1037,13 +1037,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tconst struct name_entry *n,\n \tint stage,\n \tstruct index_state *istate,\n-\tint is_transient)\n+\tint is_transient,\n+\tint is_sparse_directory)\n {\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n+\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len) :\n-\t\tmake_empty_cache_entry(istate, len);\n+\t\tmake_empty_transient_cache_entry(alloc_len) :\n+\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -1052,6 +1054,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \t/* len+1 because the cache_entry allocates space for NUL */\n \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n+\tif (is_sparse_directory) {\n+\t\tce->name[len] = '/';\n+\t\tce->name[len + 1] = 0;\n+\t\tce->ce_namelen++;\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\t}\n+\n \treturn ce;\n }\n \n@@ -1064,16 +1073,24 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t\t\t unsigned long dirmask,\n \t\t\t\t struct cache_entry **src,\n \t\t\t\t const struct name_entry *names,\n-\t\t\t\t const struct traverse_info *info)\n+\t\t\t\t const struct traverse_info *info,\n+\t\t\t\t int sparse_directory)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\t/* no-op if our cache entry doesn't match the expectations. */\n+\tif (sparse_directory) {\n+\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\t\tBUG(\"expected sparse directory entry\");\n+\t} else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n@@ -1103,7 +1120,9 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n+\t\t\t\t\t\t &o->result, o->merge,\n+\t\t\t\t\t\t sparse_directory);\n \t}\n \n \tif (o->merge) {\n@@ -1210,13 +1229,44 @@ static int find_cache_pos(struct traverse_info *info,\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n+\tstruct cache_entry *ce;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n-\telse\n+\n+\t/*\n+\t * Check for a sparse-directory entry named \"path/\".\n+\t * Due to the input p->path not having a trailing\n+\t * slash, the negative 'pos' value overshoots the\n+\t * expected position by one, hence \"-2\" here.\n+\t */\n+\tpos = -pos - 2;\n+\n+\tif (pos < 0 || pos >= o->src_index->cache_nr)\n+\t\treturn NULL;\n+\n+\tce = o->src_index->cache[pos];\n+\n+\tif (!S_ISSPARSEDIR(ce->ce_mode))\n \t\treturn NULL;\n+\n+\t/*\n+\t * Compare ce->name to info->name + '/' + p->path + '/'\n+\t * if info->name is non-empty. Compare ce->name to\n+\t * p-.path + '/' otherwise.\n+\t */\n+\tif (info->namelen) {\n+\t\tif (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n+\t\t ce->name[info->namelen] == '/' &&\n+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n+\t\t\treturn ce;\n+\t} else if (ce->ce_namelen == p->pathlen + 1 &&\n+\t\t !strncmp(ce->name, p->path, p->pathlen))\n+\t\treturn ce;\n+\treturn NULL;\n }\n \n static void debug_path(struct traverse_info *info)\n@@ -1251,6 +1301,32 @@ static void debug_unpack_callback(int n,\n \t\tdebug_name_entry(i, names + i);\n }\n \n+/*\n+ * Returns true if and only if the given cache_entry is a\n+ * sparse-directory entry that matches the given name_entry\n+ * from the tree walk at the given traverse_info.\n+ */\n+static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n+{\n+\tsize_t expected_len, name_start;\n+\n+\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n+\tif (info->namelen)\n+\t\tname_start = info->namelen + 1;\n+\telse\n+\t\tname_start = 0;\n+\texpected_len = name->pathlen + 1 + name_start;\n+\n+\tif (ce->ce_namelen != expected_len ||\n+\t strncmp(ce->name, info->name, info->namelen) ||\n+\t strncmp(ce->name + name_start, name->path, name->pathlen))\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -1307,7 +1383,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n@@ -1337,9 +1413,14 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n-\t\t\t\t\t names, info) < 0)\n+\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n+\t\t\tif (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n+\t\t}\n+\n \t\treturn mask;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"426587","messageId":"611b9f61fb2c834d01ace24b82e1d9791a21c939.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 11/14] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:09Z","receivedAt":"2021-06-07T12:34:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 851a83388e4b..f6b124e0500f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -215,6 +215,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 0c8287a023e4..0425169c1895 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e1..ab9cc9d8f032 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"426588","messageId":"856346b72f7988eed6fe41cc7f1221a221442c98.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 05/14] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:03Z","receivedAt":"2021-06-07T12:34:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 014a507d8b06..851a83388e4b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -251,6 +251,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"426589","messageId":"0c0a765dde80c75499c661d518a40a250b9809c5.1623069253.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 12/14] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:10Z","receivedAt":"2021-06-07T12:34:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex cf0c36d1dcb2..e529da7beadd 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 29ffa9ac5db9..f80e26831b36 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f6b124e0500f..099dc2bf440f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -508,12 +508,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"426590","messageId":"b99371c7dd61bad552231ac964605cf738f90c7f.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 03/14] t1092: replace incorrect 'echo' with 'cat'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:01Z","receivedAt":"2021-06-07T12:35:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis fixes the test data shape to be as expected, allowing rename\ndetection to work properly now that the 'larger-conent' file actually\nhas meaningful lines.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4f2f09b53a32..d55478a1902b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -40,7 +40,7 @@ test_expect_success 'setup' '\n \t\tdone &&\n \n \t\tgit checkout -b rename-base base &&\n-\t\techo >folder1/larger-content <<-\\EOF &&\n+\t\tcat >folder1/larger-content <<-\\EOF &&\n \t\tmatching\n \t\tlines\n \t\thelp\n-- \ngitgitgadget\n\n"},{"id":"426591","messageId":"f3f6223e955f48ea9f7d6bf70a5a230682744285.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 06/14] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:04Z","receivedAt":"2021-06-07T12:35:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dddf106d5bd4..1067db19c9d2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"426592","messageId":"46ca150c35486f6d760f03a5e1f896ac9d4e93d7.1623069253.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 14/14] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:12Z","receivedAt":"2021-06-07T12:35:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b2b3fbd75050..32ba0d17ef7c 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 45d025f96010..f70fe961902e 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"426593","messageId":"8aa41e749471df3bd9d593b8f55db6506eafea12.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:00Z","receivedAt":"2021-06-07T12:35:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but they must also be\nmarked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\ncorrectly written to disk in the case that the index is not converted\nback down to a sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 1b49898d0cb7..b2b3fbd75050 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"426594","messageId":"f4dddac1859e145bef7194cfe9c0177dcc8b4043.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 04/14] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:02Z","receivedAt":"2021-06-07T12:35:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 39 ++++++++++++++++++++++--\n 1 file changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d55478a1902b..014a507d8b06 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,20 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n+\t\tcp -r deep/deeper1/0 folder1 &&\n+\t\tcp -r deep/deeper1/0 folder2 &&\n+\t\techo >>folder1/0/0/0 &&\n+\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n@@ -56,11 +66,17 @@ test_expect_success 'setup' '\n \t\tmv folder1/a folder2/b &&\n \t\tmv folder1/larger-content folder2/edited-content &&\n \t\techo >>folder2/edited-content &&\n+\t\techo >>folder2/0/1 &&\n+\t\techo stuff >>deep/deeper1/a &&\n \t\tgit add . &&\n \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n \n \t\tgit checkout -b rename-out-to-in rename-base &&\n \t\tmv folder1/a deep/deeper1/b &&\n+\t\techo more stuff >>deep/deeper1/a &&\n+\t\trm folder2/0/1 &&\n+\t\tmkdir folder2/0/1 &&\n+\t\techo >>folder2/0/1/1 &&\n \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n \t\techo >>deep/deeper1/edited-content &&\n \t\tgit add . &&\n@@ -68,6 +84,9 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b rename-in-to-out rename-base &&\n \t\tmv deep/deeper1/a folder1/b &&\n+\t\techo >>folder2/0/1 &&\n+\t\trm -rf folder1/0/0 &&\n+\t\techo >>folder1/0/0 &&\n \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n \t\techo >>folder1/edited-content &&\n \t\tgit add . &&\n@@ -262,13 +281,29 @@ test_expect_success 'diff --staged' '\n \ttest_all_match git diff --staged\n '\n \n-test_expect_success 'diff with renames' '\n+test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- .&&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_all_match git diff --staged --no-renames &&\n+\t\ttest_all_match git diff --staged --find-renames || return 1\n+\tdone\n+'\n+\n+test_expect_success 'diff with directory/file conflicts' '\n+\tinit_repos &&\n+\n+\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tdo\n+\t\tgit -C full-checkout reset --hard &&\n+\t\ttest_sparse_match git reset --hard &&\n+\t\ttest_all_match git checkout $branch &&\n+\t\ttest_all_match git checkout rename-base -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n \t\ttest_all_match git diff --staged --no-renames &&\n \t\ttest_all_match git diff --staged --find-renames || return 1\n \tdone\n-- \ngitgitgadget\n\n"},{"id":"426595","messageId":"b9b97e0112939d1787ff1d2a13c48e5b406408db.1623069252.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:08Z","receivedAt":"2021-06-07T12:35:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile comparing an index to a tree, we may see a sparse directory entry.\nIn this case, we should compare that portion of the tree to the tree\nrepresented by that entry. This could include a new tree which needs to\nbe expanded to a full list of added files. It could also include an\nexisting tree, in which case all of the changes inside are important to\ndescribe, including the modifications, additions, and deletions. Note\nthat the case where the tree has a path and the index does not remains\nidentical to before: the lack of a cache entry is the same with a sparse\nindex.\n\nIn the case where a tree is modified, we need to expand the tree\nrecursively, and start comparing each contained entry as either an\naddition, deletion, or modification. This causes an interesting\nrecursion that did not exist before.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 188 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 188 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex b73cc1859a49..ba4c683d4bc4 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -314,6 +314,48 @@ static int get_stat_data(const struct cache_entry *ce,\n \treturn 0;\n }\n \n+struct show_new_tree_context {\n+\tstruct rev_info *revs;\n+\tunsigned added:1;\n+};\n+\n+static int show_new_file_from_tree(const struct object_id *oid,\n+\t\t\t\t struct strbuf *base, const char *path,\n+\t\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct show_new_tree_context *ctx = context;\n+\tstruct cache_entry *new_file = make_transient_cache_entry(mode, oid, path, /* stage */ 0);\n+\n+\tdiff_index_show_file(ctx->revs, ctx->added ? \"+\" : \"-\", new_file, oid, !is_null_oid(oid), mode, 0);\n+\tdiscard_cache_entry(new_file);\n+\treturn 0;\n+}\n+\n+static void show_directory(struct rev_info *revs,\n+\t\t\t const struct cache_entry *new_dir,\n+\t\t\t int added)\n+{\n+\t/*\n+\t * new_dir is a sparse directory entry, so we want to collect all\n+\t * of the new files within the tree. This requires recursively\n+\t * expanding the trees.\n+\t */\n+\tstruct show_new_tree_context ctx = { revs, added };\n+\tstruct repository *r = revs->repo;\n+\tstruct strbuf base = STRBUF_INIT;\n+\tstruct pathspec ps;\n+\tstruct tree *tree = lookup_tree(r, &new_dir->oid);\n+\n+\tmemset(&ps, 0, sizeof(ps));\n+\tps.recursive = 1;\n+\tps.has_wildcard = 1;\n+\tps.max_depth = -1;\n+\n+\tstrbuf_add(&base, new_dir->name, new_dir->ce_namelen);\n+\tread_tree_at(r, tree, &base, &ps,\n+\t\t\tshow_new_file_from_tree, &ctx);\n+}\n+\n static void show_new_file(struct rev_info *revs,\n \t\t\t const struct cache_entry *new_file,\n \t\t\t int cached, int match_missing)\n@@ -322,6 +364,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned int mode;\n \tunsigned dirty_submodule = 0;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tshow_directory(revs, new_file, /*added */ 1);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -333,6 +380,136 @@ static void show_new_file(struct rev_info *revs,\n \tdiff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n }\n \n+static int show_modified(struct rev_info *revs,\n+\t\t\t const struct cache_entry *old_entry,\n+\t\t\t const struct cache_entry *new_entry,\n+\t\t\t int report_missing,\n+\t\t\t int cached, int match_missing);\n+\n+static int compare_within_sparse_dir(int n, unsigned long mask,\n+\t\t\t\t unsigned long dirmask, struct name_entry *entry,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tstruct rev_info *revs = info->data;\n+\tstruct object_id *oid0 = &entry[0].oid;\n+\tstruct object_id *oid1 = &entry[1].oid;\n+\n+\tif (oideq(oid0, oid1))\n+\t\treturn mask;\n+\n+\t/* Directory/file conflicts are handled earlier. */\n+\tif (S_ISDIR(entry[0].mode) && S_ISDIR(entry[1].mode)) {\n+\t\tstruct tree_desc t[2];\n+\t\tvoid *buf[2];\n+\t\tstruct traverse_info info_r = { NULL, };\n+\n+\t\tinfo_r.name = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n+\t\tinfo_r.namelen = strlen(info_r.name);\n+\t\tinfo_r.traverse_path = xstrfmt(\"%s/\", info_r.name);\n+\t\tinfo_r.fn = compare_within_sparse_dir;\n+\t\tinfo_r.prev = info;\n+\t\tinfo_r.mode = entry[0].mode;\n+\t\tinfo_r.pathlen = entry[0].pathlen;\n+\t\tinfo_r.df_conflicts = 0;\n+\t\tinfo_r.data = revs;\n+\n+\t\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], oid0);\n+\t\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], oid1);\n+\n+\t\ttraverse_trees(NULL, 2, t, &info_r);\n+\n+\t\tfree((char *)info_r.name);\n+\t\tfree((char *)info_r.traverse_path);\n+\t\tfree(buf[0]);\n+\t\tfree(buf[1]);\n+\t} else {\n+\t\tchar *old_path = NULL, *new_path = NULL;\n+\t\tstruct cache_entry *old_entry = NULL, *new_entry = NULL;\n+\n+\t\tif (entry[0].path) {\n+\t\t\told_path = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n+\t\t\told_entry = make_transient_cache_entry(\n+\t\t\t\t\tentry[0].mode, &entry[0].oid,\n+\t\t\t\t\told_path, /* stage */ 0);\n+\t\t\told_entry->ce_flags |= CE_SKIP_WORKTREE;\n+\t\t}\n+\t\tif (entry[1].path) {\n+\t\t\tnew_path = xstrfmt(\"%s%s\", info->traverse_path, entry[1].path);\n+\t\t\tnew_entry = make_transient_cache_entry(\n+\t\t\t\t\tentry[1].mode, &entry[1].oid,\n+\t\t\t\t\tnew_path, /* stage */ 0);\n+\t\t\tnew_entry->ce_flags |= CE_SKIP_WORKTREE;\n+\t\t}\n+\n+\t\tif (entry[0].path && entry[1].path)\n+\t\t\tshow_modified(revs, old_entry, new_entry, 0, 1, 0);\n+\t\telse if (entry[0].path)\n+\t\t\tdiff_index_show_file(revs, revs->prefix,\n+\t\t\t\t\t old_entry, &entry[0].oid,\n+\t\t\t\t\t 0, entry[0].mode, 0);\n+\t\telse if (entry[1].path)\n+\t\t\tshow_new_file(revs, new_entry, 1, 0);\n+\n+\t\tdiscard_cache_entry(old_entry);\n+\t\tdiscard_cache_entry(new_entry);\n+\t\tfree(old_path);\n+\t\tfree(new_path);\n+\t}\n+\n+\treturn mask;\n+}\n+\n+static void show_modified_sparse_directory(struct rev_info *revs,\n+\t\t\t const struct cache_entry *old_entry,\n+\t\t\t const struct cache_entry *new_entry,\n+\t\t\t int report_missing,\n+\t\t\t int cached, int match_missing)\n+{\n+\tstruct tree_desc t[2];\n+\tvoid *buf[2];\n+\tstruct traverse_info info = { NULL };\n+\tstruct strbuf name = STRBUF_INIT;\n+\tstruct strbuf parent_path = STRBUF_INIT;\n+\tchar *last_dir_sep;\n+\n+\tif (oideq(&old_entry->oid, &new_entry->oid))\n+\t\treturn;\n+\n+\tinfo.fn = compare_within_sparse_dir;\n+\tinfo.prev = &info;\n+\n+\tstrbuf_add(&name, new_entry->name, new_entry->ce_namelen - 1);\n+\tinfo.name = name.buf;\n+\tinfo.namelen = name.len;\n+\n+\tstrbuf_add(&parent_path, new_entry->name, new_entry->ce_namelen - 1);\n+\tif ((last_dir_sep = find_last_dir_sep(parent_path.buf)) > parent_path.buf)\n+\t\tstrbuf_setlen(&parent_path, (last_dir_sep - parent_path.buf) - 1);\n+\telse\n+\t\tstrbuf_setlen(&parent_path, 0);\n+\n+\tinfo.pathlen = parent_path.len;\n+\n+\tif (parent_path.len)\n+\t\tinfo.traverse_path = parent_path.buf;\n+\telse\n+\t\tinfo.traverse_path = \"\";\n+\n+\tinfo.mode = new_entry->ce_mode;\n+\tinfo.df_conflicts = 0;\n+\tinfo.data = revs;\n+\n+\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], &old_entry->oid);\n+\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], &new_entry->oid);\n+\n+\ttraverse_trees(NULL, 2, t, &info);\n+\n+\tfree(buf[0]);\n+\tfree(buf[1]);\n+\tstrbuf_release(&name);\n+\tstrbuf_release(&parent_path);\n+}\n+\n static int show_modified(struct rev_info *revs,\n \t\t\t const struct cache_entry *old_entry,\n \t\t\t const struct cache_entry *new_entry,\n@@ -343,6 +520,17 @@ static int show_modified(struct rev_info *revs,\n \tconst struct object_id *oid;\n \tunsigned dirty_submodule = 0;\n \n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level.\n+\t */\n+\tif (old_entry && new_entry &&\n+\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n+\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tshow_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n-- \ngitgitgadget\n\n"},{"id":"426596","messageId":"02f2c7b63982cfbafc300e1cd901473d5b9b7297.1623069253.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v5 13/14] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-07T12:34:11Z","receivedAt":"2021-06-07T12:35:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 36 +++++++++++++++++\n wt-status.c | 50 ++++++++++++++++++++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 099dc2bf440f..39b86fbe2be6 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -521,4 +521,40 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\n+\t# Sparse checkouts do not agree with full checkouts about\n+\t# how to report a directory/file conflict during a reset.\n+\t# This command would fail with test_all_match because the\n+\t# full checkout reports \"T folder1/0/1\" while a sparse\n+\t# checkout reports \"D folder1/0/1\". This matches because\n+\t# the sparse checkouts skip \"adding\" the other side of\n+\t# the conflict.\n+\ttest_sparse_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 0425169c1895..90db8bd659fa 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trun_diff_index(&rev, 1);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tchar *full_name;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n+\tit = string_list_insert(&s->change, full_name);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps;\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tmemset(&ps, 0, sizeof(ps));\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"426614","messageId":"b2ce649a-00d0-8418-2a3f-9172ff69eccf@gmail.com","threadId":"55489","inReplyTo":"b9b97e0112939d1787ff1d2a13c48e5b406408db.1623069252.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-07T15:26:03Z","receivedAt":"2021-06-07T15:27:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2021 8:34 AM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n...\n> +\t\t\told_entry = make_transient_cache_entry(\n> +\t\t\t\t\tentry[0].mode, &entry[0].oid,\n> +\t\t\t\t\told_path, /* stage */ 0);\n\nI didn't realize this before I started integrating with\nv2.32.0 (which I should have done before submitting v5) that\nmake_transient_cache_entry() has changed its prototype to\ninclude a memory pool parameter.\n\nI'm working on a v6 that makes only this update and it will\nprobably be ready tomorrow.\n\nThanks,\n-Stolee\n\n"},{"id":"426683","messageId":"xmqqmts1rwr4.fsf@gitster.g","threadId":"55489","inReplyTo":"b2ce649a-00d0-8418-2a3f-9172ff69eccf@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T01:05:51Z","receivedAt":"2021-06-08T01:05:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 6/7/2021 8:34 AM, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n> ...\n>> +\t\t\told_entry = make_transient_cache_entry(\n>> +\t\t\t\t\tentry[0].mode, &entry[0].oid,\n>> +\t\t\t\t\told_path, /* stage */ 0);\n>\n> I didn't realize this before I started integrating with\n> v2.32.0 (which I should have done before submitting v5) that\n> make_transient_cache_entry() has changed its prototype to\n> include a memory pool parameter.\n\nSorry for the trouble---these are usually all known to me for topics\nI happened to have picked up in 'seen', since I try to make it a rule\nthat 'seen' must be a descendant of 'master'.\n\nHow can I usefully communicate the conflicts I find out during the\nintegration cycles to topic owners, I wonder.\n\nThanks.\n"},{"id":"426770","messageId":"b13422ee-0cc8-d77d-cdfa-c8caaf0efdbb@gmail.com","threadId":"55489","inReplyTo":"xmqqmts1rwr4.fsf@gitster.g","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-08T13:00:48Z","receivedAt":"2021-06-08T13:00:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2021 9:05 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 6/7/2021 8:34 AM, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>> ...\n>>> +\t\t\told_entry = make_transient_cache_entry(\n>>> +\t\t\t\t\tentry[0].mode, &entry[0].oid,\n>>> +\t\t\t\t\told_path, /* stage */ 0);\n>>\n>> I didn't realize this before I started integrating with\n>> v2.32.0 (which I should have done before submitting v5) that\n>> make_transient_cache_entry() has changed its prototype to\n>> include a memory pool parameter.\n> \n> Sorry for the trouble---these are usually all known to me for topics\n> I happened to have picked up in 'seen', since I try to make it a rule\n> that 'seen' must be a descendant of 'master'.\n> \n> How can I usefully communicate the conflicts I find out during the\n> integration cycles to topic owners, I wonder.\n\nThis is my fault for stacking topics. I used a GitGitGadget PR\nto target a custom merge of other topics in flight, so my\nmerges were testing against a static target. When those topics\nwere merged, I should have updated my PR to point to 'master' or\neven 'next'.\n\nThanks,\n-Stolee\n"},{"id":"426805","messageId":"CABPp-BGrAU==rEjtX5DNXs05_pcSVzAZBTFxGgGEA+xUO1oS1Q@mail.gmail.com","threadId":"55489","inReplyTo":"8aa41e749471df3bd9d593b8f55db6506eafea12.1623069252.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-08T18:56:09Z","receivedAt":"2021-06-08T19:00:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When creating a full index from a sparse one, we create cache entries\n> for every blob within a given sparse directory entry. These are\n> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n> correctly written to disk in the case that the index is not converted\n> back down to a sparse-index.\n\nIn our previous discussion on this patch from v3\n(https://lore.kernel.org/git/cb9161ca-dc6e-b77b-1a41-385ed8920bb2@gmail.com/),\nyou said you'd explain the reason for this change in a bit more\ndetail, but the commit message has not changed.\n\nCould this be corrected?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> sparse-index.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index 1b49898d0cb7..b2b3fbd75050 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid,\n> strbuf_addstr(base, path);\n>\n> ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n> - ce->ce_flags |= CE_SKIP_WORKTREE;\n> + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n> set_index_entry(istate, istate->cache_nr++, ce);\n>\n> strbuf_setlen(base, len);\n> --\n> gitgitgadget\n"},{"id":"426809","messageId":"CABPp-BGnSeACeXra8pRWEELMS5E0g4bWgBca-21duGp6+nockA@mail.gmail.com","threadId":"55489","inReplyTo":"b99371c7dd61bad552231ac964605cf738f90c7f.1623069252.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 03/14] t1092: replace incorrect 'echo' with 'cat'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-08T19:18:04Z","receivedAt":"2021-06-08T19:21:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> This fixes the test data shape to be as expected, allowing rename\n> detection to work properly now that the 'larger-conent' file actually\n\ns/conent/content/\n\n> has meaningful lines.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 4f2f09b53a32..d55478a1902b 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -40,7 +40,7 @@ test_expect_success 'setup' '\n> done &&\n>\n> git checkout -b rename-base base &&\n> - echo >folder1/larger-content <<-\\EOF &&\n> + cat >folder1/larger-content <<-\\EOF &&\n> matching\n> lines\n> help\n> --\n> gitgitgadget\n"},{"id":"426837","messageId":"CABPp-BGj0y1bKuyaZf14v==j2-mHoiEuV0Am3MBCK4hkbb085Q@mail.gmail.com","threadId":"55489","inReplyTo":"724194eef9f6d5514bcd7761daf0eb2bb87c0619.1623069252.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 08/14] unpack-trees: unpack sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T03:48:27Z","receivedAt":"2021-06-09T03:48:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> During unpack_callback(), index entries are compared against tree\n> entries. These are matched according to names and types. One goal is to\n> decide if we should recurse into subtrees or simply operate on one index\n> entry.\n>\n> In the case of a sparse-directory entry, we do not want to recurse into\n> that subtree and instead simply compare the trees. In some cases, we\n> might want to perform a merge operation on the entry, such as during\n> 'git checkout ' which wants to replace a sparse tree entry with\n> the tree for that path at the target commit. We extend the logic within\n> unpack_nondirectories() to create a sparse-directory entry in this case,\n> and then that is sent to call_unpack_fn().\n\nDoes this presume that all callbacks are prepared to accept a sparse\ndirectory entry? Or do we have an external flag that ensures we do\nnot reach this code path when using callbacks that aren't prepared to\nhandle it properly?\n\nI hope that the answer is the latter, and that the ensure_full_index()\ncalls are what prevents the code from reaching this point if a\ncallback would be used that couldn't handle a sparse directory entry.\n\nI'd be particularly concerned that merge-recursive would call this\ncode with unpack_opts.fn = threeway_merge. threeway_merge is kind of\ninteresting in that it might just happen to not die when passed a\nsparse directory entry, but would pass along data that'd just break\nstuff downstream in various subtle ways. For example, if there were\nconflicts in the sparse directory entries because both had been\nmodified, the merge should recurse and resolve individual paths\nunderneath, which the merge-recursive code would not be prepared to do\nsince unpack_trees() has already returned. Also, even if there wasn't\na \"conflict\" because only one side modified, blindly doing a trivial\ndirectory resolution will break rename detection. I mention\nmerge-recursive not because it's worth fixing (well, it was and the\nfix is called merge-ort) but because I'm most familiar with it. The\nother callbacks _might_ have similar problems, though its possible\nthat it's safe for one- and two- way merging and just fails once you\nget to three-way.\n\n> There are some subtleties in this process. For instance, we need to\n> update find_cache_entry() to allow finding a sparse-directory entry that\n> exactly matches a given path.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 101 ++++++++++++++++++++++++++++++++++++++++++++-----\n> 1 file changed, 91 insertions(+), 10 deletions(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index ef6a2b1c951c..ff448ee8424e 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1037,13 +1037,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n> const struct name_entry *n,\n> int stage,\n> struct index_state *istate,\n> - int is_transient)\n> + int is_transient,\n> + int is_sparse_directory)\n> {\n> size_t len = traverse_path_len(info, tree_entry_len(n));\n> + size_t alloc_len = is_sparse_directory ? len + 1 : len;\n> struct cache_entry *ce =\n> is_transient ?\n> - make_empty_transient_cache_entry(len) :\n> - make_empty_cache_entry(istate, len);\n> + make_empty_transient_cache_entry(alloc_len) :\n> + make_empty_cache_entry(istate, alloc_len);\n>\n> ce->ce_mode = create_ce_mode(n->mode);\n> ce->ce_flags = create_ce_flags(stage);\n> @@ -1052,6 +1054,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n> /* len+1 because the cache_entry allocates space for NUL */\n> make_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n>\n> + if (is_sparse_directory) {\n> + ce->name[len] = '/';\n> + ce->name[len + 1] = 0;\n\nShould this be '\\0', for clarity?\n\n> + ce->ce_namelen++;\n> + ce->ce_flags |= CE_SKIP_WORKTREE;\n> + }\n> +\n> return ce;\n> }\n>\n> @@ -1064,16 +1073,24 @@ static int unpack_nondirectories(int n, unsigned long mask,\n> unsigned long dirmask,\n> struct cache_entry **src,\n> const struct name_entry *names,\n> - const struct traverse_info *info)\n> + const struct traverse_info *info,\n> + int sparse_directory)\n> {\n> int i;\n> struct unpack_trees_options *o = info->data;\n> unsigned long conflicts = info->df_conflicts | dirmask;\n>\n> - /* Do we have *only* directories? Nothing to do */\n\nYou've removed the comment, but not the code. So it still returns\nimmediately if there are only directories...right? Am I missing\nsomething? Is this code still correct? Or is the comment just\nmisleading now that src[0] can be a directory?\n\n> if (mask == dirmask && !src[0])\n> return 0;\n>\n> + /* no-op if our cache entry doesn't match the expectations. */\n> + if (sparse_directory) {\n> + if (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n> + BUG(\"expected sparse directory entry\");\n> + } else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n> + return 0;\n> + }\n\nThis code reads like \"If sparse_directory is false, but the cache\nentry is a sparse directory, we'll just keep it as-is and ignore\nchanged or conflicting directories or files from the names name_entry.\nHowever, I think this has to be coupled with knowledge about changes\nto unpack_callback() you made, where you introduce an extra call to\nunpack_nondirectories() for the sparse directory case, and in the\nsecond one you would do useful work. So \"no-op\" is kind of\nmisleading, it's more deferral until the later unpack_nondirectories()\ncall.\n\nOr, at least so I think after trying to read over this patch. Am I\nunderstanding this right?\n\n> +\n> /*\n> * Ok, we've filled in up to any potential index entry in src[0],\n> * now do the rest.\n> @@ -1103,7 +1120,9 @@ static int unpack_nondirectories(int n, unsigned long mask,\n> * not stored in the index. otherwise construct the\n> * cache entry from the index aware logic.\n> */\n> - src[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n> + src[i + o->merge] = create_ce_entry(info, names + i, stage,\n> + &o->result, o->merge,\n> + sparse_directory);\n> }\n>\n> if (o->merge) {\n> @@ -1210,13 +1229,44 @@ static int find_cache_pos(struct traverse_info *info,\n> static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> const struct name_entry *p)\n> {\n> + struct cache_entry *ce;\n> int pos = find_cache_pos(info, p->path, p->pathlen);\n> struct unpack_trees_options *o = info->data;\n>\n> if (0 <= pos)\n> return o->src_index->cache[pos];\n> - else\n> +\n> + /*\n> + * Check for a sparse-directory entry named \"path/\".\n> + * Due to the input p->path not having a trailing\n> + * slash, the negative 'pos' value overshoots the\n> + * expected position by one, hence \"-2\" here.\n> + */\n> + pos = -pos - 2;\n> +\n> + if (pos < 0 || pos >= o->src_index->cache_nr)\n> + return NULL;\n> +\n> + ce = o->src_index->cache[pos];\n> +\n> + if (!S_ISSPARSEDIR(ce->ce_mode))\n> return NULL;\n> +\n> + /*\n> + * Compare ce->name to info->name + '/' + p->path + '/'\n> + * if info->name is non-empty. Compare ce->name to\n> + * p-.path + '/' otherwise.\n\np->path, not p-.path\n\nAlso, you state in both cases that you are comparing against a\ntrailing '/', but...\n\n> + */\n> + if (info->namelen) {\n> + if (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n> + ce->name[info->namelen] == '/' &&\n> + !strncmp(ce->name, info->name, info->namelen) &&\n> + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n\nYou only checked for one of the two '/' characters here. Are you\nomitting the check for the final '/' do to the S_ISSPARSEDIR() check\nabove?\n\n> + return ce;\n> + } else if (ce->ce_namelen == p->pathlen + 1 &&\n> + !strncmp(ce->name, p->path, p->pathlen))\n\nHere you didn't check for the final '/'. Is that intentional because\nof the S_ISSPARSEDIR() check above? If so, should the comment above\nthis block be corrected?\n\n> + return ce;\n> + return NULL;\n> }\n>\n> static void debug_path(struct traverse_info *info)\n> @@ -1251,6 +1301,32 @@ static void debug_unpack_callback(int n,\n> debug_name_entry(i, names + i);\n> }\n>\n> +/*\n> + * Returns true if and only if the given cache_entry is a\n> + * sparse-directory entry that matches the given name_entry\n> + * from the tree walk at the given traverse_info.\n> + */\n> +static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n> +{\n> + size_t expected_len, name_start;\n> +\n> + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n> + return 0;\n> +\n> + if (info->namelen)\n> + name_start = info->namelen + 1;\n> + else\n> + name_start = 0;\n> + expected_len = name->pathlen + 1 + name_start;\n> +\n> + if (ce->ce_namelen != expected_len ||\n> + strncmp(ce->name, info->name, info->namelen) ||\n> + strncmp(ce->name + name_start, name->path, name->pathlen))\n> + return 0;\n\nWhat about the intervening '/' character? Could we get a false hit\nbetween \"foo/bar/\" and \"foo.bar/\"?\n\nAlso, do we have to worry about the trailing '/'?\n\n> +\n> + return 1;\n> +}\n> +\n> /*\n> * Note that traverse_by_cache_tree() duplicates some logic in this function\n> * without actually calling it. If you change the logic here you may need to\n> @@ -1307,7 +1383,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n> + if (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0)\n> return -1;\n>\n> if (o->merge && src[0]) {\n> @@ -1337,9 +1413,14 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n> }\n> }\n>\n> - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> - names, info) < 0)\n> + if (is_sparse_directory_entry(src[0], names, info)) {\n> + if (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n> + return -1;\n> + } else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n> + names, info) < 0) {\n> return -1;\n> + }\n> +\n> return mask;\n> }\n>\n> --\n> gitgitgadget\n"},{"id":"426853","messageId":"CABPp-BHOcDUJbA0ZRvxhOKryp3wdXj1yOrTovQBJi3PEoAMY_Q@mail.gmail.com","threadId":"55489","inReplyTo":"02f2c7b63982cfbafc300e1cd901473d5b9b7297.1623069253.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 13/14] wt-status: expand added sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T05:27:28Z","receivedAt":"2021-06-09T05:27:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> It is difficult, but possible, to get into a state where we intend to\n> add a directory that is outside of the sparse-checkout definition. Add a\n> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n>\n> This test failed before because the output of 'git status\n> --porcelain=v2' would not match on the lines for folder1/:\n>\n> * The sparse-checkout repo (with a full index) would output each path\n> name that is intended to be added.\n>\n> * The sparse-index repo would only output that \"folder1/\" is staged for\n> addition.\n>\n> The status should report the full list of files to be added, and so this\n> sparse-directory entry should be expanded to a full list when reaching\n> it inside the wt_status_collect_changes_initial() method. Use\n> read_tree_at() to assist.\n>\n> Somehow, this loop over the cache entries was not guarded by\n> ensure_full_index() as intended.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 36 +++++++++++++++++\n> wt-status.c | 50 ++++++++++++++++++++++++\n> 2 files changed, 86 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 099dc2bf440f..39b86fbe2be6 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -521,4 +521,40 @@ test_expect_success 'sparse-index is not expanded' '\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> +test_expect_success 'reset mixed and checkout orphan' '\n> + init_repos &&\n> +\n> + test_all_match git checkout rename-out-to-in &&\n> +\n> + # Sparse checkouts do not agree with full checkouts about\n> + # how to report a directory/file conflict during a reset.\n> + # This command would fail with test_all_match because the\n> + # full checkout reports \"T folder1/0/1\" while a sparse\n> + # checkout reports \"D folder1/0/1\". This matches because\n> + # the sparse checkouts skip \"adding\" the other side of\n> + # the conflict.\n> + test_sparse_match git reset --mixed HEAD~1 &&\n\nOoh! I think you found a sparse-checkout bug here. I agree that\nsparse-checkouts and full-checkouts should give different output in\nthis case, but I don't think the current difference is the correct\none. Digging in a little closer, before running `git reset --mixed\nHEAD~1` I see:\n\n$ git ls-files -t | grep folder\nS folder1/0/0/0\nS folder1/0/1\nS folder2/0/0/0\nS folder2/0/1/1\nS folder2/a\nS folder2/larger-content\n\nand after running git reset --mixed HEAD~1, I see:\nS folder1/0/0/0\nS folder1/0/1\nH folder1/a\nH folder1/larger-content\nS folder2/0/0/0\nH folder2/0/1\nS folder2/a\nS folder2/larger-content\n\nmeaning that the reset of the index failed. It thinks some entries\nare present in the working copy, though it didn't actually check any\nof them out, leaving them to be marked as deleted. This leaves the\nsparse-checkout in a messed up state. To correct it, I need to run\neither of the following:\n\n git diff --diff-filter=D --name-only | xargs git update-index\n--skip-worktree\n\nor\n\n git sparse-checkout reapply\n\n(Though one could ask whether sparse-checkout reapply should take a\nmissing file that isn't SKIP_WORKTREE and determine it's okay to just\nmark it as SKIP_WORKTREE rather than treating it as dirty. I'm not\nsure the answer to that...)\n\nI really think that `git reset --mixed ...` should have been getting\nthe sparsity right on its own without the manual fixup afterwards that\nI needed to add.\n\n> + test_sparse_match test-tool read-cache --table --expand &&\n\nIf both the full and the sparse checkouts do a reset --mixed, I would\nthink that this step should be able to use a test_all_match...at least\nif reset --mixed weren't broken.\n\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_sparse_match git status --porcelain=v2 &&\n\nWhy is this test run twice?\n\n> +\n> + # At this point, sparse-checkouts behave differently\n> + # from the full-checkout.\n> + test_sparse_match git checkout --orphan new-branch &&\n> + test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_sparse_match git status --porcelain=v2\n\nAnd again, you run the status twice...why?\n\n> +'\n> +\n> +test_expect_success 'add everything with deep new file' '\n> + init_repos &&\n> +\n> + run_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n> +\n> + run_on_all touch deep/deeper1/x &&\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git status --porcelain=v2\n\nsame question.\n\n> +'\n> +\n> test_done\n> diff --git a/wt-status.c b/wt-status.c\n> index 0425169c1895..90db8bd659fa 100644\n> --- a/wt-status.c\n> +++ b/wt-status.c\n> @@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n> run_diff_index(&rev, 1);\n> }\n>\n> +static int add_file_to_list(const struct object_id *oid,\n> + struct strbuf *base, const char *path,\n> + unsigned int mode, void *context)\n> +{\n> + struct string_list_item *it;\n> + struct wt_status_change_data *d;\n> + struct wt_status *s = context;\n> + char *full_name;\n> +\n> + if (S_ISDIR(mode))\n> + return READ_TREE_RECURSIVE;\n> +\n> + full_name = xstrfmt(\"%s%s\", base->buf, path);\n> + it = string_list_insert(&s->change, full_name);\n> + d = it->util;\n> + if (!d) {\n> + CALLOC_ARRAY(d, 1);\n> + it->util = d;\n> + }\n> +\n> + d->index_status = DIFF_STATUS_ADDED;\n> + /* Leave {mode,oid}_head zero for adds. */\n> + d->mode_index = mode;\n> + oidcpy(&d->oid_index, oid);\n> + s->committable = 1;\n> + return 0;\n> +}\n> +\n> static void wt_status_collect_changes_initial(struct wt_status *s)\n> {\n> struct index_state *istate = s->repo->index;\n> @@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n> continue;\n> if (ce_intent_to_add(ce))\n> continue;\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + /*\n> + * This is a sparse directory entry, so we want to collect all\n> + * of the added files within the tree. This requires recursively\n> + * expanding the trees to find the elements that are new in this\n> + * tree and marking them with DIFF_STATUS_ADDED.\n> + */\n> + struct strbuf base = STRBUF_INIT;\n> + struct pathspec ps;\n> + struct tree *tree = lookup_tree(istate->repo, &ce->oid);\n> +\n> + memset(&ps, 0, sizeof(ps));\n> + ps.recursive = 1;\n> + ps.has_wildcard = 1;\n> + ps.max_depth = -1;\n> +\n> + strbuf_add(&base, ce->name, ce->ce_namelen);\n> + read_tree_at(istate->repo, tree, &base, &ps,\n> + add_file_to_list, s);\n> + continue;\n> + }\n> +\n> it = string_list_insert(&s->change, ce->name);\n> d = it->util;\n> if (!d) {\n> --\n> gitgitgadget\n>\n"},{"id":"426857","messageId":"CABPp-BEp0NzUqW_pWpTGoUvrB1JMp9fVHr28Sp8RZqV6w0-hEw@mail.gmail.com","threadId":"55489","inReplyTo":"b9b97e0112939d1787ff1d2a13c48e5b406408db.1623069252.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T05:47:27Z","receivedAt":"2021-06-09T05:47:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> While comparing an index to a tree, we may see a sparse directory entry.\n> In this case, we should compare that portion of the tree to the tree\n> represented by that entry. This could include a new tree which needs to\n> be expanded to a full list of added files. It could also include an\n> existing tree, in which case all of the changes inside are important to\n> describe, including the modifications, additions, and deletions. Note\n> that the case where the tree has a path and the index does not remains\n> identical to before: the lack of a cache entry is the same with a sparse\n> index.\n>\n> In the case where a tree is modified, we need to expand the tree\n> recursively, and start comparing each contained entry as either an\n> addition, deletion, or modification. This causes an interesting\n> recursion that did not exist before.\n\nSo, I haven't read through this in detail yet...but there's a big\nquestion I'm curious about:\n\nGit already has code for comparing an index to a tree, a tree to a\ntree, or a tree to the working directory, right? So, when comparing a\nsparse-index to a tree...can't we re-use the compare a tree to a tree\ncode when we hit a sparse directory?\n\nMaybe there's a really good reason to conceptually duplicate the\ncompare a tree to a tree code, but it seems the commit message should\nat least address that reason and why we need to reimplement that\nlogic.\n\n\n> Signed-off-by: Derrick Stolee \n> ---\n> diff-lib.c | 188 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 188 insertions(+)\n>\n> diff --git a/diff-lib.c b/diff-lib.c\n> index b73cc1859a49..ba4c683d4bc4 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -314,6 +314,48 @@ static int get_stat_data(const struct cache_entry *ce,\n> return 0;\n> }\n>\n> +struct show_new_tree_context {\n> + struct rev_info *revs;\n> + unsigned added:1;\n> +};\n> +\n> +static int show_new_file_from_tree(const struct object_id *oid,\n> + struct strbuf *base, const char *path,\n> + unsigned int mode, void *context)\n> +{\n> + struct show_new_tree_context *ctx = context;\n> + struct cache_entry *new_file = make_transient_cache_entry(mode, oid, path, /* stage */ 0);\n> +\n> + diff_index_show_file(ctx->revs, ctx->added ? \"+\" : \"-\", new_file, oid, !is_null_oid(oid), mode, 0);\n> + discard_cache_entry(new_file);\n> + return 0;\n> +}\n> +\n> +static void show_directory(struct rev_info *revs,\n> + const struct cache_entry *new_dir,\n> + int added)\n> +{\n> + /*\n> + * new_dir is a sparse directory entry, so we want to collect all\n> + * of the new files within the tree. This requires recursively\n> + * expanding the trees.\n> + */\n> + struct show_new_tree_context ctx = { revs, added };\n> + struct repository *r = revs->repo;\n> + struct strbuf base = STRBUF_INIT;\n> + struct pathspec ps;\n> + struct tree *tree = lookup_tree(r, &new_dir->oid);\n> +\n> + memset(&ps, 0, sizeof(ps));\n> + ps.recursive = 1;\n> + ps.has_wildcard = 1;\n> + ps.max_depth = -1;\n> +\n> + strbuf_add(&base, new_dir->name, new_dir->ce_namelen);\n> + read_tree_at(r, tree, &base, &ps,\n> + show_new_file_from_tree, &ctx);\n> +}\n> +\n> static void show_new_file(struct rev_info *revs,\n> const struct cache_entry *new_file,\n> int cached, int match_missing)\n> @@ -322,6 +364,11 @@ static void show_new_file(struct rev_info *revs,\n> unsigned int mode;\n> unsigned dirty_submodule = 0;\n>\n> + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n> + show_directory(revs, new_file, /*added */ 1);\n> + return;\n> + }\n> +\n> /*\n> * New file in the index: it might actually be different in\n> * the working tree.\n> @@ -333,6 +380,136 @@ static void show_new_file(struct rev_info *revs,\n> diff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n> }\n>\n> +static int show_modified(struct rev_info *revs,\n> + const struct cache_entry *old_entry,\n> + const struct cache_entry *new_entry,\n> + int report_missing,\n> + int cached, int match_missing);\n> +\n> +static int compare_within_sparse_dir(int n, unsigned long mask,\n> + unsigned long dirmask, struct name_entry *entry,\n> + struct traverse_info *info)\n> +{\n> + struct rev_info *revs = info->data;\n> + struct object_id *oid0 = &entry[0].oid;\n> + struct object_id *oid1 = &entry[1].oid;\n> +\n> + if (oideq(oid0, oid1))\n> + return mask;\n> +\n> + /* Directory/file conflicts are handled earlier. */\n> + if (S_ISDIR(entry[0].mode) && S_ISDIR(entry[1].mode)) {\n> + struct tree_desc t[2];\n> + void *buf[2];\n> + struct traverse_info info_r = { NULL, };\n> +\n> + info_r.name = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n> + info_r.namelen = strlen(info_r.name);\n> + info_r.traverse_path = xstrfmt(\"%s/\", info_r.name);\n> + info_r.fn = compare_within_sparse_dir;\n> + info_r.prev = info;\n> + info_r.mode = entry[0].mode;\n> + info_r.pathlen = entry[0].pathlen;\n> + info_r.df_conflicts = 0;\n> + info_r.data = revs;\n> +\n> + buf[0] = fill_tree_descriptor(revs->repo, &t[0], oid0);\n> + buf[1] = fill_tree_descriptor(revs->repo, &t[1], oid1);\n> +\n> + traverse_trees(NULL, 2, t, &info_r);\n> +\n> + free((char *)info_r.name);\n> + free((char *)info_r.traverse_path);\n> + free(buf[0]);\n> + free(buf[1]);\n> + } else {\n> + char *old_path = NULL, *new_path = NULL;\n> + struct cache_entry *old_entry = NULL, *new_entry = NULL;\n> +\n> + if (entry[0].path) {\n> + old_path = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n> + old_entry = make_transient_cache_entry(\n> + entry[0].mode, &entry[0].oid,\n> + old_path, /* stage */ 0);\n> + old_entry->ce_flags |= CE_SKIP_WORKTREE;\n> + }\n> + if (entry[1].path) {\n> + new_path = xstrfmt(\"%s%s\", info->traverse_path, entry[1].path);\n> + new_entry = make_transient_cache_entry(\n> + entry[1].mode, &entry[1].oid,\n> + new_path, /* stage */ 0);\n> + new_entry->ce_flags |= CE_SKIP_WORKTREE;\n> + }\n> +\n> + if (entry[0].path && entry[1].path)\n> + show_modified(revs, old_entry, new_entry, 0, 1, 0);\n> + else if (entry[0].path)\n> + diff_index_show_file(revs, revs->prefix,\n> + old_entry, &entry[0].oid,\n> + 0, entry[0].mode, 0);\n> + else if (entry[1].path)\n> + show_new_file(revs, new_entry, 1, 0);\n> +\n> + discard_cache_entry(old_entry);\n> + discard_cache_entry(new_entry);\n> + free(old_path);\n> + free(new_path);\n> + }\n> +\n> + return mask;\n> +}\n> +\n> +static void show_modified_sparse_directory(struct rev_info *revs,\n> + const struct cache_entry *old_entry,\n> + const struct cache_entry *new_entry,\n> + int report_missing,\n> + int cached, int match_missing)\n> +{\n> + struct tree_desc t[2];\n> + void *buf[2];\n> + struct traverse_info info = { NULL };\n> + struct strbuf name = STRBUF_INIT;\n> + struct strbuf parent_path = STRBUF_INIT;\n> + char *last_dir_sep;\n> +\n> + if (oideq(&old_entry->oid, &new_entry->oid))\n> + return;\n> +\n> + info.fn = compare_within_sparse_dir;\n> + info.prev = &info;\n> +\n> + strbuf_add(&name, new_entry->name, new_entry->ce_namelen - 1);\n> + info.name = name.buf;\n> + info.namelen = name.len;\n> +\n> + strbuf_add(&parent_path, new_entry->name, new_entry->ce_namelen - 1);\n> + if ((last_dir_sep = find_last_dir_sep(parent_path.buf)) > parent_path.buf)\n> + strbuf_setlen(&parent_path, (last_dir_sep - parent_path.buf) - 1);\n> + else\n> + strbuf_setlen(&parent_path, 0);\n> +\n> + info.pathlen = parent_path.len;\n> +\n> + if (parent_path.len)\n> + info.traverse_path = parent_path.buf;\n> + else\n> + info.traverse_path = \"\";\n> +\n> + info.mode = new_entry->ce_mode;\n> + info.df_conflicts = 0;\n> + info.data = revs;\n> +\n> + buf[0] = fill_tree_descriptor(revs->repo, &t[0], &old_entry->oid);\n> + buf[1] = fill_tree_descriptor(revs->repo, &t[1], &new_entry->oid);\n> +\n> + traverse_trees(NULL, 2, t, &info);\n> +\n> + free(buf[0]);\n> + free(buf[1]);\n> + strbuf_release(&name);\n> + strbuf_release(&parent_path);\n> +}\n> +\n> static int show_modified(struct rev_info *revs,\n> const struct cache_entry *old_entry,\n> const struct cache_entry *new_entry,\n> @@ -343,6 +520,17 @@ static int show_modified(struct rev_info *revs,\n> const struct object_id *oid;\n> unsigned dirty_submodule = 0;\n>\n> + /*\n> + * If both are sparse directory entries, then expand the\n> + * modifications to the file level.\n> + */\n> + if (old_entry && new_entry &&\n> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n> + show_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n> + return 0;\n> + }\n> +\n> if (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n> &dirty_submodule, &revs->diffopt) < 0) {\n> if (report_missing)\n> --\n> gitgitgadget\n>\n"},{"id":"426858","messageId":"xmqqfsxrk0oc.fsf@gitster.g","threadId":"55489","inReplyTo":"CABPp-BEp0NzUqW_pWpTGoUvrB1JMp9fVHr28Sp8RZqV6w0-hEw@mail.gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T06:32:51Z","receivedAt":"2021-06-09T06:32:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> While comparing an index to a tree, we may see a sparse directory entry.\n>> In this case, we should compare that portion of the tree to the tree\n>> represented by that entry. This could include a new tree which needs to\n>> be expanded to a full list of added files. It could also include an\n>> existing tree, in which case all of the changes inside are important to\n>> describe, including the modifications, additions, and deletions. Note\n>> that the case where the tree has a path and the index does not remains\n>> identical to before: the lack of a cache entry is the same with a sparse\n>> index.\n>>\n>> In the case where a tree is modified, we need to expand the tree\n>> recursively, and start comparing each contained entry as either an\n>> addition, deletion, or modification. This causes an interesting\n>> recursion that did not exist before.\n>\n> So, I haven't read through this in detail yet...but there's a big\n> question I'm curious about:\n>\n> Git already has code for comparing an index to a tree, a tree to a\n> tree, or a tree to the working directory, right? So, when comparing a\n> sparse-index to a tree...can't we re-use the compare a tree to a tree\n> code when we hit a sparse directory?\n\nOffhand I do not think of a reason why that cannot work.\n\nThe tree-diff machinery takes two trees, walks them in parallel and\nrepeatedly calls either diff_addremove() or diff_change(), which\nappends diff_filepair() to the diff_queue[] structure. If you see\nan unexpanded tree on the index side, you should be able to pass\nthat tree with the subtree you are comparing against to the tree-diff\nmachinery to come up with a series of filepairs, and then tweak the\npathnames of these filepairs (as such a two-tree comparison would be\ncomparing two trees representing a single subdirectory of two different\nvintages) before adding them to the diff_queue[] you are collecting\nthe index-vs-tree diff, for example.\n\nBut if a part of the index is represented as a tree because it is\noutside the cone of interest, should we even be showing the\ndifference in that part of the tree? If t/ directory is outside the\ncone of interest, should \"git diff HEAD~100 HEAD t/\" show anything\nto begin with (the same question for \"git diff --cached HEAD t/\")?\n"},{"id":"426865","messageId":"CABPp-BHEp6mH3jx6BeRk+u8C-9Q+go0=hHiaxenN_5KQKsR5iw@mail.gmail.com","threadId":"55489","inReplyTo":"xmqqfsxrk0oc.fsf@gitster.g","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T08:11:02Z","receivedAt":"2021-06-09T08:11:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 8, 2021 at 11:32 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n> >> From: Derrick Stolee \n> >>\n> >> While comparing an index to a tree, we may see a sparse directory entry.\n> >> In this case, we should compare that portion of the tree to the tree\n> >> represented by that entry. This could include a new tree which needs to\n> >> be expanded to a full list of added files. It could also include an\n> >> existing tree, in which case all of the changes inside are important to\n> >> describe, including the modifications, additions, and deletions. Note\n> >> that the case where the tree has a path and the index does not remains\n> >> identical to before: the lack of a cache entry is the same with a sparse\n> >> index.\n> >>\n> >> In the case where a tree is modified, we need to expand the tree\n> >> recursively, and start comparing each contained entry as either an\n> >> addition, deletion, or modification. This causes an interesting\n> >> recursion that did not exist before.\n> >\n> > So, I haven't read through this in detail yet...but there's a big\n> > question I'm curious about:\n> >\n> > Git already has code for comparing an index to a tree, a tree to a\n> > tree, or a tree to the working directory, right? So, when comparing a\n> > sparse-index to a tree...can't we re-use the compare a tree to a tree\n> > code when we hit a sparse directory?\n>\n> Offhand I do not think of a reason why that cannot work.\n>\n> The tree-diff machinery takes two trees, walks them in parallel and\n> repeatedly calls either diff_addremove() or diff_change(), which\n> appends diff_filepair() to the diff_queue[] structure. If you see\n> an unexpanded tree on the index side, you should be able to pass\n> that tree with the subtree you are comparing against to the tree-diff\n> machinery to come up with a series of filepairs, and then tweak the\n> pathnames of these filepairs (as such a two-tree comparison would be\n> comparing two trees representing a single subdirectory of two different\n> vintages) before adding them to the diff_queue[] you are collecting\n> the index-vs-tree diff, for example.\n\nGood to know it seems my idea might be reasonable.\n\n> But if a part of the index is represented as a tree because it is\n> outside the cone of interest, should we even be showing the\n> difference in that part of the tree? If t/ directory is outside the\n> cone of interest, should \"git diff HEAD~100 HEAD t/\" show anything\n> to begin with (the same question for \"git diff --cached HEAD t/\")?\n\nExcellent question...and not just for diff, but log, grep with\nrevisions, and other commands. We discussed this a while back[1] and\nwe seemed to lean towards eventually adding a flag because there are\nusecases both for (1) viewing full history while having sparsity paths\nrestrict just the working copy, and (2) also restricting the view of\nhistory to the sparsity paths.\n\n[1] It's been discussed a few times, but there's a relatively\ncomprehensive discussion at the \"Commands that would change for\nbehavior A\" section from\nhttps://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n"},{"id":"426910","messageId":"23508662-37f2-97a4-3cf0-916a3705a272@gmail.com","threadId":"55489","inReplyTo":"CABPp-BGrAU==rEjtX5DNXs05_pcSVzAZBTFxGgGEA+xUO1oS1Q@mail.gmail.com","subject":"Re: [PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-09T17:39:25Z","receivedAt":"2021-06-09T17:40:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/8/2021 2:56 PM, Elijah Newren wrote:\n> On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When creating a full index from a sparse one, we create cache entries\n>> for every blob within a given sparse directory entry. These are\n>> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n>> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n>> correctly written to disk in the case that the index is not converted\n>> back down to a sparse-index.\n> \n> In our previous discussion on this patch from v3\n> (https://lore.kernel.org/git/cb9161ca-dc6e-b77b-1a41-385ed8920bb2@gmail.com/),\n> you said you'd explain the reason for this change in a bit more\n> detail, but the commit message has not changed.\n\nThank you for the reminder.\n\n> Could this be corrected?\n\nHow does this sound?\n\n When creating a full index from a sparse one, we create cache entries\n for every blob within a given sparse directory entry. These are\n correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\n flag is not included. The CE_EXTENDED flag would exist if we loaded a\n full index from disk with these entries marked with CE_SKIP_WORKTREE, so\n we can add the flag here to be consistent. This allows us to directly\n compare the flags present in cache entries when testing the sparse-index\n feature, but has no significance to its correctness in the user-facing\n functionality.\n\nI have this in my local branch for now, but can update it before the next\nversion.\n\nThanks,\n-Stolee\n"},{"id":"426913","messageId":"CABPp-BEFdFngS7++u7Bw4BBAC+ja0-2AdkRjxcD21FoAJAHjzA@mail.gmail.com","threadId":"55489","inReplyTo":"23508662-37f2-97a4-3cf0-916a3705a272@gmail.com","subject":"Re: [PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T18:11:45Z","receivedAt":"2021-06-09T18:13:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jun 9, 2021 at 10:39 AM Derrick Stolee wrote:\n>\n> On 6/8/2021 2:56 PM, Elijah Newren wrote:\n> > On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n> >> From: Derrick Stolee \n> >>\n> >> When creating a full index from a sparse one, we create cache entries\n> >> for every blob within a given sparse directory entry. These are\n> >> correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n> >> marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n> >> correctly written to disk in the case that the index is not converted\n> >> back down to a sparse-index.\n> >\n> > In our previous discussion on this patch from v3\n> > (https://lore.kernel.org/git/cb9161ca-dc6e-b77b-1a41-385ed8920bb2@gmail.com/),\n> > you said you'd explain the reason for this change in a bit more\n> > detail, but the commit message has not changed.\n>\n> Thank you for the reminder.\n>\n> > Could this be corrected?\n>\n> How does this sound?\n>\n> When creating a full index from a sparse one, we create cache entries\n> for every blob within a given sparse directory entry. These are\n> correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\n> flag is not included. The CE_EXTENDED flag would exist if we loaded a\n> full index from disk with these entries marked with CE_SKIP_WORKTREE, so\n> we can add the flag here to be consistent. This allows us to directly\n> compare the flags present in cache entries when testing the sparse-index\n> feature, but has no significance to its correctness in the user-facing\n> functionality.\n>\n> I have this in my local branch for now, but can update it before the next\n> version.\n\nThanks; this looks good to me.\n"},{"id":"426930","messageId":"d7839e43-f427-e52f-7cb6-99a932b58132@gmail.com","threadId":"55489","inReplyTo":"CABPp-BGj0y1bKuyaZf14v==j2-mHoiEuV0Am3MBCK4hkbb085Q@mail.gmail.com","subject":"Re: [PATCH v5 08/14] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-09T20:21:07Z","receivedAt":"2021-06-09T20:21:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/8/2021 11:48 PM, Elijah Newren wrote:\n> On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> During unpack_callback(), index entries are compared against tree\n>> entries. These are matched according to names and types. One goal is to\n>> decide if we should recurse into subtrees or simply operate on one index\n>> entry.\n>>\n>> In the case of a sparse-directory entry, we do not want to recurse into\n>> that subtree and instead simply compare the trees. In some cases, we\n>> might want to perform a merge operation on the entry, such as during\n>> 'git checkout ' which wants to replace a sparse tree entry with\n>> the tree for that path at the target commit. We extend the logic within\n>> unpack_nondirectories() to create a sparse-directory entry in this case,\n>> and then that is sent to call_unpack_fn().\n> \n> Does this presume that all callbacks are prepared to accept a sparse\n> directory entry? Or do we have an external flag that ensures we do\n> not reach this code path when using callbacks that aren't prepared to\n> handle it properly?\n> \n> I hope that the answer is the latter, and that the ensure_full_index()\n> calls are what prevents the code from reaching this point if a\n> callback would be used that couldn't handle a sparse directory entry.\n\nTo the best of my knowledge, callbacks that are not protected have\nensure_full_index() protecting them. At minimum, the repository setting\ncommand_requires_full_index is enabled by default, causing a sparse\nindex to be expanded to a full one immediately upon parsing (and also\nafter writing) to protect cases that might be missing. That is, until\nwe can create tests for each command before disabling it for that\ncommand.\n\n> I'd be particularly concerned that merge-recursive would call this\n> code with unpack_opts.fn = threeway_merge. threeway_merge is kind of\n> interesting in that it might just happen to not die when passed a\n> sparse directory entry, but would pass along data that'd just break\n> stuff downstream in various subtle ways. For example, if there were\n> conflicts in the sparse directory entries because both had been\n> modified, the merge should recurse and resolve individual paths\n> underneath, which the merge-recursive code would not be prepared to do\n> since unpack_trees() has already returned. Also, even if there wasn't\n> a \"conflict\" because only one side modified, blindly doing a trivial\n> directory resolution will break rename detection. I mention\n> merge-recursive not because it's worth fixing (well, it was and the\n> fix is called merge-ort) but because I'm most familiar with it. The\n> other callbacks _might_ have similar problems, though its possible\n> that it's safe for one- and two- way merging and just fails once you\n> get to three-way.\n\nI also believe that threeway_merge might be difficult to integrate\nand that its use should be protected with ensure_full_index() even if\n'git merge' in general does not do it by default. We can cross that\nbridge when we get to it. Merge, rebase, and cherry-pick are next on\nmy list of \"commands to integrate with sparse-index\" but I haven't\ndone the work yet to make them work.\n>> + if (is_sparse_directory) {\n>> + ce->name[len] = '/';\n>> + ce->name[len + 1] = 0;\n> \n> Should this be '\\0', for clarity?\n\nSure.\n\n>> @@ -1064,16 +1073,24 @@ static int unpack_nondirectories(int n, unsigned long mask,\n>> unsigned long dirmask,\n>> struct cache_entry **src,\n>> const struct name_entry *names,\n>> - const struct traverse_info *info)\n>> + const struct traverse_info *info,\n>> + int sparse_directory)\n>> {\n>> int i;\n>> struct unpack_trees_options *o = info->data;\n>> unsigned long conflicts = info->df_conflicts | dirmask;\n>>\n>> - /* Do we have *only* directories? Nothing to do */\n> \n> You've removed the comment, but not the code. So it still returns\n> immediately if there are only directories...right? Am I missing\n> something? Is this code still correct? Or is the comment just\n> misleading now that src[0] can be a directory?\nYes, the comment is misleading now that we can call this method\nwith sparse-directory entries. The method name is also a bit\nmisleading: this should perhaps be renamed to unpack_single_entry()\nor something like that. That will signal that we are not recursing\nwith traverse_trees_recursive() as we do in the other case.\n \n>> if (mask == dirmask && !src[0])\n>> return 0;\n>>\n>> + /* no-op if our cache entry doesn't match the expectations. */\n>> + if (sparse_directory) {\n>> + if (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n>> + BUG(\"expected sparse directory entry\");\n>> + } else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n>> + return 0;\n>> + }\n> \n> This code reads like \"If sparse_directory is false, but the cache\n> entry is a sparse directory, we'll just keep it as-is and ignore\n> changed or conflicting directories or files from the names name_entry.\n> However, I think this has to be coupled with knowledge about changes\n> to unpack_callback() you made, where you introduce an extra call to\n> unpack_nondirectories() for the sparse directory case, and in the\n> second one you would do useful work. So \"no-op\" is kind of\n> misleading, it's more deferral until the later unpack_nondirectories()\n> call.\n> \n> Or, at least so I think after trying to read over this patch. Am I\n> understanding this right?\n\nI think they are both correct: we defer until later by doing a no-op\nright now. But using \"defer\" is more informative of the context of\nthis call.\n\n>> +\n>> + /*\n>> + * Compare ce->name to info->name + '/' + p->path + '/'\n>> + * if info->name is non-empty. Compare ce->name to\n>> + * p-.path + '/' otherwise.\n> \n> p->path, not p-.path\n\nThanks!\n \n> Also, you state in both cases that you are comparing against a\n> trailing '/', but...\n> \n>> + */\n>> + if (info->namelen) {\n>> + if (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n>> + ce->name[info->namelen] == '/' &&\n>> + !strncmp(ce->name, info->name, info->namelen) &&\n>> + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n> \n> You only checked for one of the two '/' characters here.\n\nThe first '/' check is to verify that we match \"{info->name}/{p->name}/\"\nand not \"{info->name}.{p->name}/\" ('.' means \"any character\").\n\n> Are you\n> omitting the check for the final '/' do to the S_ISSPARSEDIR() check\n> above?\n\nSince we know at this point that ce is a sparse directory entry, the\nfinal character _must_ be a trailing slash. There is not a trailing\nslash in the input p->path.\n\n>> + return ce;\n>> + } else if (ce->ce_namelen == p->pathlen + 1 &&\n>> + !strncmp(ce->name, p->path, p->pathlen))\n> \n> Here you didn't check for the final '/'. Is that intentional because\n> of the S_ISSPARSEDIR() check above? If so, should the comment above\n> this block be corrected?\n\nYes, will do.\n\n>> + return ce;\n>> + return NULL;\n>> }\n>>\n>> static void debug_path(struct traverse_info *info)\n>> @@ -1251,6 +1301,32 @@ static void debug_unpack_callback(int n,\n>> debug_name_entry(i, names + i);\n>> }\n>>\n>> +/*\n>> + * Returns true if and only if the given cache_entry is a\n>> + * sparse-directory entry that matches the given name_entry\n>> + * from the tree walk at the given traverse_info.\n>> + */\n>> +static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n>> +{\n>> + size_t expected_len, name_start;\n>> +\n>> + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n>> + return 0;\n>> +\n>> + if (info->namelen)\n>> + name_start = info->namelen + 1;\n>> + else\n>> + name_start = 0;\n>> + expected_len = name->pathlen + 1 + name_start;\n>> +\n>> + if (ce->ce_namelen != expected_len ||\n>> + strncmp(ce->name, info->name, info->namelen) ||\n>> + strncmp(ce->name + name_start, name->path, name->pathlen))\n>> + return 0;\n> \n> What about the intervening '/' character? Could we get a false hit\n> between \"foo/bar/\" and \"foo.bar/\"?\n\nHere, you are right that I missed this check. I will add it.\n\n> Also, do we have to worry about the trailing '/'?\n\nNo, the index would be malformed without it. Since this code\nis so similar to the other check (just the negation of it)\nI will add a clearly-commented helper method.\n\nThanks,\n-Stolee\n"},{"id":"426933","messageId":"60844f4b-33d4-b6d7-3611-a93cf012d3ff@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHEp6mH3jx6BeRk+u8C-9Q+go0=hHiaxenN_5KQKsR5iw@mail.gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-09T20:33:24Z","receivedAt":"2021-06-09T20:34:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2021 4:11 AM, Elijah Newren wrote:\n> On Tue, Jun 8, 2021 at 11:32 PM Junio C Hamano wrote:\n>>\n>> Elijah Newren writes:\n>>\n>>> On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n>>> wrote:\n>>>>\n>>>> From: Derrick Stolee \n>>>>\n>>>> While comparing an index to a tree, we may see a sparse directory entry.\n>>>> In this case, we should compare that portion of the tree to the tree\n>>>> represented by that entry. This could include a new tree which needs to\n>>>> be expanded to a full list of added files. It could also include an\n>>>> existing tree, in which case all of the changes inside are important to\n>>>> describe, including the modifications, additions, and deletions. Note\n>>>> that the case where the tree has a path and the index does not remains\n>>>> identical to before: the lack of a cache entry is the same with a sparse\n>>>> index.\n>>>>\n>>>> In the case where a tree is modified, we need to expand the tree\n>>>> recursively, and start comparing each contained entry as either an\n>>>> addition, deletion, or modification. This causes an interesting\n>>>> recursion that did not exist before.\n>>>\n>>> So, I haven't read through this in detail yet...but there's a big\n>>> question I'm curious about:\n>>>\n>>> Git already has code for comparing an index to a tree, a tree to a\n>>> tree, or a tree to the working directory, right? So, when comparing a\n>>> sparse-index to a tree...can't we re-use the compare a tree to a tree\n>>> code when we hit a sparse directory?\n>>\n>> Offhand I do not think of a reason why that cannot work.\n>>\n>> The tree-diff machinery takes two trees, walks them in parallel and\n>> repeatedly calls either diff_addremove() or diff_change(), which\n>> appends diff_filepair() to the diff_queue[] structure. If you see\n>> an unexpanded tree on the index side, you should be able to pass\n>> that tree with the subtree you are comparing against to the tree-diff\n>> machinery to come up with a series of filepairs, and then tweak the\n>> pathnames of these filepairs (as such a two-tree comparison would be\n>> comparing two trees representing a single subdirectory of two different\n>> vintages) before adding them to the diff_queue[] you are collecting\n>> the index-vs-tree diff, for example.\n> \n> Good to know it seems my idea might be reasonable.\n\nI agree that this is reasonable. I just didn't look hard enough\nto find existing code for this, since I found traverse_trees and\nthought that _was_ the library for this.\n\n>> But if a part of the index is represented as a tree because it is\n>> outside the cone of interest, should we even be showing the\n>> difference in that part of the tree? If t/ directory is outside the\n>> cone of interest, should \"git diff HEAD~100 HEAD t/\" show anything\n>> to begin with (the same question for \"git diff --cached HEAD t/\")?\n> \n> Excellent question...and not just for diff, but log, grep with\n> revisions, and other commands. We discussed this a while back[1] and\n> we seemed to lean towards eventually adding a flag because there are\n> usecases both for (1) viewing full history while having sparsity paths\n> restrict just the working copy, and (2) also restricting the view of\n> history to the sparsity paths.\n> \n> [1] It's been discussed a few times, but there's a relatively\n> comprehensive discussion at the \"Commands that would change for\n> behavior A\" section from\n> https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n\nYes, we could investigate this behavior change in the future. The\ngood thing is that these points that handle sparse directory\nentries create clear branching points for that future behavior\nchange.\n\nThanks,\n-Stolee\n"},{"id":"426935","messageId":"f90d241b-4b02-4f8a-a2f5-1b47fde69db4@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHOcDUJbA0ZRvxhOKryp3wdXj1yOrTovQBJi3PEoAMY_Q@mail.gmail.com","subject":"Re: [PATCH v5 13/14] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-09T20:49:26Z","receivedAt":"2021-06-09T20:50:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2021 1:27 AM, Elijah Newren wrote:\n> On Mon, Jun 7, 2021 at 5:34 AM Derrick Stolee via GitGitGadget\n> wrote:\n...\n>> +test_expect_success 'reset mixed and checkout orphan' '\n>> + init_repos &&\n>> +\n>> + test_all_match git checkout rename-out-to-in &&\n>> +\n>> + # Sparse checkouts do not agree with full checkouts about\n>> + # how to report a directory/file conflict during a reset.\n>> + # This command would fail with test_all_match because the\n>> + # full checkout reports \"T folder1/0/1\" while a sparse\n>> + # checkout reports \"D folder1/0/1\". This matches because\n>> + # the sparse checkouts skip \"adding\" the other side of\n>> + # the conflict.\n>> + test_sparse_match git reset --mixed HEAD~1 &&\n> \n> Ooh! I think you found a sparse-checkout bug here. I agree that\n> sparse-checkouts and full-checkouts should give different output in\n> this case, but I don't think the current difference is the correct\n> one. Digging in a little closer, before running `git reset --mixed\n> HEAD~1` I see:\n> \n> $ git ls-files -t | grep folder\n> S folder1/0/0/0\n> S folder1/0/1\n> S folder2/0/0/0\n> S folder2/0/1/1\n> S folder2/a\n> S folder2/larger-content\n> \n> and after running git reset --mixed HEAD~1, I see:\n> S folder1/0/0/0\n> S folder1/0/1\n> H folder1/a\n> H folder1/larger-content\n> S folder2/0/0/0\n> H folder2/0/1\n> S folder2/a\n> S folder2/larger-content\n> \n> meaning that the reset of the index failed. It thinks some entries\n> are present in the working copy, though it didn't actually check any\n> of them out, leaving them to be marked as deleted. This leaves the\n> sparse-checkout in a messed up state. To correct it, I need to run\n> either of the following:\n> \n> git diff --diff-filter=D --name-only | xargs git update-index\n> --skip-worktree\n> \n> or\n> \n> git sparse-checkout reapply\n> \n> (Though one could ask whether sparse-checkout reapply should take a\n> missing file that isn't SKIP_WORKTREE and determine it's okay to just\n> mark it as SKIP_WORKTREE rather than treating it as dirty. I'm not\n> sure the answer to that...)\n> \n> I really think that `git reset --mixed ...` should have been getting\n> the sparsity right on its own without the manual fixup afterwards that\n> I needed to add.\n> \n>> + test_sparse_match test-tool read-cache --table --expand &&\n> \n> If both the full and the sparse checkouts do a reset --mixed, I would\n> think that this step should be able to use a test_all_match...at least\n> if reset --mixed weren't broken.\n\nI will add this to my list when getting to 'git reset' integration\nwith sparse-checkout. Thanks.\n\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_sparse_match git status --porcelain=v2 &&\n> \n> Why is this test run twice?\n> \n>> +\n>> + # At this point, sparse-checkouts behave differently\n>> + # from the full-checkout.\n>> + test_sparse_match git checkout --orphan new-branch &&\n>> + test_sparse_match test-tool read-cache --table --expand &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_sparse_match git status --porcelain=v2\n> \n> And again, you run the status twice...why?\n> \n>> +'\n>> +\n>> +test_expect_success 'add everything with deep new file' '\n>> + init_repos &&\n>> +\n>> + run_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n>> +\n>> + run_on_all touch deep/deeper1/x &&\n>> + test_all_match git add . &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_all_match git status --porcelain=v2\n> \n> same question.\n\nThese double 'git status' calls are actually a bit subtle: there\nwas a bug in an earlier version that only appeared when using\n'git status' twice, because the first kept the sparse index\nwithout expanding it, and the bug actually had an incorrect\nresult when writing that index. Only the second 'git status'\nwould notice the problem. I started adding two calls to my tests,\nbut it is not necessary any more.\n\nThe reason to leave it out of the Git tests is that I'm testing\nall of my submissions against the Scalar functional tests which\nrun 'git status' multiple times throughout each test situation\nand that catches the problem as well. In the future, we will have\n'git add' keeping the sparse index in-memory; that will also\nexpose this behavior sufficiently.\n\nThanks,\n-Stolee\n"},{"id":"427052","messageId":"38d0900b-c677-a32d-cc63-e615828e9a5d@gmail.com","threadId":"55489","inReplyTo":"60844f4b-33d4-b6d7-3611-a93cf012d3ff@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-10T17:45:57Z","receivedAt":"2021-06-10T17:46:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/9/2021 4:33 PM, Derrick Stolee wrote:\n> On 6/9/2021 4:11 AM, Elijah Newren wrote:\n>> On Tue, Jun 8, 2021 at 11:32 PM Junio C Hamano wrote:\n>>>\n>>> Elijah Newren writes:\n>>>\n>>> The tree-diff machinery takes two trees, walks them in parallel and\n>>> repeatedly calls either diff_addremove() or diff_change(), which\n>>> appends diff_filepair() to the diff_queue[] structure. If you see\n>>> an unexpanded tree on the index side, you should be able to pass\n>>> that tree with the subtree you are comparing against to the tree-diff\n>>> machinery to come up with a series of filepairs, and then tweak the\n>>> pathnames of these filepairs (as such a two-tree comparison would be\n>>> comparing two trees representing a single subdirectory of two different\n>>> vintages) before adding them to the diff_queue[] you are collecting\n>>> the index-vs-tree diff, for example.\n>>\n>> Good to know it seems my idea might be reasonable.\n> \n> I agree that this is reasonable. I just didn't look hard enough\n> to find existing code for this, since I found traverse_trees and\n> thought that _was_ the library for this.\n\nThis was surprisingly simple, since most of the complicated stuff\nis built into diff_tree_oid() and its use of revs->diffopt. The\nnew patch works as shown below the cut-line.\n\nI was incredibly suspicious of how quickly this came together,\nbut it passes all the tests I have for it (including Scalar\nfunctional tests with the commit, checkout, and add integrations).\n\nI'll send a new version with this patch tomorrow, as well as the\nother recommended edits.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex c2ac9250fe9..b631df89343 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -316,6 +316,13 @@ static int get_stat_data(const struct index_state *istate,\n \treturn 0;\n }\n \n+static void show_directory(struct rev_info *revs,\n+\t\t\t const struct cache_entry *new_dir,\n+\t\t\t int added)\n+{\n+\tdiff_tree_oid(NULL, &new_dir->oid, new_dir->name, &revs->diffopt);\n+}\n+\n static void show_new_file(struct rev_info *revs,\n \t\t\t const struct cache_entry *new_file,\n \t\t\t int cached, int match_missing)\n@@ -325,6 +332,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tshow_directory(revs, new_file, /*added */ 1);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -336,6 +348,15 @@ static void show_new_file(struct rev_info *revs,\n \tdiff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n }\n \n+static void show_modified_sparse_directory(struct rev_info *revs,\n+\t\t\t const struct cache_entry *old_entry,\n+\t\t\t const struct cache_entry *new_entry,\n+\t\t\t int report_missing,\n+\t\t\t int cached, int match_missing)\n+{\n+\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n+}\n+\n static int show_modified(struct rev_info *revs,\n \t\t\t const struct cache_entry *old_entry,\n \t\t\t const struct cache_entry *new_entry,\n@@ -347,6 +368,17 @@ static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level.\n+\t */\n+\tif (old_entry && new_entry &&\n+\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n+\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tshow_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n"},{"id":"427061","messageId":"CABPp-BHFdG9=JfR7fnwSjNgJ2o=ttHn--ogD3_LQLYKe4dV9SQ@mail.gmail.com","threadId":"55489","inReplyTo":"38d0900b-c677-a32d-cc63-e615828e9a5d@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-10T21:31:32Z","receivedAt":"2021-06-10T21:32:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jun 10, 2021 at 10:45 AM Derrick Stolee wrote:\n>\n> On 6/9/2021 4:33 PM, Derrick Stolee wrote:\n> > On 6/9/2021 4:11 AM, Elijah Newren wrote:\n> >> On Tue, Jun 8, 2021 at 11:32 PM Junio C Hamano wrote:\n> >>>\n> >>> Elijah Newren writes:\n> >>>\n> >>> The tree-diff machinery takes two trees, walks them in parallel and\n> >>> repeatedly calls either diff_addremove() or diff_change(), which\n> >>> appends diff_filepair() to the diff_queue[] structure. If you see\n> >>> an unexpanded tree on the index side, you should be able to pass\n> >>> that tree with the subtree you are comparing against to the tree-diff\n> >>> machinery to come up with a series of filepairs, and then tweak the\n> >>> pathnames of these filepairs (as such a two-tree comparison would be\n> >>> comparing two trees representing a single subdirectory of two different\n> >>> vintages) before adding them to the diff_queue[] you are collecting\n> >>> the index-vs-tree diff, for example.\n> >>\n> >> Good to know it seems my idea might be reasonable.\n> >\n> > I agree that this is reasonable. I just didn't look hard enough\n> > to find existing code for this, since I found traverse_trees and\n> > thought that _was_ the library for this.\n>\n> This was surprisingly simple, since most of the complicated stuff\n> is built into diff_tree_oid() and its use of revs->diffopt. The\n> new patch works as shown below the cut-line.\n>\n> I was incredibly suspicious of how quickly this came together,\n> but it passes all the tests I have for it (including Scalar\n> functional tests with the commit, checkout, and add integrations).\n\nNice!\n\n> I'll send a new version with this patch tomorrow, as well as the\n> other recommended edits.\n>\n> Thanks,\n> -Stolee\n>\n> --- >8 ---\n>\n>\n> diff --git a/diff-lib.c b/diff-lib.c\n> index c2ac9250fe9..b631df89343 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -316,6 +316,13 @@ static int get_stat_data(const struct index_state *istate,\n> return 0;\n> }\n>\n> +static void show_directory(struct rev_info *revs,\n> + const struct cache_entry *new_dir,\n> + int added)\n> +{\n> + diff_tree_oid(NULL, &new_dir->oid, new_dir->name, &revs->diffopt);\n> +}\n> +\n> static void show_new_file(struct rev_info *revs,\n> const struct cache_entry *new_file,\n> int cached, int match_missing)\n> @@ -325,6 +332,11 @@ static void show_new_file(struct rev_info *revs,\n> unsigned dirty_submodule = 0;\n> struct index_state *istate = revs->diffopt.repo->index;\n>\n> + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n> + show_directory(revs, new_file, /*added */ 1);\n> + return;\n> + }\n> +\n> /*\n> * New file in the index: it might actually be different in\n> * the working tree.\n> @@ -336,6 +348,15 @@ static void show_new_file(struct rev_info *revs,\n> diff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n> }\n>\n> +static void show_modified_sparse_directory(struct rev_info *revs,\n> + const struct cache_entry *old_entry,\n> + const struct cache_entry *new_entry,\n> + int report_missing,\n> + int cached, int match_missing)\n> +{\n> + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n> +}\n> +\n> static int show_modified(struct rev_info *revs,\n> const struct cache_entry *old_entry,\n> const struct cache_entry *new_entry,\n> @@ -347,6 +368,17 @@ static int show_modified(struct rev_info *revs,\n> unsigned dirty_submodule = 0;\n> struct index_state *istate = revs->diffopt.repo->index;\n>\n> + /*\n> + * If both are sparse directory entries, then expand the\n> + * modifications to the file level.\n> + */\n> + if (old_entry && new_entry &&\n> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n> + show_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n> + return 0;\n> + }\n\nWhat if S_ISSPARSEDIR(old_entry->ce_mode) != S_ISSPARSEDIR(new_entry->ce_mode) ?\n\n> +\n> if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n> &dirty_submodule, &revs->diffopt) < 0) {\n> if (report_missing)\n"},{"id":"427105","messageId":"7d6ca59b-2713-5c10-88de-963efc7779fc@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHFdG9=JfR7fnwSjNgJ2o=ttHn--ogD3_LQLYKe4dV9SQ@mail.gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-11T12:57:52Z","receivedAt":"2021-06-11T12:59:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/10/2021 5:31 PM, Elijah Newren wrote:\n> On Thu, Jun 10, 2021 at 10:45 AM Derrick Stolee wrote:\n>>\n>> On 6/9/2021 4:33 PM, Derrick Stolee wrote:\n>>> On 6/9/2021 4:11 AM, Elijah Newren wrote:\n>>>> On Tue, Jun 8, 2021 at 11:32 PM Junio C Hamano wrote:\n>>>>>\n>>>>> Elijah Newren writes:\n>>>>>\n>>>>> The tree-diff machinery takes two trees, walks them in parallel and\n>>>>> repeatedly calls either diff_addremove() or diff_change(), which\n>>>>> appends diff_filepair() to the diff_queue[] structure. If you see\n>>>>> an unexpanded tree on the index side, you should be able to pass\n>>>>> that tree with the subtree you are comparing against to the tree-diff\n>>>>> machinery to come up with a series of filepairs, and then tweak the\n>>>>> pathnames of these filepairs (as such a two-tree comparison would be\n>>>>> comparing two trees representing a single subdirectory of two different\n>>>>> vintages) before adding them to the diff_queue[] you are collecting\n>>>>> the index-vs-tree diff, for example.\n>>>>\n>>>> Good to know it seems my idea might be reasonable.\n>>>\n>>> I agree that this is reasonable. I just didn't look hard enough\n>>> to find existing code for this, since I found traverse_trees and\n>>> thought that _was_ the library for this.\n>>\n>> This was surprisingly simple, since most of the complicated stuff\n>> is built into diff_tree_oid() and its use of revs->diffopt. The\n>> new patch works as shown below the cut-line.\n>>\n>> I was incredibly suspicious of how quickly this came together,\n>> but it passes all the tests I have for it (including Scalar\n>> functional tests with the commit, checkout, and add integrations).\n> \n> Nice!\n> \n>> I'll send a new version with this patch tomorrow, as well as the\n>> other recommended edits.\n\n...still planning on this today, but...\n\n>> + /*\n>> + * If both are sparse directory entries, then expand the\n>> + * modifications to the file level.\n>> + */\n>> + if (old_entry && new_entry &&\n>> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n>> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n>> + show_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n>> + return 0;\n>> + }\n> \n> What if S_ISSPARSEDIR(old_entry->ce_mode) != S_ISSPARSEDIR(new_entry->ce_mode) ?\n\nYou make a good point that something different would happen\nin the case of a directory/file conflict on the sparse checkout\nboundary. This can be as simple as the trivial \"only files at\nroot\" cone-mode sparse-checkout definition, with \"folder/\" (tree)\nchanging to \"folder\" (blob).\n\nI'll see what I can do to create a test scenario for\nthis and add the correct cases.\n\nThanks,\n-Stolee\n"},{"id":"427132","messageId":"2784d29b-b22a-2bf6-2450-7b4a0a72df54@gmail.com","threadId":"55489","inReplyTo":"7d6ca59b-2713-5c10-88de-963efc7779fc@gmail.com","subject":"Re: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-11T17:27:39Z","receivedAt":"2021-06-11T17:28:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/11/2021 8:57 AM, Derrick Stolee wrote:\n> On 6/10/2021 5:31 PM, Elijah Newren wrote:\n>> On Thu, Jun 10, 2021 at 10:45 AM Derrick Stolee wrote:\n>>>\n>>> I'll send a new version with this patch tomorrow, as well as the\n>>> other recommended edits.\n> \n> ...still planning on this today, but...\n\nSo optimistic!\n \n>>> + /*\n>>> + * If both are sparse directory entries, then expand the\n>>> + * modifications to the file level.\n>>> + */\n>>> + if (old_entry && new_entry &&\n>>> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n>>> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n>>> + show_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n>>> + return 0;\n>>> + }\n>>\n>> What if S_ISSPARSEDIR(old_entry->ce_mode) != S_ISSPARSEDIR(new_entry->ce_mode) ?\n> \n> You make a good point that something different would happen\n> in the case of a directory/file conflict on the sparse checkout\n> boundary. This can be as simple as the trivial \"only files at\n> root\" cone-mode sparse-checkout definition, with \"folder/\" (tree)\n> changing to \"folder\" (blob).\n> \n> I'll see what I can do to create a test scenario for\n> this and add the correct cases.\n\nCreating a directory/file conflict in this way exposes a bug in\na different codepath in unpack_trees(), although it isn't visible\nuntil 'git checkout' allows the index to stay sparse. It's due to\nthe code in unpack_callback() that handles blobs and trees\ndifferently, and hence the blob/tree conflict isn't handled\nappropriately there. The changes from Patch 8 are to blame for\nthese first errors.\n\nAt least, those are the first errors I have discovered with these\nconflicts. There might be other scenarios that care about this\nsection of diff-lib.c, but I have not gotten to a point where\nsuch behavior would be exposed.\n\nI don't expect to succeed in squashing this bug today, so I'll\ntry again next week.\n\nThanks,\n-Stolee\n"},{"id":"428664","messageId":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v5.git.1623069252.gitgitgadget@gmail.com","subject":"[PATCH v6 00/14] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:28Z","receivedAt":"2021-06-29T01:51:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdate in V6\n============\n\nI'm sorry that this revision took so long. Initially I was blocked on\ngetting the directory/file conflict figured out (I did), but also my team\nwas very busy with some things. Eventually, we reached an internal deadline\nto make an experimental release available [1] with initial sparse-index\nperformance boosts. Creating that included some additional review by Jeff\nHostetler and Johannes Schindelin which led to more changes in this version.\n\nThe good news is that this series has now created the basis for many Git\ncommands to integrate with the sparse-index without much additional work.\nThis effort was unfortunately overloaded on this series because the changes\nneeded for things like 'git checkout' or 'git add' all intersect with the\nchanges needed for 'git status'. Might as well get it right the first time.\n\nBecause the range-diff is a big difficult to read this time, I'll break the\nchanges down on a patch-by-patch basis.\n\n 1. sparse-index: skip indexes with unmerged entries\n \n (no change)\n\n 2. sparse-index: include EXTENDED flag when expanding\n\n * Commit message better describes the purpose of the change.\n\n 3. t1092: replace incorrect 'echo' with 'cat'\n\n * Typo fix\n\n 4. t1092: expand repository data shape\n\n * some files are added that surround \"folder1/\" immediately before and\n after, based on the sorting with the trailing slash. This provides extra\n coverage.\n\n 5. t1092: add tests for status/add and sparse files\n \n (no change)\n\n 6. unpack-trees: preserve cache_bottom\n \n (no change)\n\n 7. unpack-trees: compare sparse directories correctly\n\n * We were previosly not comparing the path lengths, which causes a problem\n (with later updates) when a sparse directory such as \"folder1/0/\" gets\n compared to a tree name \"folder1\".\n\n 8. unpack-trees: rename unpack_nondirectories()\n\n * This new commit changes the name to make more sense with its new behavior\n that could modify a sparse directory entry. The point of the method is in\n contrast to recursing into trees.\n\n 9. unpack-trees: unpack sparse directory entries\n\n * THIS is the biggest change from previous versions. There were a few\n things going on that were tricky to get right, especially with the\n directory/file conflict (handled in an update in the following, new\n patch).\n\n * The changes to create_ce_entry() regarding alloc_len missed a spot that\n was critical to getting the length right in the allocated entry.\n\n * Use '\\0' over 0 to represent the terminating character.\n\n * We don't need a \"sparse_directory\" parameter to unpack_nondirectories()\n (which was renamed to unpack_single_entry() by the previous new patch)\n because we can use dirmask to discover if src[0] (or any other value)\n should be a sparse directory entry.\n\n * Similarly, we don't need to call the method twice from unpack_callback().\n\n * The 'conflicts' variable is set to match the dirmask in the beginning,\n but it should depend on whether or not we have a sparse directory entry\n instead, and if all trees that have the path have a directory.\n\n * The implementation of find_cache_entry() uses find_cache_pos() to find an\n insertion position for a path if it doesn't find an exact match. Before,\n we subtracted one to find the sparse directory entry, but there could be\n multiple paths between the sparse directory entry and the insertion\n point, so we need to walk backwards until we find it. This requires many\n paths having the same prefix, so hopefully is a rare case. Some of the\n test data changes were added to cover the need for this logic. This uses\n a helper method, sparse_dir_matches_path, which is also used by\n is_sparse_directory_entry.\n\n 10. unpack-trees: handle dir/file conflict of sparse entries\n\n * This new logic inside twoway_merge handles the special case for dealing\n with a directory/file conflict during a 'git checkout'. The necessarily\n data and tests are also added here, though the logic will only take\n serious effect when we integrate with 'git checkout' later.\n\n 11. dir.c: accept a directory as part of cone-mode patterns\n\n * The value slash_pos was previously a pointer within a strbuf, but in some\n cases we add to that strbuf and that could reallocate the pointer, making\n slash_pos be invalid. The replacement is to have slash_pos be an integer\n position within the string, so it is consistent even if the string is\n reallocated for an append.\n\n 12. diff-lib: handle index diffs with sparse dirs\n\n * As recommended in the previous review, a simple diff_tree_oid() replaces\n the complicated use of read_tree_at() and traverse_trees() in the\n previous version.\n\n 13. status: skip sparse-checkout percentage with sparse-index\n \n (no change)\n\n 14. status: use sparse-index throughout\n \n (no change)\n\n 15. wt-status: expand added sparse directory entries\n\n * Duplicate 'git status --porcelain=v2' lines are removed from tests.\n\n * The pathspec is initialized using \"= { 0 }\" instead of memset().\n\n 16. fsmonitor: integrate with sparse index\n\n * An extra test_region is added to ensure that the filesystem monitor hook\n is still being called, and we are not simply disabling the feature\n entirely.\n\nDerrick Stolee (14):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: replace incorrect 'echo' with 'cat'\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: unpack sparse directory entries\n dir.c: accept a directory as part of cone-mode patterns\n diff-lib: handle index diffs with sparse dirs\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n diff-lib.c | 16 +++\n dir.c | 11 ++\n read-cache.c | 10 +-\n sparse-index.c | 27 +++-\n t/t1092-sparse-checkout-compatibility.sh | 155 ++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 48 +++++++\n unpack-trees.c | 130 ++++++++++++++++---\n wt-status.c | 63 ++++++++-\n wt-status.h | 1 +\n 10 files changed, 436 insertions(+), 28 deletions(-)\n\n\nbase-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v5:\n\n 1: 5a2ed3d1d70 = 1: 2a4a7256304 sparse-index: skip indexes with unmerged entries\n 2: 8aa41e74947 ! 2: f5bae86014d sparse-index: include EXTENDED flag when expanding\n @@ Commit message\n \n When creating a full index from a sparse one, we create cache entries\n for every blob within a given sparse directory entry. These are\n - correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n - marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n - correctly written to disk in the case that the index is not converted\n - back down to a sparse-index.\n + correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\n + flag is not included. The CE_EXTENDED flag would exist if we loaded a\n + full index from disk with these entries marked with CE_SKIP_WORKTREE, so\n + we can add the flag here to be consistent. This allows us to directly\n + compare the flags present in cache entries when testing the sparse-index\n + feature, but has no significance to its correctness in the user-facing\n + functionality.\n \n Signed-off-by: Derrick Stolee \n \n 3: b99371c7dd6 ! 3: d965669c766 t1092: replace incorrect 'echo' with 'cat'\n @@ Commit message\n t1092: replace incorrect 'echo' with 'cat'\n \n This fixes the test data shape to be as expected, allowing rename\n - detection to work properly now that the 'larger-conent' file actually\n + detection to work properly now that the 'larger-content' file actually\n has meaningful lines.\n \n Signed-off-by: Derrick Stolee \n 4: f4dddac1859 = 4: 44a940211b2 t1092: expand repository data shape\n 5: 856346b72f7 = 5: 701ac0e8ff6 t1092: add tests for status/add and sparse files\n 6: f3f6223e955 = 6: 587333f7c61 unpack-trees: preserve cache_bottom\n 7: 45ae96adf28 = 7: 6fc898ac23e unpack-trees: compare sparse directories correctly\n 8: 724194eef9f ! 8: b676ef4925b unpack-trees: unpack sparse directory entries\n @@ Commit message\n 'git checkout ' which wants to replace a sparse tree entry with\n the tree for that path at the target commit. We extend the logic within\n unpack_nondirectories() to create a sparse-directory entry in this case,\n - and then that is sent to call_unpack_fn().\n + and then that is sent to call_unpack_fn(). Since the name becomes\n + confusing by handling directories, rename it to unpack_single_entry()\n + since it handles a blob entry or a sparse directory entry without using\n + traverse_trees_recursive().\n \n There are some subtleties in this process. For instance, we need to\n update find_cache_entry() to allow finding a sparse-directory entry that\n - exactly matches a given path.\n + exactly matches a given path. Use the new helper method\n + sparse_dir_matches_path() for this.\n \n Signed-off-by: Derrick Stolee \n \n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n +\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n --\t\tmake_empty_transient_cache_entry(len) :\n +-\t\tmake_empty_transient_cache_entry(len, NULL) :\n -\t\tmake_empty_cache_entry(istate, len);\n -+\t\tmake_empty_transient_cache_entry(alloc_len) :\n ++\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n +\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \n +\tif (is_sparse_directory) {\n +\t\tce->name[len] = '/';\n -+\t\tce->name[len + 1] = 0;\n ++\t\tce->name[len + 1] = '\\0';\n +\t\tce->ce_namelen++;\n +\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n +\t}\n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \treturn ce;\n }\n \n -@@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n - \t\t\t\t unsigned long dirmask,\n - \t\t\t\t struct cache_entry **src,\n - \t\t\t\t const struct name_entry *names,\n +@@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n + * without actually calling it. If you change the logic here you may need to\n + * check and change there as well.\n + */\n +-static int unpack_nondirectories(int n, unsigned long mask,\n +-\t\t\t\t unsigned long dirmask,\n +-\t\t\t\t struct cache_entry **src,\n +-\t\t\t\t const struct name_entry *names,\n -\t\t\t\t const struct traverse_info *info)\n -+\t\t\t\t const struct traverse_info *info,\n -+\t\t\t\t int sparse_directory)\n ++static int unpack_single_entry(int n, unsigned long mask,\n ++\t\t\t unsigned long dirmask,\n ++\t\t\t struct cache_entry **src,\n ++\t\t\t const struct name_entry *names,\n ++\t\t\t const struct traverse_info *info,\n ++\t\t\t int sparse_directory)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n @@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n -+\t/* no-op if our cache entry doesn't match the expectations. */\n ++\t/* defer work if our cache entry doesn't match the expectations. */\n +\tif (sparse_directory) {\n +\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n +\t\t\tBUG(\"expected sparse directory entry\");\n @@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n \n \tif (o->merge) {\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n + \treturn -1;\n + }\n + \n ++/*\n ++ * Given a sparse directory entry 'ce', compare ce->name to\n ++ * info->name + '/' + p->path + '/' if info->name is non-empty.\n ++ * Compare ce->name to p->path + '/' otherwise. Note that\n ++ * ce->name must end in a trailing '/' because it is a sparse\n ++ * directory entry.\n ++ */\n ++static int sparse_dir_matches_path(const struct cache_entry *ce,\n ++\t\t\t\t struct traverse_info *info,\n ++\t\t\t\t const struct name_entry *p)\n ++{\n ++\tassert(S_ISSPARSEDIR(ce->ce_mode));\n ++\tassert(ce->name[ce->ce_namelen - 1] == '/');\n ++\n ++\tif (info->namelen)\n ++\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n ++\t\t ce->name[info->namelen] == '/' &&\n ++\t\t !strncmp(ce->name, info->name, info->namelen) &&\n ++\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n ++\treturn ce->ce_namelen == p->pathlen + 1 &&\n ++\t !strncmp(ce->name, p->path, p->pathlen);\n ++}\n ++\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n +\tpos = -pos - 2;\n +\n +\tif (pos < 0 || pos >= o->src_index->cache_nr)\n -+\t\treturn NULL;\n + \t\treturn NULL;\n +\n +\tce = o->src_index->cache[pos];\n +\n +\tif (!S_ISSPARSEDIR(ce->ce_mode))\n - \t\treturn NULL;\n ++\t\treturn NULL;\n +\n -+\t/*\n -+\t * Compare ce->name to info->name + '/' + p->path + '/'\n -+\t * if info->name is non-empty. Compare ce->name to\n -+\t * p-.path + '/' otherwise.\n -+\t */\n -+\tif (info->namelen) {\n -+\t\tif (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n -+\t\t ce->name[info->namelen] == '/' &&\n -+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n -+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n -+\t\t\treturn ce;\n -+\t} else if (ce->ce_namelen == p->pathlen + 1 &&\n -+\t\t !strncmp(ce->name, p->path, p->pathlen))\n ++\tif (sparse_dir_matches_path(ce, info, p))\n +\t\treturn ce;\n ++\n +\treturn NULL;\n }\n \n @@ unpack-trees.c: static void debug_unpack_callback(int n,\n + * sparse-directory entry that matches the given name_entry\n + * from the tree walk at the given traverse_info.\n + */\n -+static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n ++static int is_sparse_directory_entry(struct cache_entry *ce,\n ++\t\t\t\t struct name_entry *name,\n ++\t\t\t\t struct traverse_info *info)\n +{\n -+\tsize_t expected_len, name_start;\n -+\n +\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n +\t\treturn 0;\n +\n -+\tif (info->namelen)\n -+\t\tname_start = info->namelen + 1;\n -+\telse\n -+\t\tname_start = 0;\n -+\texpected_len = name->pathlen + 1 + name_start;\n -+\n -+\tif (ce->ce_namelen != expected_len ||\n -+\t strncmp(ce->name, info->name, info->namelen) ||\n -+\t strncmp(ce->name + name_start, name->path, name->pathlen))\n -+\t\treturn 0;\n -+\n -+\treturn 1;\n ++\treturn sparse_dir_matches_path(ce, info, name);\n +}\n +\n /*\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned l\n \t}\n \n -\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n -+\tif (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0)\n ++\tif (unpack_single_entry(n, mask, dirmask, src, names, info, 0) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned l\n -\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n -\t\t\t\t\t names, info) < 0)\n +\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n -+\t\t\tif (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n ++\t\t\tif (unpack_single_entry(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n +\t\t\t\treturn -1;\n +\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n +\t\t\t\t\t\t names, info) < 0) {\n 9: b8ff179f43e = 9: d693f00d9a2 dir.c: accept a directory as part of cone-mode patterns\n 10: b9b97e01129 ! 10: ed11cfc791f diff-lib: handle index diffs with sparse dirs\n @@ Commit message\n identical to before: the lack of a cache entry is the same with a sparse\n index.\n \n - In the case where a tree is modified, we need to expand the tree\n - recursively, and start comparing each contained entry as either an\n - addition, deletion, or modification. This causes an interesting\n - recursion that did not exist before.\n + Use diff_tree_oid() appropriately to appropriately compute the diff.\n \n Signed-off-by: Derrick Stolee \n \n ## diff-lib.c ##\n -@@ diff-lib.c: static int get_stat_data(const struct cache_entry *ce,\n - \treturn 0;\n - }\n - \n -+struct show_new_tree_context {\n -+\tstruct rev_info *revs;\n -+\tunsigned added:1;\n -+};\n -+\n -+static int show_new_file_from_tree(const struct object_id *oid,\n -+\t\t\t\t struct strbuf *base, const char *path,\n -+\t\t\t\t unsigned int mode, void *context)\n -+{\n -+\tstruct show_new_tree_context *ctx = context;\n -+\tstruct cache_entry *new_file = make_transient_cache_entry(mode, oid, path, /* stage */ 0);\n -+\n -+\tdiff_index_show_file(ctx->revs, ctx->added ? \"+\" : \"-\", new_file, oid, !is_null_oid(oid), mode, 0);\n -+\tdiscard_cache_entry(new_file);\n -+\treturn 0;\n -+}\n -+\n -+static void show_directory(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *new_dir,\n -+\t\t\t int added)\n -+{\n -+\t/*\n -+\t * new_dir is a sparse directory entry, so we want to collect all\n -+\t * of the new files within the tree. This requires recursively\n -+\t * expanding the trees.\n -+\t */\n -+\tstruct show_new_tree_context ctx = { revs, added };\n -+\tstruct repository *r = revs->repo;\n -+\tstruct strbuf base = STRBUF_INIT;\n -+\tstruct pathspec ps;\n -+\tstruct tree *tree = lookup_tree(r, &new_dir->oid);\n -+\n -+\tmemset(&ps, 0, sizeof(ps));\n -+\tps.recursive = 1;\n -+\tps.has_wildcard = 1;\n -+\tps.max_depth = -1;\n -+\n -+\tstrbuf_add(&base, new_dir->name, new_dir->ce_namelen);\n -+\tread_tree_at(r, tree, &base, &ps,\n -+\t\t\tshow_new_file_from_tree, &ctx);\n -+}\n -+\n - static void show_new_file(struct rev_info *revs,\n - \t\t\t const struct cache_entry *new_file,\n - \t\t\t int cached, int match_missing)\n @@ diff-lib.c: static void show_new_file(struct rev_info *revs,\n - \tunsigned int mode;\n \tunsigned dirty_submodule = 0;\n + \tstruct index_state *istate = revs->diffopt.repo->index;\n \n +\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n -+\t\tshow_directory(revs, new_file, /*added */ 1);\n ++\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n +\t\treturn;\n +\t}\n +\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n -@@ diff-lib.c: static void show_new_file(struct rev_info *revs,\n - \tdiff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n - }\n - \n -+static int show_modified(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *old_entry,\n -+\t\t\t const struct cache_entry *new_entry,\n -+\t\t\t int report_missing,\n -+\t\t\t int cached, int match_missing);\n -+\n -+static int compare_within_sparse_dir(int n, unsigned long mask,\n -+\t\t\t\t unsigned long dirmask, struct name_entry *entry,\n -+\t\t\t\t struct traverse_info *info)\n -+{\n -+\tstruct rev_info *revs = info->data;\n -+\tstruct object_id *oid0 = &entry[0].oid;\n -+\tstruct object_id *oid1 = &entry[1].oid;\n -+\n -+\tif (oideq(oid0, oid1))\n -+\t\treturn mask;\n -+\n -+\t/* Directory/file conflicts are handled earlier. */\n -+\tif (S_ISDIR(entry[0].mode) && S_ISDIR(entry[1].mode)) {\n -+\t\tstruct tree_desc t[2];\n -+\t\tvoid *buf[2];\n -+\t\tstruct traverse_info info_r = { NULL, };\n -+\n -+\t\tinfo_r.name = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n -+\t\tinfo_r.namelen = strlen(info_r.name);\n -+\t\tinfo_r.traverse_path = xstrfmt(\"%s/\", info_r.name);\n -+\t\tinfo_r.fn = compare_within_sparse_dir;\n -+\t\tinfo_r.prev = info;\n -+\t\tinfo_r.mode = entry[0].mode;\n -+\t\tinfo_r.pathlen = entry[0].pathlen;\n -+\t\tinfo_r.df_conflicts = 0;\n -+\t\tinfo_r.data = revs;\n -+\n -+\t\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], oid0);\n -+\t\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], oid1);\n -+\n -+\t\ttraverse_trees(NULL, 2, t, &info_r);\n -+\n -+\t\tfree((char *)info_r.name);\n -+\t\tfree((char *)info_r.traverse_path);\n -+\t\tfree(buf[0]);\n -+\t\tfree(buf[1]);\n -+\t} else {\n -+\t\tchar *old_path = NULL, *new_path = NULL;\n -+\t\tstruct cache_entry *old_entry = NULL, *new_entry = NULL;\n -+\n -+\t\tif (entry[0].path) {\n -+\t\t\told_path = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n -+\t\t\told_entry = make_transient_cache_entry(\n -+\t\t\t\t\tentry[0].mode, &entry[0].oid,\n -+\t\t\t\t\told_path, /* stage */ 0);\n -+\t\t\told_entry->ce_flags |= CE_SKIP_WORKTREE;\n -+\t\t}\n -+\t\tif (entry[1].path) {\n -+\t\t\tnew_path = xstrfmt(\"%s%s\", info->traverse_path, entry[1].path);\n -+\t\t\tnew_entry = make_transient_cache_entry(\n -+\t\t\t\t\tentry[1].mode, &entry[1].oid,\n -+\t\t\t\t\tnew_path, /* stage */ 0);\n -+\t\t\tnew_entry->ce_flags |= CE_SKIP_WORKTREE;\n -+\t\t}\n -+\n -+\t\tif (entry[0].path && entry[1].path)\n -+\t\t\tshow_modified(revs, old_entry, new_entry, 0, 1, 0);\n -+\t\telse if (entry[0].path)\n -+\t\t\tdiff_index_show_file(revs, revs->prefix,\n -+\t\t\t\t\t old_entry, &entry[0].oid,\n -+\t\t\t\t\t 0, entry[0].mode, 0);\n -+\t\telse if (entry[1].path)\n -+\t\t\tshow_new_file(revs, new_entry, 1, 0);\n -+\n -+\t\tdiscard_cache_entry(old_entry);\n -+\t\tdiscard_cache_entry(new_entry);\n -+\t\tfree(old_path);\n -+\t\tfree(new_path);\n -+\t}\n -+\n -+\treturn mask;\n -+}\n -+\n -+static void show_modified_sparse_directory(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *old_entry,\n -+\t\t\t const struct cache_entry *new_entry,\n -+\t\t\t int report_missing,\n -+\t\t\t int cached, int match_missing)\n -+{\n -+\tstruct tree_desc t[2];\n -+\tvoid *buf[2];\n -+\tstruct traverse_info info = { NULL };\n -+\tstruct strbuf name = STRBUF_INIT;\n -+\tstruct strbuf parent_path = STRBUF_INIT;\n -+\tchar *last_dir_sep;\n -+\n -+\tif (oideq(&old_entry->oid, &new_entry->oid))\n -+\t\treturn;\n -+\n -+\tinfo.fn = compare_within_sparse_dir;\n -+\tinfo.prev = &info;\n -+\n -+\tstrbuf_add(&name, new_entry->name, new_entry->ce_namelen - 1);\n -+\tinfo.name = name.buf;\n -+\tinfo.namelen = name.len;\n -+\n -+\tstrbuf_add(&parent_path, new_entry->name, new_entry->ce_namelen - 1);\n -+\tif ((last_dir_sep = find_last_dir_sep(parent_path.buf)) > parent_path.buf)\n -+\t\tstrbuf_setlen(&parent_path, (last_dir_sep - parent_path.buf) - 1);\n -+\telse\n -+\t\tstrbuf_setlen(&parent_path, 0);\n -+\n -+\tinfo.pathlen = parent_path.len;\n -+\n -+\tif (parent_path.len)\n -+\t\tinfo.traverse_path = parent_path.buf;\n -+\telse\n -+\t\tinfo.traverse_path = \"\";\n -+\n -+\tinfo.mode = new_entry->ce_mode;\n -+\tinfo.df_conflicts = 0;\n -+\tinfo.data = revs;\n -+\n -+\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], &old_entry->oid);\n -+\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], &new_entry->oid);\n -+\n -+\ttraverse_trees(NULL, 2, t, &info);\n -+\n -+\tfree(buf[0]);\n -+\tfree(buf[1]);\n -+\tstrbuf_release(&name);\n -+\tstrbuf_release(&parent_path);\n -+}\n -+\n - static int show_modified(struct rev_info *revs,\n - \t\t\t const struct cache_entry *old_entry,\n - \t\t\t const struct cache_entry *new_entry,\n @@ diff-lib.c: static int show_modified(struct rev_info *revs,\n - \tconst struct object_id *oid;\n \tunsigned dirty_submodule = 0;\n + \tstruct index_state *istate = revs->diffopt.repo->index;\n \n +\t/*\n +\t * If both are sparse directory entries, then expand the\n @@ diff-lib.c: static int show_modified(struct rev_info *revs,\n +\tif (old_entry && new_entry &&\n +\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n +\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n -+\t\tshow_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n ++\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n +\t\treturn 0;\n +\t}\n +\n - \tif (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n + \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n 11: 611b9f61fb2 = 11: 48fd25aacbe status: skip sparse-checkout percentage with sparse-index\n 12: 0c0a765dde8 = 12: 3499105eb67 status: use sparse-index throughout\n 13: 02f2c7b6398 ! 13: 08225483d69 wt-status: expand added sparse directory entries\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n +\ttest_sparse_match git reset --mixed HEAD~1 &&\n +\ttest_sparse_match test-tool read-cache --table --expand &&\n +\ttest_sparse_match git status --porcelain=v2 &&\n -+\ttest_sparse_match git status --porcelain=v2 &&\n +\n +\t# At this point, sparse-checkouts behave differently\n +\t# from the full-checkout.\n +\ttest_sparse_match git checkout --orphan new-branch &&\n +\ttest_sparse_match test-tool read-cache --table --expand &&\n -+\ttest_sparse_match git status --porcelain=v2 &&\n +\ttest_sparse_match git status --porcelain=v2\n +'\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n +\n +\trun_on_all touch deep/deeper1/x &&\n +\ttest_all_match git add . &&\n -+\ttest_all_match git status --porcelain=v2 &&\n +\ttest_all_match git status --porcelain=v2\n +'\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n \n ## wt-status.c ##\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n - \trun_diff_index(&rev, 1);\n + \tclear_pathspec(&rev.prune_data);\n }\n \n +static int add_file_to_list(const struct object_id *oid,\n @@ wt-status.c: static void wt_status_collect_changes_initial(struct wt_status *s)\n +\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n +\t\t\t */\n +\t\t\tstruct strbuf base = STRBUF_INIT;\n -+\t\t\tstruct pathspec ps;\n ++\t\t\tstruct pathspec ps = { 0 };\n +\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n +\n -+\t\t\tmemset(&ps, 0, sizeof(ps));\n +\t\t\tps.recursive = 1;\n +\t\t\tps.has_wildcard = 1;\n +\t\t\tps.max_depth = -1;\n 14: 46ca150c354 = 14: 711e403a63a fsmonitor: integrate with sparse index\n\n-- \ngitgitgadget\n"},{"id":"428665","messageId":"2a4a7256304c44f168baf54ff1bdc191be67408f.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 01/14] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:29Z","receivedAt":"2021-06-29T01:51:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex affc4048f27..2c695930275 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e9a815ca7aa..ba2fd94adaf 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"428666","messageId":"f5bae86014d0f76e4dc39560a5e73d5911255bd6.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 02/14] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:30Z","receivedAt":"2021-06-29T01:51:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\nflag is not included. The CE_EXTENDED flag would exist if we loaded a\nfull index from disk with these entries marked with CE_SKIP_WORKTREE, so\nwe can add the flag here to be consistent. This allows us to directly\ncompare the flags present in cache entries when testing the sparse-index\nfeature, but has no significance to its correctness in the user-facing\nfunctionality.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2c695930275..ef53bd2198b 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"428667","messageId":"d965669c76663ebe6b172c4046c9c517ba9102f0.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 03/14] t1092: replace incorrect 'echo' with 'cat'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:31Z","receivedAt":"2021-06-29T01:51:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis fixes the test data shape to be as expected, allowing rename\ndetection to work properly now that the 'larger-content' file actually\nhas meaningful lines.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ba2fd94adaf..ebbba044f77 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -40,7 +40,7 @@ test_expect_success 'setup' '\n \t\tdone &&\n \n \t\tgit checkout -b rename-base base &&\n-\t\techo >folder1/larger-content <<-\\EOF &&\n+\t\tcat >folder1/larger-content <<-\\EOF &&\n \t\tmatching\n \t\tlines\n \t\thelp\n-- \ngitgitgadget\n\n"},{"id":"428668","messageId":"44a940211b2fedc4a65b6dc2379e99c29cc5b64f.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 04/14] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:32Z","receivedAt":"2021-06-29T01:51:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 39 ++++++++++++++++++++++--\n 1 file changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ebbba044f77..4a7b48e8d3f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,20 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n+\t\tcp -r deep/deeper1/0 folder1 &&\n+\t\tcp -r deep/deeper1/0 folder2 &&\n+\t\techo >>folder1/0/0/0 &&\n+\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n@@ -56,11 +66,17 @@ test_expect_success 'setup' '\n \t\tmv folder1/a folder2/b &&\n \t\tmv folder1/larger-content folder2/edited-content &&\n \t\techo >>folder2/edited-content &&\n+\t\techo >>folder2/0/1 &&\n+\t\techo stuff >>deep/deeper1/a &&\n \t\tgit add . &&\n \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n \n \t\tgit checkout -b rename-out-to-in rename-base &&\n \t\tmv folder1/a deep/deeper1/b &&\n+\t\techo more stuff >>deep/deeper1/a &&\n+\t\trm folder2/0/1 &&\n+\t\tmkdir folder2/0/1 &&\n+\t\techo >>folder2/0/1/1 &&\n \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n \t\techo >>deep/deeper1/edited-content &&\n \t\tgit add . &&\n@@ -68,6 +84,9 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b rename-in-to-out rename-base &&\n \t\tmv deep/deeper1/a folder1/b &&\n+\t\techo >>folder2/0/1 &&\n+\t\trm -rf folder1/0/0 &&\n+\t\techo >>folder1/0/0 &&\n \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n \t\techo >>folder1/edited-content &&\n \t\tgit add . &&\n@@ -262,13 +281,29 @@ test_expect_success 'diff --staged' '\n \ttest_all_match git diff --staged\n '\n \n-test_expect_success 'diff with renames' '\n+test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- .&&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_all_match git diff --staged --no-renames &&\n+\t\ttest_all_match git diff --staged --find-renames || return 1\n+\tdone\n+'\n+\n+test_expect_success 'diff with directory/file conflicts' '\n+\tinit_repos &&\n+\n+\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tdo\n+\t\tgit -C full-checkout reset --hard &&\n+\t\ttest_sparse_match git reset --hard &&\n+\t\ttest_all_match git checkout $branch &&\n+\t\ttest_all_match git checkout rename-base -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n \t\ttest_all_match git diff --staged --no-renames &&\n \t\ttest_all_match git diff --staged --find-renames || return 1\n \tdone\n-- \ngitgitgadget\n\n"},{"id":"428669","messageId":"701ac0e8ff6331284e63047bceed98d1219daa47.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 05/14] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:33Z","receivedAt":"2021-06-29T01:52:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4a7b48e8d3f..7c78e40b861 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -251,6 +251,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"428670","messageId":"6fc898ac23e244b3f9c1406badb89441945c6e18.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 07/14] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:35Z","receivedAt":"2021-06-29T01:52:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 13 ++++++++++++-\n 1 file changed, 12 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 87c1ed204c8..7a507ddfe05 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1014,6 +1016,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"428671","messageId":"587333f7c618f8eb65f7a25d1a4ab02f9ace92fd.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 06/14] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:34Z","receivedAt":"2021-06-29T01:52:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f88a69f8e71..87c1ed204c8 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"428672","messageId":"b676ef4925ba214cffcfd1f4ea8185cdc5e7a0ed.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 08/14] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:36Z","receivedAt":"2021-06-29T01:52:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring unpack_callback(), index entries are compared against tree\nentries. These are matched according to names and types. One goal is to\ndecide if we should recurse into subtrees or simply operate on one index\nentry.\n\nIn the case of a sparse-directory entry, we do not want to recurse into\nthat subtree and instead simply compare the trees. In some cases, we\nmight want to perform a merge operation on the entry, such as during\n'git checkout ' which wants to replace a sparse tree entry with\nthe tree for that path at the target commit. We extend the logic within\nunpack_nondirectories() to create a sparse-directory entry in this case,\nand then that is sent to call_unpack_fn(). Since the name becomes\nconfusing by handling directories, rename it to unpack_single_entry()\nsince it handles a blob entry or a sparse directory entry without using\ntraverse_trees_recursive().\n\nThere are some subtleties in this process. For instance, we need to\nupdate find_cache_entry() to allow finding a sparse-directory entry that\nexactly matches a given path. Use the new helper method\nsparse_dir_matches_path() for this.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 110 ++++++++++++++++++++++++++++++++++++++++++-------\n 1 file changed, 96 insertions(+), 14 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7a507ddfe05..d205a15d61f 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1051,13 +1051,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tconst struct name_entry *n,\n \tint stage,\n \tstruct index_state *istate,\n-\tint is_transient)\n+\tint is_transient,\n+\tint is_sparse_directory)\n {\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n+\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len, NULL) :\n-\t\tmake_empty_cache_entry(istate, len);\n+\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n+\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -1066,6 +1068,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \t/* len+1 because the cache_entry allocates space for NUL */\n \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n+\tif (is_sparse_directory) {\n+\t\tce->name[len] = '/';\n+\t\tce->name[len + 1] = '\\0';\n+\t\tce->ce_namelen++;\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\t}\n+\n \treturn ce;\n }\n \n@@ -1074,20 +1083,28 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n * without actually calling it. If you change the logic here you may need to\n * check and change there as well.\n */\n-static int unpack_nondirectories(int n, unsigned long mask,\n-\t\t\t\t unsigned long dirmask,\n-\t\t\t\t struct cache_entry **src,\n-\t\t\t\t const struct name_entry *names,\n-\t\t\t\t const struct traverse_info *info)\n+static int unpack_single_entry(int n, unsigned long mask,\n+\t\t\t unsigned long dirmask,\n+\t\t\t struct cache_entry **src,\n+\t\t\t const struct name_entry *names,\n+\t\t\t const struct traverse_info *info,\n+\t\t\t int sparse_directory)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\t/* defer work if our cache entry doesn't match the expectations. */\n+\tif (sparse_directory) {\n+\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\t\tBUG(\"expected sparse directory entry\");\n+\t} else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n+\t\treturn 0;\n+\t}\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n@@ -1117,7 +1134,9 @@ static int unpack_nondirectories(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n+\t\t\t\t\t\t &o->result, o->merge,\n+\t\t\t\t\t\t sparse_directory);\n \t}\n \n \tif (o->merge) {\n@@ -1221,16 +1240,59 @@ static int find_cache_pos(struct traverse_info *info,\n \treturn -1;\n }\n \n+/*\n+ * Given a sparse directory entry 'ce', compare ce->name to\n+ * info->name + '/' + p->path + '/' if info->name is non-empty.\n+ * Compare ce->name to p->path + '/' otherwise. Note that\n+ * ce->name must end in a trailing '/' because it is a sparse\n+ * directory entry.\n+ */\n+static int sparse_dir_matches_path(const struct cache_entry *ce,\n+\t\t\t\t struct traverse_info *info,\n+\t\t\t\t const struct name_entry *p)\n+{\n+\tassert(S_ISSPARSEDIR(ce->ce_mode));\n+\tassert(ce->name[ce->ce_namelen - 1] == '/');\n+\n+\tif (info->namelen)\n+\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n+\t\t ce->name[info->namelen] == '/' &&\n+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\treturn ce->ce_namelen == p->pathlen + 1 &&\n+\t !strncmp(ce->name, p->path, p->pathlen);\n+}\n+\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n+\tstruct cache_entry *ce;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n-\telse\n+\n+\t/*\n+\t * Check for a sparse-directory entry named \"path/\".\n+\t * Due to the input p->path not having a trailing\n+\t * slash, the negative 'pos' value overshoots the\n+\t * expected position by one, hence \"-2\" here.\n+\t */\n+\tpos = -pos - 2;\n+\n+\tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n+\n+\tce = o->src_index->cache[pos];\n+\n+\tif (!S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn NULL;\n+\n+\tif (sparse_dir_matches_path(ce, info, p))\n+\t\treturn ce;\n+\n+\treturn NULL;\n }\n \n static void debug_path(struct traverse_info *info)\n@@ -1265,6 +1327,21 @@ static void debug_unpack_callback(int n,\n \t\tdebug_name_entry(i, names + i);\n }\n \n+/*\n+ * Returns true if and only if the given cache_entry is a\n+ * sparse-directory entry that matches the given name_entry\n+ * from the tree walk at the given traverse_info.\n+ */\n+static int is_sparse_directory_entry(struct cache_entry *ce,\n+\t\t\t\t struct name_entry *name,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n+\treturn sparse_dir_matches_path(ce, info, name);\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -1321,7 +1398,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_single_entry(n, mask, dirmask, src, names, info, 0) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n@@ -1351,9 +1428,14 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n-\t\t\t\t\t names, info) < 0)\n+\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n+\t\t\tif (unpack_single_entry(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n+\t\t}\n+\n \t\treturn mask;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"428673","messageId":"d693f00d9a28fff049a3e1d469ac3024b9f99bb4.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 09/14] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:37Z","receivedAt":"2021-06-29T01:52:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e0..2155107b1d7 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1397,6 +1397,17 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n-- \ngitgitgadget\n\n"},{"id":"428674","messageId":"ed11cfc791f65fe79db3d39471a6779138370c61.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 10/14] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:38Z","receivedAt":"2021-06-29T01:52:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile comparing an index to a tree, we may see a sparse directory entry.\nIn this case, we should compare that portion of the tree to the tree\nrepresented by that entry. This could include a new tree which needs to\nbe expanded to a full list of added files. It could also include an\nexisting tree, in which case all of the changes inside are important to\ndescribe, including the modifications, additions, and deletions. Note\nthat the case where the tree has a path and the index does not remains\nidentical to before: the lack of a cache entry is the same with a sparse\nindex.\n\nUse diff_tree_oid() appropriately to appropriately compute the diff.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex c2ac9250fe9..3f32f038371 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level.\n+\t */\n+\tif (old_entry && new_entry &&\n+\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n+\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n-- \ngitgitgadget\n\n"},{"id":"428675","messageId":"48fd25aacbeb1baeca637aec40f3f4cba0da0f52.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 11/14] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:39Z","receivedAt":"2021-06-29T01:52:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 7c78e40b861..9035adcb7db 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -215,6 +215,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 42b67357169..96db3e74962 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e..ab9cc9d8f03 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"428676","messageId":"08225483d699514426d4ddaf268338e09ea4e8bb.1624931503.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 13/14] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:41Z","receivedAt":"2021-06-29T01:52:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 33 ++++++++++++++++\n wt-status.c | 49 ++++++++++++++++++++++++\n 2 files changed, 82 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 1e9737cb4b7..ef918437908 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -521,4 +521,37 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\n+\t# Sparse checkouts do not agree with full checkouts about\n+\t# how to report a directory/file conflict during a reset.\n+\t# This command would fail with test_all_match because the\n+\t# full checkout reports \"T folder1/0/1\" while a sparse\n+\t# checkout reports \"D folder1/0/1\". This matches because\n+\t# the sparse checkouts skip \"adding\" the other side of\n+\t# the conflict.\n+\ttest_sparse_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 96db3e74962..a90c7b6aa8a 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -657,6 +657,34 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \tclear_pathspec(&rev.prune_data);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tchar *full_name;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n+\tit = string_list_insert(&s->change, full_name);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -671,6 +699,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps = { 0 };\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"428677","messageId":"3499105eb67f8514589b5839c0661b4049714fb7.1624931502.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 12/14] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:40Z","receivedAt":"2021-06-29T01:52:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 190d215d43b..12f51db158a 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 1b3c2eb408b..277c2970a03 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1584,8 +1584,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1600,6 +1599,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9035adcb7db..1e9737cb4b7 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -508,12 +508,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"428678","messageId":"711e403a63a7815d0581b469a9f0d52d0c8054ca.1624931503.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v6 14/14] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T01:51:42Z","receivedAt":"2021-06-29T01:52:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex ef53bd2198b..53c8f711ccc 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 637391c6ce4..8f9240b9b1a 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"428682","messageId":"dec05200-75a9-fd60-8ab3-1c0b51d2c946@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 00/14] Sparse-index: integrate with status","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-29T02:02:33Z","receivedAt":"2021-06-29T02:02:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/28/2021 9:51 PM, Derrick Stolee via GitGitGadget wrote:\n...\n> Update in V6\n> ============\n\nI'm very sorry for the noise. I was working on my cover letter and\nadjusting the patches relative to some nits I discovered, including\nthe ordering of some patches in my larger topic on this subject,\nand forgot to push those before submitting the v6. Please ignore\nthis version. A correct v7 is coming soon.\n\nAgain, sorry for the noise.\n\n-Stolee\n"},{"id":"428683","messageId":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v6.git.1624931502.gitgitgadget@gmail.com","subject":"[PATCH v7 00/16] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:37Z","receivedAt":"2021-06-29T02:04:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdate in V7 (relative to v5)\n=============================\n\nAPOLOGIES: As I was working on this cover letter, I was still organizing my\nbig list of patches, including reordering some into this series. I forgot to\nactually include them in my v6 submission, so here is a re-submission.\nPlease ignore v6.\n\nI'm sorry that this revision took so long. Initially I was blocked on\ngetting the directory/file conflict figured out (I did), but also my team\nwas very busy with some things. Eventually, we reached an internal deadline\nto make an experimental release available [1] with initial sparse-index\nperformance boosts. Creating that included some additional review by Jeff\nHostetler and Johannes Schindelin which led to more changes in this version.\n\nThe good news is that this series has now created the basis for many Git\ncommands to integrate with the sparse-index without much additional work.\nThis effort was unfortunately overloaded on this series because the changes\nneeded for things like 'git checkout' or 'git add' all intersect with the\nchanges needed for 'git status'. Might as well get it right the first time.\n\nBecause the range-diff is a big difficult to read this time, I'll break the\nchanges down on a patch-by-patch basis.\n\n 1. sparse-index: skip indexes with unmerged entries\n \n (no change)\n\n 2. sparse-index: include EXTENDED flag when expanding\n\n * Commit message better describes the purpose of the change.\n\n 3. t1092: replace incorrect 'echo' with 'cat'\n\n * Typo fix\n\n 4. t1092: expand repository data shape\n\n * some files are added that surround \"folder1/\" immediately before and\n after, based on the sorting with the trailing slash. This provides extra\n coverage.\n\n 5. t1092: add tests for status/add and sparse files\n \n (no change)\n\n 6. unpack-trees: preserve cache_bottom\n \n (no change)\n\n 7. unpack-trees: compare sparse directories correctly\n\n * We were previosly not comparing the path lengths, which causes a problem\n (with later updates) when a sparse directory such as \"folder1/0/\" gets\n compared to a tree name \"folder1\".\n\n 8. unpack-trees: rename unpack_nondirectories()\n\n * This new commit changes the name to make more sense with its new behavior\n that could modify a sparse directory entry. The point of the method is in\n contrast to recursing into trees.\n\n 9. unpack-trees: unpack sparse directory entries\n\n * THIS is the biggest change from previous versions. There were a few\n things going on that were tricky to get right, especially with the\n directory/file conflict (handled in an update in the following, new\n patch).\n\n * The changes to create_ce_entry() regarding alloc_len missed a spot that\n was critical to getting the length right in the allocated entry.\n\n * Use '\\0' over 0 to represent the terminating character.\n\n * We don't need a \"sparse_directory\" parameter to unpack_nondirectories()\n (which was renamed to unpack_single_entry() by the previous new patch)\n because we can use dirmask to discover if src[0] (or any other value)\n should be a sparse directory entry.\n\n * Similarly, we don't need to call the method twice from unpack_callback().\n\n * The 'conflicts' variable is set to match the dirmask in the beginning,\n but it should depend on whether or not we have a sparse directory entry\n instead, and if all trees that have the path have a directory.\n\n * The implementation of find_cache_entry() uses find_cache_pos() to find an\n insertion position for a path if it doesn't find an exact match. Before,\n we subtracted one to find the sparse directory entry, but there could be\n multiple paths between the sparse directory entry and the insertion\n point, so we need to walk backwards until we find it. This requires many\n paths having the same prefix, so hopefully is a rare case. Some of the\n test data changes were added to cover the need for this logic. This uses\n a helper method, sparse_dir_matches_path, which is also used by\n is_sparse_directory_entry.\n\n 10. unpack-trees: handle dir/file conflict of sparse entries\n\n * This new logic inside twoway_merge handles the special case for dealing\n with a directory/file conflict during a 'git checkout'. The necessarily\n data and tests are also added here, though the logic will only take\n serious effect when we integrate with 'git checkout' later.\n\n 11. dir.c: accept a directory as part of cone-mode patterns\n\n * The value slash_pos was previously a pointer within a strbuf, but in some\n cases we add to that strbuf and that could reallocate the pointer, making\n slash_pos be invalid. The replacement is to have slash_pos be an integer\n position within the string, so it is consistent even if the string is\n reallocated for an append.\n\n 12. diff-lib: handle index diffs with sparse dirs\n\n * As recommended in the previous review, a simple diff_tree_oid() replaces\n the complicated use of read_tree_at() and traverse_trees() in the\n previous version.\n\n 13. status: skip sparse-checkout percentage with sparse-index\n \n (no change)\n\n 14. status: use sparse-index throughout\n \n (no change)\n\n 15. wt-status: expand added sparse directory entries\n\n * Duplicate 'git status --porcelain=v2' lines are removed from tests.\n\n * The pathspec is initialized using \"= { 0 }\" instead of memset().\n\n 16. fsmonitor: integrate with sparse index\n\n * An extra test_region is added to ensure that the filesystem monitor hook\n is still being called, and we are not simply disabling the feature\n entirely.\n\nDerrick Stolee (16):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: replace incorrect 'echo' with 'cat'\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: rename unpack_nondirectories()\n unpack-trees: unpack sparse directory entries\n unpack-trees: handle dir/file conflict of sparse entries\n dir.c: accept a directory as part of cone-mode patterns\n diff-lib: handle index diffs with sparse dirs\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n diff-lib.c | 16 ++\n dir.c | 24 ++-\n read-cache.c | 10 +-\n sparse-index.c | 27 +++-\n t/t1092-sparse-checkout-compatibility.sh | 181 ++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 49 ++++++\n unpack-trees.c | 145 +++++++++++++++---\n wt-status.c | 65 +++++++-\n wt-status.h | 1 +\n 10 files changed, 485 insertions(+), 36 deletions(-)\n\n\nbase-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v7\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v6:\n\n 1: 2a4a7256304 = 1: 2a4a7256304 sparse-index: skip indexes with unmerged entries\n 2: f5bae86014d = 2: f5bae86014d sparse-index: include EXTENDED flag when expanding\n 3: d965669c766 = 3: d965669c766 t1092: replace incorrect 'echo' with 'cat'\n 4: 44a940211b2 ! 4: e10fa11cfdb t1092: expand repository data shape\n @@ Commit message\n one entry and are the first entry of a directory with multiple\n entries.\n \n + * Add filenames adjacent to a sparse directory entry that sort before\n + and after the trailing slash.\n +\n Later tests will take advantage of these shapes, but they also deepen\n the tests that already exist.\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'setup' '\n +\t\tmkdir deep/deeper1/0/0 &&\n +\t\ttouch deep/deeper1/0/1 &&\n +\t\ttouch deep/deeper1/0/0/0 &&\n ++\t\t>folder1- &&\n ++\t\t>folder1.x &&\n ++\t\t>folder10 &&\n +\t\tcp -r deep/deeper1/0 folder1 &&\n +\t\tcp -r deep/deeper1/0 folder2 &&\n +\t\techo >>folder1/0/0/0 &&\n 5: 701ac0e8ff6 = 5: e94ffa07d46 t1092: add tests for status/add and sparse files\n 6: 587333f7c61 = 6: a8dda933567 unpack-trees: preserve cache_bottom\n 7: 6fc898ac23e ! 7: e52166f6e4c unpack-trees: compare sparse directories correctly\n @@ Commit message\n Within compare_entry(), it first calls do_compare_entry() to check the\n leading portion of the name. When the input path is a directory name, we\n could match exactly already. Thus, we should return 0 if we have an\n - exact string match on a sparse directory entry.\n + exact string match on a sparse directory entry. The final check is a\n + length comparison between the strings.\n \n Signed-off-by: Derrick Stolee \n \n @@ unpack-trees.c: static int compare_entry(const struct cache_entry *ce, const str\n +\t * works when the input name is a directory, since ce->name\n +\t * ends in a directory separator.\n +\t */\n -+\tif (S_ISSPARSEDIR(ce->ce_mode))\n ++\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n ++\t ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1)\n +\t\treturn 0;\n +\n \t/*\n -: ----------- > 8: d04b62381b8 unpack-trees: rename unpack_nondirectories()\n 8: b676ef4925b ! 9: 237ccf4e43d unpack-trees: unpack sparse directory entries\n @@ Commit message\n might want to perform a merge operation on the entry, such as during\n 'git checkout ' which wants to replace a sparse tree entry with\n the tree for that path at the target commit. We extend the logic within\n - unpack_nondirectories() to create a sparse-directory entry in this case,\n - and then that is sent to call_unpack_fn(). Since the name becomes\n - confusing by handling directories, rename it to unpack_single_entry()\n - since it handles a blob entry or a sparse directory entry without using\n - traverse_trees_recursive().\n + unpack_single_entry() to create a sparse-directory entry in this case,\n + and then that is sent to call_unpack_fn().\n \n There are some subtleties in this process. For instance, we need to\n update find_cache_entry() to allow finding a sparse-directory entry that\n exactly matches a given path. Use the new helper method\n - sparse_dir_matches_path() for this.\n + sparse_dir_matches_path() for this. We also need to ignore conflict\n + markers in the case that the entries correspond to directories and we\n + already have a sparse directory entry.\n \n Signed-off-by: Derrick Stolee \n \n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \treturn ce;\n }\n \n -@@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n - * without actually calling it. If you change the logic here you may need to\n - * check and change there as well.\n - */\n --static int unpack_nondirectories(int n, unsigned long mask,\n --\t\t\t\t unsigned long dirmask,\n --\t\t\t\t struct cache_entry **src,\n --\t\t\t\t const struct name_entry *names,\n --\t\t\t\t const struct traverse_info *info)\n -+static int unpack_single_entry(int n, unsigned long mask,\n -+\t\t\t unsigned long dirmask,\n -+\t\t\t struct cache_entry **src,\n -+\t\t\t const struct name_entry *names,\n -+\t\t\t const struct traverse_info *info,\n -+\t\t\t int sparse_directory)\n - {\n - \tint i;\n +@@ unpack-trees.c: static int unpack_single_entry(int n, unsigned long mask,\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n -+\t/* defer work if our cache entry doesn't match the expectations. */\n -+\tif (sparse_directory) {\n -+\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n -+\t\t\tBUG(\"expected sparse directory entry\");\n -+\t} else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n -+\t\treturn 0;\n -+\t}\n ++\t/*\n ++\t * When we have a sparse directory entry for src[0],\n ++\t * then this isn't necessarily a directory-file conflict.\n ++\t */\n ++\tif (mask == dirmask && src[0] &&\n ++\t S_ISSPARSEDIR(src[0]->ce_mode))\n ++\t\tconflicts = 0;\n +\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n -@@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n +@@ unpack-trees.c: static int unpack_single_entry(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n -\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n +\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n +\t\t\t\t\t\t &o->result, o->merge,\n -+\t\t\t\t\t\t sparse_directory);\n ++\t\t\t\t\t\t bit & dirmask);\n \t}\n \n \tif (o->merge) {\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n +\t * Check for a sparse-directory entry named \"path/\".\n +\t * Due to the input p->path not having a trailing\n +\t * slash, the negative 'pos' value overshoots the\n -+\t * expected position by one, hence \"-2\" here.\n ++\t * expected position by at least one, hence \"-2\" here.\n +\t */\n +\tpos = -pos - 2;\n +\n +\tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n +\n -+\tce = o->src_index->cache[pos];\n ++\t/*\n ++\t * We might have multiple entries between 'pos' and\n ++\t * the actual sparse-directory entry, so start walking\n ++\t * back until finding it or passing where it would be.\n ++\t */\n ++\twhile (pos >= 0) {\n ++\t\tce = o->src_index->cache[pos];\n ++\n ++\t\tif (strncmp(ce->name, p->path, p->pathlen))\n ++\t\t\treturn NULL;\n +\n -+\tif (!S_ISSPARSEDIR(ce->ce_mode))\n -+\t\treturn NULL;\n ++\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n ++\t\t sparse_dir_matches_path(ce, info, p))\n ++\t\t\treturn ce;\n +\n -+\tif (sparse_dir_matches_path(ce, info, p))\n -+\t\treturn ce;\n ++\t\tpos--;\n ++\t}\n +\n +\treturn NULL;\n }\n @@ unpack-trees.c: static void debug_unpack_callback(int n,\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n -@@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n - \t\t}\n - \t}\n - \n --\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n -+\tif (unpack_single_entry(n, mask, dirmask, src, names, info, 0) < 0)\n - \t\treturn -1;\n - \n - \tif (o->merge && src[0]) {\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n -\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n -\t\t\t\t\t names, info) < 0)\n -+\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n -+\t\t\tif (unpack_single_entry(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n -+\t\t\t\treturn -1;\n -+\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n ++\t\tif (!is_sparse_directory_entry(src[0], names, info) &&\n ++\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n +\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n +\t\t}\n -: ----------- > 10: 9f31c691af6 unpack-trees: handle dir/file conflict of sparse entries\n 9: d693f00d9a2 ! 11: 2a43287c47e dir.c: accept a directory as part of cone-mode patterns\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## dir.c ##\n +@@ dir.c: enum pattern_match_result path_matches_pattern_list(\n + \tstruct path_pattern *pattern;\n + \tstruct strbuf parent_pathname = STRBUF_INIT;\n + \tint result = NOT_MATCHED;\n +-\tconst char *slash_pos;\n ++\tsize_t slash_pos;\n + \n + \tif (!pl->use_cone_patterns) {\n + \t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n @@ dir.c: enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n @@ dir.c: enum pattern_match_result path_matches_pattern_list(\n +\t * use the file-base matching logic in an equivalent way.\n +\t */\n +\tif (parent_pathname.len > 0 &&\n -+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n ++\t parent_pathname.buf[parent_pathname.len - 1] == '/') {\n ++\t\tslash_pos = parent_pathname.len - 1;\n +\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n ++\t} else {\n ++\t\tconst char *slash_ptr = strrchr(parent_pathname.buf, '/');\n ++\t\tslash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0;\n ++\t}\n +\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n + \t\tgoto done;\n + \t}\n + \n +-\tslash_pos = strrchr(parent_pathname.buf, '/');\n +-\n +-\tif (slash_pos == parent_pathname.buf) {\n ++\tif (!slash_pos) {\n + \t\t/* include every file in root */\n + \t\tresult = MATCHED;\n + \t\tgoto done;\n + \t}\n + \n +-\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n ++\tstrbuf_setlen(&parent_pathname, slash_pos);\n + \n + \tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n + \t\tresult = MATCHED;\n 10: ed11cfc791f ! 12: f83aa08ff6b diff-lib: handle index diffs with sparse dirs\n @@ Commit message\n identical to before: the lack of a cache entry is the same with a sparse\n index.\n \n - Use diff_tree_oid() appropriately to appropriately compute the diff.\n + Use diff_tree_oid() appropriately to compute the diff.\n \n Signed-off-by: Derrick Stolee \n \n 11: 48fd25aacbe = 13: 35063ffb8ed status: skip sparse-checkout percentage with sparse-index\n 12: 3499105eb67 = 14: b4033a9bf36 status: use sparse-index throughout\n 13: 08225483d69 ! 15: 717a3f49f97 wt-status: expand added sparse directory entries\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n +\tstruct string_list_item *it;\n +\tstruct wt_status_change_data *d;\n +\tstruct wt_status *s = context;\n -+\tchar *full_name;\n ++\tstruct strbuf full_name = STRBUF_INIT;\n +\n +\tif (S_ISDIR(mode))\n +\t\treturn READ_TREE_RECURSIVE;\n +\n -+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n -+\tit = string_list_insert(&s->change, full_name);\n ++\tstrbuf_add(&full_name, base->buf, base->len);\n ++\tstrbuf_addstr(&full_name, path);\n ++\tit = string_list_insert(&s->change, full_name.buf);\n +\td = it->util;\n +\tif (!d) {\n +\t\tCALLOC_ARRAY(d, 1);\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n +\td->mode_index = mode;\n +\toidcpy(&d->oid_index, oid);\n +\ts->committable = 1;\n ++\tstrbuf_release(&full_name);\n +\treturn 0;\n +}\n +\n 14: 711e403a63a ! 16: 1d744848ee6 fsmonitor: integrate with sparse index\n @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n +\t\tgit status --porcelain=v2 >actual &&\n +\ttest_region $1 index ensure_full_index trace2.txt &&\n ++\ttest_region fsm_hook query trace2.txt &&\n +\ttest_cmp expect actual &&\n +\trm trace2.txt &&\n +\tgit sparse-checkout disable\n\n-- \ngitgitgadget\n"},{"id":"428684","messageId":"2a4a7256304c44f168baf54ff1bdc191be67408f.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 01/16] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:38Z","receivedAt":"2021-06-29T02:05:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex affc4048f27..2c695930275 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e9a815ca7aa..ba2fd94adaf 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"428685","messageId":"f5bae86014d0f76e4dc39560a5e73d5911255bd6.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 02/16] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:39Z","receivedAt":"2021-06-29T02:05:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\nflag is not included. The CE_EXTENDED flag would exist if we loaded a\nfull index from disk with these entries marked with CE_SKIP_WORKTREE, so\nwe can add the flag here to be consistent. This allows us to directly\ncompare the flags present in cache entries when testing the sparse-index\nfeature, but has no significance to its correctness in the user-facing\nfunctionality.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2c695930275..ef53bd2198b 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"428686","messageId":"d965669c76663ebe6b172c4046c9c517ba9102f0.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 03/16] t1092: replace incorrect 'echo' with 'cat'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:40Z","receivedAt":"2021-06-29T02:05:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis fixes the test data shape to be as expected, allowing rename\ndetection to work properly now that the 'larger-content' file actually\nhas meaningful lines.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ba2fd94adaf..ebbba044f77 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -40,7 +40,7 @@ test_expect_success 'setup' '\n \t\tdone &&\n \n \t\tgit checkout -b rename-base base &&\n-\t\techo >folder1/larger-content <<-\\EOF &&\n+\t\tcat >folder1/larger-content <<-\\EOF &&\n \t\tmatching\n \t\tlines\n \t\thelp\n-- \ngitgitgadget\n\n"},{"id":"428687","messageId":"e10fa11cfdb79071099e41c5f4328590e535ac1d.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 04/16] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:41Z","receivedAt":"2021-06-29T02:05:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\n* Add filenames adjacent to a sparse directory entry that sort before\n and after the trailing slash.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 42 ++++++++++++++++++++++--\n 1 file changed, 40 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ebbba044f77..363d605c209 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,23 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n+\t\t>folder1- &&\n+\t\t>folder1.x &&\n+\t\t>folder10 &&\n+\t\tcp -r deep/deeper1/0 folder1 &&\n+\t\tcp -r deep/deeper1/0 folder2 &&\n+\t\techo >>folder1/0/0/0 &&\n+\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n@@ -56,11 +69,17 @@ test_expect_success 'setup' '\n \t\tmv folder1/a folder2/b &&\n \t\tmv folder1/larger-content folder2/edited-content &&\n \t\techo >>folder2/edited-content &&\n+\t\techo >>folder2/0/1 &&\n+\t\techo stuff >>deep/deeper1/a &&\n \t\tgit add . &&\n \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n \n \t\tgit checkout -b rename-out-to-in rename-base &&\n \t\tmv folder1/a deep/deeper1/b &&\n+\t\techo more stuff >>deep/deeper1/a &&\n+\t\trm folder2/0/1 &&\n+\t\tmkdir folder2/0/1 &&\n+\t\techo >>folder2/0/1/1 &&\n \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n \t\techo >>deep/deeper1/edited-content &&\n \t\tgit add . &&\n@@ -68,6 +87,9 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b rename-in-to-out rename-base &&\n \t\tmv deep/deeper1/a folder1/b &&\n+\t\techo >>folder2/0/1 &&\n+\t\trm -rf folder1/0/0 &&\n+\t\techo >>folder1/0/0 &&\n \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n \t\techo >>folder1/edited-content &&\n \t\tgit add . &&\n@@ -262,13 +284,29 @@ test_expect_success 'diff --staged' '\n \ttest_all_match git diff --staged\n '\n \n-test_expect_success 'diff with renames' '\n+test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- .&&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_all_match git diff --staged --no-renames &&\n+\t\ttest_all_match git diff --staged --find-renames || return 1\n+\tdone\n+'\n+\n+test_expect_success 'diff with directory/file conflicts' '\n+\tinit_repos &&\n+\n+\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tdo\n+\t\tgit -C full-checkout reset --hard &&\n+\t\ttest_sparse_match git reset --hard &&\n+\t\ttest_all_match git checkout $branch &&\n+\t\ttest_all_match git checkout rename-base -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n \t\ttest_all_match git diff --staged --no-renames &&\n \t\ttest_all_match git diff --staged --find-renames || return 1\n \tdone\n-- \ngitgitgadget\n\n"},{"id":"428688","messageId":"a8dda933567788c6fbe53ff97bbf65851a10bcc6.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 06/16] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:43Z","receivedAt":"2021-06-29T02:05:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f88a69f8e71..87c1ed204c8 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"428689","messageId":"e94ffa07d4671c4f2ab87ab0a1abba53b5906678.1624932293.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 05/16] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:42Z","receivedAt":"2021-06-29T02:05:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 363d605c209..3f61e5686b5 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -254,6 +254,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"428690","messageId":"d04b62381b8c18cce8ec499d7213041772bfd4b2.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 08/16] unpack-trees: rename unpack_nondirectories()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:45Z","receivedAt":"2021-06-29T02:05:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn the next change, we will use this method to unpack a sparse directory\nentry, so change the name to unpack_single_entry() so these entries\napply. The new name reflects that we will not recurse into trees in\norder to resolve the conflicts.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b113cc750f2..d26386ce8b2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -804,7 +804,7 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,\n \t\tBUG(\"We need cache-tree to do this optimization\");\n \n \t/*\n-\t * Do what unpack_callback() and unpack_nondirectories() normally\n+\t * Do what unpack_callback() and unpack_single_entry() normally\n \t * do. But we walk all paths in an iterative loop instead.\n \t *\n \t * D/F conflicts and higher stage entries are not a concern\n@@ -1075,11 +1075,11 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n * without actually calling it. If you change the logic here you may need to\n * check and change there as well.\n */\n-static int unpack_nondirectories(int n, unsigned long mask,\n-\t\t\t\t unsigned long dirmask,\n-\t\t\t\t struct cache_entry **src,\n-\t\t\t\t const struct name_entry *names,\n-\t\t\t\t const struct traverse_info *info)\n+static int unpack_single_entry(int n, unsigned long mask,\n+\t\t\t unsigned long dirmask,\n+\t\t\t struct cache_entry **src,\n+\t\t\t const struct name_entry *names,\n+\t\t\t const struct traverse_info *info)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n@@ -1322,7 +1322,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_single_entry(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n-- \ngitgitgadget\n\n"},{"id":"428691","messageId":"e52166f6e4c0a7316214971c462d66f4c2c9cad9.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 07/16] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:44Z","receivedAt":"2021-06-29T02:05:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry. The final check is a\nlength comparison between the strings.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 87c1ed204c8..b113cc750f2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1014,6 +1016,16 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1)\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"428692","messageId":"237ccf4e43dd461250b3e6d609a475c1d675ea86.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 09/16] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:46Z","receivedAt":"2021-06-29T02:05:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring unpack_callback(), index entries are compared against tree\nentries. These are matched according to names and types. One goal is to\ndecide if we should recurse into subtrees or simply operate on one index\nentry.\n\nIn the case of a sparse-directory entry, we do not want to recurse into\nthat subtree and instead simply compare the trees. In some cases, we\nmight want to perform a merge operation on the entry, such as during\n'git checkout ' which wants to replace a sparse tree entry with\nthe tree for that path at the target commit. We extend the logic within\nunpack_single_entry() to create a sparse-directory entry in this case,\nand then that is sent to call_unpack_fn().\n\nThere are some subtleties in this process. For instance, we need to\nupdate find_cache_entry() to allow finding a sparse-directory entry that\nexactly matches a given path. Use the new helper method\nsparse_dir_matches_path() for this. We also need to ignore conflict\nmarkers in the case that the entries correspond to directories and we\nalready have a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 105 +++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 97 insertions(+), 8 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d26386ce8b2..d141dffbd94 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1052,13 +1052,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tconst struct name_entry *n,\n \tint stage,\n \tstruct index_state *istate,\n-\tint is_transient)\n+\tint is_transient,\n+\tint is_sparse_directory)\n {\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n+\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len, NULL) :\n-\t\tmake_empty_cache_entry(istate, len);\n+\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n+\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -1067,6 +1069,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \t/* len+1 because the cache_entry allocates space for NUL */\n \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n+\tif (is_sparse_directory) {\n+\t\tce->name[len] = '/';\n+\t\tce->name[len + 1] = '\\0';\n+\t\tce->ce_namelen++;\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\t}\n+\n \treturn ce;\n }\n \n@@ -1085,10 +1094,17 @@ static int unpack_single_entry(int n, unsigned long mask,\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\t/*\n+\t * When we have a sparse directory entry for src[0],\n+\t * then this isn't necessarily a directory-file conflict.\n+\t */\n+\tif (mask == dirmask && src[0] &&\n+\t S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\tconflicts = 0;\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n@@ -1118,7 +1134,9 @@ static int unpack_single_entry(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n+\t\t\t\t\t\t &o->result, o->merge,\n+\t\t\t\t\t\t bit & dirmask);\n \t}\n \n \tif (o->merge) {\n@@ -1222,16 +1240,69 @@ static int find_cache_pos(struct traverse_info *info,\n \treturn -1;\n }\n \n+/*\n+ * Given a sparse directory entry 'ce', compare ce->name to\n+ * info->name + '/' + p->path + '/' if info->name is non-empty.\n+ * Compare ce->name to p->path + '/' otherwise. Note that\n+ * ce->name must end in a trailing '/' because it is a sparse\n+ * directory entry.\n+ */\n+static int sparse_dir_matches_path(const struct cache_entry *ce,\n+\t\t\t\t struct traverse_info *info,\n+\t\t\t\t const struct name_entry *p)\n+{\n+\tassert(S_ISSPARSEDIR(ce->ce_mode));\n+\tassert(ce->name[ce->ce_namelen - 1] == '/');\n+\n+\tif (info->namelen)\n+\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n+\t\t ce->name[info->namelen] == '/' &&\n+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\treturn ce->ce_namelen == p->pathlen + 1 &&\n+\t !strncmp(ce->name, p->path, p->pathlen);\n+}\n+\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n+\tstruct cache_entry *ce;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n-\telse\n+\n+\t/*\n+\t * Check for a sparse-directory entry named \"path/\".\n+\t * Due to the input p->path not having a trailing\n+\t * slash, the negative 'pos' value overshoots the\n+\t * expected position by at least one, hence \"-2\" here.\n+\t */\n+\tpos = -pos - 2;\n+\n+\tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n+\n+\t/*\n+\t * We might have multiple entries between 'pos' and\n+\t * the actual sparse-directory entry, so start walking\n+\t * back until finding it or passing where it would be.\n+\t */\n+\twhile (pos >= 0) {\n+\t\tce = o->src_index->cache[pos];\n+\n+\t\tif (strncmp(ce->name, p->path, p->pathlen))\n+\t\t\treturn NULL;\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t sparse_dir_matches_path(ce, info, p))\n+\t\t\treturn ce;\n+\n+\t\tpos--;\n+\t}\n+\n+\treturn NULL;\n }\n \n static void debug_path(struct traverse_info *info)\n@@ -1266,6 +1337,21 @@ static void debug_unpack_callback(int n,\n \t\tdebug_name_entry(i, names + i);\n }\n \n+/*\n+ * Returns true if and only if the given cache_entry is a\n+ * sparse-directory entry that matches the given name_entry\n+ * from the tree walk at the given traverse_info.\n+ */\n+static int is_sparse_directory_entry(struct cache_entry *ce,\n+\t\t\t\t struct name_entry *name,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n+\treturn sparse_dir_matches_path(ce, info, name);\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -1352,9 +1438,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n-\t\t\t\t\t names, info) < 0)\n+\t\tif (!is_sparse_directory_entry(src[0], names, info) &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n+\t\t}\n+\n \t\treturn mask;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"428693","messageId":"2a43287c47ed361fcae962d508a8a9354c654fcb.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 11/16] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:48Z","receivedAt":"2021-06-29T02:05:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 24 +++++++++++++++++++-----\n 1 file changed, 19 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e0..0c5264b3b20 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1376,7 +1376,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tstruct path_pattern *pattern;\n \tstruct strbuf parent_pathname = STRBUF_INIT;\n \tint result = NOT_MATCHED;\n-\tconst char *slash_pos;\n+\tsize_t slash_pos;\n \n \tif (!pl->use_cone_patterns) {\n \t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n@@ -1397,21 +1397,35 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/') {\n+\t\tslash_pos = parent_pathname.len - 1;\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\t} else {\n+\t\tconst char *slash_ptr = strrchr(parent_pathname.buf, '/');\n+\t\tslash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0;\n+\t}\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n-\tslash_pos = strrchr(parent_pathname.buf, '/');\n-\n-\tif (slash_pos == parent_pathname.buf) {\n+\tif (!slash_pos) {\n \t\t/* include every file in root */\n \t\tresult = MATCHED;\n \t\tgoto done;\n \t}\n \n-\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\tstrbuf_setlen(&parent_pathname, slash_pos);\n \n \tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n \t\tresult = MATCHED;\n-- \ngitgitgadget\n\n"},{"id":"428694","messageId":"9f31c691af6780f0ea48bdcb5ff6d56b628f1a81.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 10/16] unpack-trees: handle dir/file conflict of sparse entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:47Z","receivedAt":"2021-06-29T02:05:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 25 ++++++++++++++++++++++--\n unpack-trees.c | 5 ++++-\n 2 files changed, 27 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 3f61e5686b5..4e6446e7545 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -95,6 +95,19 @@ test_expect_success 'setup' '\n \t\tgit add . &&\n \t\tgit commit -m \"rename deep/deeper1/... to folder1/...\" &&\n \n+\t\tgit checkout -b df-conflict base &&\n+\t\trm -rf folder1 &&\n+\t\techo content >folder1 &&\n+\t\tgit add . &&\n+\t\tgit commit -m df &&\n+\n+\t\tgit checkout -b fd-conflict base &&\n+\t\trm a &&\n+\t\tmkdir a &&\n+\t\techo content >a/a &&\n+\t\tgit add . &&\n+\t\tgit commit -m fd &&\n+\n \t\tgit checkout -b deepest base &&\n \t\techo \"updated deepest\" >deep/deeper1/deepest/a &&\n \t\tgit commit -a -m \"update deepest\" &&\n@@ -325,7 +338,11 @@ test_expect_success 'diff --staged' '\n test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n-\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tfor branch in rename-out-to-out \\\n+\t\t rename-out-to-in \\\n+\t\t rename-in-to-out \\\n+\t\t df-conflict \\\n+\t\t fd-conflict\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- .&&\n@@ -338,7 +355,11 @@ test_expect_success 'diff with renames and conflicts' '\n test_expect_success 'diff with directory/file conflicts' '\n \tinit_repos &&\n \n-\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tfor branch in rename-out-to-out \\\n+\t\t rename-out-to-in \\\n+\t\t rename-in-to-out \\\n+\t\t df-conflict \\\n+\t\t fd-conflict\n \tdo\n \t\tgit -C full-checkout reset --hard &&\n \t\ttest_sparse_match git reset --hard &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d141dffbd94..e63b2dcacbc 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2617,7 +2617,10 @@ int twoway_merge(const struct cache_entry * const *src,\n \t\t\t same(current, oldtree) && !same(current, newtree)) {\n \t\t\t/* 20 or 21 */\n \t\t\treturn merged_entry(newtree, current, o);\n-\t\t} else\n+\t\t} else if (current && !oldtree && newtree &&\n+\t\t\t S_ISSPARSEDIR(current->ce_mode) != S_ISSPARSEDIR(newtree->ce_mode))\n+\t\t\treturn merged_entry(newtree, current, o);\n+\t\telse\n \t\t\treturn reject_merge(current, o);\n \t}\n \telse if (newtree) {\n-- \ngitgitgadget\n\n"},{"id":"428695","messageId":"f83aa08ff6b0fd18d6f9f3ce5ee993523a7f1759.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 12/16] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:49Z","receivedAt":"2021-06-29T02:05:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile comparing an index to a tree, we may see a sparse directory entry.\nIn this case, we should compare that portion of the tree to the tree\nrepresented by that entry. This could include a new tree which needs to\nbe expanded to a full list of added files. It could also include an\nexisting tree, in which case all of the changes inside are important to\ndescribe, including the modifications, additions, and deletions. Note\nthat the case where the tree has a path and the index does not remains\nidentical to before: the lack of a cache entry is the same with a sparse\nindex.\n\nUse diff_tree_oid() appropriately to compute the diff.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 16 ++++++++++++++++\n 1 file changed, 16 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex c2ac9250fe9..3f32f038371 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level.\n+\t */\n+\tif (old_entry && new_entry &&\n+\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n+\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n-- \ngitgitgadget\n\n"},{"id":"428696","messageId":"35063ffb8ed6c60c5777d27e4e36a802234200d2.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 13/16] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:50Z","receivedAt":"2021-06-29T02:05:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4e6446e7545..d372932cd12 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -231,6 +231,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 42b67357169..96db3e74962 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e..ab9cc9d8f03 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"428697","messageId":"717a3f49f97753149d5c435a83b3f1773dd4b1bb.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 15/16] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:52Z","receivedAt":"2021-06-29T02:05:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++\n wt-status.c | 51 ++++++++++++++++++++++++\n 2 files changed, 84 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex fed0440bafe..df217a2d10b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -545,4 +545,37 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\n+\t# Sparse checkouts do not agree with full checkouts about\n+\t# how to report a directory/file conflict during a reset.\n+\t# This command would fail with test_all_match because the\n+\t# full checkout reports \"T folder1/0/1\" while a sparse\n+\t# checkout reports \"D folder1/0/1\". This matches because\n+\t# the sparse checkouts skip \"adding\" the other side of\n+\t# the conflict.\n+\ttest_sparse_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 96db3e74962..0317baef87e 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \tclear_pathspec(&rev.prune_data);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tstruct strbuf full_name = STRBUF_INIT;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tstrbuf_add(&full_name, base->buf, base->len);\n+\tstrbuf_addstr(&full_name, path);\n+\tit = string_list_insert(&s->change, full_name.buf);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\tstrbuf_release(&full_name);\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps = { 0 };\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"428698","messageId":"1d744848ee6b58ccaf3a30f20abe9797ed5d2ce7.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 16/16] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:53Z","receivedAt":"2021-06-29T02:05:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 49 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex ef53bd2198b..53c8f711ccc 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 637391c6ce4..deea88d4431 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,52 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_region fsm_hook query trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"428699","messageId":"b4033a9bf364a45f9b75a58b17dfe8a6930fc992.1624932294.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v7 14/16] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-06-29T02:04:51Z","receivedAt":"2021-06-29T02:05:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 190d215d43b..12f51db158a 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 1b3c2eb408b..277c2970a03 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1584,8 +1584,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1600,6 +1599,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d372932cd12..fed0440bafe 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -532,12 +532,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"428708","messageId":"b49f8b53-ccbf-b156-3054-03d1273cf9dd@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 00/16] Sparse-index: integrate with status","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-29T02:16:04Z","receivedAt":"2021-06-29T02:16:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/28/2021 10:04 PM, Derrick Stolee via GitGitGadget wrote:\n...\n> Update in V7 (relative to v5)\n> =============================\n> \n> APOLOGIES: As I was working on this cover letter, I was still organizing my\n> big list of patches, including reordering some into this series. I forgot to\n> actually include them in my v6 submission, so here is a re-submission.\n> Please ignore v6.\n\nSince v6 was a mistake, here is the full range-diff of v5 versus v7:\n\n 1: 5a2ed3d1d70 = 1: 2a4a7256304 sparse-index: skip indexes with unmerged entries\n 2: 8aa41e74947 ! 2: f5bae86014d sparse-index: include EXTENDED flag when expanding\n @@ Commit message\n \n When creating a full index from a sparse one, we create cache entries\n for every blob within a given sparse directory entry. These are\n - correctly marked with the CE_SKIP_WORKTREE flag, but they must also be\n - marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is\n - correctly written to disk in the case that the index is not converted\n - back down to a sparse-index.\n + correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\n + flag is not included. The CE_EXTENDED flag would exist if we loaded a\n + full index from disk with these entries marked with CE_SKIP_WORKTREE, so\n + we can add the flag here to be consistent. This allows us to directly\n + compare the flags present in cache entries when testing the sparse-index\n + feature, but has no significance to its correctness in the user-facing\n + functionality.\n \n Signed-off-by: Derrick Stolee \n \n 3: b99371c7dd6 ! 3: d965669c766 t1092: replace incorrect 'echo' with 'cat'\n @@ Commit message\n t1092: replace incorrect 'echo' with 'cat'\n \n This fixes the test data shape to be as expected, allowing rename\n - detection to work properly now that the 'larger-conent' file actually\n + detection to work properly now that the 'larger-content' file actually\n has meaningful lines.\n \n Signed-off-by: Derrick Stolee \n 4: f4dddac1859 ! 4: e10fa11cfdb t1092: expand repository data shape\n @@ Commit message\n one entry and are the first entry of a directory with multiple\n entries.\n \n + * Add filenames adjacent to a sparse directory entry that sort before\n + and after the trailing slash.\n +\n Later tests will take advantage of these shapes, but they also deepen\n the tests that already exist.\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'setup' '\n +\t\tmkdir deep/deeper1/0/0 &&\n +\t\ttouch deep/deeper1/0/1 &&\n +\t\ttouch deep/deeper1/0/0/0 &&\n ++\t\t>folder1- &&\n ++\t\t>folder1.x &&\n ++\t\t>folder10 &&\n +\t\tcp -r deep/deeper1/0 folder1 &&\n +\t\tcp -r deep/deeper1/0 folder2 &&\n +\t\techo >>folder1/0/0/0 &&\n 5: 856346b72f7 = 5: e94ffa07d46 t1092: add tests for status/add and sparse files\n 6: f3f6223e955 = 6: a8dda933567 unpack-trees: preserve cache_bottom\n 7: 45ae96adf28 ! 7: e52166f6e4c unpack-trees: compare sparse directories correctly\n @@ Commit message\n Within compare_entry(), it first calls do_compare_entry() to check the\n leading portion of the name. When the input path is a directory name, we\n could match exactly already. Thus, we should return 0 if we have an\n - exact string match on a sparse directory entry.\n + exact string match on a sparse directory entry. The final check is a\n + length comparison between the strings.\n \n Signed-off-by: Derrick Stolee \n \n @@ unpack-trees.c: static int compare_entry(const struct cache_entry *ce, const str\n +\t * works when the input name is a directory, since ce->name\n +\t * ends in a directory separator.\n +\t */\n -+\tif (S_ISSPARSEDIR(ce->ce_mode))\n ++\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n ++\t ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1)\n +\t\treturn 0;\n +\n \t/*\n -: ----------- > 8: d04b62381b8 unpack-trees: rename unpack_nondirectories()\n 8: 724194eef9f ! 9: 237ccf4e43d unpack-trees: unpack sparse directory entries\n @@ Commit message\n might want to perform a merge operation on the entry, such as during\n 'git checkout ' which wants to replace a sparse tree entry with\n the tree for that path at the target commit. We extend the logic within\n - unpack_nondirectories() to create a sparse-directory entry in this case,\n + unpack_single_entry() to create a sparse-directory entry in this case,\n and then that is sent to call_unpack_fn().\n \n There are some subtleties in this process. For instance, we need to\n update find_cache_entry() to allow finding a sparse-directory entry that\n - exactly matches a given path.\n + exactly matches a given path. Use the new helper method\n + sparse_dir_matches_path() for this. We also need to ignore conflict\n + markers in the case that the entries correspond to directories and we\n + already have a sparse directory entry.\n \n Signed-off-by: Derrick Stolee \n \n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n +\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n --\t\tmake_empty_transient_cache_entry(len) :\n +-\t\tmake_empty_transient_cache_entry(len, NULL) :\n -\t\tmake_empty_cache_entry(istate, len);\n -+\t\tmake_empty_transient_cache_entry(alloc_len) :\n ++\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n +\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \n +\tif (is_sparse_directory) {\n +\t\tce->name[len] = '/';\n -+\t\tce->name[len + 1] = 0;\n ++\t\tce->name[len + 1] = '\\0';\n +\t\tce->ce_namelen++;\n +\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n +\t}\n @@ unpack-trees.c: static struct cache_entry *create_ce_entry(const struct traverse\n \treturn ce;\n }\n \n -@@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n - \t\t\t\t unsigned long dirmask,\n - \t\t\t\t struct cache_entry **src,\n - \t\t\t\t const struct name_entry *names,\n --\t\t\t\t const struct traverse_info *info)\n -+\t\t\t\t const struct traverse_info *info,\n -+\t\t\t\t int sparse_directory)\n - {\n - \tint i;\n +@@ unpack-trees.c: static int unpack_single_entry(int n, unsigned long mask,\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n @@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n -+\t/* no-op if our cache entry doesn't match the expectations. */\n -+\tif (sparse_directory) {\n -+\t\tif (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode))\n -+\t\t\tBUG(\"expected sparse directory entry\");\n -+\t} else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) {\n -+\t\treturn 0;\n -+\t}\n ++\t/*\n ++\t * When we have a sparse directory entry for src[0],\n ++\t * then this isn't necessarily a directory-file conflict.\n ++\t */\n ++\tif (mask == dirmask && src[0] &&\n ++\t S_ISSPARSEDIR(src[0]->ce_mode))\n ++\t\tconflicts = 0;\n +\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n -@@ unpack-trees.c: static int unpack_nondirectories(int n, unsigned long mask,\n +@@ unpack-trees.c: static int unpack_single_entry(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n -\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n +\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n +\t\t\t\t\t\t &o->result, o->merge,\n -+\t\t\t\t\t\t sparse_directory);\n ++\t\t\t\t\t\t bit & dirmask);\n \t}\n \n \tif (o->merge) {\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n + \treturn -1;\n + }\n + \n ++/*\n ++ * Given a sparse directory entry 'ce', compare ce->name to\n ++ * info->name + '/' + p->path + '/' if info->name is non-empty.\n ++ * Compare ce->name to p->path + '/' otherwise. Note that\n ++ * ce->name must end in a trailing '/' because it is a sparse\n ++ * directory entry.\n ++ */\n ++static int sparse_dir_matches_path(const struct cache_entry *ce,\n ++\t\t\t\t struct traverse_info *info,\n ++\t\t\t\t const struct name_entry *p)\n ++{\n ++\tassert(S_ISSPARSEDIR(ce->ce_mode));\n ++\tassert(ce->name[ce->ce_namelen - 1] == '/');\n ++\n ++\tif (info->namelen)\n ++\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n ++\t\t ce->name[info->namelen] == '/' &&\n ++\t\t !strncmp(ce->name, info->name, info->namelen) &&\n ++\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n ++\treturn ce->ce_namelen == p->pathlen + 1 &&\n ++\t !strncmp(ce->name, p->path, p->pathlen);\n ++}\n ++\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n +\t * Check for a sparse-directory entry named \"path/\".\n +\t * Due to the input p->path not having a trailing\n +\t * slash, the negative 'pos' value overshoots the\n -+\t * expected position by one, hence \"-2\" here.\n ++\t * expected position by at least one, hence \"-2\" here.\n +\t */\n +\tpos = -pos - 2;\n +\n +\tif (pos < 0 || pos >= o->src_index->cache_nr)\n -+\t\treturn NULL;\n -+\n -+\tce = o->src_index->cache[pos];\n -+\n -+\tif (!S_ISSPARSEDIR(ce->ce_mode))\n \t\treturn NULL;\n +\n +\t/*\n -+\t * Compare ce->name to info->name + '/' + p->path + '/'\n -+\t * if info->name is non-empty. Compare ce->name to\n -+\t * p-.path + '/' otherwise.\n ++\t * We might have multiple entries between 'pos' and\n ++\t * the actual sparse-directory entry, so start walking\n ++\t * back until finding it or passing where it would be.\n +\t */\n -+\tif (info->namelen) {\n -+\t\tif (ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n -+\t\t ce->name[info->namelen] == '/' &&\n -+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n -+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen))\n ++\twhile (pos >= 0) {\n ++\t\tce = o->src_index->cache[pos];\n ++\n ++\t\tif (strncmp(ce->name, p->path, p->pathlen))\n ++\t\t\treturn NULL;\n ++\n ++\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n ++\t\t sparse_dir_matches_path(ce, info, p))\n +\t\t\treturn ce;\n -+\t} else if (ce->ce_namelen == p->pathlen + 1 &&\n -+\t\t !strncmp(ce->name, p->path, p->pathlen))\n -+\t\treturn ce;\n ++\n ++\t\tpos--;\n ++\t}\n ++\n +\treturn NULL;\n }\n \n @@ unpack-trees.c: static void debug_unpack_callback(int n,\n + * sparse-directory entry that matches the given name_entry\n + * from the tree walk at the given traverse_info.\n + */\n -+static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info)\n ++static int is_sparse_directory_entry(struct cache_entry *ce,\n ++\t\t\t\t struct name_entry *name,\n ++\t\t\t\t struct traverse_info *info)\n +{\n -+\tsize_t expected_len, name_start;\n -+\n +\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n +\t\treturn 0;\n +\n -+\tif (info->namelen)\n -+\t\tname_start = info->namelen + 1;\n -+\telse\n -+\t\tname_start = 0;\n -+\texpected_len = name->pathlen + 1 + name_start;\n -+\n -+\tif (ce->ce_namelen != expected_len ||\n -+\t strncmp(ce->name, info->name, info->namelen) ||\n -+\t strncmp(ce->name + name_start, name->path, name->pathlen))\n -+\t\treturn 0;\n -+\n -+\treturn 1;\n ++\treturn sparse_dir_matches_path(ce, info, name);\n +}\n +\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n -@@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n - \t\t}\n - \t}\n - \n --\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n -+\tif (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0)\n - \t\treturn -1;\n - \n - \tif (o->merge && src[0]) {\n @@ unpack-trees.c: static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n -\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n -\t\t\t\t\t names, info) < 0)\n -+\t\tif (is_sparse_directory_entry(src[0], names, info)) {\n -+\t\t\tif (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0)\n -+\t\t\t\treturn -1;\n -+\t\t} else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n ++\t\tif (!is_sparse_directory_entry(src[0], names, info) &&\n ++\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n +\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n +\t\t}\n -: ----------- > 10: 9f31c691af6 unpack-trees: handle dir/file conflict of sparse entries\n 9: b8ff179f43e ! 11: 2a43287c47e dir.c: accept a directory as part of cone-mode patterns\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## dir.c ##\n +@@ dir.c: enum pattern_match_result path_matches_pattern_list(\n + \tstruct path_pattern *pattern;\n + \tstruct strbuf parent_pathname = STRBUF_INIT;\n + \tint result = NOT_MATCHED;\n +-\tconst char *slash_pos;\n ++\tsize_t slash_pos;\n + \n + \tif (!pl->use_cone_patterns) {\n + \t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n @@ dir.c: enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n @@ dir.c: enum pattern_match_result path_matches_pattern_list(\n +\t * use the file-base matching logic in an equivalent way.\n +\t */\n +\tif (parent_pathname.len > 0 &&\n -+\t parent_pathname.buf[parent_pathname.len - 1] == '/')\n ++\t parent_pathname.buf[parent_pathname.len - 1] == '/') {\n ++\t\tslash_pos = parent_pathname.len - 1;\n +\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n ++\t} else {\n ++\t\tconst char *slash_ptr = strrchr(parent_pathname.buf, '/');\n ++\t\tslash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0;\n ++\t}\n +\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n + \t\tgoto done;\n + \t}\n + \n +-\tslash_pos = strrchr(parent_pathname.buf, '/');\n +-\n +-\tif (slash_pos == parent_pathname.buf) {\n ++\tif (!slash_pos) {\n + \t\t/* include every file in root */\n + \t\tresult = MATCHED;\n + \t\tgoto done;\n + \t}\n + \n +-\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n ++\tstrbuf_setlen(&parent_pathname, slash_pos);\n + \n + \tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n + \t\tresult = MATCHED;\n10: b9b97e01129 ! 12: f83aa08ff6b diff-lib: handle index diffs with sparse dirs\n @@ Commit message\n identical to before: the lack of a cache entry is the same with a sparse\n index.\n \n - In the case where a tree is modified, we need to expand the tree\n - recursively, and start comparing each contained entry as either an\n - addition, deletion, or modification. This causes an interesting\n - recursion that did not exist before.\n + Use diff_tree_oid() appropriately to compute the diff.\n \n Signed-off-by: Derrick Stolee \n \n ## diff-lib.c ##\n -@@ diff-lib.c: static int get_stat_data(const struct cache_entry *ce,\n - \treturn 0;\n - }\n - \n -+struct show_new_tree_context {\n -+\tstruct rev_info *revs;\n -+\tunsigned added:1;\n -+};\n -+\n -+static int show_new_file_from_tree(const struct object_id *oid,\n -+\t\t\t\t struct strbuf *base, const char *path,\n -+\t\t\t\t unsigned int mode, void *context)\n -+{\n -+\tstruct show_new_tree_context *ctx = context;\n -+\tstruct cache_entry *new_file = make_transient_cache_entry(mode, oid, path, /* stage */ 0);\n -+\n -+\tdiff_index_show_file(ctx->revs, ctx->added ? \"+\" : \"-\", new_file, oid, !is_null_oid(oid), mode, 0);\n -+\tdiscard_cache_entry(new_file);\n -+\treturn 0;\n -+}\n -+\n -+static void show_directory(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *new_dir,\n -+\t\t\t int added)\n -+{\n -+\t/*\n -+\t * new_dir is a sparse directory entry, so we want to collect all\n -+\t * of the new files within the tree. This requires recursively\n -+\t * expanding the trees.\n -+\t */\n -+\tstruct show_new_tree_context ctx = { revs, added };\n -+\tstruct repository *r = revs->repo;\n -+\tstruct strbuf base = STRBUF_INIT;\n -+\tstruct pathspec ps;\n -+\tstruct tree *tree = lookup_tree(r, &new_dir->oid);\n -+\n -+\tmemset(&ps, 0, sizeof(ps));\n -+\tps.recursive = 1;\n -+\tps.has_wildcard = 1;\n -+\tps.max_depth = -1;\n -+\n -+\tstrbuf_add(&base, new_dir->name, new_dir->ce_namelen);\n -+\tread_tree_at(r, tree, &base, &ps,\n -+\t\t\tshow_new_file_from_tree, &ctx);\n -+}\n -+\n - static void show_new_file(struct rev_info *revs,\n - \t\t\t const struct cache_entry *new_file,\n - \t\t\t int cached, int match_missing)\n @@ diff-lib.c: static void show_new_file(struct rev_info *revs,\n - \tunsigned int mode;\n \tunsigned dirty_submodule = 0;\n + \tstruct index_state *istate = revs->diffopt.repo->index;\n \n +\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n -+\t\tshow_directory(revs, new_file, /*added */ 1);\n ++\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n +\t\treturn;\n +\t}\n +\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n -@@ diff-lib.c: static void show_new_file(struct rev_info *revs,\n - \tdiff_index_show_file(revs, \"+\", new_file, oid, !is_null_oid(oid), mode, dirty_submodule);\n - }\n - \n -+static int show_modified(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *old_entry,\n -+\t\t\t const struct cache_entry *new_entry,\n -+\t\t\t int report_missing,\n -+\t\t\t int cached, int match_missing);\n -+\n -+static int compare_within_sparse_dir(int n, unsigned long mask,\n -+\t\t\t\t unsigned long dirmask, struct name_entry *entry,\n -+\t\t\t\t struct traverse_info *info)\n -+{\n -+\tstruct rev_info *revs = info->data;\n -+\tstruct object_id *oid0 = &entry[0].oid;\n -+\tstruct object_id *oid1 = &entry[1].oid;\n -+\n -+\tif (oideq(oid0, oid1))\n -+\t\treturn mask;\n -+\n -+\t/* Directory/file conflicts are handled earlier. */\n -+\tif (S_ISDIR(entry[0].mode) && S_ISDIR(entry[1].mode)) {\n -+\t\tstruct tree_desc t[2];\n -+\t\tvoid *buf[2];\n -+\t\tstruct traverse_info info_r = { NULL, };\n -+\n -+\t\tinfo_r.name = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n -+\t\tinfo_r.namelen = strlen(info_r.name);\n -+\t\tinfo_r.traverse_path = xstrfmt(\"%s/\", info_r.name);\n -+\t\tinfo_r.fn = compare_within_sparse_dir;\n -+\t\tinfo_r.prev = info;\n -+\t\tinfo_r.mode = entry[0].mode;\n -+\t\tinfo_r.pathlen = entry[0].pathlen;\n -+\t\tinfo_r.df_conflicts = 0;\n -+\t\tinfo_r.data = revs;\n -+\n -+\t\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], oid0);\n -+\t\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], oid1);\n -+\n -+\t\ttraverse_trees(NULL, 2, t, &info_r);\n -+\n -+\t\tfree((char *)info_r.name);\n -+\t\tfree((char *)info_r.traverse_path);\n -+\t\tfree(buf[0]);\n -+\t\tfree(buf[1]);\n -+\t} else {\n -+\t\tchar *old_path = NULL, *new_path = NULL;\n -+\t\tstruct cache_entry *old_entry = NULL, *new_entry = NULL;\n -+\n -+\t\tif (entry[0].path) {\n -+\t\t\told_path = xstrfmt(\"%s%s\", info->traverse_path, entry[0].path);\n -+\t\t\told_entry = make_transient_cache_entry(\n -+\t\t\t\t\tentry[0].mode, &entry[0].oid,\n -+\t\t\t\t\told_path, /* stage */ 0);\n -+\t\t\told_entry->ce_flags |= CE_SKIP_WORKTREE;\n -+\t\t}\n -+\t\tif (entry[1].path) {\n -+\t\t\tnew_path = xstrfmt(\"%s%s\", info->traverse_path, entry[1].path);\n -+\t\t\tnew_entry = make_transient_cache_entry(\n -+\t\t\t\t\tentry[1].mode, &entry[1].oid,\n -+\t\t\t\t\tnew_path, /* stage */ 0);\n -+\t\t\tnew_entry->ce_flags |= CE_SKIP_WORKTREE;\n -+\t\t}\n -+\n -+\t\tif (entry[0].path && entry[1].path)\n -+\t\t\tshow_modified(revs, old_entry, new_entry, 0, 1, 0);\n -+\t\telse if (entry[0].path)\n -+\t\t\tdiff_index_show_file(revs, revs->prefix,\n -+\t\t\t\t\t old_entry, &entry[0].oid,\n -+\t\t\t\t\t 0, entry[0].mode, 0);\n -+\t\telse if (entry[1].path)\n -+\t\t\tshow_new_file(revs, new_entry, 1, 0);\n -+\n -+\t\tdiscard_cache_entry(old_entry);\n -+\t\tdiscard_cache_entry(new_entry);\n -+\t\tfree(old_path);\n -+\t\tfree(new_path);\n -+\t}\n -+\n -+\treturn mask;\n -+}\n -+\n -+static void show_modified_sparse_directory(struct rev_info *revs,\n -+\t\t\t const struct cache_entry *old_entry,\n -+\t\t\t const struct cache_entry *new_entry,\n -+\t\t\t int report_missing,\n -+\t\t\t int cached, int match_missing)\n -+{\n -+\tstruct tree_desc t[2];\n -+\tvoid *buf[2];\n -+\tstruct traverse_info info = { NULL };\n -+\tstruct strbuf name = STRBUF_INIT;\n -+\tstruct strbuf parent_path = STRBUF_INIT;\n -+\tchar *last_dir_sep;\n -+\n -+\tif (oideq(&old_entry->oid, &new_entry->oid))\n -+\t\treturn;\n -+\n -+\tinfo.fn = compare_within_sparse_dir;\n -+\tinfo.prev = &info;\n -+\n -+\tstrbuf_add(&name, new_entry->name, new_entry->ce_namelen - 1);\n -+\tinfo.name = name.buf;\n -+\tinfo.namelen = name.len;\n -+\n -+\tstrbuf_add(&parent_path, new_entry->name, new_entry->ce_namelen - 1);\n -+\tif ((last_dir_sep = find_last_dir_sep(parent_path.buf)) > parent_path.buf)\n -+\t\tstrbuf_setlen(&parent_path, (last_dir_sep - parent_path.buf) - 1);\n -+\telse\n -+\t\tstrbuf_setlen(&parent_path, 0);\n -+\n -+\tinfo.pathlen = parent_path.len;\n -+\n -+\tif (parent_path.len)\n -+\t\tinfo.traverse_path = parent_path.buf;\n -+\telse\n -+\t\tinfo.traverse_path = \"\";\n -+\n -+\tinfo.mode = new_entry->ce_mode;\n -+\tinfo.df_conflicts = 0;\n -+\tinfo.data = revs;\n -+\n -+\tbuf[0] = fill_tree_descriptor(revs->repo, &t[0], &old_entry->oid);\n -+\tbuf[1] = fill_tree_descriptor(revs->repo, &t[1], &new_entry->oid);\n -+\n -+\ttraverse_trees(NULL, 2, t, &info);\n -+\n -+\tfree(buf[0]);\n -+\tfree(buf[1]);\n -+\tstrbuf_release(&name);\n -+\tstrbuf_release(&parent_path);\n -+}\n -+\n - static int show_modified(struct rev_info *revs,\n - \t\t\t const struct cache_entry *old_entry,\n - \t\t\t const struct cache_entry *new_entry,\n @@ diff-lib.c: static int show_modified(struct rev_info *revs,\n - \tconst struct object_id *oid;\n \tunsigned dirty_submodule = 0;\n + \tstruct index_state *istate = revs->diffopt.repo->index;\n \n +\t/*\n +\t * If both are sparse directory entries, then expand the\n @@ diff-lib.c: static int show_modified(struct rev_info *revs,\n +\tif (old_entry && new_entry &&\n +\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n +\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n -+\t\tshow_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing);\n ++\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n +\t\treturn 0;\n +\t}\n +\n - \tif (get_stat_data(new_entry, &oid, &mode, cached, match_missing,\n + \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n11: 611b9f61fb2 = 13: 35063ffb8ed status: skip sparse-checkout percentage with sparse-index\n12: 0c0a765dde8 = 14: b4033a9bf36 status: use sparse-index throughout\n13: 02f2c7b6398 ! 15: 717a3f49f97 wt-status: expand added sparse directory entries\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n +\ttest_sparse_match git reset --mixed HEAD~1 &&\n +\ttest_sparse_match test-tool read-cache --table --expand &&\n +\ttest_sparse_match git status --porcelain=v2 &&\n -+\ttest_sparse_match git status --porcelain=v2 &&\n +\n +\t# At this point, sparse-checkouts behave differently\n +\t# from the full-checkout.\n +\ttest_sparse_match git checkout --orphan new-branch &&\n +\ttest_sparse_match test-tool read-cache --table --expand &&\n -+\ttest_sparse_match git status --porcelain=v2 &&\n +\ttest_sparse_match git status --porcelain=v2\n +'\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n +\n +\trun_on_all touch deep/deeper1/x &&\n +\ttest_all_match git add . &&\n -+\ttest_all_match git status --porcelain=v2 &&\n +\ttest_all_match git status --porcelain=v2\n +'\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n \n ## wt-status.c ##\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n - \trun_diff_index(&rev, 1);\n + \tclear_pathspec(&rev.prune_data);\n }\n \n +static int add_file_to_list(const struct object_id *oid,\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n +\tstruct string_list_item *it;\n +\tstruct wt_status_change_data *d;\n +\tstruct wt_status *s = context;\n -+\tchar *full_name;\n ++\tstruct strbuf full_name = STRBUF_INIT;\n +\n +\tif (S_ISDIR(mode))\n +\t\treturn READ_TREE_RECURSIVE;\n +\n -+\tfull_name = xstrfmt(\"%s%s\", base->buf, path);\n -+\tit = string_list_insert(&s->change, full_name);\n ++\tstrbuf_add(&full_name, base->buf, base->len);\n ++\tstrbuf_addstr(&full_name, path);\n ++\tit = string_list_insert(&s->change, full_name.buf);\n +\td = it->util;\n +\tif (!d) {\n +\t\tCALLOC_ARRAY(d, 1);\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n +\td->mode_index = mode;\n +\toidcpy(&d->oid_index, oid);\n +\ts->committable = 1;\n ++\tstrbuf_release(&full_name);\n +\treturn 0;\n +}\n +\n @@ wt-status.c: static void wt_status_collect_changes_initial(struct wt_status *s)\n +\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n +\t\t\t */\n +\t\t\tstruct strbuf base = STRBUF_INIT;\n -+\t\t\tstruct pathspec ps;\n ++\t\t\tstruct pathspec ps = { 0 };\n +\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n +\n -+\t\t\tmemset(&ps, 0, sizeof(ps));\n +\t\t\tps.recursive = 1;\n +\t\t\tps.has_wildcard = 1;\n +\t\t\tps.max_depth = -1;\n14: 46ca150c354 ! 16: 1d744848ee6 fsmonitor: integrate with sparse index\n @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/\n +\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n +\t\tgit status --porcelain=v2 >actual &&\n +\ttest_region $1 index ensure_full_index trace2.txt &&\n ++\ttest_region fsm_hook query trace2.txt &&\n +\ttest_cmp expect actual &&\n +\trm trace2.txt &&\n +\tgit sparse-checkout disable\n \n"},{"id":"428825","messageId":"CABPp-BHqoJjF9f6NKZ8jjQdj1bgUNgrwek5jcnGTRk2m-m8dBg@mail.gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 00/16] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-30T14:32:17Z","receivedAt":"2021-06-30T14:32:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 28, 2021 at 7:04 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' very fast when a sparse-index is enabled on a repository with\n> cone-mode sparse-checkout (and a small populated set).\n>\n> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n> The latter branch is needed because it changes the behavior of 'git add'\n> around sparse entries, which changes the expectations of a test added in\n> patch 1.\n>\n> The approach here is to audit the places where ensure_full_index() pops up\n> while doing normal commands with pathspecs within the sparse-checkout\n> definition. Each of these are checked and tested. In the end, the\n> sparse-index is integrated with these features:\n>\n> * git status\n> * FS Monitor index extension.\n>\n> The performance tests in p2000-sparse-operations.sh improve by 95% or more,\n> even when compared with the full-index cases, not just the sparse-index\n> cases that previously had extra overhead.\n>\n> Hopefully this is the first example of how ds/sparse-index-protections has\n> done the basic work to do these conversions safely, making them look easier\n> than they seemed when starting this adventure.\n>\n> Thanks, -Stolee\n>\n>\n> Update in V7 (relative to v5)\n> =============================\n>\n> APOLOGIES: As I was working on this cover letter, I was still organizing my\n> big list of patches, including reordering some into this series. I forgot to\n> actually include them in my v6 submission, so here is a re-submission.\n> Please ignore v6.\n>\n> I'm sorry that this revision took so long. Initially I was blocked on\n> getting the directory/file conflict figured out (I did), but also my team\n> was very busy with some things. Eventually, we reached an internal deadline\n> to make an experimental release available [1] with initial sparse-index\n> performance boosts. Creating that included some additional review by Jeff\n> Hostetler and Johannes Schindelin which led to more changes in this version.\n>\n> The good news is that this series has now created the basis for many Git\n> commands to integrate with the sparse-index without much additional work.\n> This effort was unfortunately overloaded on this series because the changes\n> needed for things like 'git checkout' or 'git add' all intersect with the\n> changes needed for 'git status'. Might as well get it right the first time.\n>\n> Because the range-diff is a big difficult to read this time, I'll break the\n> changes down on a patch-by-patch basis.\n>\n> 1. sparse-index: skip indexes with unmerged entries\n>\n> (no change)\n>\n> 2. sparse-index: include EXTENDED flag when expanding\n>\n> * Commit message better describes the purpose of the change.\n>\n> 3. t1092: replace incorrect 'echo' with 'cat'\n>\n> * Typo fix\n>\n> 4. t1092: expand repository data shape\n>\n> * some files are added that surround \"folder1/\" immediately before and\n> after, based on the sorting with the trailing slash. This provides extra\n> coverage.\n>\n> 5. t1092: add tests for status/add and sparse files\n>\n> (no change)\n>\n> 6. unpack-trees: preserve cache_bottom\n>\n> (no change)\n>\n> 7. unpack-trees: compare sparse directories correctly\n>\n> * We were previosly not comparing the path lengths, which causes a problem\n> (with later updates) when a sparse directory such as \"folder1/0/\" gets\n> compared to a tree name \"folder1\".\n>\n> 8. unpack-trees: rename unpack_nondirectories()\n>\n> * This new commit changes the name to make more sense with its new behavior\n> that could modify a sparse directory entry. The point of the method is in\n> contrast to recursing into trees.\n>\n> 9. unpack-trees: unpack sparse directory entries\n>\n> * THIS is the biggest change from previous versions. There were a few\n> things going on that were tricky to get right, especially with the\n> directory/file conflict (handled in an update in the following, new\n> patch).\n>\n> * The changes to create_ce_entry() regarding alloc_len missed a spot that\n> was critical to getting the length right in the allocated entry.\n>\n> * Use '\\0' over 0 to represent the terminating character.\n>\n> * We don't need a \"sparse_directory\" parameter to unpack_nondirectories()\n> (which was renamed to unpack_single_entry() by the previous new patch)\n> because we can use dirmask to discover if src[0] (or any other value)\n> should be a sparse directory entry.\n>\n> * Similarly, we don't need to call the method twice from unpack_callback().\n>\n> * The 'conflicts' variable is set to match the dirmask in the beginning,\n> but it should depend on whether or not we have a sparse directory entry\n> instead, and if all trees that have the path have a directory.\n>\n> * The implementation of find_cache_entry() uses find_cache_pos() to find an\n> insertion position for a path if it doesn't find an exact match. Before,\n> we subtracted one to find the sparse directory entry, but there could be\n> multiple paths between the sparse directory entry and the insertion\n> point, so we need to walk backwards until we find it. This requires many\n> paths having the same prefix, so hopefully is a rare case. Some of the\n> test data changes were added to cover the need for this logic. This uses\n> a helper method, sparse_dir_matches_path, which is also used by\n> is_sparse_directory_entry.\n>\n> 10. unpack-trees: handle dir/file conflict of sparse entries\n>\n> * This new logic inside twoway_merge handles the special case for dealing\n> with a directory/file conflict during a 'git checkout'. The necessarily\n> data and tests are also added here, though the logic will only take\n> serious effect when we integrate with 'git checkout' later.\n>\n> 11. dir.c: accept a directory as part of cone-mode patterns\n>\n> * The value slash_pos was previously a pointer within a strbuf, but in some\n> cases we add to that strbuf and that could reallocate the pointer, making\n> slash_pos be invalid. The replacement is to have slash_pos be an integer\n> position within the string, so it is consistent even if the string is\n> reallocated for an append.\n>\n> 12. diff-lib: handle index diffs with sparse dirs\n>\n> * As recommended in the previous review, a simple diff_tree_oid() replaces\n> the complicated use of read_tree_at() and traverse_trees() in the\n> previous version.\n>\n> 13. status: skip sparse-checkout percentage with sparse-index\n>\n> (no change)\n>\n> 14. status: use sparse-index throughout\n>\n> (no change)\n>\n> 15. wt-status: expand added sparse directory entries\n>\n> * Duplicate 'git status --porcelain=v2' lines are removed from tests.\n>\n> * The pathspec is initialized using \"= { 0 }\" instead of memset().\n>\n> 16. fsmonitor: integrate with sparse index\n>\n> * An extra test_region is added to ensure that the filesystem monitor hook\n> is still being called, and we are not simply disabling the feature\n> entirely.\n\nThis is SUPER exciting. I've only read the cover letter, but it\nstrongly suggests you've not only handled all my feedback in previous\nrounds, but got things pretty solidly nailed away. I'll try to make\nsome time to go over it all soon.\n"},{"id":"429451","messageId":"CABPp-BE_vGHv7GTESwQbbJ-YSSzkDjapKScEhgYq9QZkVVEg5w@mail.gmail.com","threadId":"55489","inReplyTo":"237ccf4e43dd461250b3e6d609a475c1d675ea86.1624932294.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 09/16] unpack-trees: unpack sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-07T22:25:55Z","receivedAt":"2021-07-07T22:26:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n wrote:\n> - else\n> +\n> + /*\n> + * Check for a sparse-directory entry named \"path/\".\n> + * Due to the input p->path not having a trailing\n> + * slash, the negative 'pos' value overshoots the\n> + * expected position by at least one, hence \"-2\" here.\n\nYou added the qualifier \"at least\" to this comment since v5. I think\nit's slightly misleading because it sounds like -2 is the end of the\nspecial handling of the \"at least\" one overshoot. Perhaps if you\nended with... \"hence '-2' instead of '-1' here, and we also need to\ncheck below if we overshot more than one\".\n\n> + */\n> + pos = -pos - 2;\n> +\n> + if (pos < 0 || pos >= o->src_index->cache_nr)\n> return NULL;\n> +\n> + /*\n> + * We might have multiple entries between 'pos' and\n> + * the actual sparse-directory entry, so start walking\n> + * back until finding it or passing where it would be.\n\nIt might be helpful to add a quick comment about the scenario where\nthis comes up. e.g.\n\n This arises due to lexicographic sort ordering and sparse\ndirectory entries coming with a trailing slash, causing there to be\nmultiple entries between \"subdir\" and \"subdir/\" (such as anything\nbeginning with \"subdir.\" or \"subdir-\"). We are trying to walk back\nfrom \"subdir/\" to \"subdir\" here.\n\n\n> + */\n> + while (pos >= 0) {\n> + ce = o->src_index->cache[pos];\n> +\n> + if (strncmp(ce->name, p->path, p->pathlen))\n> + return NULL;\n> +\n> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n> + sparse_dir_matches_path(ce, info, p))\n> + return ce;\n> +\n> + pos--;\n> + }\n> +\n> + return NULL;\n> }\n"},{"id":"429464","messageId":"CABPp-BEcELkxaVT1nPca7VL=OcWaMQ4XYsbjNBuw6xbUw9x42w@mail.gmail.com","threadId":"55489","inReplyTo":"9f31c691af6780f0ea48bdcb5ff6d56b628f1a81.1624932294.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 10/16] unpack-trees: handle dir/file conflict of sparse entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-07T23:19:20Z","receivedAt":"2021-07-07T23:19:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 25 ++++++++++++++++++++++--\n> unpack-trees.c | 5 ++++-\n> 2 files changed, 27 insertions(+), 3 deletions(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 3f61e5686b5..4e6446e7545 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -95,6 +95,19 @@ test_expect_success 'setup' '\n> git add . &&\n> git commit -m \"rename deep/deeper1/... to folder1/...\" &&\n>\n> + git checkout -b df-conflict base &&\n> + rm -rf folder1 &&\n> + echo content >folder1 &&\n> + git add . &&\n> + git commit -m df &&\n> +\n> + git checkout -b fd-conflict base &&\n> + rm a &&\n> + mkdir a &&\n> + echo content >a/a &&\n> + git add . &&\n> + git commit -m fd &&\n> +\n> git checkout -b deepest base &&\n> echo \"updated deepest\" >deep/deeper1/deepest/a &&\n> git commit -a -m \"update deepest\" &&\n> @@ -325,7 +338,11 @@ test_expect_success 'diff --staged' '\n> test_expect_success 'diff with renames and conflicts' '\n> init_repos &&\n>\n> - for branch in rename-out-to-out rename-out-to-in rename-in-to-out\n> + for branch in rename-out-to-out \\\n> + rename-out-to-in \\\n> + rename-in-to-out \\\n> + df-conflict \\\n> + fd-conflict\n> do\n> test_all_match git checkout rename-base &&\n> test_all_match git checkout $branch -- .&&\n> @@ -338,7 +355,11 @@ test_expect_success 'diff with renames and conflicts' '\n> test_expect_success 'diff with directory/file conflicts' '\n> init_repos &&\n>\n> - for branch in rename-out-to-out rename-out-to-in rename-in-to-out\n> + for branch in rename-out-to-out \\\n> + rename-out-to-in \\\n> + rename-in-to-out \\\n> + df-conflict \\\n> + fd-conflict\n> do\n> git -C full-checkout reset --hard &&\n> test_sparse_match git reset --hard &&\n\nTests look good...\n\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index d141dffbd94..e63b2dcacbc 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -2617,7 +2617,10 @@ int twoway_merge(const struct cache_entry * const *src,\n> same(current, oldtree) && !same(current, newtree)) {\n> /* 20 or 21 */\n> return merged_entry(newtree, current, o);\n> - } else\n> + } else if (current && !oldtree && newtree &&\n> + S_ISSPARSEDIR(current->ce_mode) != S_ISSPARSEDIR(newtree->ce_mode))\n> + return merged_entry(newtree, current, o);\n> + else\n> return reject_merge(current, o);\n> }\n> else if (newtree) {\n\nThis seems wrong to me but I'm having a hard time nailing down a\ntestcase to prove it. The logic looks to me like \"if the old tree as\nnothing in the index at the given path, and the newtree has something,\nand the index had something staged, but the newtree and staged index\nentry disagree on the type of the object, do some weird merged_entry()\nlogic on both types of trees that tends to just take the newer I\nthought but who knows what functions like verify_uptodate(entry) do\nwhen entry is a sparse directory...\".\n\nSo, I'm not so sure about this. Could you explain this a bit more?\n\nHowever, I did find a testcase that aborts with a fatal error...though\nI can't tell if it's even triggering the above logic; I think it isn't\nbecause I have an \"ignoreme\" on both sides of the history. Here's the\ntestcase:\n\n# Make a little test repo\ngit init dumb\ncd dumb\n\n# Setup old commit\ntouch tracked\necho foo >ignoreme\ngit add .\ngit commit -m \"Initial\"\ngit branch orig\n\n# Setup new commit\ngit rm ignoreme\nmkdir ignoreme\ntouch ignoreme/file\ngit add ignoreme/file\ngit commit -m \"whatever\"\n\n# Switch to old commit\ngit checkout orig\n\n# Make index != new (and index != old)\ngit rm ignoreme\nmkdir ignoreme\necho user-data >ignoreme/file\ngit add ignoreme/file\n\n# Sparsify\nGIT_TEST_SPARSE_INDEX=0 # GIT_TEST_SPARSE_INDEX is documented as a boolean;\n # but the traditional boolean value is ignored and it\n # really only cares about set/unset. Confusing.\ngit sparse-checkout init --cone --sparse-index\ngit sparse-checkout set tracked\n\n# Check status and dirs/paths in index\ngit status --porcelain\ntest-tool read-cache --table\ntest-tool read-cache --table --expand\n\n# Run a command that aborts with a fatal error\ngit checkout -m master\n"},{"id":"429513","messageId":"CABPp-BFaWxuBTgztpmso9zCOz3AC62HQuAi5XVAEwsfevYv8gw@mail.gmail.com","threadId":"55489","inReplyTo":"f83aa08ff6b0fd18d6f9f3ce5ee993523a7f1759.1624932294.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 12/16] diff-lib: handle index diffs with sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-08T23:10:04Z","receivedAt":"2021-07-08T23:10:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> While comparing an index to a tree, we may see a sparse directory entry.\n> In this case, we should compare that portion of the tree to the tree\n> represented by that entry. This could include a new tree which needs to\n> be expanded to a full list of added files. It could also include an\n> existing tree, in which case all of the changes inside are important to\n> describe, including the modifications, additions, and deletions. Note\n> that the case where the tree has a path and the index does not remains\n> identical to before: the lack of a cache entry is the same with a sparse\n> index.\n>\n> Use diff_tree_oid() appropriately to compute the diff.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff-lib.c | 16 ++++++++++++++++\n> 1 file changed, 16 insertions(+)\n>\n> diff --git a/diff-lib.c b/diff-lib.c\n> index c2ac9250fe9..3f32f038371 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n> unsigned dirty_submodule = 0;\n> struct index_state *istate = revs->diffopt.repo->index;\n>\n> + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n> + diff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n> + return;\n> + }\n> +\n> /*\n> * New file in the index: it might actually be different in\n> * the working tree.\n> @@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs,\n> unsigned dirty_submodule = 0;\n> struct index_state *istate = revs->diffopt.repo->index;\n>\n> + /*\n> + * If both are sparse directory entries, then expand the\n> + * modifications to the file level.\n> + */\n> + if (old_entry && new_entry &&\n> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n> + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n> + return 0;\n> + }\n> +\n> if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n> &dirty_submodule, &revs->diffopt) < 0) {\n> if (report_missing)\n\nLove the simpler patch.\n\nI'm curious about the case where S_ISSPARSEDIR(old_entry->ce_mode) !=\nS_ISSPARSEDIR(new_entry->ce_mode), though; how is that handled?\n"},{"id":"429515","messageId":"CABPp-BHnjB5H4cJ_mx=3PNVss0Z+d-aEE5Hw4AcuLTC0PC+3qg@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BFaWxuBTgztpmso9zCOz3AC62HQuAi5XVAEwsfevYv8gw@mail.gmail.com","subject":"Re: [PATCH v7 12/16] diff-lib: handle index diffs with sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-08T23:51:55Z","receivedAt":"2021-07-08T23:52:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 8, 2021 at 4:10 PM Elijah Newren wrote:\n>\n> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > From: Derrick Stolee \n> >\n> > While comparing an index to a tree, we may see a sparse directory entry.\n> > In this case, we should compare that portion of the tree to the tree\n> > represented by that entry. This could include a new tree which needs to\n> > be expanded to a full list of added files. It could also include an\n> > existing tree, in which case all of the changes inside are important to\n> > describe, including the modifications, additions, and deletions. Note\n> > that the case where the tree has a path and the index does not remains\n> > identical to before: the lack of a cache entry is the same with a sparse\n> > index.\n> >\n> > Use diff_tree_oid() appropriately to compute the diff.\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > diff-lib.c | 16 ++++++++++++++++\n> > 1 file changed, 16 insertions(+)\n> >\n> > diff --git a/diff-lib.c b/diff-lib.c\n> > index c2ac9250fe9..3f32f038371 100644\n> > --- a/diff-lib.c\n> > +++ b/diff-lib.c\n> > @@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n> > unsigned dirty_submodule = 0;\n> > struct index_state *istate = revs->diffopt.repo->index;\n> >\n> > + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n> > + diff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n> > + return;\n> > + }\n> > +\n> > /*\n> > * New file in the index: it might actually be different in\n> > * the working tree.\n> > @@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs,\n> > unsigned dirty_submodule = 0;\n> > struct index_state *istate = revs->diffopt.repo->index;\n> >\n> > + /*\n> > + * If both are sparse directory entries, then expand the\n> > + * modifications to the file level.\n> > + */\n> > + if (old_entry && new_entry &&\n> > + S_ISSPARSEDIR(old_entry->ce_mode) &&\n> > + S_ISSPARSEDIR(new_entry->ce_mode)) {\n> > + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n> > + return 0;\n> > + }\n> > +\n> > if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n> > &dirty_submodule, &revs->diffopt) < 0) {\n> > if (report_missing)\n>\n> Love the simpler patch.\n>\n> I'm curious about the case where S_ISSPARSEDIR(old_entry->ce_mode) !=\n> S_ISSPARSEDIR(new_entry->ce_mode), though; how is that handled?\n\nDigging a little deeper, it appears that we could add this just before\nyour new if-block:\n\n assert(S_ISSPARSEDIR(old_entry->ce_mode) ==\n S_ISSPARSEDIR(new_entry->ce_mode));\n\nAnd the code still functions, while that also removes some of the\nsurprise factor. I'm guessing that the difference between \"folder1\"\nand \"folder1/\" cause us to never try to directly compare a file to a\ndirectory...but if that's accurate, a comment of some effect might\nhelp make this code be a little clearer and make readers less likely\nto wonder why you need to check that both old and new are sparse\ndirectories.\n"},{"id":"429518","messageId":"CABPp-BHwTAKwFiWQ0-2P=_g+7HLK5FfOAz-uujRjLou1fXT3zw@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BEcELkxaVT1nPca7VL=OcWaMQ4XYsbjNBuw6xbUw9x42w@mail.gmail.com","subject":"Re: [PATCH v7 10/16] unpack-trees: handle dir/file conflict of sparse entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-09T00:58:39Z","receivedAt":"2021-07-09T00:58:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jul 7, 2021 at 4:19 PM Elijah Newren wrote:\n>\n> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > From: Derrick Stolee \n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > t/t1092-sparse-checkout-compatibility.sh | 25 ++++++++++++++++++++++--\n> > unpack-trees.c | 5 ++++-\n> > 2 files changed, 27 insertions(+), 3 deletions(-)\n> >\n> > diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> > index 3f61e5686b5..4e6446e7545 100755\n> > --- a/t/t1092-sparse-checkout-compatibility.sh\n> > +++ b/t/t1092-sparse-checkout-compatibility.sh\n> > @@ -95,6 +95,19 @@ test_expect_success 'setup' '\n> > git add . &&\n> > git commit -m \"rename deep/deeper1/... to folder1/...\" &&\n> >\n> > + git checkout -b df-conflict base &&\n> > + rm -rf folder1 &&\n> > + echo content >folder1 &&\n> > + git add . &&\n> > + git commit -m df &&\n> > +\n> > + git checkout -b fd-conflict base &&\n> > + rm a &&\n> > + mkdir a &&\n> > + echo content >a/a &&\n> > + git add . &&\n> > + git commit -m fd &&\n> > +\n> > git checkout -b deepest base &&\n> > echo \"updated deepest\" >deep/deeper1/deepest/a &&\n> > git commit -a -m \"update deepest\" &&\n> > @@ -325,7 +338,11 @@ test_expect_success 'diff --staged' '\n> > test_expect_success 'diff with renames and conflicts' '\n> > init_repos &&\n> >\n> > - for branch in rename-out-to-out rename-out-to-in rename-in-to-out\n> > + for branch in rename-out-to-out \\\n> > + rename-out-to-in \\\n> > + rename-in-to-out \\\n> > + df-conflict \\\n> > + fd-conflict\n> > do\n> > test_all_match git checkout rename-base &&\n> > test_all_match git checkout $branch -- .&&\n> > @@ -338,7 +355,11 @@ test_expect_success 'diff with renames and conflicts' '\n> > test_expect_success 'diff with directory/file conflicts' '\n> > init_repos &&\n> >\n> > - for branch in rename-out-to-out rename-out-to-in rename-in-to-out\n> > + for branch in rename-out-to-out \\\n> > + rename-out-to-in \\\n> > + rename-in-to-out \\\n> > + df-conflict \\\n> > + fd-conflict\n> > do\n> > git -C full-checkout reset --hard &&\n> > test_sparse_match git reset --hard &&\n>\n> Tests look good...\n>\n> > diff --git a/unpack-trees.c b/unpack-trees.c\n> > index d141dffbd94..e63b2dcacbc 100644\n> > --- a/unpack-trees.c\n> > +++ b/unpack-trees.c\n> > @@ -2617,7 +2617,10 @@ int twoway_merge(const struct cache_entry * const *src,\n> > same(current, oldtree) && !same(current, newtree)) {\n> > /* 20 or 21 */\n> > return merged_entry(newtree, current, o);\n> > - } else\n> > + } else if (current && !oldtree && newtree &&\n> > + S_ISSPARSEDIR(current->ce_mode) != S_ISSPARSEDIR(newtree->ce_mode))\n> > + return merged_entry(newtree, current, o);\n> > + else\n> > return reject_merge(current, o);\n> > }\n> > else if (newtree) {\n\nt1092 still passes if you replace the\n return merged_entry(newtree, current, o);\nline with\n die(\"This line is never hit.\");\n\nIs it possible that you thought you needed this block but further\nrefactoring removed the need? Or that it is only needed by the later\nds/commit-and-checkout-with-sparse-index topic (which I haven't yet\nreviewed, because I was reviewing this topic first)? It seems this\ncode change should either be dropped, or moved out to the relevant\nseries that uses it.\n\n> This seems wrong to me but I'm having a hard time nailing down a\n> testcase to prove it. The logic looks to me like \"if the old tree as\n> nothing in the index at the given path, and the newtree has something,\n> and the index had something staged, but the newtree and staged index\n> entry disagree on the type of the object, do some weird merged_entry()\n> logic on both types of trees that tends to just take the newer I\n> thought but who knows what functions like verify_uptodate(entry) do\n> when entry is a sparse directory...\".\n>\n> So, I'm not so sure about this. Could you explain this a bit more?\n>\n> However, I did find a testcase that aborts with a fatal error...though\n> I can't tell if it's even triggering the above logic; I think it isn't\n> because I have an \"ignoreme\" on both sides of the history. Here's the\n> testcase:\n>\n> # Make a little test repo\n> git init dumb\n> cd dumb\n>\n> # Setup old commit\n> touch tracked\n> echo foo >ignoreme\n> git add .\n> git commit -m \"Initial\"\n> git branch orig\n>\n> # Setup new commit\n> git rm ignoreme\n> mkdir ignoreme\n> touch ignoreme/file\n> git add ignoreme/file\n> git commit -m \"whatever\"\n>\n> # Switch to old commit\n> git checkout orig\n>\n> # Make index != new (and index != old)\n> git rm ignoreme\n> mkdir ignoreme\n> echo user-data >ignoreme/file\n> git add ignoreme/file\n>\n> # Sparsify\n> GIT_TEST_SPARSE_INDEX=0 # GIT_TEST_SPARSE_INDEX is documented as a boolean;\n> # but the traditional boolean value is ignored and it\n> # really only cares about set/unset. Confusing.\n> git sparse-checkout init --cone --sparse-index\n> git sparse-checkout set tracked\n>\n> # Check status and dirs/paths in index\n> git status --porcelain\n> test-tool read-cache --table\n> test-tool read-cache --table --expand\n>\n> # Run a command that aborts with a fatal error\n> git checkout -m master\n\nIt turns out that this testcase I provided still triggers the same\nfatal error if you omit the --sparse-index flag, so it's not a\nsparse-index-specific bug.\n\nSo, perhaps it shouldn't hold up this series, but given that a lot of\nyour correctness verification in t1092 relies on comparisons between\nsparse checkouts and sparse indexes, it might be worth trying to get\nto the root of this.\n"},{"id":"429519","messageId":"CABPp-BE=qw9pO89hggUFbu=eovfL=7Os5BY8DZNd5Z=qU==wFg@mail.gmail.com","threadId":"55489","inReplyTo":"717a3f49f97753149d5c435a83b3f1773dd4b1bb.1624932294.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 15/16] wt-status: expand added sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-09T01:03:54Z","receivedAt":"2021-07-09T01:04:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> It is difficult, but possible, to get into a state where we intend to\n> add a directory that is outside of the sparse-checkout definition. Add a\n> test to t1092-sparse-checkout-compatibility.sh that demonstrates this\n> using a combination of 'git reset --mixed' and 'git checkout --orphan'.\n>\n> This test failed before because the output of 'git status\n> --porcelain=v2' would not match on the lines for folder1/:\n>\n> * The sparse-checkout repo (with a full index) would output each path\n> name that is intended to be added.\n>\n> * The sparse-index repo would only output that \"folder1/\" is staged for\n> addition.\n>\n> The status should report the full list of files to be added, and so this\n> sparse-directory entry should be expanded to a full list when reaching\n> it inside the wt_status_collect_changes_initial() method. Use\n> read_tree_at() to assist.\n>\n> Somehow, this loop over the cache entries was not guarded by\n> ensure_full_index() as intended.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++\n> wt-status.c | 51 ++++++++++++++++++++++++\n> 2 files changed, 84 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index fed0440bafe..df217a2d10b 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -545,4 +545,37 @@ test_expect_success 'sparse-index is not expanded' '\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> +test_expect_success 'reset mixed and checkout orphan' '\n> + init_repos &&\n> +\n> + test_all_match git checkout rename-out-to-in &&\n> +\n> + # Sparse checkouts do not agree with full checkouts about\n> + # how to report a directory/file conflict during a reset.\n> + # This command would fail with test_all_match because the\n> + # full checkout reports \"T folder1/0/1\" while a sparse\n> + # checkout reports \"D folder1/0/1\". This matches because\n> + # the sparse checkouts skip \"adding\" the other side of\n> + # the conflict.\n\nThe same issue I highlighted last time is still present. If you\ninsert an \"exit 1\" right here, then run\n ./t1092-sparse-checkout-compatibility.sh --ver --imm -x\nuntil it stops, then\n cd t/trash directory.t1092-sparse-checkout-compatibility/sparse-checkout\n git ls-files -t | grep folder # Note the files that are sparse\n git reset --mixed HEAD~1\n git ls-files -t | grep folder # Note the files that are sparse --\nthere are some that aren't that should be\n git sparse-checkout reapply\n git ls-files -t | grep folder # Note the files that are sparse\n\nGranted, this is a bug with sparse-checkout without sparse-index, so\nnot something new to your series. But since you are using comparisons\nbetween regular sparse-checkouts and sparse-index to verify\ncorrectness, this seems problematic to me.\n\n> + test_sparse_match git reset --mixed HEAD~1 &&\n> + test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + # At this point, sparse-checkouts behave differently\n> + # from the full-checkout.\n> + test_sparse_match git checkout --orphan new-branch &&\n> + test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'add everything with deep new file' '\n> + init_repos &&\n> +\n> + run_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n> +\n> + run_on_all touch deep/deeper1/x &&\n> + test_all_match git add . &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> test_done\n> diff --git a/wt-status.c b/wt-status.c\n> index 96db3e74962..0317baef87e 100644\n> --- a/wt-status.c\n> +++ b/wt-status.c\n> @@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n> clear_pathspec(&rev.prune_data);\n> }\n>\n> +static int add_file_to_list(const struct object_id *oid,\n> + struct strbuf *base, const char *path,\n> + unsigned int mode, void *context)\n> +{\n> + struct string_list_item *it;\n> + struct wt_status_change_data *d;\n> + struct wt_status *s = context;\n> + struct strbuf full_name = STRBUF_INIT;\n> +\n> + if (S_ISDIR(mode))\n> + return READ_TREE_RECURSIVE;\n> +\n> + strbuf_add(&full_name, base->buf, base->len);\n> + strbuf_addstr(&full_name, path);\n> + it = string_list_insert(&s->change, full_name.buf);\n> + d = it->util;\n> + if (!d) {\n> + CALLOC_ARRAY(d, 1);\n> + it->util = d;\n> + }\n> +\n> + d->index_status = DIFF_STATUS_ADDED;\n> + /* Leave {mode,oid}_head zero for adds. */\n> + d->mode_index = mode;\n> + oidcpy(&d->oid_index, oid);\n> + s->committable = 1;\n> + strbuf_release(&full_name);\n> + return 0;\n> +}\n> +\n> static void wt_status_collect_changes_initial(struct wt_status *s)\n> {\n> struct index_state *istate = s->repo->index;\n> @@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n> continue;\n> if (ce_intent_to_add(ce))\n> continue;\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + /*\n> + * This is a sparse directory entry, so we want to collect all\n> + * of the added files within the tree. This requires recursively\n> + * expanding the trees to find the elements that are new in this\n> + * tree and marking them with DIFF_STATUS_ADDED.\n> + */\n> + struct strbuf base = STRBUF_INIT;\n> + struct pathspec ps = { 0 };\n> + struct tree *tree = lookup_tree(istate->repo, &ce->oid);\n> +\n> + ps.recursive = 1;\n> + ps.has_wildcard = 1;\n> + ps.max_depth = -1;\n> +\n> + strbuf_add(&base, ce->name, ce->ce_namelen);\n> + read_tree_at(istate->repo, tree, &base, &ps,\n> + add_file_to_list, s);\n> + continue;\n> + }\n> +\n> it = string_list_insert(&s->change, ce->name);\n> d = it->util;\n> if (!d) {\n> --\n> gitgitgadget\n"},{"id":"429520","messageId":"CABPp-BF2BLQm6864m+esquJ7og3fWHvTZeX_R-y4pnx9ApZ6XA@mail.gmail.com","threadId":"55489","inReplyTo":"CABPp-BHqoJjF9f6NKZ8jjQdj1bgUNgrwek5jcnGTRk2m-m8dBg@mail.gmail.com","subject":"Re: [PATCH v7 00/16] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-09T01:16:46Z","receivedAt":"2021-07-09T01:17:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jun 30, 2021 at 7:32 AM Elijah Newren wrote:\n>\n> On Mon, Jun 28, 2021 at 7:04 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> > status' very fast when a sparse-index is enabled on a repository with\n> > cone-mode sparse-checkout (and a small populated set).\n> >\n...\n> > Because the range-diff is a big difficult to read this time, I'll break the\n> > changes down on a patch-by-patch basis.\n\nThanks for doing this; it was helpful.\n\n> This is SUPER exciting. I've only read the cover letter, but it\n> strongly suggests you've not only handled all my feedback in previous\n> rounds, but got things pretty solidly nailed away. I'll try to make\n> some time to go over it all soon.\n\nYou have indeed addressed nearly all my feedback in previous rounds,\nand I found few problems with all the new code in this round.\nOverall, this round is looking really good, though there are a couple\nthings I called out in comments on individual patches that I'll\nsummarize here:\n\nPatch 9: a few minor suggestions for improving comments\n\nPatch 10: the new code is never triggered and probably should either\nbe dropped or made part of a later series if the later series needs\nit. Also, although it doesn't necessarily need to hold up this\nseries, I found a bug affecting sparse-checkouts with or without\nsparse-index while trying to understand this patch.\n\nPatch 12: the code was slightly confusing to me, but I found that\nthere seems to be an invariant it is based upon. Adding an assert\nwith a comment or just a comment about this invariant might help make\nthe code more readable.\n\nPatch 15: since the new tests in t1092 are written to compare\nsparse-checkout and sparse-index, it seems we should investigate a bug\nwhere the testsuite commands we invoke are giving incorrect behavior\nin both sparse-checkout and sparse-index.\n"},{"id":"429749","messageId":"84b045bc-2c00-7516-5796-7812af77c8dc@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHwTAKwFiWQ0-2P=_g+7HLK5FfOAz-uujRjLou1fXT3zw@mail.gmail.com","subject":"Re: [PATCH v7 10/16] unpack-trees: handle dir/file conflict of sparse entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-12T13:46:08Z","receivedAt":"2021-07-12T13:46:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2021 8:58 PM, Elijah Newren wrote:\n> On Wed, Jul 7, 2021 at 4:19 PM Elijah Newren wrote:\n>>\n>> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n>> wrote:\n...\n>>> + } else if (current && !oldtree && newtree &&\n>>> + S_ISSPARSEDIR(current->ce_mode) != S_ISSPARSEDIR(newtree->ce_mode))\n>>> + return merged_entry(newtree, current, o);\n>>> + else\n>>> return reject_merge(current, o);\n>>> }\n>>> else if (newtree) {\n> \n> t1092 still passes if you replace the\n> return merged_entry(newtree, current, o);\n> line with\n> die(\"This line is never hit.\");\n> \n> Is it possible that you thought you needed this block but further\n> refactoring removed the need? Or that it is only needed by the later\n> ds/commit-and-checkout-with-sparse-index topic (which I haven't yet\n> reviewed, because I was reviewing this topic first)? It seems this\n> code change should either be dropped, or moved out to the relevant\n> series that uses it.\n\nI have been working with the whole stack of patches (including the\nones that update 'git add') and trying to make the necessary changes\nin this series, especially when updating the test data shape or\nmodifying methods that were necessary for 'git status'.\n\nIt is likely I was just overcautious thinking this was necessary so\nearly. It might also have been necessary due to some strange case\nthat is only exposed in a Scalar functional test. I'll try moving it\nto the next series and double check those tests.\n\n>> This seems wrong to me but I'm having a hard time nailing down a\n>> testcase to prove it. The logic looks to me like \"if the old tree as\n>> nothing in the index at the given path, and the newtree has something,\n>> and the index had something staged, but the newtree and staged index\n>> entry disagree on the type of the object, do some weird merged_entry()\n>> logic on both types of trees that tends to just take the newer I\n>> thought but who knows what functions like verify_uptodate(entry) do\n>> when entry is a sparse directory...\".\n>>\n>> So, I'm not so sure about this. Could you explain this a bit more?\n\nThe most important point is that 'current' and 'newtree' are both\npresent but have different types (blob and tree) and the tree is\nnecessarily at the edge of the sparse-checkout cone. In the cases\nwhere I was able to trigger this logic in the debugger, 'oldtree'\nwas NULL, so I added as a condition to be extra cautious around\nunexpected initialization. It is possible that the '!oldtree'\ncondition is unnecessary. I will investigate.\n\nAs for verify_uptodate(), it ignores the sparse directory unless\no->skip_sparse_checkout is set. I wonder if it is possible to\nhave that set and hit this case.\n\n>> However, I did find a testcase that aborts with a fatal error...though\n>> I can't tell if it's even triggering the above logic; I think it isn't\n>> because I have an \"ignoreme\" on both sides of the history. Here's the\n>> testcase:\n>>\n...\n>> # Run a command that aborts with a fatal error\n>> git checkout -m master\n\nThank you for finding an interesting test!\n\n> It turns out that this testcase I provided still triggers the same\n> fatal error if you omit the --sparse-index flag, so it's not a\n> sparse-index-specific bug.\n> \n> So, perhaps it shouldn't hold up this series, but given that a lot of\n> your correctness verification in t1092 relies on comparisons between\n> sparse checkouts and sparse indexes, it might be worth trying to get\n> to the root of this.\n \nI have some upcoming thoughts on changing how sparse-checkout works\nfor other complicated cases, so I will add this to that pile.\n\nThanks,\n-Stolee\n"},{"id":"429752","messageId":"bfc422fd-7040-fed4-3284-8ea66b33bcd1@gmail.com","threadId":"55489","inReplyTo":"CABPp-BHnjB5H4cJ_mx=3PNVss0Z+d-aEE5Hw4AcuLTC0PC+3qg@mail.gmail.com","subject":"Re: [PATCH v7 12/16] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-12T13:52:41Z","receivedAt":"2021-07-12T13:52:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2021 7:51 PM, Elijah Newren wrote:\n> On Thu, Jul 8, 2021 at 4:10 PM Elijah Newren wrote:\n>>\n>> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n...\n>>> + /*\n>>> + * If both are sparse directory entries, then expand the\n>>> + * modifications to the file level.\n>>> + */\n>>> + if (old_entry && new_entry &&\n>>> + S_ISSPARSEDIR(old_entry->ce_mode) &&\n>>> + S_ISSPARSEDIR(new_entry->ce_mode)) {\n>>> + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n>>> + return 0;\n>>> + }\n>>> +\n>>> if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n>>> &dirty_submodule, &revs->diffopt) < 0) {\n>>> if (report_missing)\n>>\n>> Love the simpler patch.\n>>\n>> I'm curious about the case where S_ISSPARSEDIR(old_entry->ce_mode) !=\n>> S_ISSPARSEDIR(new_entry->ce_mode), though; how is that handled?\n> \n> Digging a little deeper, it appears that we could add this just before\n> your new if-block:\n> \n> assert(S_ISSPARSEDIR(old_entry->ce_mode) ==\n> S_ISSPARSEDIR(new_entry->ce_mode));\n> \n> And the code still functions, while that also removes some of the\n> surprise factor. I'm guessing that the difference between \"folder1\"\n> and \"folder1/\" cause us to never try to directly compare a file to a\n> directory...but if that's accurate, a comment of some effect might\n> help make this code be a little clearer and make readers less likely\n> to wonder why you need to check that both old and new are sparse\n> directories.\n\nI was surprised that this worked, because my patch conditioned on\nold_entry and new_entry being non-NULL. But of course show_modified()\nrequires them to be non-NULL. That can be further simplified.\n\nAdding the assert helps demonstrate this expectation, but also I will\nexpand upon the comment.\n\nThanks,\n-Stolee\n"},{"id":"429753","messageId":"3cf96b5f-3a8d-c00c-5d9e-c7095c058059@gmail.com","threadId":"55489","inReplyTo":"CABPp-BE=qw9pO89hggUFbu=eovfL=7Os5BY8DZNd5Z=qU==wFg@mail.gmail.com","subject":"Re: [PATCH v7 15/16] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-12T13:56:25Z","receivedAt":"2021-07-12T13:56:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2021 9:03 PM, Elijah Newren wrote:\n> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n> wrote:\n...\n>> +test_expect_success 'reset mixed and checkout orphan' '\n>> + init_repos &&\n>> +\n>> + test_all_match git checkout rename-out-to-in &&\n>> +\n>> + # Sparse checkouts do not agree with full checkouts about\n>> + # how to report a directory/file conflict during a reset.\n>> + # This command would fail with test_all_match because the\n>> + # full checkout reports \"T folder1/0/1\" while a sparse\n>> + # checkout reports \"D folder1/0/1\". This matches because\n>> + # the sparse checkouts skip \"adding\" the other side of\n>> + # the conflict.\n> \n> The same issue I highlighted last time is still present. If you\n> insert an \"exit 1\" right here, then run\n> ./t1092-sparse-checkout-compatibility.sh --ver --imm -x\n> until it stops, then\n> cd t/trash directory.t1092-sparse-checkout-compatibility/sparse-checkout\n> git ls-files -t | grep folder # Note the files that are sparse\n> git reset --mixed HEAD~1\n> git ls-files -t | grep folder # Note the files that are sparse --\n> there are some that aren't that should be\n> git sparse-checkout reapply\n> git ls-files -t | grep folder # Note the files that are sparse\n> \n> Granted, this is a bug with sparse-checkout without sparse-index, so\n> not something new to your series. But since you are using comparisons\n> between regular sparse-checkouts and sparse-index to verify\n> correctness, this seems problematic to me.\n\nI'll add it to the pile, but I want to continue having this series\nfocus on making the sparse-index work quickly without a change in\nbehavior from a normal index. Changing the behavior of the sparse-\ncheckout feature should be a separate series.\n\nThanks,\n-Stolee\n"},{"id":"429754","messageId":"6cdb4ba8-fe85-2a65-8e6c-74d3b5c3ef96@gmail.com","threadId":"55489","inReplyTo":"CABPp-BF2BLQm6864m+esquJ7og3fWHvTZeX_R-y4pnx9ApZ6XA@mail.gmail.com","subject":"Re: [PATCH v7 00/16] Sparse-index: integrate with status","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-12T14:46:30Z","receivedAt":"2021-07-12T14:46:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2021 9:16 PM, Elijah Newren wrote:\n> On Wed, Jun 30, 2021 at 7:32 AM Elijah Newren wrote:\n>>\n>> On Mon, Jun 28, 2021 at 7:04 PM Derrick Stolee via GitGitGadget\n>> wrote:\n>>>\n>>> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n>>> status' very fast when a sparse-index is enabled on a repository with\n>>> cone-mode sparse-checkout (and a small populated set).\n>>>\n> ...\n>>> Because the range-diff is a big difficult to read this time, I'll break the\n>>> changes down on a patch-by-patch basis.\n> \n> Thanks for doing this; it was helpful.\n> \n>> This is SUPER exciting. I've only read the cover letter, but it\n>> strongly suggests you've not only handled all my feedback in previous\n>> rounds, but got things pretty solidly nailed away. I'll try to make\n>> some time to go over it all soon.\n> \n> You have indeed addressed nearly all my feedback in previous rounds,\n> and I found few problems with all the new code in this round.\n> Overall, this round is looking really good, though there are a couple\n> things I called out in comments on individual patches that I'll\n> summarize here:\n\nThis summary is nice. I will try to do a similar thing myself\nin the future.\n\n> Patch 9: a few minor suggestions for improving comments\n\nDone.\n\n> Patch 10: the new code is never triggered and probably should either\n> be dropped or made part of a later series if the later series needs\n> it. Also, although it doesn't necessarily need to hold up this\n> series, I found a bug affecting sparse-checkouts with or without\n> sparse-index while trying to understand this patch.\n\nI checked and this patch is not necessary until the next series,\nso I will move it to that one.\n\n> Patch 12: the code was slightly confusing to me, but I found that\n> there seems to be an invariant it is based upon. Adding an assert\n> with a comment or just a comment about this invariant might help make\n> the code more readable.\n\nThe assert() plus a better comment makes this patch cleaner.\n\n> Patch 15: since the new tests in t1092 are written to compare\n> sparse-checkout and sparse-index, it seems we should investigate a bug\n> where the testsuite commands we invoke are giving incorrect behavior\n> in both sparse-checkout and sparse-index.\n\nI have made a note in an internal tracker to follow-up on this\nalong with some other behavior things that currently exist in\nsparse-checkout as they should be pursued in a separate series.\n\nI'm not ignoring your comments, but I would like to keep these\npatches that change the underlying data structure from being\nconflated with behavior changes. The fact that I am discovering\n(and documenting in tests) the changes is only highlighting that\nthey already exist.\n\nThanks,\n-Stolee\n"},{"id":"429790","messageId":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v7.git.1624932293.gitgitgadget@gmail.com","subject":"[PATCH v8 00/15] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:41Z","receivedAt":"2021-07-12T17:56:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdate in V8\n============\n\n * The directory/file conflict patch is removed and delayed to the next\n series where it will be required. (It will also be improved in that\n series.)\n\n * Some comments have been improved, including a new assert() that helps\n document the situation.\n\nDerrick Stolee (15):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: replace incorrect 'echo' with 'cat'\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: rename unpack_nondirectories()\n unpack-trees: unpack sparse directory entries\n dir.c: accept a directory as part of cone-mode patterns\n diff-lib: handle index diffs with sparse dirs\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n\n builtin/commit.c | 3 +\n diff-lib.c | 19 +++\n dir.c | 24 +++-\n read-cache.c | 10 +-\n sparse-index.c | 27 +++-\n t/t1092-sparse-checkout-compatibility.sh | 158 ++++++++++++++++++++++-\n t/t7519-status-fsmonitor.sh | 49 +++++++\n unpack-trees.c | 142 +++++++++++++++++---\n wt-status.c | 65 +++++++++-\n wt-status.h | 1 +\n 10 files changed, 464 insertions(+), 34 deletions(-)\n\n\nbase-commit: d486ca60a51c9cb1fe068803c3f540724e95e83a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v8\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v7:\n\n 1: 2a4a7256304 = 1: 1815c148e8c sparse-index: skip indexes with unmerged entries\n 2: f5bae86014d = 2: 7bcde075d8d sparse-index: include EXTENDED flag when expanding\n 3: d965669c766 = 3: 05981e30b97 t1092: replace incorrect 'echo' with 'cat'\n 4: e10fa11cfdb ! 4: d38b66e9ee4 t1092: expand repository data shape\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'diff --staged' '\n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n - \t\ttest_all_match git checkout $branch -- .&&\n + \t\ttest_all_match git checkout $branch -- . &&\n +\t\ttest_all_match git status --porcelain=v2 &&\n +\t\ttest_all_match git diff --staged --no-renames &&\n +\t\ttest_all_match git diff --staged --find-renames || return 1\n 5: e94ffa07d46 = 5: 95ddd3abe4e t1092: add tests for status/add and sparse files\n 6: a8dda933567 = 6: b182b456613 unpack-trees: preserve cache_bottom\n 7: e52166f6e4c = 7: 988ddce4d45 unpack-trees: compare sparse directories correctly\n 8: d04b62381b8 = 8: d67ad048b08 unpack-trees: rename unpack_nondirectories()\n 9: 237ccf4e43d ! 9: c0b0b58584c unpack-trees: unpack sparse directory entries\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n +\t * Check for a sparse-directory entry named \"path/\".\n +\t * Due to the input p->path not having a trailing\n +\t * slash, the negative 'pos' value overshoots the\n -+\t * expected position by at least one, hence \"-2\" here.\n ++\t * expected position, hence \"-2\" instead of \"-1\".\n +\t */\n +\tpos = -pos - 2;\n +\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n \t\treturn NULL;\n +\n +\t/*\n -+\t * We might have multiple entries between 'pos' and\n -+\t * the actual sparse-directory entry, so start walking\n -+\t * back until finding it or passing where it would be.\n ++\t * Due to lexicographic sorting and sparse directory\n ++\t * entried ending with a trailing slash, our path as a\n ++\t * sparse directory (e.g \"subdir/\") and\tour path as a\n ++\t * file (e.g. \"subdir\") might be separated by other\n ++\t * paths (e.g. \"subdir-\").\n +\t */\n +\twhile (pos >= 0) {\n +\t\tce = o->src_index->cache[pos];\n 10: 9f31c691af6 < -: ----------- unpack-trees: handle dir/file conflict of sparse entries\n 11: 2a43287c47e = 10: 76c7528f78f dir.c: accept a directory as part of cone-mode patterns\n 12: f83aa08ff6b ! 11: d875a7f8585 diff-lib: handle index diffs with sparse dirs\n @@ diff-lib.c: static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n ++\tassert(S_ISSPARSEDIR(old_entry->ce_mode) ==\n ++\t S_ISSPARSEDIR(new_entry->ce_mode));\n ++\n +\t/*\n +\t * If both are sparse directory entries, then expand the\n -+\t * modifications to the file level.\n ++\t * modifications to the file level. If only one was a sparse\n ++\t * directory, then they appear as an add and delete instead of\n ++\t * a modification.\n +\t */\n -+\tif (old_entry && new_entry &&\n -+\t S_ISSPARSEDIR(old_entry->ce_mode) &&\n -+\t S_ISSPARSEDIR(new_entry->ce_mode)) {\n ++\tif (S_ISSPARSEDIR(new_entry->ce_mode)) {\n +\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n +\t\treturn 0;\n +\t}\n 13: 35063ffb8ed = 12: 2b72cc2d985 status: skip sparse-checkout percentage with sparse-index\n 14: b4033a9bf36 = 13: 1c1feef3733 status: use sparse-index throughout\n 15: 717a3f49f97 = 14: dada1b91bdc wt-status: expand added sparse directory entries\n 16: 1d744848ee6 = 15: bdc771cf373 fsmonitor: integrate with sparse index\n\n-- \ngitgitgadget\n"},{"id":"429791","messageId":"1815c148e8c36d86de2ac5f27986f5f8d850f6c4.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 01/15] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:42Z","receivedAt":"2021-07-12T17:56:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex affc4048f27..2c695930275 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d028b73eba1..b8617ceef71 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"429792","messageId":"7bcde075d8db1f89c21213fb92f1d280b8adc032.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 02/15] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:43Z","receivedAt":"2021-07-12T17:56:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\nflag is not included. The CE_EXTENDED flag would exist if we loaded a\nfull index from disk with these entries marked with CE_SKIP_WORKTREE, so\nwe can add the flag here to be consistent. This allows us to directly\ncompare the flags present in cache entries when testing the sparse-index\nfeature, but has no significance to its correctness in the user-facing\nfunctionality.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2c695930275..ef53bd2198b 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"429793","messageId":"05981e30b97a3e668b3a9675508135d42f15450f.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 03/15] t1092: replace incorrect 'echo' with 'cat'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:44Z","receivedAt":"2021-07-12T17:56:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis fixes the test data shape to be as expected, allowing rename\ndetection to work properly now that the 'larger-content' file actually\nhas meaningful lines.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b8617ceef71..87f1014a1c9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -40,7 +40,7 @@ test_expect_success 'setup' '\n \t\tdone &&\n \n \t\tgit checkout -b rename-base base &&\n-\t\techo >folder1/larger-content <<-\\EOF &&\n+\t\tcat >folder1/larger-content <<-\\EOF &&\n \t\tmatching\n \t\tlines\n \t\thelp\n-- \ngitgitgadget\n\n"},{"id":"429794","messageId":"d38b66e9ee4607b22c0473f23243a979e4880ab5.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 04/15] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:45Z","receivedAt":"2021-07-12T17:56:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\n* Add filenames adjacent to a sparse directory entry that sort before\n and after the trailing slash.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 42 ++++++++++++++++++++++--\n 1 file changed, 40 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 87f1014a1c9..0e71a623619 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,23 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n+\t\t>folder1- &&\n+\t\t>folder1.x &&\n+\t\t>folder10 &&\n+\t\tcp -r deep/deeper1/0 folder1 &&\n+\t\tcp -r deep/deeper1/0 folder2 &&\n+\t\techo >>folder1/0/0/0 &&\n+\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n@@ -56,11 +69,17 @@ test_expect_success 'setup' '\n \t\tmv folder1/a folder2/b &&\n \t\tmv folder1/larger-content folder2/edited-content &&\n \t\techo >>folder2/edited-content &&\n+\t\techo >>folder2/0/1 &&\n+\t\techo stuff >>deep/deeper1/a &&\n \t\tgit add . &&\n \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n \n \t\tgit checkout -b rename-out-to-in rename-base &&\n \t\tmv folder1/a deep/deeper1/b &&\n+\t\techo more stuff >>deep/deeper1/a &&\n+\t\trm folder2/0/1 &&\n+\t\tmkdir folder2/0/1 &&\n+\t\techo >>folder2/0/1/1 &&\n \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n \t\techo >>deep/deeper1/edited-content &&\n \t\tgit add . &&\n@@ -68,6 +87,9 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b rename-in-to-out rename-base &&\n \t\tmv deep/deeper1/a folder1/b &&\n+\t\techo >>folder2/0/1 &&\n+\t\trm -rf folder1/0/0 &&\n+\t\techo >>folder1/0/0 &&\n \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n \t\techo >>folder1/edited-content &&\n \t\tgit add . &&\n@@ -262,13 +284,29 @@ test_expect_success 'diff --staged' '\n \ttest_all_match git diff --staged\n '\n \n-test_expect_success 'diff with renames' '\n+test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_all_match git diff --staged --no-renames &&\n+\t\ttest_all_match git diff --staged --find-renames || return 1\n+\tdone\n+'\n+\n+test_expect_success 'diff with directory/file conflicts' '\n+\tinit_repos &&\n+\n+\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tdo\n+\t\tgit -C full-checkout reset --hard &&\n+\t\ttest_sparse_match git reset --hard &&\n+\t\ttest_all_match git checkout $branch &&\n+\t\ttest_all_match git checkout rename-base -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n \t\ttest_all_match git diff --staged --no-renames &&\n \t\ttest_all_match git diff --staged --find-renames || return 1\n \tdone\n-- \ngitgitgadget\n\n"},{"id":"429795","messageId":"95ddd3abe4e5c85e4c911ba6c2d4bd86099085b4.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 05/15] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:46Z","receivedAt":"2021-07-12T17:56:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0e71a623619..2269f44e033 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -254,6 +254,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"429796","messageId":"988ddce4d45d994488ca074039a9b8a8ed2ca285.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 07/15] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:48Z","receivedAt":"2021-07-12T17:56:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry. The final check is a\nlength comparison between the strings.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 87c1ed204c8..b113cc750f2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1014,6 +1016,16 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1)\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"429797","messageId":"b182b4566135c0cc9aa59f244d63d5e296d4297d.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 06/15] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:47Z","receivedAt":"2021-07-12T17:56:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f88a69f8e71..87c1ed204c8 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"429798","messageId":"d67ad048b08aa168495fa89e007d905a79c1c5db.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 08/15] unpack-trees: rename unpack_nondirectories()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:49Z","receivedAt":"2021-07-12T17:56:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn the next change, we will use this method to unpack a sparse directory\nentry, so change the name to unpack_single_entry() so these entries\napply. The new name reflects that we will not recurse into trees in\norder to resolve the conflicts.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b113cc750f2..d26386ce8b2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -804,7 +804,7 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,\n \t\tBUG(\"We need cache-tree to do this optimization\");\n \n \t/*\n-\t * Do what unpack_callback() and unpack_nondirectories() normally\n+\t * Do what unpack_callback() and unpack_single_entry() normally\n \t * do. But we walk all paths in an iterative loop instead.\n \t *\n \t * D/F conflicts and higher stage entries are not a concern\n@@ -1075,11 +1075,11 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n * without actually calling it. If you change the logic here you may need to\n * check and change there as well.\n */\n-static int unpack_nondirectories(int n, unsigned long mask,\n-\t\t\t\t unsigned long dirmask,\n-\t\t\t\t struct cache_entry **src,\n-\t\t\t\t const struct name_entry *names,\n-\t\t\t\t const struct traverse_info *info)\n+static int unpack_single_entry(int n, unsigned long mask,\n+\t\t\t unsigned long dirmask,\n+\t\t\t struct cache_entry **src,\n+\t\t\t const struct name_entry *names,\n+\t\t\t const struct traverse_info *info)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n@@ -1322,7 +1322,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_single_entry(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n-- \ngitgitgadget\n\n"},{"id":"429799","messageId":"c0b0b58584c3abbd2a3ed29e5e7fb3df29b7db25.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 09/15] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:50Z","receivedAt":"2021-07-12T17:56:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring unpack_callback(), index entries are compared against tree\nentries. These are matched according to names and types. One goal is to\ndecide if we should recurse into subtrees or simply operate on one index\nentry.\n\nIn the case of a sparse-directory entry, we do not want to recurse into\nthat subtree and instead simply compare the trees. In some cases, we\nmight want to perform a merge operation on the entry, such as during\n'git checkout ' which wants to replace a sparse tree entry with\nthe tree for that path at the target commit. We extend the logic within\nunpack_single_entry() to create a sparse-directory entry in this case,\nand then that is sent to call_unpack_fn().\n\nThere are some subtleties in this process. For instance, we need to\nupdate find_cache_entry() to allow finding a sparse-directory entry that\nexactly matches a given path. Use the new helper method\nsparse_dir_matches_path() for this. We also need to ignore conflict\nmarkers in the case that the entries correspond to directories and we\nalready have a sparse directory entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 107 +++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 99 insertions(+), 8 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d26386ce8b2..62ccd5a0ff6 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1052,13 +1052,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tconst struct name_entry *n,\n \tint stage,\n \tstruct index_state *istate,\n-\tint is_transient)\n+\tint is_transient,\n+\tint is_sparse_directory)\n {\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n+\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len, NULL) :\n-\t\tmake_empty_cache_entry(istate, len);\n+\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n+\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -1067,6 +1069,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \t/* len+1 because the cache_entry allocates space for NUL */\n \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n+\tif (is_sparse_directory) {\n+\t\tce->name[len] = '/';\n+\t\tce->name[len + 1] = '\\0';\n+\t\tce->ce_namelen++;\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\t}\n+\n \treturn ce;\n }\n \n@@ -1085,10 +1094,17 @@ static int unpack_single_entry(int n, unsigned long mask,\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\t/*\n+\t * When we have a sparse directory entry for src[0],\n+\t * then this isn't necessarily a directory-file conflict.\n+\t */\n+\tif (mask == dirmask && src[0] &&\n+\t S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\tconflicts = 0;\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n@@ -1118,7 +1134,9 @@ static int unpack_single_entry(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n+\t\t\t\t\t\t &o->result, o->merge,\n+\t\t\t\t\t\t bit & dirmask);\n \t}\n \n \tif (o->merge) {\n@@ -1222,16 +1240,71 @@ static int find_cache_pos(struct traverse_info *info,\n \treturn -1;\n }\n \n+/*\n+ * Given a sparse directory entry 'ce', compare ce->name to\n+ * info->name + '/' + p->path + '/' if info->name is non-empty.\n+ * Compare ce->name to p->path + '/' otherwise. Note that\n+ * ce->name must end in a trailing '/' because it is a sparse\n+ * directory entry.\n+ */\n+static int sparse_dir_matches_path(const struct cache_entry *ce,\n+\t\t\t\t struct traverse_info *info,\n+\t\t\t\t const struct name_entry *p)\n+{\n+\tassert(S_ISSPARSEDIR(ce->ce_mode));\n+\tassert(ce->name[ce->ce_namelen - 1] == '/');\n+\n+\tif (info->namelen)\n+\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n+\t\t ce->name[info->namelen] == '/' &&\n+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\treturn ce->ce_namelen == p->pathlen + 1 &&\n+\t !strncmp(ce->name, p->path, p->pathlen);\n+}\n+\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n+\tstruct cache_entry *ce;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n-\telse\n+\n+\t/*\n+\t * Check for a sparse-directory entry named \"path/\".\n+\t * Due to the input p->path not having a trailing\n+\t * slash, the negative 'pos' value overshoots the\n+\t * expected position, hence \"-2\" instead of \"-1\".\n+\t */\n+\tpos = -pos - 2;\n+\n+\tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n+\n+\t/*\n+\t * Due to lexicographic sorting and sparse directory\n+\t * entried ending with a trailing slash, our path as a\n+\t * sparse directory (e.g \"subdir/\") and\tour path as a\n+\t * file (e.g. \"subdir\") might be separated by other\n+\t * paths (e.g. \"subdir-\").\n+\t */\n+\twhile (pos >= 0) {\n+\t\tce = o->src_index->cache[pos];\n+\n+\t\tif (strncmp(ce->name, p->path, p->pathlen))\n+\t\t\treturn NULL;\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t sparse_dir_matches_path(ce, info, p))\n+\t\t\treturn ce;\n+\n+\t\tpos--;\n+\t}\n+\n+\treturn NULL;\n }\n \n static void debug_path(struct traverse_info *info)\n@@ -1266,6 +1339,21 @@ static void debug_unpack_callback(int n,\n \t\tdebug_name_entry(i, names + i);\n }\n \n+/*\n+ * Returns true if and only if the given cache_entry is a\n+ * sparse-directory entry that matches the given name_entry\n+ * from the tree walk at the given traverse_info.\n+ */\n+static int is_sparse_directory_entry(struct cache_entry *ce,\n+\t\t\t\t struct name_entry *name,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n+\treturn sparse_dir_matches_path(ce, info, name);\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -1352,9 +1440,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n-\t\t\t\t\t names, info) < 0)\n+\t\tif (!is_sparse_directory_entry(src[0], names, info) &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n+\t\t}\n+\n \t\treturn mask;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"429800","messageId":"76c7528f78f0be2ed0a9fc0428f715b36fd451ba.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 10/15] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:51Z","receivedAt":"2021-07-12T17:56:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 24 +++++++++++++++++++-----\n 1 file changed, 19 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e0..0c5264b3b20 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1376,7 +1376,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tstruct path_pattern *pattern;\n \tstruct strbuf parent_pathname = STRBUF_INIT;\n \tint result = NOT_MATCHED;\n-\tconst char *slash_pos;\n+\tsize_t slash_pos;\n \n \tif (!pl->use_cone_patterns) {\n \t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n@@ -1397,21 +1397,35 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/') {\n+\t\tslash_pos = parent_pathname.len - 1;\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\t} else {\n+\t\tconst char *slash_ptr = strrchr(parent_pathname.buf, '/');\n+\t\tslash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0;\n+\t}\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n-\tslash_pos = strrchr(parent_pathname.buf, '/');\n-\n-\tif (slash_pos == parent_pathname.buf) {\n+\tif (!slash_pos) {\n \t\t/* include every file in root */\n \t\tresult = MATCHED;\n \t\tgoto done;\n \t}\n \n-\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\tstrbuf_setlen(&parent_pathname, slash_pos);\n \n \tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n \t\tresult = MATCHED;\n-- \ngitgitgadget\n\n"},{"id":"429801","messageId":"2b72cc2d98505a635d902ab1be66bd1af763a99d.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 12/15] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:53Z","receivedAt":"2021-07-12T17:56:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 2269f44e033..375b0d35565 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -218,6 +218,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 42b67357169..96db3e74962 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e..ab9cc9d8f03 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"429802","messageId":"dada1b91bdc8ce671e949c4a2f32068d2ba4adab.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 14/15] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:55Z","receivedAt":"2021-07-12T17:56:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++\n wt-status.c | 51 ++++++++++++++++++++++++\n 2 files changed, 84 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 751f397cc7f..2394c36d881 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -524,4 +524,37 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\n+\t# Sparse checkouts do not agree with full checkouts about\n+\t# how to report a directory/file conflict during a reset.\n+\t# This command would fail with test_all_match because the\n+\t# full checkout reports \"T folder1/0/1\" while a sparse\n+\t# checkout reports \"D folder1/0/1\". This matches because\n+\t# the sparse checkouts skip \"adding\" the other side of\n+\t# the conflict.\n+\ttest_sparse_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 96db3e74962..0317baef87e 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \tclear_pathspec(&rev.prune_data);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tstruct strbuf full_name = STRBUF_INIT;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tstrbuf_add(&full_name, base->buf, base->len);\n+\tstrbuf_addstr(&full_name, path);\n+\tit = string_list_insert(&s->change, full_name.buf);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\tstrbuf_release(&full_name);\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps = { 0 };\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"429803","messageId":"bdc771cf37383896d546f592b6847d95d666672a.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 15/15] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:56Z","receivedAt":"2021-07-12T17:56:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 49 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex ef53bd2198b..53c8f711ccc 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 637391c6ce4..deea88d4431 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,52 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_region fsm_hook query trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"429804","messageId":"d875a7f8585107591e1811ad744601bebb3df9bc.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 11/15] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:52Z","receivedAt":"2021-07-12T17:56:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile comparing an index to a tree, we may see a sparse directory entry.\nIn this case, we should compare that portion of the tree to the tree\nrepresented by that entry. This could include a new tree which needs to\nbe expanded to a full list of added files. It could also include an\nexisting tree, in which case all of the changes inside are important to\ndescribe, including the modifications, additions, and deletions. Note\nthat the case where the tree has a path and the index does not remains\nidentical to before: the lack of a cache entry is the same with a sparse\nindex.\n\nUse diff_tree_oid() appropriately to compute the diff.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex c2ac9250fe9..f9eadc4fc1a 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -347,6 +352,20 @@ static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tassert(S_ISSPARSEDIR(old_entry->ce_mode) ==\n+\t S_ISSPARSEDIR(new_entry->ce_mode));\n+\n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level. If only one was a sparse\n+\t * directory, then they appear as an add and delete instead of\n+\t * a modification.\n+\t */\n+\tif (S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n-- \ngitgitgadget\n\n"},{"id":"429805","messageId":"1c1feef37339ff19ebfcdaf87a29d7adae73aa06.1626112556.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v8 13/15] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-12T17:55:54Z","receivedAt":"2021-07-12T17:56:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 190d215d43b..12f51db158a 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 0c3ac3cefc0..6a1337cc905 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1585,8 +1585,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1601,6 +1600,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 375b0d35565..751f397cc7f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -511,12 +511,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"429816","messageId":"CABPp-BGJ+LTubgS=zvGJjk3kgyfW-7UFEa=qg-0mdyrY32j0pQ@mail.gmail.com","threadId":"55489","inReplyTo":"3cf96b5f-3a8d-c00c-5d9e-c7095c058059@gmail.com","subject":"Re: [PATCH v7 15/16] wt-status: expand added sparse directory entries","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-12T19:32:22Z","receivedAt":"2021-07-12T19:32:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jul 12, 2021 at 6:56 AM Derrick Stolee wrote:\n>\n> On 7/8/2021 9:03 PM, Elijah Newren wrote:\n> > On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n> > wrote:\n> ...\n> >> +test_expect_success 'reset mixed and checkout orphan' '\n> >> + init_repos &&\n> >> +\n> >> + test_all_match git checkout rename-out-to-in &&\n> >> +\n> >> + # Sparse checkouts do not agree with full checkouts about\n> >> + # how to report a directory/file conflict during a reset.\n> >> + # This command would fail with test_all_match because the\n> >> + # full checkout reports \"T folder1/0/1\" while a sparse\n> >> + # checkout reports \"D folder1/0/1\". This matches because\n> >> + # the sparse checkouts skip \"adding\" the other side of\n> >> + # the conflict.\n> >\n> > The same issue I highlighted last time is still present. If you\n> > insert an \"exit 1\" right here, then run\n> > ./t1092-sparse-checkout-compatibility.sh --ver --imm -x\n> > until it stops, then\n> > cd t/trash directory.t1092-sparse-checkout-compatibility/sparse-checkout\n> > git ls-files -t | grep folder # Note the files that are sparse\n> > git reset --mixed HEAD~1\n> > git ls-files -t | grep folder # Note the files that are sparse --\n> > there are some that aren't that should be\n> > git sparse-checkout reapply\n> > git ls-files -t | grep folder # Note the files that are sparse\n> >\n> > Granted, this is a bug with sparse-checkout without sparse-index, so\n> > not something new to your series. But since you are using comparisons\n> > between regular sparse-checkouts and sparse-index to verify\n> > correctness, this seems problematic to me.\n>\n> I'll add it to the pile, but I want to continue having this series\n> focus on making the sparse-index work quickly without a change in\n> behavior from a normal index. Changing the behavior of the sparse-\n> checkout feature should be a separate series.\n\nHmm..perhaps there's some middle ground? I appreciate that you want\nto have this series focus on making the sparse-index work without\nworrying about behavioral changes to sparse-checkout. I'm concerned,\nthough, that testcases tend to be treated as documentation of intended\nbehavior, even when the tests are testing something else. These tests\nare clearly triggering buggy behavior, and I think your comments and\nsubsequent command may be affected by it. I don't want to leave\nfuture folks (even ourselves) to have to try to explain away why the\nbehavior expected in this test should not be expected.\n\nPerhaps we can just add a comment that this testcase is triggering a\nbug in both sparse-checkout and sparse-index but we're only checking\nthat the two match, and that once the bug is fix, the testcase itself\nmay need tweaking?\n"},{"id":"429817","messageId":"CABPp-BGZ0VNkAcScXLKAX0M3FwmXuZ5ei0o7Nu31L=rpSt6iQA@mail.gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 00/15] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-12T19:38:42Z","receivedAt":"2021-07-12T19:38:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jul 12, 2021 at 10:55 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' very fast when a sparse-index is enabled on a repository with\n> cone-mode sparse-checkout (and a small populated set).\n>\n...\n> Update in V8\n> ============\n>\n> * The directory/file conflict patch is removed and delayed to the next\n> series where it will be required. (It will also be improved in that\n> series.)\n>\n> * Some comments have been improved, including a new assert() that helps\n> document the situation.\n>\n\nThis one looks really good. Just two minor comments:\n\n> Range-diff vs v7:\n>\n...\n> 9: 237ccf4e43d ! 9: c0b0b58584c unpack-trees: unpack sparse directory entries\n> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n> + * Check for a sparse-directory entry named \"path/\".\n> + * Due to the input p->path not having a trailing\n> + * slash, the negative 'pos' value overshoots the\n> -+ * expected position by at least one, hence \"-2\" here.\n> ++ * expected position, hence \"-2\" instead of \"-1\".\n> + */\n> + pos = -pos - 2;\n> +\n> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n> return NULL;\n> +\n> + /*\n> -+ * We might have multiple entries between 'pos' and\n> -+ * the actual sparse-directory entry, so start walking\n> -+ * back until finding it or passing where it would be.\n> ++ * Due to lexicographic sorting and sparse directory\n> ++ * entried ending with a trailing slash, our path as a\n\ns/entried/entries/ ?\n\n\n> ++ * sparse directory (e.g \"subdir/\") and our path as a\n> ++ * file (e.g. \"subdir\") might be separated by other\n> ++ * paths (e.g. \"subdir-\").\n> + */\n> + while (pos >= 0) {\n> + ce = o->src_index->cache[pos];\n...\n> 15: 717a3f49f97 = 14: dada1b91bdc wt-status: expand added sparse directory entries\n\nAs I commented over at [1], I would appreciate if we could at least\nadd a comment in the testcase that we know this testcase triggers a\nbug for both sparse-index and sparse-checkout...and that fixing it\nmight affect the other comments and commands within that testcase in\nthe future...but for now, we're just testing as best we can that the\ntwo give the same behavior.\n\n[1] https://lore.kernel.org/git/CABPp-BGJ+LTubgS=zvGJjk3kgyfW-7UFEa=qg-0mdyrY32j0pQ@mail.gmail.com/\n\n\nIf you agree and include the two fixups above, the entire series is:\nReviewed-by: Elijah Newren \n\nIf you disagree, then all patches other than 9 and 14 can have my\nReviewed-by tag. :-)\n\n\nThanks for all the awesome work!\n"},{"id":"429819","messageId":"d5b6f7c0-4837-45a0-5fab-68fd321e21ff@gmail.com","threadId":"55489","inReplyTo":"CABPp-BGJ+LTubgS=zvGJjk3kgyfW-7UFEa=qg-0mdyrY32j0pQ@mail.gmail.com","subject":"Re: [PATCH v7 15/16] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-12T19:41:44Z","receivedAt":"2021-07-12T19:41:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2021 3:32 PM, Elijah Newren wrote:\n> On Mon, Jul 12, 2021 at 6:56 AM Derrick Stolee wrote:\n>>\n>> On 7/8/2021 9:03 PM, Elijah Newren wrote:\n>>> On Mon, Jun 28, 2021 at 7:05 PM Derrick Stolee via GitGitGadget\n>>> wrote:\n>> ...\n>>>> +test_expect_success 'reset mixed and checkout orphan' '\n>>>> + init_repos &&\n>>>> +\n>>>> + test_all_match git checkout rename-out-to-in &&\n>>>> +\n>>>> + # Sparse checkouts do not agree with full checkouts about\n>>>> + # how to report a directory/file conflict during a reset.\n>>>> + # This command would fail with test_all_match because the\n>>>> + # full checkout reports \"T folder1/0/1\" while a sparse\n>>>> + # checkout reports \"D folder1/0/1\". This matches because\n>>>> + # the sparse checkouts skip \"adding\" the other side of\n>>>> + # the conflict.\n>>>\n>>> The same issue I highlighted last time is still present. If you\n>>> insert an \"exit 1\" right here, then run\n>>> ./t1092-sparse-checkout-compatibility.sh --ver --imm -x\n>>> until it stops, then\n>>> cd t/trash directory.t1092-sparse-checkout-compatibility/sparse-checkout\n>>> git ls-files -t | grep folder # Note the files that are sparse\n>>> git reset --mixed HEAD~1\n>>> git ls-files -t | grep folder # Note the files that are sparse --\n>>> there are some that aren't that should be\n>>> git sparse-checkout reapply\n>>> git ls-files -t | grep folder # Note the files that are sparse\n>>>\n>>> Granted, this is a bug with sparse-checkout without sparse-index, so\n>>> not something new to your series. But since you are using comparisons\n>>> between regular sparse-checkouts and sparse-index to verify\n>>> correctness, this seems problematic to me.\n>>\n>> I'll add it to the pile, but I want to continue having this series\n>> focus on making the sparse-index work quickly without a change in\n>> behavior from a normal index. Changing the behavior of the sparse-\n>> checkout feature should be a separate series.\n> \n> Hmm..perhaps there's some middle ground? I appreciate that you want\n> to have this series focus on making the sparse-index work without\n> worrying about behavioral changes to sparse-checkout. I'm concerned,\n> though, that testcases tend to be treated as documentation of intended\n> behavior, even when the tests are testing something else. These tests\n> are clearly triggering buggy behavior, and I think your comments and\n> subsequent command may be affected by it. I don't want to leave\n> future folks (even ourselves) to have to try to explain away why the\n> behavior expected in this test should not be expected.\n> \n> Perhaps we can just add a comment that this testcase is triggering a\n> bug in both sparse-checkout and sparse-index but we're only checking\n> that the two match, and that once the bug is fix, the testcase itself\n> may need tweaking?\n \nI can get behind that approach: document the bug, but comment that it\n_is_ a bug and should be changed in the future.\n\nThanks,\n-Stolee\n"},{"id":"429891","messageId":"e7199fc6-bc7b-f556-c10c-12dee04af287@gmail.com","threadId":"55489","inReplyTo":"CABPp-BGZ0VNkAcScXLKAX0M3FwmXuZ5ei0o7Nu31L=rpSt6iQA@mail.gmail.com","subject":"Re: [PATCH v8 00/15] Sparse-index: integrate with status","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-07-13T12:57:44Z","receivedAt":"2021-07-13T12:57:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/12/2021 3:38 PM, Elijah Newren wrote:\n> On Mon, Jul 12, 2021 at 10:55 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> 9: 237ccf4e43d ! 9: c0b0b58584c unpack-trees: unpack sparse directory entries\n>> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n>> + * Check for a sparse-directory entry named \"path/\".\n>> + * Due to the input p->path not having a trailing\n>> + * slash, the negative 'pos' value overshoots the\n>> -+ * expected position by at least one, hence \"-2\" here.\n>> ++ * expected position, hence \"-2\" instead of \"-1\".\n>> + */\n>> + pos = -pos - 2;\n>> +\n>> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n>> return NULL;\n>> +\n>> + /*\n>> -+ * We might have multiple entries between 'pos' and\n>> -+ * the actual sparse-directory entry, so start walking\n>> -+ * back until finding it or passing where it would be.\n>> ++ * Due to lexicographic sorting and sparse directory\n>> ++ * entried ending with a trailing slash, our path as a\n> \n> s/entried/entries/ ?\n\nOops! Yes, that would be a valuable fixup. Thanks for catching it.\n\n> \n>> ++ * sparse directory (e.g \"subdir/\") and our path as a\n>> ++ * file (e.g. \"subdir\") might be separated by other\n>> ++ * paths (e.g. \"subdir-\").\n>> + */\n>> + while (pos >= 0) {\n>> + ce = o->src_index->cache[pos];\n> ...\n>> 15: 717a3f49f97 = 14: dada1b91bdc wt-status: expand added sparse directory entries\n> \n> As I commented over at [1], I would appreciate if we could at least\n> add a comment in the testcase that we know this testcase triggers a\n> bug for both sparse-index and sparse-checkout...and that fixing it\n> might affect the other comments and commands within that testcase in\n> the future...but for now, we're just testing as best we can that the\n> two give the same behavior.\n> \n> [1] https://lore.kernel.org/git/CABPp-BGJ+LTubgS=zvGJjk3kgyfW-7UFEa=qg-0mdyrY32j0pQ@mail.gmail.com/\n\nHow do you feel about a new patch that focuses on adding these\ncomments, including an older test that had a similar documentation\nof the behavior change? A patch that could be queued on top of\nthis series is pasted below the cutline.\n\nThanks,\n-Stolee\n\n\n-- >8 --\n\nFrom 8e69def90f5844c117cc1e9efd673c92b85c9238 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 13 Jul 2021 08:50:24 -0400\nSubject: [PATCH 16/15] t1092: document bad sparse-checkout behavior\n\nThere are several situations where a repository with sparse-checkout\nenabled will act differently than a normal repository, and in ways that\nare not intentional. The test t1092-sparse-checkout-compatibility.sh\ndocuments some of these deviations, but a casual reader might think\nthese are intentional behavior changes.\n\nAdd comments on these tests that make it clear that these behaviors\nshould be updated. Using 'NEEDSWORK' helps contributors find that these\nare potential areas for improvement.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 2394c36d881..cabbd42e339 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -392,8 +392,8 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# TODO: reset currently does not behave as expected when in a\n-# sparse-checkout.\n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_failure 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n@@ -403,8 +403,8 @@ test_expect_failure 'checkout and reset (mixed)' '\n \ttest_all_match git reset update-folder2\n '\n \n-# Ensure that sparse-index behaves identically to\n-# sparse-checkout with a full index.\n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_success 'checkout and reset (mixed) [sparse]' '\n \tinit_repos &&\n \n@@ -524,6 +524,8 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n \tinit_repos &&\n \n-- \n2.32.0.15.g108d1b86e21\n\n\n\n"},{"id":"429906","messageId":"CABPp-BGs_X0=th9a_sgDPS36WQZaGO-r5DE0p+xgf7vHKUy1uQ@mail.gmail.com","threadId":"55489","inReplyTo":"e7199fc6-bc7b-f556-c10c-12dee04af287@gmail.com","subject":"Re: [PATCH v8 00/15] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-13T17:37:24Z","receivedAt":"2021-07-13T17:37:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jul 13, 2021 at 5:57 AM Derrick Stolee wrote:\n>\n> On 7/12/2021 3:38 PM, Elijah Newren wrote:\n> > On Mon, Jul 12, 2021 at 10:55 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >> 9: 237ccf4e43d ! 9: c0b0b58584c unpack-trees: unpack sparse directory entries\n> >> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n> >> + * Check for a sparse-directory entry named \"path/\".\n> >> + * Due to the input p->path not having a trailing\n> >> + * slash, the negative 'pos' value overshoots the\n> >> -+ * expected position by at least one, hence \"-2\" here.\n> >> ++ * expected position, hence \"-2\" instead of \"-1\".\n> >> + */\n> >> + pos = -pos - 2;\n> >> +\n> >> @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n> >> return NULL;\n> >> +\n> >> + /*\n> >> -+ * We might have multiple entries between 'pos' and\n> >> -+ * the actual sparse-directory entry, so start walking\n> >> -+ * back until finding it or passing where it would be.\n> >> ++ * Due to lexicographic sorting and sparse directory\n> >> ++ * entried ending with a trailing slash, our path as a\n> >\n> > s/entried/entries/ ?\n>\n> Oops! Yes, that would be a valuable fixup. Thanks for catching it.\n>\n> >\n> >> ++ * sparse directory (e.g \"subdir/\") and our path as a\n> >> ++ * file (e.g. \"subdir\") might be separated by other\n> >> ++ * paths (e.g. \"subdir-\").\n> >> + */\n> >> + while (pos >= 0) {\n> >> + ce = o->src_index->cache[pos];\n> > ...\n> >> 15: 717a3f49f97 = 14: dada1b91bdc wt-status: expand added sparse directory entries\n> >\n> > As I commented over at [1], I would appreciate if we could at least\n> > add a comment in the testcase that we know this testcase triggers a\n> > bug for both sparse-index and sparse-checkout...and that fixing it\n> > might affect the other comments and commands within that testcase in\n> > the future...but for now, we're just testing as best we can that the\n> > two give the same behavior.\n> >\n> > [1] https://lore.kernel.org/git/CABPp-BGJ+LTubgS=zvGJjk3kgyfW-7UFEa=qg-0mdyrY32j0pQ@mail.gmail.com/\n>\n> How do you feel about a new patch that focuses on adding these\n> comments, including an older test that had a similar documentation\n> of the behavior change? A patch that could be queued on top of\n> this series is pasted below the cutline.\n\nLooks good to me. Re-roll with my Reviewed-by and let's get this\nseries merged down to next. :-)\n\n>\n> Thanks,\n> -Stolee\n>\n>\n> -- >8 --\n>\n> From 8e69def90f5844c117cc1e9efd673c92b85c9238 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Tue, 13 Jul 2021 08:50:24 -0400\n> Subject: [PATCH 16/15] t1092: document bad sparse-checkout behavior\n>\n> There are several situations where a repository with sparse-checkout\n> enabled will act differently than a normal repository, and in ways that\n> are not intentional. The test t1092-sparse-checkout-compatibility.sh\n> documents some of these deviations, but a casual reader might think\n> these are intentional behavior changes.\n>\n> Add comments on these tests that make it clear that these behaviors\n> should be updated. Using 'NEEDSWORK' helps contributors find that these\n> are potential areas for improvement.\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 10 ++++++----\n> 1 file changed, 6 insertions(+), 4 deletions(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 2394c36d881..cabbd42e339 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -392,8 +392,8 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n> test_all_match git blame deep/deeper2/deepest/a\n> '\n>\n> -# TODO: reset currently does not behave as expected when in a\n> -# sparse-checkout.\n> +# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> +# in this scenario, but it shouldn't.\n> test_expect_failure 'checkout and reset (mixed)' '\n> init_repos &&\n>\n> @@ -403,8 +403,8 @@ test_expect_failure 'checkout and reset (mixed)' '\n> test_all_match git reset update-folder2\n> '\n>\n> -# Ensure that sparse-index behaves identically to\n> -# sparse-checkout with a full index.\n> +# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> +# in this scenario, but it shouldn't.\n> test_expect_success 'checkout and reset (mixed) [sparse]' '\n> init_repos &&\n>\n> @@ -524,6 +524,8 @@ test_expect_success 'sparse-index is not expanded' '\n> test_region ! index ensure_full_index trace2.txt\n> '\n>\n> +# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> +# in this scenario, but it shouldn't.\n> test_expect_success 'reset mixed and checkout orphan' '\n> init_repos &&\n>\n> --\n"},{"id":"429955","messageId":"06de6811-0d04-c39f-589f-d5563b3149cf@gmail.com","threadId":"55489","inReplyTo":"05981e30b97a3e668b3a9675508135d42f15450f.1626112556.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 03/15] t1092: replace incorrect 'echo' with 'cat'","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-07-14T00:02:46Z","receivedAt":"2021-07-14T00:02:54Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 13/07/21 00.55, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> This fixes the test data shape to be as expected, allowing rename\n> detection to work properly now that the 'larger-content' file actually\n> has meaningful lines.\n> \n\nWhat's the connection with s/echo/cat/ ?\n\n> @@ -40,7 +40,7 @@ test_expect_success 'setup' '\n> \t\tdone &&\n> \n> \t\tgit checkout -b rename-base base &&\n> -\t\techo >folder1/larger-content <<-\\EOF &&\n> +\t\tcat >folder1/larger-content <<-\\EOF &&\n> \t\tmatching\n> \t\tlines\n> \t\thelp\n> \n\nOK, because to write multi-line file in scripts one must use cat, while \necho can only write single line.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"430038","messageId":"ecab56fd57facf0ac8d5ec8abcb43cd509e6e303.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 01/16] sparse-index: skip indexes with unmerged entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:25Z","receivedAt":"2021-07-14T13:12:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-index format is designed to be compatible with merge\nconflicts, even those outside the sparse-checkout definition. The reason\nis that when converting a full index to a sparse one, a cache entry with\nnonzero stage will not be collapsed into a sparse directory entry.\n\nHowever, this behavior was not tested, and a different behavior within\nconvert_to_sparse() fails in this scenario. Specifically,\ncache_tree_update() will fail when unmerged entries exist.\nconvert_to_sparse_rec() uses the cache-tree data to recursively walk the\ntree structure, but also to compute the OIDs used in the\nsparse-directory entries.\n\nAdd an index scan to convert_to_sparse() that will detect if these merge\nconflict entries exist and skip the conversion before trying to update\nthe cache-tree. This is marked as NEEDSWORK because this can be removed\nwith a suitable update to cache_tree_update() or a similar method that\ncan construct a cache-tree with invalid nodes, but still allow creating\nthe nodes necessary for creating sparse directory entries.\n\nIt is possible that in the future we will not need to make such an\nupdate, since if we do not expand a sparse-index into a full one, this\nconversion does not need to happen. Thus, this can be deferred until the\nmerge machinery is made to integrate with the sparse-index.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 18 ++++++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex affc4048f27..2c695930275 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable)\n \treturn res;\n }\n \n+static int index_has_unmerged_entries(struct index_state *istate)\n+{\n+\tint i;\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tif (ce_stage(istate->cache[i]))\n+\t\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n@@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * NEEDSWORK: If we have unmerged entries, then stay full.\n+\t * Unmerged entries prevent the cache-tree extension from working.\n+\t */\n+\tif (index_has_unmerged_entries(istate))\n+\t\treturn 0;\n+\n \tif (cache_tree_update(istate, 0)) {\n \t\twarning(_(\"unable to update cache-tree, staying full\"));\n \t\treturn -1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d028b73eba1..b8617ceef71 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' '\n \tdone\n '\n \n+# Sparse-index fails to convert the index in the\n+# final 'git cherry-pick' command.\n+test_expect_success 'cherry-pick with conflicts' '\n+\tinit_repos &&\n+\n+\twrite_script edit-conflict <<-\\EOF &&\n+\techo $1 >conflict\n+\tEOF\n+\n+\ttest_all_match git checkout -b to-cherry-pick &&\n+\trun_on_all ../edit-conflict ABC &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict to pick\" &&\n+\n+\ttest_all_match git checkout -B base HEAD~1 &&\n+\trun_on_all ../edit-conflict DEF &&\n+\ttest_all_match git add conflict &&\n+\ttest_all_match git commit -m \"conflict in base\" &&\n+\n+\ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"430039","messageId":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v8.git.1626112556.gitgitgadget@gmail.com","subject":"[PATCH v9 00/16] Sparse-index: integrate with status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:24Z","receivedAt":"2021-07-14T13:12:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first \"payoff\" series in the sparse-index work. It makes 'git\nstatus' very fast when a sparse-index is enabled on a repository with\ncone-mode sparse-checkout (and a small populated set).\n\nThis is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\nThe latter branch is needed because it changes the behavior of 'git add'\naround sparse entries, which changes the expectations of a test added in\npatch 1.\n\nThe approach here is to audit the places where ensure_full_index() pops up\nwhile doing normal commands with pathspecs within the sparse-checkout\ndefinition. Each of these are checked and tested. In the end, the\nsparse-index is integrated with these features:\n\n * git status\n * FS Monitor index extension.\n\nThe performance tests in p2000-sparse-operations.sh improve by 95% or more,\neven when compared with the full-index cases, not just the sparse-index\ncases that previously had extra overhead.\n\nHopefully this is the first example of how ds/sparse-index-protections has\ndone the basic work to do these conversions safely, making them look easier\nthan they seemed when starting this adventure.\n\nThanks, -Stolee\n\n\nUpdate in V9\n============\n\n * Fixed typo.\n\n * All patches are marked as Reviewed-by Elijah. Thanks for the careful\n review!\n\nDerrick Stolee (16):\n sparse-index: skip indexes with unmerged entries\n sparse-index: include EXTENDED flag when expanding\n t1092: replace incorrect 'echo' with 'cat'\n t1092: expand repository data shape\n t1092: add tests for status/add and sparse files\n unpack-trees: preserve cache_bottom\n unpack-trees: compare sparse directories correctly\n unpack-trees: rename unpack_nondirectories()\n unpack-trees: unpack sparse directory entries\n dir.c: accept a directory as part of cone-mode patterns\n diff-lib: handle index diffs with sparse dirs\n status: skip sparse-checkout percentage with sparse-index\n status: use sparse-index throughout\n wt-status: expand added sparse directory entries\n fsmonitor: integrate with sparse index\n t1092: document bad sparse-checkout behavior\n\n builtin/commit.c | 3 +\n diff-lib.c | 19 +++\n dir.c | 24 +++-\n read-cache.c | 10 +-\n sparse-index.c | 27 +++-\n t/t1092-sparse-checkout-compatibility.sh | 168 +++++++++++++++++++++--\n t/t7519-status-fsmonitor.sh | 49 +++++++\n unpack-trees.c | 142 ++++++++++++++++---\n wt-status.c | 65 ++++++++-\n wt-status.h | 1 +\n 10 files changed, 470 insertions(+), 38 deletions(-)\n\n\nbase-commit: d486ca60a51c9cb1fe068803c3f540724e95e83a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-932%2Fderrickstolee%2Fsparse-index%2Fstatus-and-add-v9\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-932/derrickstolee/sparse-index/status-and-add-v9\nPull-Request: https://github.com/gitgitgadget/git/pull/932\n\nRange-diff vs v8:\n\n 1: 1815c148e8c ! 1: ecab56fd57f sparse-index: skip indexes with unmerged entries\n @@ Commit message\n conversion does not need to happen. Thus, this can be deferred until the\n merge machinery is made to integrate with the sparse-index.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## sparse-index.c ##\n 2: 7bcde075d8d ! 2: f3de9ce7baa sparse-index: include EXTENDED flag when expanding\n @@ Commit message\n feature, but has no significance to its correctness in the user-facing\n functionality.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## sparse-index.c ##\n 3: 05981e30b97 ! 3: 5b59436011c t1092: replace incorrect 'echo' with 'cat'\n @@ Commit message\n detection to work properly now that the 'larger-content' file actually\n has meaningful lines.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n 4: d38b66e9ee4 ! 4: 4d06d972911 t1092: expand repository data shape\n @@ Commit message\n Later tests will take advantage of these shapes, but they also deepen\n the tests that already exist.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n 5: 95ddd3abe4e ! 5: cc83a0cd307 t1092: add tests for status/add and sparse files\n @@ Commit message\n be more sensible, and this test could be modified to satisfy the new\n expected behavior.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n 6: b182b456613 ! 6: 18cb1f6ea9b unpack-trees: preserve cache_bottom\n @@ Commit message\n the cache_bottom will be modified as the cache-tree walk advances. Do\n not update it as well within mark_ce_used().\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n 7: 988ddce4d45 ! 7: bdd47bd30e0 unpack-trees: compare sparse directories correctly\n @@ Commit message\n exact string match on a sparse directory entry. The final check is a\n length comparison between the strings.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n 8: d67ad048b08 ! 8: 0e4b7390f47 unpack-trees: rename unpack_nondirectories()\n @@ Commit message\n apply. The new name reflects that we will not recurse into trees in\n order to resolve the conflicts.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n 9: c0b0b58584c ! 9: 602525f5d71 unpack-trees: unpack sparse directory entries\n @@ Commit message\n markers in the case that the entries correspond to directories and we\n already have a sparse directory entry.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n @@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n +\n +\t/*\n +\t * Due to lexicographic sorting and sparse directory\n -+\t * entried ending with a trailing slash, our path as a\n ++\t * entries ending with a trailing slash, our path as a\n +\t * sparse directory (e.g \"subdir/\") and\tour path as a\n +\t * file (e.g. \"subdir\") might be separated by other\n +\t * paths (e.g. \"subdir-\").\n 10: 76c7528f78f ! 10: b051c0847a5 dir.c: accept a directory as part of cone-mode patterns\n @@ Commit message\n assuming we are in cone mode. Since sparse index requires cone mode\n patterns, this is an acceptable assumption.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## dir.c ##\n 11: d875a7f8585 ! 11: e749fd41cda diff-lib: handle index diffs with sparse dirs\n @@ Commit message\n \n Use diff_tree_oid() appropriately to compute the diff.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## diff-lib.c ##\n 12: 2b72cc2d985 ! 12: 7f782e3fe50 status: skip sparse-checkout percentage with sparse-index\n @@ Commit message\n this message is equal across both modes, but instead just the important\n information about staged, modified, and untracked files are compared.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n 13: 1c1feef3733 ! 13: ad1715e3319 status: use sparse-index throughout\n @@ Commit message\n other way. Correct integration with FS Monitor will be validated in\n later changes.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## builtin/commit.c ##\n 14: dada1b91bdc ! 14: aa2258be302 wt-status: expand added sparse directory entries\n @@ Commit message\n Somehow, this loop over the cache entries was not guarded by\n ensure_full_index() as intended.\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n 15: bdc771cf373 ! 15: 1d4b1f8aea0 fsmonitor: integrate with sparse index\n @@ Commit message\n to a tracked directory outside of the sparse cone will trigger\n ensure_full_index().\n \n + Reviewed-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n ## sparse-index.c ##\n -: ----------- > 16: 45861118991 t1092: document bad sparse-checkout behavior\n\n-- \ngitgitgadget\n"},{"id":"430040","messageId":"f3de9ce7baac9667f5f946bcdf4f0efcddd24c64.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 02/16] sparse-index: include EXTENDED flag when expanding","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:26Z","receivedAt":"2021-07-14T13:12:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen creating a full index from a sparse one, we create cache entries\nfor every blob within a given sparse directory entry. These are\ncorrectly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED\nflag is not included. The CE_EXTENDED flag would exist if we loaded a\nfull index from disk with these entries marked with CE_SKIP_WORKTREE, so\nwe can add the flag here to be consistent. This allows us to directly\ncompare the flags present in cache entries when testing the sparse-index\nfeature, but has no significance to its correctness in the user-facing\nfunctionality.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2c695930275..ef53bd2198b 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid,\n \tstrbuf_addstr(base, path);\n \n \tce = make_cache_entry(istate, mode, oid, base->buf, 0, 0);\n-\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED;\n \tset_index_entry(istate, istate->cache_nr++, ce);\n \n \tstrbuf_setlen(base, len);\n-- \ngitgitgadget\n\n"},{"id":"430041","messageId":"5b59436011c0ba8a86cbbc28954166a79e49a4eb.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 03/16] t1092: replace incorrect 'echo' with 'cat'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:27Z","receivedAt":"2021-07-14T13:12:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis fixes the test data shape to be as expected, allowing rename\ndetection to work properly now that the 'larger-content' file actually\nhas meaningful lines.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b8617ceef71..87f1014a1c9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -40,7 +40,7 @@ test_expect_success 'setup' '\n \t\tdone &&\n \n \t\tgit checkout -b rename-base base &&\n-\t\techo >folder1/larger-content <<-\\EOF &&\n+\t\tcat >folder1/larger-content <<-\\EOF &&\n \t\tmatching\n \t\tlines\n \t\thelp\n-- \ngitgitgadget\n\n"},{"id":"430042","messageId":"cc83a0cd3079f1400bcb5405e914206fa422404f.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 05/16] t1092: add tests for status/add and sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:29Z","receivedAt":"2021-07-14T13:12:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore moving to update 'git status' and 'git add' to work with sparse\nindexes, add an explicit test that ensures the sparse-index works the\nsame as a normal sparse-checkout when the worktree contains directories\nand files outside of the sparse cone.\n\nSpecifically, 'folder1/a' is a file in our test repo, but 'folder1' is\nnot in the sparse cone. When 'folder1/a' is modified, the file is not\nshown as modified and adding it will fail. This is new behavior as of\na20f704 (add: warn when asked to update SKIP_WORKTREE entries,\n2021-04-08). Before that change, these adds would be silently ignored.\n\nUntracked files are fine: adding new files both with 'git add .' and\n'git add folder1/' works just as in a full checkout. This may not be\nentirely desirable, but we are not intending to change behavior at the\nmoment, only document it. A future change could alter the behavior to\nbe more sensible, and this test could be modified to satisfy the new\nexpected behavior.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++\n 1 file changed, 38 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0e71a623619..2269f44e033 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -254,6 +254,44 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_success 'status/add: outside sparse cone' '\n+\tinit_repos &&\n+\n+\t# adding a \"missing\" file outside the cone should fail\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\n+\t# folder1 is at HEAD, but outside the sparse cone\n+\trun_on_sparse mkdir folder1 &&\n+\tcp initial-repo/folder1/a sparse-checkout/folder1/a &&\n+\tcp initial-repo/folder1/a sparse-index/folder1/a &&\n+\n+\ttest_sparse_match git status &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/new &&\n+\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# This \"git add folder1/a\" fails with a warning\n+\t# in the sparse repos, differing from the full\n+\t# repo. This is intentional.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/new &&\n+\n+\trun_on_all ../edit-contents folder1/newer &&\n+\ttest_all_match git add folder1/ &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git commit -m folder1/newer\n+'\n+\n test_expect_success 'checkout and reset --hard' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"430043","messageId":"4d06d97291105093c768a42e35b455d59ec791ab.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 04/16] t1092: expand repository data shape","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:28Z","receivedAt":"2021-07-14T13:12:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs more features integrate with the sparse-index feature, more and more\nspecial cases arise that require different data shapes within the tree\nstructure of the repository in order to demonstrate those cases.\n\nAdd several interesting special cases all at once instead of sprinkling\nthem across several commits. The interesting cases being added here are:\n\n* Add sparse-directory entries on both sides of directories within the\n sparse-checkout definition.\n\n* Add directories outside the sparse-checkout definition who have only\n one entry and are the first entry of a directory with multiple\n entries.\n\n* Add filenames adjacent to a sparse directory entry that sort before\n and after the trailing slash.\n\nLater tests will take advantage of these shapes, but they also deepen\nthe tests that already exist.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 42 ++++++++++++++++++++++--\n 1 file changed, 40 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 87f1014a1c9..0e71a623619 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -17,7 +17,7 @@ test_expect_success 'setup' '\n \t\techo \"after folder1\" >g &&\n \t\techo \"after x\" >z &&\n \t\tmkdir folder1 folder2 deep x &&\n-\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n@@ -25,10 +25,23 @@ test_expect_success 'setup' '\n \t\tcp a folder2 &&\n \t\tcp a x &&\n \t\tcp a deep &&\n+\t\tcp a deep/before &&\n \t\tcp a deep/deeper1 &&\n \t\tcp a deep/deeper2 &&\n+\t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n \t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tmkdir deep/deeper1/0 &&\n+\t\tmkdir deep/deeper1/0/0 &&\n+\t\ttouch deep/deeper1/0/1 &&\n+\t\ttouch deep/deeper1/0/0/0 &&\n+\t\t>folder1- &&\n+\t\t>folder1.x &&\n+\t\t>folder10 &&\n+\t\tcp -r deep/deeper1/0 folder1 &&\n+\t\tcp -r deep/deeper1/0 folder2 &&\n+\t\techo >>folder1/0/0/0 &&\n+\t\techo >>folder2/0/1 &&\n \t\tgit add . &&\n \t\tgit commit -m \"initial commit\" &&\n \t\tgit checkout -b base &&\n@@ -56,11 +69,17 @@ test_expect_success 'setup' '\n \t\tmv folder1/a folder2/b &&\n \t\tmv folder1/larger-content folder2/edited-content &&\n \t\techo >>folder2/edited-content &&\n+\t\techo >>folder2/0/1 &&\n+\t\techo stuff >>deep/deeper1/a &&\n \t\tgit add . &&\n \t\tgit commit -m \"rename folder1/... to folder2/...\" &&\n \n \t\tgit checkout -b rename-out-to-in rename-base &&\n \t\tmv folder1/a deep/deeper1/b &&\n+\t\techo more stuff >>deep/deeper1/a &&\n+\t\trm folder2/0/1 &&\n+\t\tmkdir folder2/0/1 &&\n+\t\techo >>folder2/0/1/1 &&\n \t\tmv folder1/larger-content deep/deeper1/edited-content &&\n \t\techo >>deep/deeper1/edited-content &&\n \t\tgit add . &&\n@@ -68,6 +87,9 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b rename-in-to-out rename-base &&\n \t\tmv deep/deeper1/a folder1/b &&\n+\t\techo >>folder2/0/1 &&\n+\t\trm -rf folder1/0/0 &&\n+\t\techo >>folder1/0/0 &&\n \t\tmv deep/deeper1/larger-content folder1/edited-content &&\n \t\techo >>folder1/edited-content &&\n \t\tgit add . &&\n@@ -262,13 +284,29 @@ test_expect_success 'diff --staged' '\n \ttest_all_match git diff --staged\n '\n \n-test_expect_success 'diff with renames' '\n+test_expect_success 'diff with renames and conflicts' '\n \tinit_repos &&\n \n \tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n \tdo\n \t\ttest_all_match git checkout rename-base &&\n \t\ttest_all_match git checkout $branch -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_all_match git diff --staged --no-renames &&\n+\t\ttest_all_match git diff --staged --find-renames || return 1\n+\tdone\n+'\n+\n+test_expect_success 'diff with directory/file conflicts' '\n+\tinit_repos &&\n+\n+\tfor branch in rename-out-to-out rename-out-to-in rename-in-to-out\n+\tdo\n+\t\tgit -C full-checkout reset --hard &&\n+\t\ttest_sparse_match git reset --hard &&\n+\t\ttest_all_match git checkout $branch &&\n+\t\ttest_all_match git checkout rename-base -- . &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n \t\ttest_all_match git diff --staged --no-renames &&\n \t\ttest_all_match git diff --staged --find-renames || return 1\n \tdone\n-- \ngitgitgadget\n\n"},{"id":"430044","messageId":"18cb1f6ea9b145e7ab664c1f9de5886ffba732a2.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 06/16] unpack-trees: preserve cache_bottom","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:30Z","receivedAt":"2021-07-14T13:12:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe cache_bottom member of 'struct unpack_trees_options' is used to\ntrack the range of index entries corresponding to a node of the cache\ntree. While recursing with traverse_by_cache_tree(), this value is\npreserved on the call stack using a local and then restored as that\nmethod returns.\n\nThe mark_ce_used() method normally modifies the cache_bottom member when\nit refers to the marked cache entry. However, sparse directory entries\nare stored as nodes in the cache-tree data structure as of 2de37c53\n(cache-tree: integrate with sparse directory entries, 2021-03-30). Thus,\nthe cache_bottom will be modified as the cache-tree walk advances. Do\nnot update it as well within mark_ce_used().\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f88a69f8e71..87c1ed204c8 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o)\n {\n \tce->ce_flags |= CE_UNPACKED;\n \n+\t/*\n+\t * If this is a sparse directory, don't advance cache_bottom.\n+\t * That will be advanced later using the cache-tree data.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn;\n+\n \tif (o->cache_bottom < o->src_index->cache_nr &&\n \t o->src_index->cache[o->cache_bottom] == ce) {\n \t\tint bottom = o->cache_bottom;\n-- \ngitgitgadget\n\n"},{"id":"430045","messageId":"bdd47bd30e02a2ffd92ca7371e1d990f08935ba7.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 07/16] unpack-trees: compare sparse directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:31Z","receivedAt":"2021-07-14T13:12:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we further integrate the sparse-index into unpack-trees, we need to\nensure that we compare sparse directory entries correctly with other\nentries. This affects searching for an exact path as well as sorting\nindex entries.\n\nSparse directory entries contain the trailing directory separator. This\nis important for the sorting, in particular. Thus, within\ndo_compare_entry() we stop using S_IFREG in all cases, since sparse\ndirectories should use S_IFDIR to indicate that the comparison should\ntreat the entry name as a dirctory.\n\nWithin compare_entry(), it first calls do_compare_entry() to check the\nleading portion of the name. When the input path is a directory name, we\ncould match exactly already. Thus, we should return 0 if we have an\nexact string match on a sparse directory entry. The final check is a\nlength comparison between the strings.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 87c1ed204c8..b113cc750f2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tint pathlen, ce_len;\n \tconst char *ce_name;\n \tint cmp;\n+\tunsigned ce_mode;\n \n \t/*\n \t * If we have not precomputed the traverse path, it is quicker\n@@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce,\n \tce_len -= pathlen;\n \tce_name = ce->name + pathlen;\n \n-\treturn df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode);\n+\tce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG;\n+\treturn df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode);\n }\n \n static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n)\n@@ -1014,6 +1016,16 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf\n \tif (cmp)\n \t\treturn cmp;\n \n+\t/*\n+\t * At this point, we know that we have a prefix match. If ce\n+\t * is a sparse directory, then allow an exact match. This only\n+\t * works when the input name is a directory, since ce->name\n+\t * ends in a directory separator.\n+\t */\n+\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1)\n+\t\treturn 0;\n+\n \t/*\n \t * Even if the beginning compared identically, the ce should\n \t * compare as bigger than a directory leading up to it!\n-- \ngitgitgadget\n\n"},{"id":"430046","messageId":"0e4b7390f478e89d20e33894b69567e489d69c9b.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 08/16] unpack-trees: rename unpack_nondirectories()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:32Z","receivedAt":"2021-07-14T13:12:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn the next change, we will use this method to unpack a sparse directory\nentry, so change the name to unpack_single_entry() so these entries\napply. The new name reflects that we will not recurse into trees in\norder to resolve the conflicts.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 14 +++++++-------\n 1 file changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b113cc750f2..d26386ce8b2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -804,7 +804,7 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names,\n \t\tBUG(\"We need cache-tree to do this optimization\");\n \n \t/*\n-\t * Do what unpack_callback() and unpack_nondirectories() normally\n+\t * Do what unpack_callback() and unpack_single_entry() normally\n \t * do. But we walk all paths in an iterative loop instead.\n \t *\n \t * D/F conflicts and higher stage entries are not a concern\n@@ -1075,11 +1075,11 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n * without actually calling it. If you change the logic here you may need to\n * check and change there as well.\n */\n-static int unpack_nondirectories(int n, unsigned long mask,\n-\t\t\t\t unsigned long dirmask,\n-\t\t\t\t struct cache_entry **src,\n-\t\t\t\t const struct name_entry *names,\n-\t\t\t\t const struct traverse_info *info)\n+static int unpack_single_entry(int n, unsigned long mask,\n+\t\t\t unsigned long dirmask,\n+\t\t\t struct cache_entry **src,\n+\t\t\t const struct name_entry *names,\n+\t\t\t const struct traverse_info *info)\n {\n \tint i;\n \tstruct unpack_trees_options *o = info->data;\n@@ -1322,7 +1322,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t}\n \t}\n \n-\tif (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0)\n+\tif (unpack_single_entry(n, mask, dirmask, src, names, info) < 0)\n \t\treturn -1;\n \n \tif (o->merge && src[0]) {\n-- \ngitgitgadget\n\n"},{"id":"430047","messageId":"602525f5d71df71ff5f8071f9d87f6e3a7b7f5c6.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 09/16] unpack-trees: unpack sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:33Z","receivedAt":"2021-07-14T13:12:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring unpack_callback(), index entries are compared against tree\nentries. These are matched according to names and types. One goal is to\ndecide if we should recurse into subtrees or simply operate on one index\nentry.\n\nIn the case of a sparse-directory entry, we do not want to recurse into\nthat subtree and instead simply compare the trees. In some cases, we\nmight want to perform a merge operation on the entry, such as during\n'git checkout ' which wants to replace a sparse tree entry with\nthe tree for that path at the target commit. We extend the logic within\nunpack_single_entry() to create a sparse-directory entry in this case,\nand then that is sent to call_unpack_fn().\n\nThere are some subtleties in this process. For instance, we need to\nupdate find_cache_entry() to allow finding a sparse-directory entry that\nexactly matches a given path. Use the new helper method\nsparse_dir_matches_path() for this. We also need to ignore conflict\nmarkers in the case that the entries correspond to directories and we\nalready have a sparse directory entry.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 107 +++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 99 insertions(+), 8 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d26386ce8b2..0a5135ab397 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1052,13 +1052,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \tconst struct name_entry *n,\n \tint stage,\n \tstruct index_state *istate,\n-\tint is_transient)\n+\tint is_transient,\n+\tint is_sparse_directory)\n {\n \tsize_t len = traverse_path_len(info, tree_entry_len(n));\n+\tsize_t alloc_len = is_sparse_directory ? len + 1 : len;\n \tstruct cache_entry *ce =\n \t\tis_transient ?\n-\t\tmake_empty_transient_cache_entry(len, NULL) :\n-\t\tmake_empty_cache_entry(istate, len);\n+\t\tmake_empty_transient_cache_entry(alloc_len, NULL) :\n+\t\tmake_empty_cache_entry(istate, alloc_len);\n \n \tce->ce_mode = create_ce_mode(n->mode);\n \tce->ce_flags = create_ce_flags(stage);\n@@ -1067,6 +1069,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info,\n \t/* len+1 because the cache_entry allocates space for NUL */\n \tmake_traverse_path(ce->name, len + 1, info, n->path, n->pathlen);\n \n+\tif (is_sparse_directory) {\n+\t\tce->name[len] = '/';\n+\t\tce->name[len + 1] = '\\0';\n+\t\tce->ce_namelen++;\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\t}\n+\n \treturn ce;\n }\n \n@@ -1085,10 +1094,17 @@ static int unpack_single_entry(int n, unsigned long mask,\n \tstruct unpack_trees_options *o = info->data;\n \tunsigned long conflicts = info->df_conflicts | dirmask;\n \n-\t/* Do we have *only* directories? Nothing to do */\n \tif (mask == dirmask && !src[0])\n \t\treturn 0;\n \n+\t/*\n+\t * When we have a sparse directory entry for src[0],\n+\t * then this isn't necessarily a directory-file conflict.\n+\t */\n+\tif (mask == dirmask && src[0] &&\n+\t S_ISSPARSEDIR(src[0]->ce_mode))\n+\t\tconflicts = 0;\n+\n \t/*\n \t * Ok, we've filled in up to any potential index entry in src[0],\n \t * now do the rest.\n@@ -1118,7 +1134,9 @@ static int unpack_single_entry(int n, unsigned long mask,\n \t\t * not stored in the index. otherwise construct the\n \t\t * cache entry from the index aware logic.\n \t\t */\n-\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge);\n+\t\tsrc[i + o->merge] = create_ce_entry(info, names + i, stage,\n+\t\t\t\t\t\t &o->result, o->merge,\n+\t\t\t\t\t\t bit & dirmask);\n \t}\n \n \tif (o->merge) {\n@@ -1222,16 +1240,71 @@ static int find_cache_pos(struct traverse_info *info,\n \treturn -1;\n }\n \n+/*\n+ * Given a sparse directory entry 'ce', compare ce->name to\n+ * info->name + '/' + p->path + '/' if info->name is non-empty.\n+ * Compare ce->name to p->path + '/' otherwise. Note that\n+ * ce->name must end in a trailing '/' because it is a sparse\n+ * directory entry.\n+ */\n+static int sparse_dir_matches_path(const struct cache_entry *ce,\n+\t\t\t\t struct traverse_info *info,\n+\t\t\t\t const struct name_entry *p)\n+{\n+\tassert(S_ISSPARSEDIR(ce->ce_mode));\n+\tassert(ce->name[ce->ce_namelen - 1] == '/');\n+\n+\tif (info->namelen)\n+\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n+\t\t ce->name[info->namelen] == '/' &&\n+\t\t !strncmp(ce->name, info->name, info->namelen) &&\n+\t\t !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\treturn ce->ce_namelen == p->pathlen + 1 &&\n+\t !strncmp(ce->name, p->path, p->pathlen);\n+}\n+\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n+\tstruct cache_entry *ce;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n-\telse\n+\n+\t/*\n+\t * Check for a sparse-directory entry named \"path/\".\n+\t * Due to the input p->path not having a trailing\n+\t * slash, the negative 'pos' value overshoots the\n+\t * expected position, hence \"-2\" instead of \"-1\".\n+\t */\n+\tpos = -pos - 2;\n+\n+\tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n+\n+\t/*\n+\t * Due to lexicographic sorting and sparse directory\n+\t * entries ending with a trailing slash, our path as a\n+\t * sparse directory (e.g \"subdir/\") and\tour path as a\n+\t * file (e.g. \"subdir\") might be separated by other\n+\t * paths (e.g. \"subdir-\").\n+\t */\n+\twhile (pos >= 0) {\n+\t\tce = o->src_index->cache[pos];\n+\n+\t\tif (strncmp(ce->name, p->path, p->pathlen))\n+\t\t\treturn NULL;\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t sparse_dir_matches_path(ce, info, p))\n+\t\t\treturn ce;\n+\n+\t\tpos--;\n+\t}\n+\n+\treturn NULL;\n }\n \n static void debug_path(struct traverse_info *info)\n@@ -1266,6 +1339,21 @@ static void debug_unpack_callback(int n,\n \t\tdebug_name_entry(i, names + i);\n }\n \n+/*\n+ * Returns true if and only if the given cache_entry is a\n+ * sparse-directory entry that matches the given name_entry\n+ * from the tree walk at the given traverse_info.\n+ */\n+static int is_sparse_directory_entry(struct cache_entry *ce,\n+\t\t\t\t struct name_entry *name,\n+\t\t\t\t struct traverse_info *info)\n+{\n+\tif (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode))\n+\t\treturn 0;\n+\n+\treturn sparse_dir_matches_path(ce, info, name);\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -1352,9 +1440,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \t\t\t}\n \t\t}\n \n-\t\tif (traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n-\t\t\t\t\t names, info) < 0)\n+\t\tif (!is_sparse_directory_entry(src[0], names, info) &&\n+\t\t traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n+\t\t\t\t\t\t names, info) < 0) {\n \t\t\treturn -1;\n+\t\t}\n+\n \t\treturn mask;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"430048","messageId":"ad1715e3319b988fc6979970892a9f9c6ff5bf4a.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 13/16] status: use sparse-index throughout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:37Z","receivedAt":"2021-07-14T13:12:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBy testing 'git -c core.fsmonitor= status -uno', we can check for the\nsimplest index operations that can be made sparse-aware. The necessary\nimplementation details are already integrated with sparse-checkout, so\nmodify command_requires_full_index to be zero for cmd_status().\n\nIn refresh_index(), we loop through the index entries to refresh their\nstat() information. However, sparse directories have no stat()\ninformation to populate. Ignore these entries.\n\nThis allows 'git status' to no longer expand a sparse index to a full\none. This is further tested by dropping the \"-uno\" option and adding an\nuntracked file into the worktree.\n\nThe performance test p2000-sparse-checkout-operations.sh demonstrates\nthese improvements:\n\nTest HEAD~1 HEAD\n-----------------------------------------------------------------------------\n2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0%\n2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7%\n2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3%\n2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9%\n\nNote that since HEAD~1 was expanding the sparse index by parsing trees,\nit was artificially slower than the full index case. Thus, the 98%\nimprovement is misleading, and instead we should celebrate the 0.34s to\n0.05s improvement of 85%. This is more indicative of the peformance\ngains we are expecting by using a sparse index.\n\nNote: we are dropping the assignment of core.fsmonitor here. This is not\nnecessary for the test script as we are not altering the config any\nother way. Correct integration with FS Monitor will be validated in\nlater changes.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 3 +++\n read-cache.c | 10 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++----\n 3 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 190d215d43b..12f51db158a 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(builtin_status_usage, builtin_status_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tstatus_init_config(&s, git_status_config);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_status_options,\ndiff --git a/read-cache.c b/read-cache.c\nindex 0c3ac3cefc0..6a1337cc905 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1585,8 +1585,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(istate);\n+\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -1601,6 +1600,13 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t\tif (ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n+\t\t/*\n+\t\t * If this entry is a sparse directory, then there isn't\n+\t\t * any stat() information to update. Ignore the entry.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tcontinue;\n+\n \t\tif (pathspec && !ce_path_match(istate, ce, pathspec, seen))\n \t\t\tfiltered = 1;\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 375b0d35565..751f397cc7f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -511,12 +511,17 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n \ttest_region index convert_to_sparse trace2.txt &&\n-\ttest_region index ensure_full_index trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt\n+'\n \n-\trm trace2.txt &&\n+test_expect_success 'sparse-index is not expanded' '\n+\tinit_repos &&\n+\n+\trm -f trace2.txt &&\n+\techo >>sparse-index/untracked.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" status -uno &&\n-\ttest_region index ensure_full_index trace2.txt\n+\t\tgit -C sparse-index status &&\n+\ttest_region ! index ensure_full_index trace2.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"430049","messageId":"aa2258be302fc42763c2da2e3aeb7f66123ca5b9.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 14/16] wt-status: expand added sparse directory entries","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:38Z","receivedAt":"2021-07-14T13:12:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt is difficult, but possible, to get into a state where we intend to\nadd a directory that is outside of the sparse-checkout definition. Add a\ntest to t1092-sparse-checkout-compatibility.sh that demonstrates this\nusing a combination of 'git reset --mixed' and 'git checkout --orphan'.\n\nThis test failed before because the output of 'git status\n--porcelain=v2' would not match on the lines for folder1/:\n\n* The sparse-checkout repo (with a full index) would output each path\n name that is intended to be added.\n\n* The sparse-index repo would only output that \"folder1/\" is staged for\n addition.\n\nThe status should report the full list of files to be added, and so this\nsparse-directory entry should be expanded to a full list when reaching\nit inside the wt_status_collect_changes_initial() method. Use\nread_tree_at() to assist.\n\nSomehow, this loop over the cache entries was not guarded by\nensure_full_index() as intended.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++\n wt-status.c | 51 ++++++++++++++++++++++++\n 2 files changed, 84 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 751f397cc7f..2394c36d881 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -524,4 +524,37 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+test_expect_success 'reset mixed and checkout orphan' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout rename-out-to-in &&\n+\n+\t# Sparse checkouts do not agree with full checkouts about\n+\t# how to report a directory/file conflict during a reset.\n+\t# This command would fail with test_all_match because the\n+\t# full checkout reports \"T folder1/0/1\" while a sparse\n+\t# checkout reports \"D folder1/0/1\". This matches because\n+\t# the sparse checkouts skip \"adding\" the other side of\n+\t# the conflict.\n+\ttest_sparse_match git reset --mixed HEAD~1 &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\t# At this point, sparse-checkouts behave differently\n+\t# from the full-checkout.\n+\ttest_sparse_match git checkout --orphan new-branch &&\n+\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'add everything with deep new file' '\n+\tinit_repos &&\n+\n+\trun_on_sparse git sparse-checkout set deep/deeper1/deepest &&\n+\n+\trun_on_all touch deep/deeper1/x &&\n+\ttest_all_match git add . &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_done\ndiff --git a/wt-status.c b/wt-status.c\nindex 96db3e74962..0317baef87e 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \tclear_pathspec(&rev.prune_data);\n }\n \n+static int add_file_to_list(const struct object_id *oid,\n+\t\t\t struct strbuf *base, const char *path,\n+\t\t\t unsigned int mode, void *context)\n+{\n+\tstruct string_list_item *it;\n+\tstruct wt_status_change_data *d;\n+\tstruct wt_status *s = context;\n+\tstruct strbuf full_name = STRBUF_INIT;\n+\n+\tif (S_ISDIR(mode))\n+\t\treturn READ_TREE_RECURSIVE;\n+\n+\tstrbuf_add(&full_name, base->buf, base->len);\n+\tstrbuf_addstr(&full_name, path);\n+\tit = string_list_insert(&s->change, full_name.buf);\n+\td = it->util;\n+\tif (!d) {\n+\t\tCALLOC_ARRAY(d, 1);\n+\t\tit->util = d;\n+\t}\n+\n+\td->index_status = DIFF_STATUS_ADDED;\n+\t/* Leave {mode,oid}_head zero for adds. */\n+\td->mode_index = mode;\n+\toidcpy(&d->oid_index, oid);\n+\ts->committable = 1;\n+\tstrbuf_release(&full_name);\n+\treturn 0;\n+}\n+\n static void wt_status_collect_changes_initial(struct wt_status *s)\n {\n \tstruct index_state *istate = s->repo->index;\n@@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s)\n \t\t\tcontinue;\n \t\tif (ce_intent_to_add(ce))\n \t\t\tcontinue;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * This is a sparse directory entry, so we want to collect all\n+\t\t\t * of the added files within the tree. This requires recursively\n+\t\t\t * expanding the trees to find the elements that are new in this\n+\t\t\t * tree and marking them with DIFF_STATUS_ADDED.\n+\t\t\t */\n+\t\t\tstruct strbuf base = STRBUF_INIT;\n+\t\t\tstruct pathspec ps = { 0 };\n+\t\t\tstruct tree *tree = lookup_tree(istate->repo, &ce->oid);\n+\n+\t\t\tps.recursive = 1;\n+\t\t\tps.has_wildcard = 1;\n+\t\t\tps.max_depth = -1;\n+\n+\t\t\tstrbuf_add(&base, ce->name, ce->ce_namelen);\n+\t\t\tread_tree_at(istate->repo, tree, &base, &ps,\n+\t\t\t\t add_file_to_list, s);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tit = string_list_insert(&s->change, ce->name);\n \t\td = it->util;\n \t\tif (!d) {\n-- \ngitgitgadget\n\n"},{"id":"430050","messageId":"e749fd41cda25936812d1a6ab7354d4b41524375.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 11/16] diff-lib: handle index diffs with sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:35Z","receivedAt":"2021-07-14T13:13:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile comparing an index to a tree, we may see a sparse directory entry.\nIn this case, we should compare that portion of the tree to the tree\nrepresented by that entry. This could include a new tree which needs to\nbe expanded to a full list of added files. It could also include an\nexisting tree, in which case all of the changes inside are important to\ndescribe, including the modifications, additions, and deletions. Note\nthat the case where the tree has a path and the index does not remains\nidentical to before: the lack of a cache entry is the same with a sparse\nindex.\n\nUse diff_tree_oid() appropriately to compute the diff.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex c2ac9250fe9..f9eadc4fc1a 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tif (new_file && S_ISSPARSEDIR(new_file->ce_mode)) {\n+\t\tdiff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * New file in the index: it might actually be different in\n \t * the working tree.\n@@ -347,6 +352,20 @@ static int show_modified(struct rev_info *revs,\n \tunsigned dirty_submodule = 0;\n \tstruct index_state *istate = revs->diffopt.repo->index;\n \n+\tassert(S_ISSPARSEDIR(old_entry->ce_mode) ==\n+\t S_ISSPARSEDIR(new_entry->ce_mode));\n+\n+\t/*\n+\t * If both are sparse directory entries, then expand the\n+\t * modifications to the file level. If only one was a sparse\n+\t * directory, then they appear as an add and delete instead of\n+\t * a modification.\n+\t */\n+\tif (S_ISSPARSEDIR(new_entry->ce_mode)) {\n+\t\tdiff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt);\n+\t\treturn 0;\n+\t}\n+\n \tif (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing,\n \t\t\t &dirty_submodule, &revs->diffopt) < 0) {\n \t\tif (report_missing)\n-- \ngitgitgadget\n\n"},{"id":"430051","messageId":"b051c0847a53eed206493a296ef5c24718c7eb51.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 10/16] dir.c: accept a directory as part of cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:34Z","receivedAt":"2021-07-14T13:13:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen we have sparse directory entries in the index, we want to compare\nthat directory against sparse-checkout patterns. Those pattern matching\nalgorithms are built expecting a file path, not a directory path. This\nis especially important in the \"cone mode\" patterns which will match\nfiles that exist within the \"parent directories\" as well as the\nrecursive directory matches.\n\nIf path_matches_pattern_list() is given a directory, we can add a fake\nfilename (\"-\") to the directory and get the same results as before,\nassuming we are in cone mode. Since sparse index requires cone mode\npatterns, this is an acceptable assumption.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n dir.c | 24 +++++++++++++++++++-----\n 1 file changed, 19 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ebe5ec046e0..0c5264b3b20 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1376,7 +1376,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tstruct path_pattern *pattern;\n \tstruct strbuf parent_pathname = STRBUF_INIT;\n \tint result = NOT_MATCHED;\n-\tconst char *slash_pos;\n+\tsize_t slash_pos;\n \n \tif (!pl->use_cone_patterns) {\n \t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n@@ -1397,21 +1397,35 @@ enum pattern_match_result path_matches_pattern_list(\n \tstrbuf_addch(&parent_pathname, '/');\n \tstrbuf_add(&parent_pathname, pathname, pathlen);\n \n+\t/*\n+\t * Directory entries are matched if and only if a file\n+\t * contained immediately within them is matched. For the\n+\t * case of a directory entry, modify the path to create\n+\t * a fake filename within this directory, allowing us to\n+\t * use the file-base matching logic in an equivalent way.\n+\t */\n+\tif (parent_pathname.len > 0 &&\n+\t parent_pathname.buf[parent_pathname.len - 1] == '/') {\n+\t\tslash_pos = parent_pathname.len - 1;\n+\t\tstrbuf_add(&parent_pathname, \"-\", 1);\n+\t} else {\n+\t\tconst char *slash_ptr = strrchr(parent_pathname.buf, '/');\n+\t\tslash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0;\n+\t}\n+\n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n \t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n-\tslash_pos = strrchr(parent_pathname.buf, '/');\n-\n-\tif (slash_pos == parent_pathname.buf) {\n+\tif (!slash_pos) {\n \t\t/* include every file in root */\n \t\tresult = MATCHED;\n \t\tgoto done;\n \t}\n \n-\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\tstrbuf_setlen(&parent_pathname, slash_pos);\n \n \tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n \t\tresult = MATCHED;\n-- \ngitgitgadget\n\n"},{"id":"430052","messageId":"7f782e3fe5061e90e7130d463accfc1f0f59a020.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 12/16] status: skip sparse-checkout percentage with sparse-index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:36Z","receivedAt":"2021-07-14T13:13:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n'git status' began reporting a percentage of populated paths when\nsparse-checkout is enabled in 051df3cf (wt-status: show sparse\ncheckout status as well, 2020-07-18). This percentage is incorrect when\nthe index has sparse directories. It would also be expensive to\ncalculate as we would need to parse trees to count the total number of\npossible paths.\n\nAvoid the expensive computation by simplifying the output to only report\nthat a sparse checkout exists, without the percentage.\n\nThis change is the reason we use 'git status --porcelain=v2' in\nt1092-sparse-checkout-compatibility.sh. We don't want to ensure that\nthis message is equal across both modes, but instead just the important\ninformation about staged, modified, and untracked files are compared.\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++\n wt-status.c | 14 +++++++++++---\n wt-status.h | 1 +\n 3 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 2269f44e033..375b0d35565 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -218,6 +218,14 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status reports sparse-checkout' '\n+\tinit_repos &&\n+\tgit -C sparse-checkout status >full &&\n+\tgit -C sparse-index status >sparse &&\n+\ttest_i18ngrep \"You are in a sparse checkout with \" full &&\n+\ttest_i18ngrep \"You are in a sparse checkout.\" sparse\n+'\n+\n test_expect_success 'add, commit, checkout' '\n \tinit_repos &&\n \ndiff --git a/wt-status.c b/wt-status.c\nindex 42b67357169..96db3e74962 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s,\n \tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED)\n \t\treturn;\n \n-\tstatus_printf_ln(s, color,\n-\t\t\t _(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n-\t\t\t s->state.sparse_checkout_percentage);\n+\tif (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX)\n+\t\tstatus_printf_ln(s, color, _(\"You are in a sparse checkout.\"));\n+\telse\n+\t\tstatus_printf_ln(s, color,\n+\t\t\t\t_(\"You are in a sparse checkout with %d%% of tracked files present.\"),\n+\t\t\t\ts->state.sparse_checkout_percentage);\n \twt_longstatus_print_trailer(s);\n }\n \n@@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r,\n \t\treturn;\n \t}\n \n+\tif (r->index->sparse_index) {\n+\t\tstate->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX;\n+\t\treturn;\n+\t}\n+\n \tfor (i = 0; i < r->index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = r->index->cache[i];\n \t\tif (ce_skip_worktree(ce))\ndiff --git a/wt-status.h b/wt-status.h\nindex 0d32799b28e..ab9cc9d8f03 100644\n--- a/wt-status.h\n+++ b/wt-status.h\n@@ -78,6 +78,7 @@ enum wt_status_format {\n };\n \n #define SPARSE_CHECKOUT_DISABLED -1\n+#define SPARSE_CHECKOUT_SPARSE_INDEX -2\n \n struct wt_status_state {\n \tint merge_in_progress;\n-- \ngitgitgadget\n\n"},{"id":"430053","messageId":"1d4b1f8aea0183a00f8c20226cca5e9c387e4adc.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 15/16] fsmonitor: integrate with sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:39Z","receivedAt":"2021-07-14T13:13:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we need to expand a sparse-index into a full one, then the FS Monitor\nbitmap is going to be incorrect. Ensure that we start fresh at such an\nevent.\n\nWhile this is currently a performance drawback, the eventual hope of the\nsparse-index feature is that these expansions will be rare and hence we\nwill be able to keep the FS Monitor data accurate across multiple Git\ncommands.\n\nThese tests are added to demonstrate that the behavior is the same\nacross a full index and a sparse index, but also that file modifications\nto a tracked directory outside of the sparse cone will trigger\nensure_full_index().\n\nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 7 ++++++\n t/t7519-status-fsmonitor.sh | 49 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex ef53bd2198b..53c8f711ccc 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate)\n \tcache_tree_free(&istate->cache_tree);\n \tcache_tree_update(istate, 0);\n \n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n+\n \tistate->sparse_index = 1;\n \ttrace2_region_leave(\"index\", \"convert_to_sparse\", istate->repo);\n \treturn 0;\n@@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate)\n \tistate->cache = full->cache;\n \tistate->cache_nr = full->cache_nr;\n \tistate->cache_alloc = full->cache_alloc;\n+\tistate->fsmonitor_has_run_once = 0;\n+\tFREE_AND_NULL(istate->fsmonitor_dirty);\n+\tFREE_AND_NULL(istate->fsmonitor_last_update);\n \n \tstrbuf_release(&base);\n \tfree(full);\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 637391c6ce4..deea88d4431 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -73,6 +73,7 @@ test_expect_success 'setup' '\n \texpect*\n \tactual*\n \tmarker*\n+\ttrace2*\n \tEOF\n '\n \n@@ -383,4 +384,52 @@ test_expect_success 'status succeeds after staging/unstaging' '\n \t)\n '\n \n+# Usage:\n+# check_sparse_index_behavior [!]\n+# If \"!\" is supplied, then we verify that we do not call ensure_full_index\n+# during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n+check_sparse_index_behavior () {\n+\tgit status --porcelain=v2 >expect &&\n+\tgit sparse-checkout init --cone --sparse-index &&\n+\tgit sparse-checkout set dir1 dir2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit status --porcelain=v2 >actual &&\n+\ttest_region $1 index ensure_full_index trace2.txt &&\n+\ttest_region fsm_hook query trace2.txt &&\n+\ttest_cmp expect actual &&\n+\trm trace2.txt &&\n+\tgit sparse-checkout disable\n+}\n+\n+test_expect_success 'status succeeds with sparse index' '\n+\tgit reset --hard &&\n+\n+\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\tEOF\n+\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tcheck_sparse_index_behavior ! &&\n+\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior ! &&\n+\n+\tcp -r dir1 dir1a &&\n+\tgit add dir1a &&\n+\tgit commit -m \"add dir1a\" &&\n+\n+\t# This one modifies outside the sparse-checkout definition\n+\t# and hence we expect to expand the sparse-index.\n+\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\t\tprintf \"last_update_token\\0\"\n+\t\tprintf \"dir1a/modified\\0\"\n+\tEOF\n+\tcheck_sparse_index_behavior\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"430054","messageId":"45861118991c20b0aa415b4fb1205a3a2738cbe3.1626268360.git.gitgitgadget@gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"[PATCH v9 16/16] t1092: document bad sparse-checkout behavior","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-14T13:12:40Z","receivedAt":"2021-07-14T13:13:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThere are several situations where a repository with sparse-checkout\nenabled will act differently than a normal repository, and in ways that\nare not intentional. The test t1092-sparse-checkout-compatibility.sh\ndocuments some of these deviations, but a casual reader might think\nthese are intentional behavior changes.\n\nAdd comments on these tests that make it clear that these behaviors\nshould be updated. Using 'NEEDSWORK' helps contributors find that these\nare potential areas for improvement.\n\nHelped-by: Elijah Newren \nReviewed-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 2394c36d881..cabbd42e339 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -392,8 +392,8 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# TODO: reset currently does not behave as expected when in a\n-# sparse-checkout.\n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_failure 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n@@ -403,8 +403,8 @@ test_expect_failure 'checkout and reset (mixed)' '\n \ttest_all_match git reset update-folder2\n '\n \n-# Ensure that sparse-index behaves identically to\n-# sparse-checkout with a full index.\n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_success 'checkout and reset (mixed) [sparse]' '\n \tinit_repos &&\n \n@@ -524,6 +524,8 @@ test_expect_success 'sparse-index is not expanded' '\n \ttest_region ! index ensure_full_index trace2.txt\n '\n \n+# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n+# in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n"},{"id":"430057","messageId":"CABPp-BEJFfTpmekgcsEji4hH+zswPCPxTZS+3sbnDhwGy2kvqA@mail.gmail.com","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 00/16] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-14T15:08:27Z","receivedAt":"2021-07-14T15:08:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jul 14, 2021 at 6:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' very fast when a sparse-index is enabled on a repository with\n> cone-mode sparse-checkout (and a small populated set).\n>\n> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n> The latter branch is needed because it changes the behavior of 'git add'\n> around sparse entries, which changes the expectations of a test added in\n> patch 1.\n>\n> The approach here is to audit the places where ensure_full_index() pops up\n> while doing normal commands with pathspecs within the sparse-checkout\n> definition. Each of these are checked and tested. In the end, the\n> sparse-index is integrated with these features:\n>\n> * git status\n> * FS Monitor index extension.\n>\n> The performance tests in p2000-sparse-operations.sh improve by 95% or more,\n> even when compared with the full-index cases, not just the sparse-index\n> cases that previously had extra overhead.\n>\n> Hopefully this is the first example of how ds/sparse-index-protections has\n> done the basic work to do these conversions safely, making them look easier\n> than they seemed when starting this adventure.\n>\n> Thanks, -Stolee\n>\n>\n> Update in V9\n> ============\n>\n> * Fixed typo.\n>\n> * All patches are marked as Reviewed-by Elijah. Thanks for the careful\n> review!\n\nThanks for all the hard work and pushing this feature forward!\n"},{"id":"430124","messageId":"xmqqim1cvdh6.fsf@gitster.g","threadId":"55489","inReplyTo":"pull.932.v9.git.1626268360.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 00/16] Sparse-index: integrate with status","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-07-14T20:37:25Z","receivedAt":"2021-07-14T20:37:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> status' very fast when a sparse-index is enabled on a repository with\n> cone-mode sparse-checkout (and a small populated set).\n\nThe first payoff turning out to be a long time coming ;-)\n\n> This is based on ds/sparse-index-protections AND mt/add-rm-sparse-checkout.\n> The latter branch is needed because it changes the behavior of 'git add'\n> around sparse entries, which changes the expectations of a test added in\n> patch 1.\n>\n> The approach here is to audit the places where ensure_full_index() pops up\n> while doing normal commands with pathspecs within the sparse-checkout\n> definition. Each of these are checked and tested. In the end, the\n> sparse-index is integrated with these features:\n>\n> * git status\n> * FS Monitor index extension.\n\nI said this alreayd but I think the approach makes sense. One\nrelated tangent that makes me wonder is if we can teach \"ls-files\"\nto help scriptors with a mode that does not pretend we have the full\nindex (i.e. when asked to show sparse index state, \"ls-files\n--sparse\" would show the same output as \"ls-files --stage\" without\nexpanding each tree entry in the index into its flattend list of\npaths).\n\nThanks.\n"},{"id":"430178","messageId":"CABPp-BHP_jbQQX=5XRF3fVYvDGBsvWpuXeFg0RFePD47L_8U8w@mail.gmail.com","threadId":"55489","inReplyTo":"xmqqim1cvdh6.fsf@gitster.g","subject":"Re: [PATCH v9 00/16] Sparse-index: integrate with status","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-15T02:41:38Z","receivedAt":"2021-07-15T02:41:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jul 14, 2021 at 1:37 PM Junio C Hamano wrote:\n>\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n> > This is the first \"payoff\" series in the sparse-index work. It makes 'git\n> > status' very fast when a sparse-index is enabled on a repository with\n> > cone-mode sparse-checkout (and a small populated set).\n>\n> The first payoff turning out to be a long time coming ;-)\n\nTrue, but much, much less time than it took between when I started the\nmerge machinery overhaul and its first big payoffs. ;-)\n"}]}