{"thread":{"id":"56614","subject":"[PATCH 0/7] Sparse Index: integrate with reset","startedAt":"2021-09-30T14:51:05Z","lastAt":"2021-12-01T20:35:24Z","messageCount":114,"participants":["Victoria Dye via GitGitGadget","Kevin Willford via GitGitGadget","Junio C Hamano","Taylor Blau","Victoria Dye","Bagas Sanjaya","Ævar Arnfjörð Bjarmason","Elijah Newren","Phillip Wood","Derrick Stolee","Emily Shaffer"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"437545","messageId":"pull.1048.git.1633013461.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":null,"subject":"[PATCH 0/7] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:54Z","receivedAt":"2021-09-30T14:51:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate an overall ~70% execution time reduction across\nall tested usages of git reset using a sparse index:\n\nTest before after \n------------------------------------------------------------------------\n2000.22: git reset (full-v3) 0.48 0.51 +6.3% \n2000.23: git reset (full-v4) 0.47 0.50 +6.4% \n2000.24: git reset (sparse-v3) 0.93 0.30 -67.7%\n2000.25: git reset (sparse-v4) 0.94 0.29 -69.1%\n2000.26: git reset --hard (full-v3) 0.69 0.68 -1.4% \n2000.27: git reset --hard (full-v4) 0.75 0.68 -9.3% \n2000.28: git reset --hard (sparse-v3) 1.29 0.34 -73.6%\n2000.29: git reset --hard (sparse-v4) 1.31 0.34 -74.0%\n2000.30: git reset -- does-not-exist (full-v3) 0.54 0.51 -5.6% \n2000.31: git reset -- does-not-exist (full-v4) 0.54 0.52 -3.7% \n2000.32: git reset -- does-not-exist (sparse-v3) 1.02 0.31 -69.6%\n2000.33: git reset -- does-not-exist (sparse-v4) 1.07 0.30 -72.0%\n\n\n[1] microsoft@6b8a074\n\nThanks! -Victoria\n\nKevin Willford (1):\n reset: behave correctly with sparse-checkout\n\nVictoria Dye (6):\n sparse-index: update command for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n builtin/reset.c | 77 ++++++++++++-\n cache-tree.c | 43 ++++++-\n cache.h | 10 ++\n read-cache.c | 22 ++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 139 ++++++++++++++++++++++-\n t/t7114-reset-sparse-checkout.sh | 61 ++++++++++\n unpack-trees.c | 23 +++-\n 8 files changed, 353 insertions(+), 25 deletions(-)\n create mode 100755 t/t7114-reset-sparse-checkout.sh\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n-- \ngitgitgadget\n"},{"id":"437547","messageId":"65905bf4e001118e8b9ced95c1bcecbacb6334ac.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 1/7] reset: behave correctly with sparse-checkout","fromName":"Kevin Willford via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:55Z","receivedAt":"2021-09-30T14:51:06Z","isPatch":true,"sender":{"key":"kewillf@microsoft.com","avatar":null},"body":"From: Kevin Willford \n\nWhen using the sparse checkout feature, 'git reset' will add entries to\nthe index that will have the skip-worktree bit off but will leave the\nworking directory empty. File data is lost because the index version of\nthe files has been changed but there is nothing that is in the working\ndirectory. This will cause the next 'git status' call to show either\ndeleted for files modified or deleting or nothing for files added. The\nadded files should be shown as untracked and modified files should be\nshown as modified.\n\nTo fix this when the reset is running if there is not a file in the\nworking directory and if it will be missing with the new index entry or\nwas not missing in the previous version, we create the previous index\nversion of the file in the working directory so that status will report\ncorrectly and the files will be availble for the user to deal with.\n\nThis fixes a documented failure from t1092 that was created in 19a0acc\n(t1092: test interesting sparse-checkout scenarios, 2021-01-23).\n\nSigned-off-by: Kevin Willford \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 39 +++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 4 +-\n t/t7114-reset-sparse-checkout.sh | 61 ++++++++++++++++++++++++\n 3 files changed, 101 insertions(+), 3 deletions(-)\n create mode 100755 t/t7114-reset-sparse-checkout.sh\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..8ffcd713720 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,8 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n+#include \"entry.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -127,12 +129,49 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tstruct diff_options *opt, void *data)\n {\n \tint i;\n+\tint pos;\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n+\t\tstruct diff_filespec *two = q->queue[i]->two;\n \t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tint was_missing = !two->mode && is_null_oid(&two->oid);\n \t\tstruct cache_entry *ce;\n+\t\tstruct cache_entry *ce_before;\n+\t\tstruct checkout state = CHECKOUT_INIT;\n+\n+\t\t/*\n+\t\t * When using the sparse-checkout feature the cache entries\n+\t\t * that are added here will not have the skip-worktree bit\n+\t\t * set. Without this code there is data that is lost because\n+\t\t * the files that would normally be in the working directory\n+\t\t * are not there and show as deleted for the next status.\n+\t\t * In the case of added files, they just disappear.\n+\t\t *\n+\t\t * We need to create the previous version of the files in\n+\t\t * the working directory so that they will have the right\n+\t\t * content and the next status call will show modified or\n+\t\t * untracked files correctly.\n+\t\t */\n+\t\tif (core_apply_sparse_checkout && !file_exists(two->path)) {\n+\t\t\tpos = cache_name_pos(two->path, strlen(two->path));\n+\t\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) &&\n+\t\t\t (is_missing || !was_missing)) {\n+\t\t\t\tstate.force = 1;\n+\t\t\t\tstate.refresh_cache = 1;\n+\t\t\t\tstate.istate = &the_index;\n+\n+\t\t\t\tce_before = make_cache_entry(&the_index, two->mode,\n+\t\t\t\t\t\t\t &two->oid, two->path,\n+\t\t\t\t\t\t\t 0, 0);\n+\t\t\t\tif (!ce_before)\n+\t\t\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n+\t\t\t\t\t\ttwo->path);\n+\n+\t\t\t\tcheckout_entry(ce_before, &state, NULL, NULL);\n+\t\t\t}\n+\t\t}\n \n \t\tif (is_missing && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..c5977152661 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,9 +459,7 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\ndiff --git a/t/t7114-reset-sparse-checkout.sh b/t/t7114-reset-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..a8029707fb1\n--- /dev/null\n+++ b/t/t7114-reset-sparse-checkout.sh\n@@ -0,0 +1,61 @@\n+#!/bin/sh\n+\n+test_description='reset when using a sparse-checkout'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_tick &&\n+\techo \"checkout file\" >c &&\n+\techo \"modify file\" >m &&\n+\techo \"delete file\" >d &&\n+\tgit add . &&\n+\tgit commit -m \"initial commit\" &&\n+\techo \"added file\" >a &&\n+\techo \"modification of a file\" >m &&\n+\tgit rm d &&\n+\tgit add . &&\n+\tgit commit -m \"second commit\" &&\n+\tgit checkout -b endCommit\n+'\n+\n+test_expect_success 'reset when there is a sparse-checkout' '\n+\techo \"/c\" >.git/info/sparse-checkout &&\n+\ttest_config core.sparsecheckout true &&\n+\tgit checkout -B resetBranch &&\n+\ttest_path_is_missing m &&\n+\ttest_path_is_missing a &&\n+\ttest_path_is_missing d &&\n+\tgit reset HEAD~1 &&\n+\techo \"checkout file\" >expect &&\n+\ttest_cmp expect c &&\n+\techo \"added file\" >expect &&\n+\ttest_cmp expect a &&\n+\techo \"modification of a file\" >expect &&\n+\ttest_cmp expect m &&\n+\ttest_path_is_missing d\n+'\n+\n+test_expect_success 'reset after deleting file without skip-worktree bit' '\n+\tgit checkout -f endCommit &&\n+\tgit clean -xdf &&\n+\tcat >.git/info/sparse-checkout <<-\\EOF &&\n+\t/c\n+\t/m\n+\tEOF\n+\ttest_config core.sparsecheckout true &&\n+\tgit checkout -B resetAfterDelete &&\n+\ttest_path_is_file m &&\n+\ttest_path_is_missing a &&\n+\ttest_path_is_missing d &&\n+\trm -f m &&\n+\tgit reset HEAD~1 &&\n+\techo \"checkout file\" >expect &&\n+\ttest_cmp expect c &&\n+\techo \"added file\" >expect &&\n+\ttest_cmp expect a &&\n+\ttest_path_is_missing m &&\n+\ttest_path_is_missing d\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"437546","messageId":"a1fa7c080aed2056afaad6415186c125c04a80cb.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:56Z","receivedAt":"2021-09-30T14:51:07Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn anticipation of multiple commands being fully integrated with sparse\nindex, update the test for index expand and collapse for non-sparse index\nintegrated commands to use `mv`.\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c5977152661..aed8683e629 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"437548","messageId":"d033c5e365fec8e3e0c5c67c2fadbc87e15dd16c.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 3/7] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:57Z","receivedAt":"2021-09-30T14:51:14Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs both inside and outside of the sparse checkout definition. New\nperformance test cases exercise various execution paths for `reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 107 +++++++++++++++++++++++\n 2 files changed, 110 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex aed8683e629..e36fb18098d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -479,6 +479,113 @@ test_expect_success 'checkout and reset (mixed) [sparse]' '\n \ttest_sparse_match git reset update-folder2\n '\n \n+# NEEDSWORK: with mixed reset, files with differences between HEAD and \n+# will be added to the work tree even if outside the sparse checkout\n+# definition, and even if the file is modified to a state of having no local\n+# changes. The file is \"re-ignored\" if a hard reset is executed. We may want to\n+# change this behavior in the future and enforce that files are not written\n+# outside of the sparse checkout definition.\n+test_expect_success 'checkout and mixed reset file tracking [sparse]' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset update-folder1 &&\n+\ttest_all_match git reset update-deep &&\n+\n+\t# At this point, there are no changes in the working tree. However,\n+\t# folder1/a now exists locally (even though it is outside of the sparse\n+\t# paths).\n+\trun_on_sparse test_path_exists folder1 &&\n+\n+\trun_on_all rm folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_sparse test_path_is_missing folder1 &&\n+\ttest_path_exists full-checkout/folder1\n+'\n+\n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with sparse directory pathspec outside definition' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- folder1 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with pathspec match in sparse directory' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- folder1/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- \\*/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"437549","messageId":"2d63a250637d62f8966b8b78b4f57b7c6784c967.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 4/7] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:58Z","receivedAt":"2021-09-30T14:51:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\n`reset --soft` does not modify the index, so no compatibility changes are\nneeded for it to function without expanding the index. For all other reset\nmodes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is\nexplicitly expanded with `ensure_full_index` to maintain current behavior.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` does need to\nbe run after enabling sparse index for the command and before resetting.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 8ffcd713720..92b9a3815c7 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -205,6 +205,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -282,9 +283,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -430,6 +428,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"437550","messageId":"e919e6d3270889d14a32ff49c391ec484bab6836.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 5/7] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:50:59Z","receivedAt":"2021-09-30T14:51:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn order to accurately reconstruct the cache tree in `prime_cache_tree_rec`,\nthe function must determine whether the currently-processing directory in\nthe tree is sparse or not. If it is not sparse, the tree is parsed and\nsubtree recursively constructed. If it is sparse, no subtrees are added to\nthe tree and the entry count is set to 1 (representing the sparse directory\nitself).\n\nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 44 +++++++++++++++++++++---\n cache.h | 10 ++++++\n read-cache.c | 22 ++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 ++++++--\n 4 files changed, 78 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..9021669d682 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,29 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct repository *r,\n+\t\t\t\t\tstruct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree,\n+\t\t\t\t\tstruct strbuf *tree_path)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+\treturn;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n+\tstruct strbuf subtree_path = STRBUF_INIT;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,27 +771,49 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\t\t\tstrbuf_reset(&subtree_path);\n+\t\t\tstrbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1);\n+\t\t\tstrbuf_addbuf(&subtree_path, tree_path);\n+\t\t\tstrbuf_add(&subtree_path, entry.path, entry.pathlen);\n+\t\t\tstrbuf_addch(&subtree_path, '/');\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, subtree_path.buf, subtree_path.len))\n+\t\t\t\tprime_cache_tree_sparse_dir(r, sub->cache_tree, subtree, &subtree_path);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, &subtree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n \tit->entry_count = cnt;\n+\n+\tstrbuf_release(&subtree_path);\n }\n \n void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..ea1166895f8 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -551,7 +551,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tint search_sparse)\n {\n \tint first, last;\n \n@@ -570,7 +573,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_sparse && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +589,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_sparse);\n \t\t}\n \t}\n \n@@ -595,7 +598,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, 1);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, 0) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1230,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, 1);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1330,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1365,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e36fb18098d..0b6ff0de17d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -786,9 +786,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -798,6 +798,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"437551","messageId":"e7cda32efb63c63cf60513ec447418fd97ab3fa9.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 6/7] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:51:00Z","receivedAt":"2021-09-30T14:51:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nSparse directory entries are \"diffed\" as trees in `diff_cache` (used\ninternally by `reset --mixed`), following a code path separate from\nindividual file handling. The use of `diff_tree_oid` there requires setting\nexplicit `change` and `add_remove` functions to process the internal\ncontents of a sparse directory.\n\nAdditionally, the `recursive` diff option handles cases in which `reset\n--mixed` must diff/merge files that are nested multiple levels deep in a\nsparse directory.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 30 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++-\n 2 files changed, 41 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 92b9a3815c7..2d95ce76f20 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -196,6 +196,8 @@ static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t int intent_to_add)\n {\n \tstruct diff_options opt;\n+\tunsigned int i;\n+\tchar *skip_worktree_seen = NULL;\n \n \tmemset(&opt, 0, sizeof(opt));\n \tcopy_pathspec(&opt.pathspec, pathspec);\n@@ -203,9 +205,35 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\t/*\n+\t * When pathspec is given for resetting a cone-mode sparse checkout, it may\n+\t * identify entries that are nested in sparse directories, in which case the\n+\t * index should be expanded. For the sake of efficiency, this check is\n+\t * overly-cautious: anything with a wildcard or a magic prefix requires\n+\t * expansion, as well as literal paths that aren't in the sparse checkout\n+\t * definition AND don't match any directory in the index.\n+\t */\n+\tif (pathspec->nr && the_index.sparse_index) {\n+\t\tif (pathspec->magic || pathspec->has_wildcard) {\n+\t\t\tensure_full_index(&the_index);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\t\t\tif (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) &&\n+\t\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) {\n+\t\t\t\t\tensure_full_index(&the_index);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfree(skip_worktree_seen);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0b6ff0de17d..c9b9ef4992c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -801,14 +801,25 @@ test_expect_success 'sparse-index is not expanded' '\n \tfor ref in update-deep update-folder1 update-folder2 update-deep\n \tdo\n \t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --mixed $ref\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n-\tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so it will be reset without needing to\n+\t# expand the full index.\n+\tensure_not_expanded reset --hard update-folder1 &&\n+\tensure_not_expanded reset base -- folder1 &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"437552","messageId":"8637ec1660ebb1b19f1f73c38debb7b098187e6f.1633013461.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH 7/7] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-30T14:51:01Z","receivedAt":"2021-09-30T14:51:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain\nthe benefit `cache_bottom` provides in non-sparse index cases, a separate\n`hint` position indicates the first position `next_cache_entry` should\nsearch, updated each execution with a new position. The performance of `git\nreset -- does-not-exist` (testing the \"worst case\" in which all entries in\nthe index are unpacked with `next_cache_entry`) is significantly improved\nfor the sparse index case:\n\nTest before after\n------------------------------------------------------\n(full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n(full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n(sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1%\n(sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3%\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"437565","messageId":"xmqqpmspc3pn.fsf@gitster.g","threadId":"56614","inReplyTo":"65905bf4e001118e8b9ced95c1bcecbacb6334ac.1633013461.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] reset: behave correctly with sparse-checkout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-30T18:34:28Z","receivedAt":"2021-09-30T18:34:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Kevin Willford via GitGitGadget\" writes:\n\n> @@ -127,12 +129,49 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> \t\tstruct diff_options *opt, void *data)\n> {\n> \tint i;\n> +\tint pos;\n> \tint intent_to_add = *(int *)data;\n> \n> \tfor (i = 0; i < q->nr; i++) {\n> \t\tstruct diff_filespec *one = q->queue[i]->one;\n> +\t\tstruct diff_filespec *two = q->queue[i]->two;\n> \t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n> +\t\tint was_missing = !two->mode && is_null_oid(&two->oid);\n\nNot a problem introduced by this patch per-se, but is_missing is a\ncounter-intuitive name for what the boolean wants to represent, I\nthink, which was brought in by b4b313f9 (reset: support \"--mixed\n--intent-to-add\" mode, 2014-02-04). Before the commit, we used to\nsay\n\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n\t\tif (one->mode && !is_null_sha1(one->sha1)) {\n\t\t\t... create ce out of one and add to the\tindex ...\n\t\t} else\n \t\t\tremove_file_from_cache(one->path);\n\t\t...\n\ni.e. \"if one is not missing, create a ce and add it, otherwise\nremove the path\".\n\nIt should have been called \"one_is_missing\" if we wanted to\nliterally express the condition the code checked, but an even better\nname would have been given after the intent of what the code wants\nto do with the information. If the resetted-to tree (that is what\n'one' side of the comparison in diff_cache() is) has a valid blob,\nwe want it to be in the index, and otherwise, we do not want it in\nthe index.\n\nNow, the patch makes things worse and I had to do the above digging\nto see why the new code is even more confusing. The 'two' side of\nthe comparison is what is in the to-be-corrected-by-reset index.\n\"was_missing\" in contrast to \"is_missing\" makes it sound as if it\nwas the state before whatever \"is_missing\" tries to represent, but\nthat is not what is happening. \"is_missing\" does not mean \"the\nentry is currently not there in the index\", but \"was_missing\" does\nmean exactly that: \"the entry is currently not there in the index\".\n\nThere isn't any \"was\" missing about it. It \"is\" missing in the\nindex. Instead of renaming, I wonder if we can do without this new\nvariable. Let's read on the patch.\n\nAlso, now the code uses both sides of the filepair, we must double\ncheck that our do_diff_cache() is *not* doing any rename detection.\nIt might be even prudent to ensure that \n\n\tif (strcmp(one->path, two->path))\n\t\tBUG(\"reset drove diff-cache with rename detection\");\n\nbut it might be with too much paranoia. I dunno.\n\n> \t\tstruct cache_entry *ce;\n> +\t\tstruct cache_entry *ce_before;\n> +\t\tstruct checkout state = CHECKOUT_INIT;\n\nThese two new variables do not need this wide a scope, I would\nthink. Shouldn't it be inside the body of the new \"if\" statement\nthis patch adds?\n\n> +\t\t/*\n> +\t\t * When using the sparse-checkout feature the cache entries\n> +\t\t * that are added here will not have the skip-worktree bit\n> +\t\t * set. Without this code there is data that is lost because\n> +\t\t * the files that would normally be in the working directory\n> +\t\t * are not there and show as deleted for the next status.\n> +\t\t * In the case of added files, they just disappear.\n> +\t\t *\n> +\t\t * We need to create the previous version of the files in\n> +\t\t * the working directory so that they will have the right\n> +\t\t * content and the next status call will show modified or\n> +\t\t * untracked files correctly.\n> +\t\t */\n> +\t\tif (core_apply_sparse_checkout && !file_exists(two->path)) {\n\nIn a sparsely checked out working tree, there is nothing in the\nworking tree at the path. It may be because it is sparse and we\ndidn't want to have anything there, or it may be because the user\nwanted to get rid of it and said \"rm path\" (not \"git rm path\") and\nthis part of the tree were of interest even if the sparse checkout\nfeature was used to hide other parts of the tree. With the above\ntwo checks alone, we cannot tell which. Let's read on.\n\n> +\t\t\tpos = cache_name_pos(two->path, strlen(two->path));\n\nWe check the index to see if there is an entry for it. I suspect\nthat because we need to do this check anyway, we shouldn't even have\nto look at 'two' (and add a new 'was_missing' variable), because\n'one' and 'two' came from a comparison between the resetted-to tree\nobject and the current index, and if cache_name_pos() for the path\n(we can use 'one->path') says there is an entry in the index, by\ndefinition, 'two' would not be showing a \"removed\" state (i.e. \"the\nresetted-to tree had it, the index does not\" is what \"was_missing\"\nwants to say).\n\nSo I wonder if it is better to\n\n - use one->path for !file_exists() above and cache_name_pos() here\n instead of two->path.\n\n - drop the confusingly named 'was_missing', because (pos < 0) is\n equivalent to it after this point, and we didn't need it up to\n this point.\n\n> +\t\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) &&\n\nAnd we do find an entry for it. So this path is not something\nsparse cone specifies not to check out (otherwise we would have a\ntree-like entry that covers this path in the index and not an entry\nfor this specific path)?\n\nAnyway, if it is marked with the skip-worktree bit, does that mean\nthere is no risk that the reason why two->path does not exist in the\nworking tree is because we earlier gave it in the working tree but\nit was later removed by the user? Just making sure that we are not\nbreaking the end-user's wish that the path should be removed by\nresurrecting it in the working tree with a new call to\ncheckout_entry().\n\n> +\t\t\t (is_missing || !was_missing)) {\n\nAnd in such a case, if the resetted-to tree says we shouldn't have\nthe path in the resulting index, or if the original state in the\nindex had this path (but because (0 <= pos) must be true for us to \nreach this point, I am not sure if \"was_missing\" can ever be true\nhere), then do the following, which is ...\n\n> +\t\t\t\tstate.force = 1;\n> +\t\t\t\tstate.refresh_cache = 1;\n> +\t\t\t\tstate.istate = &the_index;\n> +\n> +\t\t\t\tce_before = make_cache_entry(&the_index, two->mode,\n> +\t\t\t\t\t\t\t &two->oid, two->path,\n> +\t\t\t\t\t\t\t 0, 0);\n> +\t\t\t\tif (!ce_before)\n> +\t\t\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n> +\t\t\t\t\t\ttwo->path);\n> +\n> +\t\t\t\tcheckout_entry(ce_before, &state, NULL, NULL);\n\n... to resurrect the last \"git add\"ed state from the index and write\nit out to the working tree. As I suspected, ce_before and state\nshould be scoped inside this block and not visible outside, no?\n\nI am not sure why this behaviour is desirable. The \"mixed\" reset\nshould not have to touch the working tree in the first place.\n\nThe large comment before this block says \"... will not have the\nskip-worktree bit set\", but we are dealing with a case where the\noriginal index had a cache entry there with skip-worktree bit set,\nso isn't the more desirable outcome that the cache entry added back\nto the index has the skip-worktree bit still set and there is no\nworking tree file that the user did not desire to have?\n\nAnd isn't it the matter of preserving the skip-worktree bit when the\ncode in the post context of this hunk this patch did not touch adds\nthe entry back to the index?\n\n> +\t\t\t}\n> +\t\t}\n> \n> \t\tif (is_missing && !intent_to_add) {\n> \t\t\tremove_file_from_cache(one->path);\n\nIf we look at the code after this point, we do use \"is_missing\"\ninformation to tweak ce->ce_flags with the intent-to-add bit.\n\nPerhaps we can do a similar tweak to the cache entry to mark it with\nskip-worktree bit if the index had a cache entry at the path with\nthe bit set? The code that needs to do so would only have to\nremember if the one->path is in the current index and the cache\nentry for the path has the skip-worktree bit in the body of the new\nif() statement about (core_apply_sparse_checkout && !file_exists())\nadded by this patch (I am not sure if !file_exists() even matters,\nthough, as the approach I am suggesting is to preserve the skip bit\nand not disturb the working tree files at all).\n\nThanks.\n\n\n\n\n"},{"id":"437569","messageId":"YVYNW13aVADaR+g6@nand.local","threadId":"56614","inReplyTo":"a1fa7c080aed2056afaad6415186c125c04a80cb.1633013461.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-09-30T19:17:47Z","receivedAt":"2021-09-30T19:17:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 30, 2021 at 02:50:56PM +0000, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n>\n> In anticipation of multiple commands being fully integrated with sparse\n> index, update the test for index expand and collapse for non-sparse index\n> integrated commands to use `mv`.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index c5977152661..aed8683e629 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> \tinit_repos &&\n>\n> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n> +\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n\nDouble-checking my understanding as somebody who is not so familiar with\nt1092: this test is to ensure that commands which don't yet understand\nthe sparse index can temporarily expand it in order to do their work?\n\nIf so, makes sense to me. And renaming 'a' to 'b' is arbitrary and fine\nto do since we end up recreating the sparse-index repository each time\nvia init_repos.\n\nLooks good to me.\n\nThanks,\nTaylor\n"},{"id":"437574","messageId":"6e9843d5-bc04-e69c-9e53-7bf18ca30fc8@github.com","threadId":"56614","inReplyTo":"YVYNW13aVADaR+g6@nand.local","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-09-30T20:11:20Z","receivedAt":"2021-09-30T20:11:24Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Taylor Blau wrote:\n> On Thu, Sep 30, 2021 at 02:50:56PM +0000, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye \n>>\n>> In anticipation of multiple commands being fully integrated with sparse\n>> index, update the test for index expand and collapse for non-sparse index\n>> integrated commands to use `mv`.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> t/t1092-sparse-checkout-compatibility.sh | 2 +-\n>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index c5977152661..aed8683e629 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n>> \tinit_repos &&\n>>\n>> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n>> -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n>> +\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n> \n> Double-checking my understanding as somebody who is not so familiar with\n> t1092: this test is to ensure that commands which don't yet understand\n> the sparse index can temporarily expand it in order to do their work?\n\nExactly - if a command doesn't explicitly enable use of the sparse index by\nsetting `command_requires_full_index` to 0, the index is expanded if/when it\nis first read during the command's execution and collapsed if/when it is\nwritten to disk. This test makes sure that mechanism works as intended.\n\n-Victoria\n\n\n\n"},{"id":"437580","messageId":"xmqqk0ixagw8.fsf@gitster.g","threadId":"56614","inReplyTo":"6e9843d5-bc04-e69c-9e53-7bf18ca30fc8@github.com","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-30T21:32:39Z","receivedAt":"2021-09-30T21:32:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Taylor Blau wrote:\n>> On Thu, Sep 30, 2021 at 02:50:56PM +0000, Victoria Dye via GitGitGadget wrote:\n>>> From: Victoria Dye \n>>>\n>>> In anticipation of multiple commands being fully integrated with sparse\n>>> index, update the test for index expand and collapse for non-sparse index\n>>> integrated commands to use `mv`.\n>>> ...\n>>> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n>>> -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n>>> +\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n>> \n>> Double-checking my understanding as somebody who is not so familiar with\n>> t1092: this test is to ensure that commands which don't yet understand\n>> the sparse index can temporarily expand it in order to do their work?\n>\n> Exactly - if a command doesn't explicitly enable use of the sparse index by\n> setting `command_requires_full_index` to 0, the index is expanded if/when it\n> is first read during the command's execution and collapsed if/when it is\n> written to disk. This test makes sure that mechanism works as intended.\n\nSorry, I do not quite follow. \n\nSo is this \"before this series of patches, 'reset --hard' can be\nused to as a sample of a command that expands and then collapses,\nbut because it no longer is a good sample of a command so we replace\nit with 'mv a b'\"? Do we need to update this further when \"mv a b\"\nlearns to expand and then collapse?\n"},{"id":"437584","messageId":"e6049a17-c721-c596-51a6-101cb6065829@github.com","threadId":"56614","inReplyTo":"xmqqk0ixagw8.fsf@gitster.g","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-09-30T22:59:57Z","receivedAt":"2021-09-30T23:00:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n>> Taylor Blau wrote:\n>>> On Thu, Sep 30, 2021 at 02:50:56PM +0000, Victoria Dye via GitGitGadget wrote:\n>>>> From: Victoria Dye \n>>>>\n>>>> In anticipation of multiple commands being fully integrated with sparse\n>>>> index, update the test for index expand and collapse for non-sparse index\n>>>> integrated commands to use `mv`.\n>>>> ...\n>>>> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n>>>> -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n>>>> +\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n>>>\n>>> Double-checking my understanding as somebody who is not so familiar with\n>>> t1092: this test is to ensure that commands which don't yet understand\n>>> the sparse index can temporarily expand it in order to do their work?\n>>\n>> Exactly - if a command doesn't explicitly enable use of the sparse index by\n>> setting `command_requires_full_index` to 0, the index is expanded if/when it\n>> is first read during the command's execution and collapsed if/when it is\n>> written to disk. This test makes sure that mechanism works as intended.\n> \n> Sorry, I do not quite follow. \n> \n> So is this \"before this series of patches, 'reset --hard' can be\n> used to as a sample of a command that expands and then collapses,\n> but because it no longer is a good sample of a command so we replace\n> it with 'mv a b'\"?\n\nYes, because this series enables sparse index integration in `git reset`,\nthe test no longer applies to that command (but it does apply to `git mv`).\n\n> Do we need to update this further when \"mv a b\"\n> learns to expand and then collapse?\n\nUnfortunately, yes. `git mv` was picked more-or-less at random from the set\nof commands that read the index and don't already have sparse index\nintegrations (excluding those I know are planned for sparse index\nintegration in the near future). If `git mv` were to be updated to disable\n`command_requires_full_index`, the command in the test would need to change\nagain.\n\nFor what it's worth, I do think the test itself is valuable, since it makes\nsure a command's capability to use the sparse index is always the result of\nan intentional update to (and review of) the code.\n"},{"id":"437588","messageId":"xmqqr1d58v9x.fsf@gitster.g","threadId":"56614","inReplyTo":"e6049a17-c721-c596-51a6-101cb6065829@github.com","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-01T00:04:58Z","receivedAt":"2021-10-01T00:05:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n>> Do we need to update this further when \"mv a b\"\n>> learns to expand and then collapse?\n>\n> Unfortunately, yes. `git mv` was picked more-or-less at random from the set\n> of commands that read the index and don't already have sparse index\n> integrations (excluding those I know are planned for sparse index\n> integration in the near future). If `git mv` were to be updated to disable\n> `command_requires_full_index`, the command in the test would need to change\n> again.\n>\n> For what it's worth, I do think the test itself is valuable, since it makes\n> sure a command's capability to use the sparse index is always the result of\n> an intentional update to (and review of) the code.\n\nOh, of course. \n\nI was actually wondering if it woudl be a good idea to leave a\ncommand that will never be \"converted\" so that we can keep using it\nfor testing.\n\nPerhaps a new option that is invented exactly for the purpose added\nto a plumbing e.g. \"git update-index --expand-collapse\"?\n"},{"id":"437630","messageId":"3de9e873-681a-2bcb-ed56-fc247177cb37@gmail.com","threadId":"56614","inReplyTo":"a1fa7c080aed2056afaad6415186c125c04a80cb.1633013461.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-01T09:14:29Z","receivedAt":"2021-10-01T09:14:36Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 30/09/21 21.50, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> In anticipation of multiple commands being fully integrated with sparse\n> index, update the test for index expand and collapse for non-sparse index\n> integrated commands to use `mv`.\n> \n\nWe can say \"use git sparse-index mv instead of git sparse-index reset\".\n\nWhy is mv used for this case?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"437702","messageId":"53263148-84b1-f76a-263d-afd00819e7a7@github.com","threadId":"56614","inReplyTo":"xmqqpmspc3pn.fsf@gitster.g","subject":"Re: [PATCH 1/7] reset: behave correctly with sparse-checkout","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-01T14:55:22Z","receivedAt":"2021-10-01T14:55:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Kevin Willford via GitGitGadget\" writes:\n> \n>> @@ -127,12 +129,49 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> \t\tstruct diff_options *opt, void *data)\n>> {\n>> \tint i;\n>> +\tint pos;\n>> \tint intent_to_add = *(int *)data;\n>> \n>> \tfor (i = 0; i < q->nr; i++) {\n>> \t\tstruct diff_filespec *one = q->queue[i]->one;\n>> +\t\tstruct diff_filespec *two = q->queue[i]->two;\n>> \t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n>> +\t\tint was_missing = !two->mode && is_null_oid(&two->oid);\n> \n> Not a problem introduced by this patch per-se, but is_missing is a\n> counter-intuitive name for what the boolean wants to represent, I\n> think, which was brought in by b4b313f9 (reset: support \"--mixed\n> --intent-to-add\" mode, 2014-02-04). Before the commit, we used to\n> say\n> \n> \tfor (i = 0; i < q->nr; i++) {\n> \t\tstruct diff_filespec *one = q->queue[i]->one;\n> \t\tif (one->mode && !is_null_sha1(one->sha1)) {\n> \t\t\t... create ce out of one and add to the\tindex ...\n> \t\t} else\n> \t\t\tremove_file_from_cache(one->path);\n> \t\t...\n> \n> i.e. \"if one is not missing, create a ce and add it, otherwise\n> remove the path\".\n> \n> It should have been called \"one_is_missing\" if we wanted to\n> literally express the condition the code checked, but an even better\n> name would have been given after the intent of what the code wants\n> to do with the information. If the resetted-to tree (that is what\n> 'one' side of the comparison in diff_cache() is) has a valid blob,\n> we want it to be in the index, and otherwise, we do not want it in\n> the index.\n> \n> Now, the patch makes things worse and I had to do the above digging\n> to see why the new code is even more confusing. The 'two' side of\n> the comparison is what is in the to-be-corrected-by-reset index.\n> \"was_missing\" in contrast to \"is_missing\" makes it sound as if it\n> was the state before whatever \"is_missing\" tries to represent, but\n> that is not what is happening. \"is_missing\" does not mean \"the\n> entry is currently not there in the index\", but \"was_missing\" does\n> mean exactly that: \"the entry is currently not there in the index\".\n> \n> There isn't any \"was\" missing about it. It \"is\" missing in the\n> index. Instead of renaming, I wonder if we can do without this new\n> variable. Let's read on the patch.\n\nThe new variable can most likely be refactored away, but based on this it's\nprobably worth renaming \"is_missing\" to \"is_missing_in_reset_tree\" (or\ninverting the boolean and using \"is_in_reset_tree\").\n\n> Also, now the code uses both sides of the filepair, we must double\n> check that our do_diff_cache() is *not* doing any rename detection.\n> It might be even prudent to ensure that \n> \n> \tif (strcmp(one->path, two->path))\n> \t\tBUG(\"reset drove diff-cache with rename detection\");\n> \n> but it might be with too much paranoia. I dunno.\n\nI don't think a rename would break what this change intends to do (although\nit does break some of the current assumptions in the patch). I'll make sure\nto verify the rename case works before submitting a new version, just in \ncase.\n\n>> \t\tstruct cache_entry *ce;\n>> +\t\tstruct cache_entry *ce_before;\n>> +\t\tstruct checkout state = CHECKOUT_INIT;\n> \n> These two new variables do not need this wide a scope, I would\n> think. Shouldn't it be inside the body of the new \"if\" statement\n> this patch adds?\n\nI will likely need to make other changes to this patch and re-roll, so I'll\nfix the scoping of all of the variables added here when I do.\n\n>> +\t\t/*\n>> +\t\t * When using the sparse-checkout feature the cache entries\n>> +\t\t * that are added here will not have the skip-worktree bit\n>> +\t\t * set. Without this code there is data that is lost because\n>> +\t\t * the files that would normally be in the working directory\n>> +\t\t * are not there and show as deleted for the next status.\n>> +\t\t * In the case of added files, they just disappear.\n>> +\t\t *\n>> +\t\t * We need to create the previous version of the files in\n>> +\t\t * the working directory so that they will have the right\n>> +\t\t * content and the next status call will show modified or\n>> +\t\t * untracked files correctly.\n>> +\t\t */\n>> +\t\tif (core_apply_sparse_checkout && !file_exists(two->path)) {\n> \n> In a sparsely checked out working tree, there is nothing in the\n> working tree at the path. It may be because it is sparse and we\n> didn't want to have anything there, or it may be because the user\n> wanted to get rid of it and said \"rm path\" (not \"git rm path\") and\n> this part of the tree were of interest even if the sparse checkout\n> feature was used to hide other parts of the tree. With the above\n> two checks alone, we cannot tell which. Let's read on.\n> \n>> +\t\t\tpos = cache_name_pos(two->path, strlen(two->path));\n> \n> We check the index to see if there is an entry for it. I suspect\n> that because we need to do this check anyway, we shouldn't even have\n> to look at 'two' (and add a new 'was_missing' variable), because\n> 'one' and 'two' came from a comparison between the resetted-to tree\n> object and the current index, and if cache_name_pos() for the path\n> (we can use 'one->path') says there is an entry in the index, by\n> definition, 'two' would not be showing a \"removed\" state (i.e. \"the\n> resetted-to tree had it, the index does not\" is what \"was_missing\"\n> wants to say).\n> \n> So I wonder if it is better to\n> \n> - use one->path for !file_exists() above and cache_name_pos() here\n> instead of two->path.\n> \n> - drop the confusingly named 'was_missing', because (pos < 0) is\n> equivalent to it after this point, and we didn't need it up to\n> this point.\n> \n>> +\t\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) &&\n> \n> And we do find an entry for it. So this path is not something\n> sparse cone specifies not to check out (otherwise we would have a\n> tree-like entry that covers this path in the index and not an entry\n> for this specific path)?\n> \n> Anyway, if it is marked with the skip-worktree bit, does that mean\n> there is no risk that the reason why two->path does not exist in the\n> working tree is because we earlier gave it in the working tree but\n> it was later removed by the user? Just making sure that we are not\n> breaking the end-user's wish that the path should be removed by\n> resurrecting it in the working tree with a new call to\n> checkout_entry().\n> \n>> +\t\t\t (is_missing || !was_missing)) {\n> \n> And in such a case, if the resetted-to tree says we shouldn't have\n> the path in the resulting index, or if the original state in the\n> index had this path (but because (0 <= pos) must be true for us to \n> reach this point, I am not sure if \"was_missing\" can ever be true\n> here), then do the following, which is ...\n> \n>> +\t\t\t\tstate.force = 1;\n>> +\t\t\t\tstate.refresh_cache = 1;\n>> +\t\t\t\tstate.istate = &the_index;\n>> +\n>> +\t\t\t\tce_before = make_cache_entry(&the_index, two->mode,\n>> +\t\t\t\t\t\t\t &two->oid, two->path,\n>> +\t\t\t\t\t\t\t 0, 0);\n>> +\t\t\t\tif (!ce_before)\n>> +\t\t\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n>> +\t\t\t\t\t\ttwo->path);\n>> +\n>> +\t\t\t\tcheckout_entry(ce_before, &state, NULL, NULL);\n> \n> ... to resurrect the last \"git add\"ed state from the index and write\n> it out to the working tree. As I suspected, ce_before and state\n> should be scoped inside this block and not visible outside, no?\n> \n> I am not sure why this behaviour is desirable. The \"mixed\" reset\n> should not have to touch the working tree in the first place.\n> \n> The large comment before this block says \"... will not have the\n> skip-worktree bit set\", but we are dealing with a case where the\n> original index had a cache entry there with skip-worktree bit set,\n> so isn't the more desirable outcome that the cache entry added back\n> to the index has the skip-worktree bit still set and there is no\n> working tree file that the user did not desire to have?\n> \n> And isn't it the matter of preserving the skip-worktree bit when the\n> code in the post context of this hunk this patch did not touch adds\n> the entry back to the index?\n> \n>> +\t\t\t}\n>> +\t\t}\n>> \n>> \t\tif (is_missing && !intent_to_add) {\n>> \t\t\tremove_file_from_cache(one->path);\n> \n> If we look at the code after this point, we do use \"is_missing\"\n> information to tweak ce->ce_flags with the intent-to-add bit.\n> \n> Perhaps we can do a similar tweak to the cache entry to mark it with\n> skip-worktree bit if the index had a cache entry at the path with\n> the bit set? The code that needs to do so would only have to\n> remember if the one->path is in the current index and the cache\n> entry for the path has the skip-worktree bit in the body of the new\n> if() statement about (core_apply_sparse_checkout && !file_exists())\n> added by this patch (I am not sure if !file_exists() even matters,\n> though, as the approach I am suggesting is to preserve the skip bit\n> and not disturb the working tree files at all).\n\nI think it might easier to address these points as a whole rather than\ninline.\n\nThe problem this patch is attempting to solve is that, while (as you noted)\n`git reset --mixed` should not touch the working tree, it is *also* expected\nto preserve the files of the pre-reset state (both statements paraphrased\nfrom the `--mixed` option doc). Normally these statements don't conflict,\nbut if `skip-worktree` is respected and nothing is done to the working tree\nbefore resetting the index, `skip-worktree` files will effectively be `reset\n--hard`. So, to force preservation of the pre-reset state, the files are\nchecked out.\n\nBased on that high-level intent, the implementation here can be simplified\n(and clarified). The condition on checking out a file (to avoid the `reset \n--hard`) would be \"if the path exists in the current index and the entry \nin the index has `skip-worktree` enabled\".\n\n* \"if the path exists in the current index\" - if it does not exist in the\n index, there's nothing to preserve.\n* \"if the entry in the index has `skip-worktree` enabled\" - if it does not,\n it's already in the working tree so we don't need to checkout.\n\nThen, `checkout_entry()` can then be run on the index entry found (rather\nthan a \"fake\" one created with `make_cache_entry`). This eliminates a lot of\nunnecessary usage of `one` and `two`, which hopefully addresses some of your\nconcerns about them. After that, the index reset proceeds as normal (without\nmanual changes to the `skip-worktree` bit).\n\nAs for the issue of ignoring `skip-worktree`: all of this could be\nconditioned on a \"--ignore-skip-worktree-bits\" flag (or something like it)\nif you'd prefer the default behavior is \"don't touch the working tree\".\n"},{"id":"437705","messageId":"9fe518fc-59dc-d12f-01ba-eca53777d629@github.com","threadId":"56614","inReplyTo":"e7cda32efb63c63cf60513ec447418fd97ab3fa9.1633013461.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/7] reset: make --mixed sparse-aware","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-01T15:03:44Z","receivedAt":"2021-10-01T15:03:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Victoria Dye via GitGitGadget wrote:\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 0b6ff0de17d..c9b9ef4992c 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -801,14 +801,25 @@ test_expect_success 'sparse-index is not expanded' '\n> \tfor ref in update-deep update-folder1 update-folder2 update-deep\n> \tdo\n> \t\techo >>sparse-index/README.md &&\n> +\t\tensure_not_expanded reset --mixed $ref\n> \t\tensure_not_expanded reset --hard $ref || return 1\n> \tdone &&\nThis is a bug - it's missing `&&` at the end of the line (and adding it will\ncause the test to fail). The index is expanded if a mixed reset modifies an\nentry outside the sparse cone, so I'll update the test in V2 to verify reset\nbetween two refs with only in-cone files changed between them. \n"},{"id":"437892","messageId":"a1722c44-9788-0700-3d76-4199369d1c9f@github.com","threadId":"56614","inReplyTo":"xmqqr1d58v9x.fsf@gitster.g","subject":"Re: [PATCH 2/7] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-04T13:47:29Z","receivedAt":"2021-10-04T13:55:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n>>> Do we need to update this further when \"mv a b\"\n>>> learns to expand and then collapse?\n>>\n>> Unfortunately, yes. `git mv` was picked more-or-less at random from the set\n>> of commands that read the index and don't already have sparse index\n>> integrations (excluding those I know are planned for sparse index\n>> integration in the near future). If `git mv` were to be updated to disable\n>> `command_requires_full_index`, the command in the test would need to change\n>> again.\n>>\n>> For what it's worth, I do think the test itself is valuable, since it makes\n>> sure a command's capability to use the sparse index is always the result of\n>> an intentional update to (and review of) the code.\n> \n> Oh, of course. \n> \n> I was actually wondering if it woudl be a good idea to leave a\n> command that will never be \"converted\" so that we can keep using it\n> for testing.\n> \n> Perhaps a new option that is invented exactly for the purpose added\n> to a plumbing e.g. \"git update-index --expand-collapse\"?\n> \n\nThat sounds good to me! I'll add an `update-index --expand-collapse`\nimplementation and update the test in v2 of this series.\n"},{"id":"437969","messageId":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.git.1633013461.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:50Z","receivedAt":"2021-10-05T13:21:04Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate an overall ~70% execution time reduction across\nall tested usages of git reset using a sparse index:\n\nTest before after \n------------------------------------------------------------------------\n2000.22: git reset (full-v3) 0.48 0.51 +6.3% \n2000.23: git reset (full-v4) 0.47 0.50 +6.4% \n2000.24: git reset (sparse-v3) 0.93 0.30 -67.7%\n2000.25: git reset (sparse-v4) 0.94 0.29 -69.1%\n2000.26: git reset --hard (full-v3) 0.69 0.68 -1.4% \n2000.27: git reset --hard (full-v4) 0.75 0.68 -9.3% \n2000.28: git reset --hard (sparse-v3) 1.29 0.34 -73.6%\n2000.29: git reset --hard (sparse-v4) 1.31 0.34 -74.0%\n2000.30: git reset -- does-not-exist (full-v3) 0.54 0.51 -5.6% \n2000.31: git reset -- does-not-exist (full-v4) 0.54 0.52 -3.7% \n2000.32: git reset -- does-not-exist (sparse-v3) 1.02 0.31 -69.6%\n2000.33: git reset -- does-not-exist (sparse-v4) 1.07 0.30 -72.0%\n\n\n\nChanges since V1\n================\n\n * Add --force-full-index option to update-index. The option is used\n circumvent changing command_requires_full_index from its default value -\n right now this is effectively a no-op, but will change once update-index\n is integrated with sparse index. By using this option in the t1092\n expand/collapse test, the command used to test will not need to be\n updated with subsequent sparse index integrations.\n * Update implementation of mixed reset for entries outside sparse checkout\n definition. The condition in which a file should be checked out before\n index reset is simplified to \"if it has skip-worktree enabled and a reset\n would change the file, check it out\".\n * After checking the behavior of update_index_from_diff with renames,\n found that the diff used by reset does not produce diff queue entries\n with different pathnames for one and two. Because of this, and that\n nothing in the implementation seems to rely on identical path names, no\n BUG check is added.\n * Correct a bug in the sparse index is not expanded tests in t1092 where\n failure of a git reset --mixed test was not being reported. Test now\n verifies an appropriate scenario with corrected failure-checking.\n\nThanks! -Victoria\n\n[1] microsoft@6b8a074\n\nKevin Willford (1):\n reset: behave correctly with sparse-checkout\n\nVictoria Dye (6):\n update-index: add --force-full-index option for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n Documentation/git-update-index.txt | 5 +\n builtin/reset.c | 62 +++++++++-\n builtin/update-index.c | 11 ++\n cache-tree.c | 43 ++++++-\n cache.h | 10 ++\n read-cache.c | 22 ++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 139 ++++++++++++++++++++++-\n t/t7114-reset-sparse-checkout.sh | 61 ++++++++++\n unpack-trees.c | 23 +++-\n 10 files changed, 351 insertions(+), 28 deletions(-)\n create mode 100755 t/t7114-reset-sparse-checkout.sh\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n\nRange-diff vs v1:\n\n 1: 65905bf4e00 ! 1: 22c69bc6030 reset: behave correctly with sparse-checkout\n @@ builtin/reset.c\n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n - \t\tstruct diff_options *opt, void *data)\n - {\n - \tint i;\n -+\tint pos;\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n ++\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n +-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n +\t\tstruct diff_filespec *two = q->queue[i]->two;\n - \t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n -+\t\tint was_missing = !two->mode && is_null_oid(&two->oid);\n ++\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n -+\t\tstruct cache_entry *ce_before;\n -+\t\tstruct checkout state = CHECKOUT_INIT;\n -+\n + \n +-\t\tif (is_missing && !intent_to_add) {\n +\t\t/*\n -+\t\t * When using the sparse-checkout feature the cache entries\n -+\t\t * that are added here will not have the skip-worktree bit\n -+\t\t * set. Without this code there is data that is lost because\n -+\t\t * the files that would normally be in the working directory\n -+\t\t * are not there and show as deleted for the next status.\n -+\t\t * In the case of added files, they just disappear.\n -+\t\t *\n -+\t\t * We need to create the previous version of the files in\n -+\t\t * the working directory so that they will have the right\n -+\t\t * content and the next status call will show modified or\n -+\t\t * untracked files correctly.\n ++\t\t * If the file being reset has `skip-worktree` enabled, we need\n ++\t\t * to check it out to prevent the file from being hard reset.\n +\t\t */\n -+\t\tif (core_apply_sparse_checkout && !file_exists(two->path)) {\n -+\t\t\tpos = cache_name_pos(two->path, strlen(two->path));\n -+\t\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) &&\n -+\t\t\t (is_missing || !was_missing)) {\n -+\t\t\t\tstate.force = 1;\n -+\t\t\t\tstate.refresh_cache = 1;\n -+\t\t\t\tstate.istate = &the_index;\n -+\n -+\t\t\t\tce_before = make_cache_entry(&the_index, two->mode,\n -+\t\t\t\t\t\t\t &two->oid, two->path,\n -+\t\t\t\t\t\t\t 0, 0);\n -+\t\t\t\tif (!ce_before)\n -+\t\t\t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n -+\t\t\t\t\t\ttwo->path);\n ++\t\tpos = cache_name_pos(two->path, strlen(two->path));\n ++\t\tif (pos >= 0 && ce_skip_worktree(active_cache[pos])) {\n ++\t\t\tstruct checkout state = CHECKOUT_INIT;\n ++\t\t\tstate.force = 1;\n ++\t\t\tstate.refresh_cache = 1;\n ++\t\t\tstate.istate = &the_index;\n +\n -+\t\t\t\tcheckout_entry(ce_before, &state, NULL, NULL);\n -+\t\t\t}\n ++\t\t\tcheckout_entry(active_cache[pos], &state, NULL, NULL);\n +\t\t}\n - \n - \t\tif (is_missing && !intent_to_add) {\n ++\n ++\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n + \t\t\tcontinue;\n + \t\t}\n +@@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n + \t\tif (!ce)\n + \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n + \t\t\t one->path);\n +-\t\tif (is_missing) {\n ++\t\tif (!is_in_reset_tree) {\n + \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n + \t\t\tset_object_name_for_intent_to_add_entry(ce);\n + \t\t}\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'blame with pathspec outside sparse definition' '\n 2: a1fa7c080ae ! 2: f7cb9013d46 sparse-index: update command for expand/collapse test\n @@ Metadata\n Author: Victoria Dye \n \n ## Commit message ##\n - sparse-index: update command for expand/collapse test\n + update-index: add --force-full-index option for expand/collapse test\n \n - In anticipation of multiple commands being fully integrated with sparse\n - index, update the test for index expand and collapse for non-sparse index\n - integrated commands to use `mv`.\n + Add a new `--force-full-index` option to `git update-index`, which skips\n + explicitly setting `command_requires_full_index`. This lets `git\n + update-index --force-full-index` run as a command without sparse index\n + compatibility implemented, even after it receives sparse index compatibility\n + updates.\n +\n + By using `git update-index --force-full-index` in the `t1092` test\n + `sparse-index is expanded and converted back`, commands can continue to\n + integrate with the sparse index without the need to keep modifying the\n + command used in the test.\n \n Signed-off-by: Victoria Dye \n \n + ## Documentation/git-update-index.txt ##\n +@@ Documentation/git-update-index.txt: SYNOPSIS\n + \t [--[no-]fsmonitor]\n + \t [--really-refresh] [--unresolve] [--again | -g]\n + \t [--info-only] [--index-info]\n ++\t [--force-full-index]\n + \t [-z] [--stdin] [--index-version ]\n + \t [--verbose]\n + \t [--] [...]\n +@@ Documentation/git-update-index.txt: time. Version 4 is relatively young (first released in 1.8.0 in\n + October 2012). Other Git implementations such as JGit and libgit2\n + may not support it yet.\n + \n ++--force-full-index::\n ++\tForce the command to operate on a full index, expanding a sparse\n ++\tindex if necessary.\n ++\n + -z::\n + \tOnly meaningful with `--stdin` or `--index-info`; paths are\n + \tseparated with NUL character instead of LF.\n +\n + ## builtin/update-index.c ##\n +@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)\n + \tint split_index = -1;\n + \tint force_write = 0;\n + \tint fsmonitor = -1;\n ++\tint use_default_full_index = 0;\n + \tstruct lock_file lock_file = LOCK_INIT;\n + \tstruct parse_opt_ctx_t ctx;\n + \tstrbuf_getline_fn getline_fn;\n +@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)\n + \t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n + \t\t\tN_(\"clear fsmonitor valid bit\"),\n + \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n ++\t\tOPT_SET_INT(0, \"force-full-index\", &use_default_full_index,\n ++\t\t\tN_(\"run with full index explicitly required\"), 1),\n + \t\tOPT_END()\n + \t};\n + \n +@@ builtin/update-index.c: int cmd_update_index(int argc, const char **argv, const char *prefix)\n + \tif (newfd < 0)\n + \t\tlock_error = errno;\n + \n ++\t/*\n ++\t * If --force-full-index is set, the command should skip manually\n ++\t * setting `command_requires_full_index`.\n ++\t */\n ++\tprepare_repo_settings(r);\n ++\tif (!use_default_full_index)\n ++\t\tr->settings.command_requires_full_index = 1;\n ++\n + \tentries = read_cache();\n + \tif (entries < 0)\n + \t\tdie(\"cache corrupted\");\n +\n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n -+\t\tgit -C sparse-index -c core.fsmonitor=\"\" mv a b &&\n ++\t\tgit -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n 3: d033c5e365f = 3: c7e9d9f4e03 reset: expand test coverage for sparse checkouts\n 4: 2d63a250637 = 4: 49813c8d9ed reset: integrate with sparse index\n 5: e919e6d3270 = 5: 78cd85d8dcc reset: make sparse-aware (except --mixed)\n 6: e7cda32efb6 ! 6: 5eaae0825af reset: make --mixed sparse-aware\n @@ builtin/reset.c: static int read_from_tree(const struct pathspec *pathspec,\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is not expanded' '\n - \tfor ref in update-deep update-folder1 update-folder2 update-deep\n - \tdo\n - \t\techo >>sparse-index/README.md &&\n -+\t\tensure_not_expanded reset --mixed $ref\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n ++\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n 7: 8637ec1660e = 7: aa963eefae7 unpack-trees: improve performance of next_cache_entry\n\n-- \ngitgitgadget\n"},{"id":"437970","messageId":"22c69bc60308fef13acd7c3aab4e11e175c89440.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Kevin Willford via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:51Z","receivedAt":"2021-10-05T13:21:05Z","isPatch":true,"sender":{"key":"kewillf@microsoft.com","avatar":null},"body":"From: Kevin Willford \n\nWhen using the sparse checkout feature, 'git reset' will add entries to\nthe index that will have the skip-worktree bit off but will leave the\nworking directory empty. File data is lost because the index version of\nthe files has been changed but there is nothing that is in the working\ndirectory. This will cause the next 'git status' call to show either\ndeleted for files modified or deleting or nothing for files added. The\nadded files should be shown as untracked and modified files should be\nshown as modified.\n\nTo fix this when the reset is running if there is not a file in the\nworking directory and if it will be missing with the new index entry or\nwas not missing in the previous version, we create the previous index\nversion of the file in the working directory so that status will report\ncorrectly and the files will be availble for the user to deal with.\n\nThis fixes a documented failure from t1092 that was created in 19a0acc\n(t1092: test interesting sparse-checkout scenarios, 2021-01-23).\n\nSigned-off-by: Kevin Willford \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 24 ++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 4 +-\n t/t7114-reset-sparse-checkout.sh | 61 ++++++++++++++++++++++++\n 3 files changed, 83 insertions(+), 6 deletions(-)\n create mode 100755 t/t7114-reset-sparse-checkout.sh\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..3b75d3b2f20 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,8 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n+#include \"entry.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -130,11 +132,27 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n+\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tstruct diff_filespec *two = q->queue[i]->two;\n+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n \n-\t\tif (is_missing && !intent_to_add) {\n+\t\t/*\n+\t\t * If the file being reset has `skip-worktree` enabled, we need\n+\t\t * to check it out to prevent the file from being hard reset.\n+\t\t */\n+\t\tpos = cache_name_pos(two->path, strlen(two->path));\n+\t\tif (pos >= 0 && ce_skip_worktree(active_cache[pos])) {\n+\t\t\tstruct checkout state = CHECKOUT_INIT;\n+\t\t\tstate.force = 1;\n+\t\t\tstate.refresh_cache = 1;\n+\t\t\tstate.istate = &the_index;\n+\n+\t\t\tcheckout_entry(active_cache[pos], &state, NULL, NULL);\n+\t\t}\n+\n+\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n \t\t\tcontinue;\n \t\t}\n@@ -144,7 +162,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n-\t\tif (is_missing) {\n+\t\tif (!is_in_reset_tree) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n \t\t\tset_object_name_for_intent_to_add_entry(ce);\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..c5977152661 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,9 +459,7 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\ndiff --git a/t/t7114-reset-sparse-checkout.sh b/t/t7114-reset-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..a8029707fb1\n--- /dev/null\n+++ b/t/t7114-reset-sparse-checkout.sh\n@@ -0,0 +1,61 @@\n+#!/bin/sh\n+\n+test_description='reset when using a sparse-checkout'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_tick &&\n+\techo \"checkout file\" >c &&\n+\techo \"modify file\" >m &&\n+\techo \"delete file\" >d &&\n+\tgit add . &&\n+\tgit commit -m \"initial commit\" &&\n+\techo \"added file\" >a &&\n+\techo \"modification of a file\" >m &&\n+\tgit rm d &&\n+\tgit add . &&\n+\tgit commit -m \"second commit\" &&\n+\tgit checkout -b endCommit\n+'\n+\n+test_expect_success 'reset when there is a sparse-checkout' '\n+\techo \"/c\" >.git/info/sparse-checkout &&\n+\ttest_config core.sparsecheckout true &&\n+\tgit checkout -B resetBranch &&\n+\ttest_path_is_missing m &&\n+\ttest_path_is_missing a &&\n+\ttest_path_is_missing d &&\n+\tgit reset HEAD~1 &&\n+\techo \"checkout file\" >expect &&\n+\ttest_cmp expect c &&\n+\techo \"added file\" >expect &&\n+\ttest_cmp expect a &&\n+\techo \"modification of a file\" >expect &&\n+\ttest_cmp expect m &&\n+\ttest_path_is_missing d\n+'\n+\n+test_expect_success 'reset after deleting file without skip-worktree bit' '\n+\tgit checkout -f endCommit &&\n+\tgit clean -xdf &&\n+\tcat >.git/info/sparse-checkout <<-\\EOF &&\n+\t/c\n+\t/m\n+\tEOF\n+\ttest_config core.sparsecheckout true &&\n+\tgit checkout -B resetAfterDelete &&\n+\ttest_path_is_file m &&\n+\ttest_path_is_missing a &&\n+\ttest_path_is_missing d &&\n+\trm -f m &&\n+\tgit reset HEAD~1 &&\n+\techo \"checkout file\" >expect &&\n+\ttest_cmp expect c &&\n+\techo \"added file\" >expect &&\n+\ttest_cmp expect a &&\n+\ttest_path_is_missing m &&\n+\ttest_path_is_missing d\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"437973","messageId":"c7e9d9f4e03fe0dbd8a112460f0ac0864f087c8f.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:53Z","receivedAt":"2021-10-05T13:21:06Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs both inside and outside of the sparse checkout definition. New\nperformance test cases exercise various execution paths for `reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 107 +++++++++++++++++++++++\n 2 files changed, 110 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b3c0d3b98ee..f0723a6ac97 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -479,6 +479,113 @@ test_expect_success 'checkout and reset (mixed) [sparse]' '\n \ttest_sparse_match git reset update-folder2\n '\n \n+# NEEDSWORK: with mixed reset, files with differences between HEAD and \n+# will be added to the work tree even if outside the sparse checkout\n+# definition, and even if the file is modified to a state of having no local\n+# changes. The file is \"re-ignored\" if a hard reset is executed. We may want to\n+# change this behavior in the future and enforce that files are not written\n+# outside of the sparse checkout definition.\n+test_expect_success 'checkout and mixed reset file tracking [sparse]' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset update-folder1 &&\n+\ttest_all_match git reset update-deep &&\n+\n+\t# At this point, there are no changes in the working tree. However,\n+\t# folder1/a now exists locally (even though it is outside of the sparse\n+\t# paths).\n+\trun_on_sparse test_path_exists folder1 &&\n+\n+\trun_on_all rm folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_sparse test_path_is_missing folder1 &&\n+\ttest_path_exists full-checkout/folder1\n+'\n+\n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with sparse directory pathspec outside definition' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- folder1 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with pathspec match in sparse directory' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- folder1/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset --hard update-folder1 &&\n+\ttest_all_match git reset base -- \\*/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"437971","messageId":"49813c8d9ed94fd56f30eb204d346eb5a30473ca.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:54Z","receivedAt":"2021-10-05T13:21:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\n`reset --soft` does not modify the index, so no compatibility changes are\nneeded for it to function without expanding the index. For all other reset\nmodes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is\nexplicitly expanded with `ensure_full_index` to maintain current behavior.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` does need to\nbe run after enabling sparse index for the command and before resetting.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 3b75d3b2f20..e1f2a2bb2c4 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -184,6 +184,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -261,9 +262,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -409,6 +407,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"437972","messageId":"f7cb9013d46731855c3ed42add62b021c0ad0c73.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:52Z","receivedAt":"2021-10-05T13:21:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd a new `--force-full-index` option to `git update-index`, which skips\nexplicitly setting `command_requires_full_index`. This lets `git\nupdate-index --force-full-index` run as a command without sparse index\ncompatibility implemented, even after it receives sparse index compatibility\nupdates.\n\nBy using `git update-index --force-full-index` in the `t1092` test\n`sparse-index is expanded and converted back`, commands can continue to\nintegrate with the sparse index without the need to keep modifying the\ncommand used in the test.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/git-update-index.txt | 5 +++++\n builtin/update-index.c | 11 +++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 3 files changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..06255e321a3 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -24,6 +24,7 @@ SYNOPSIS\n \t [--[no-]fsmonitor]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n+\t [--force-full-index]\n \t [-z] [--stdin] [--index-version ]\n \t [--verbose]\n \t [--] [...]\n@@ -170,6 +171,10 @@ time. Version 4 is relatively young (first released in 1.8.0 in\n October 2012). Other Git implementations such as JGit and libgit2\n may not support it yet.\n \n+--force-full-index::\n+\tForce the command to operate on a full index, expanding a sparse\n+\tindex if necessary.\n+\n -z::\n \tOnly meaningful with `--stdin` or `--index-info`; paths are\n \tseparated with NUL character instead of LF.\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..32ada3ead77 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -964,6 +964,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tint split_index = -1;\n \tint force_write = 0;\n \tint fsmonitor = -1;\n+\tint use_default_full_index = 0;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1069,6 +1070,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n \t\t\tN_(\"clear fsmonitor valid bit\"),\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n+\t\tOPT_SET_INT(0, \"force-full-index\", &use_default_full_index,\n+\t\t\tN_(\"run with full index explicitly required\"), 1),\n \t\tOPT_END()\n \t};\n \n@@ -1082,6 +1085,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tif (newfd < 0)\n \t\tlock_error = errno;\n \n+\t/*\n+\t * If --force-full-index is set, the command should skip manually\n+\t * setting `command_requires_full_index`.\n+\t */\n+\tprepare_repo_settings(r);\n+\tif (!use_default_full_index)\n+\t\tr->settings.command_requires_full_index = 1;\n+\n \tentries = read_cache();\n \tif (entries < 0)\n \t\tdie(\"cache corrupted\");\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c5977152661..b3c0d3b98ee 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"437974","messageId":"78cd85d8dcc790251ce8235e649902cf6adf091a.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:55Z","receivedAt":"2021-10-05T13:21:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn order to accurately reconstruct the cache tree in `prime_cache_tree_rec`,\nthe function must determine whether the currently-processing directory in\nthe tree is sparse or not. If it is not sparse, the tree is parsed and\nsubtree recursively constructed. If it is sparse, no subtrees are added to\nthe tree and the entry count is set to 1 (representing the sparse directory\nitself).\n\nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 44 +++++++++++++++++++++---\n cache.h | 10 ++++++\n read-cache.c | 22 ++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 ++++++--\n 4 files changed, 78 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..9021669d682 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,29 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct repository *r,\n+\t\t\t\t\tstruct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree,\n+\t\t\t\t\tstruct strbuf *tree_path)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+\treturn;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n+\tstruct strbuf subtree_path = STRBUF_INIT;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,27 +771,49 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\t\t\tstrbuf_reset(&subtree_path);\n+\t\t\tstrbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1);\n+\t\t\tstrbuf_addbuf(&subtree_path, tree_path);\n+\t\t\tstrbuf_add(&subtree_path, entry.path, entry.pathlen);\n+\t\t\tstrbuf_addch(&subtree_path, '/');\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, subtree_path.buf, subtree_path.len))\n+\t\t\t\tprime_cache_tree_sparse_dir(r, sub->cache_tree, subtree, &subtree_path);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, &subtree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n \tit->entry_count = cnt;\n+\n+\tstrbuf_release(&subtree_path);\n }\n \n void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..ea1166895f8 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -551,7 +551,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tint search_sparse)\n {\n \tint first, last;\n \n@@ -570,7 +573,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_sparse && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +589,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_sparse);\n \t\t}\n \t}\n \n@@ -595,7 +598,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, 1);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, 0) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1230,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, 1);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1330,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1365,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f0723a6ac97..e301ef5633a 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -786,9 +786,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -798,6 +798,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"437975","messageId":"5eaae0825af4cee84040b784c32190bb1de2f919.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:56Z","receivedAt":"2021-10-05T13:21:12Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nSparse directory entries are \"diffed\" as trees in `diff_cache` (used\ninternally by `reset --mixed`), following a code path separate from\nindividual file handling. The use of `diff_tree_oid` there requires setting\nexplicit `change` and `add_remove` functions to process the internal\ncontents of a sparse directory.\n\nAdditionally, the `recursive` diff option handles cases in which `reset\n--mixed` must diff/merge files that are nested multiple levels deep in a\nsparse directory.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 30 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++-\n 2 files changed, 41 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex e1f2a2bb2c4..ceb9b122897 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -175,6 +175,8 @@ static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t int intent_to_add)\n {\n \tstruct diff_options opt;\n+\tunsigned int i;\n+\tchar *skip_worktree_seen = NULL;\n \n \tmemset(&opt, 0, sizeof(opt));\n \tcopy_pathspec(&opt.pathspec, pathspec);\n@@ -182,9 +184,35 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\t/*\n+\t * When pathspec is given for resetting a cone-mode sparse checkout, it may\n+\t * identify entries that are nested in sparse directories, in which case the\n+\t * index should be expanded. For the sake of efficiency, this check is\n+\t * overly-cautious: anything with a wildcard or a magic prefix requires\n+\t * expansion, as well as literal paths that aren't in the sparse checkout\n+\t * definition AND don't match any directory in the index.\n+\t */\n+\tif (pathspec->nr && the_index.sparse_index) {\n+\t\tif (pathspec->magic || pathspec->has_wildcard) {\n+\t\t\tensure_full_index(&the_index);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\t\t\tif (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) &&\n+\t\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) {\n+\t\t\t\t\tensure_full_index(&the_index);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tfree(skip_worktree_seen);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e301ef5633a..4afcbc2d673 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -804,11 +804,22 @@ test_expect_success 'sparse-index is not expanded' '\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n+\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n-\tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so it will be reset without needing to\n+\t# expand the full index.\n+\tensure_not_expanded reset --hard update-folder1 &&\n+\tensure_not_expanded reset base -- folder1 &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"437976","messageId":"aa963eefae75d41983201e24398bc5692267b91b.1633440057.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-05T13:20:57Z","receivedAt":"2021-10-05T13:21:13Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain\nthe benefit `cache_bottom` provides in non-sparse index cases, a separate\n`hint` position indicates the first position `next_cache_entry` should\nsearch, updated each execution with a new position. The performance of `git\nreset -- does-not-exist` (testing the \"worst case\" in which all entries in\nthe index are unpacked with `next_cache_entry`) is significantly improved\nfor the sparse index case:\n\nTest before after\n------------------------------------------------------\n(full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n(full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n(sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1%\n(sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3%\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"437977","messageId":"878rz7lbm6.fsf@evledraar.gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] Sparse Index: integrate with reset","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-10-05T15:34:48Z","receivedAt":"2021-10-05T15:44:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 05 2021, Victoria Dye via GitGitGadget wrote:\n\n> The p2000 tests demonstrate an overall ~70% execution time reduction across\n> all tested usages of git reset using a sparse index:\n\n[...]\n\n> Test before after \n> ------------------------------------------------------------------------\n> 2000.22: git reset (full-v3) 0.48 0.51 +6.3% \n> 2000.23: git reset (full-v4) 0.47 0.50 +6.4% \n> 2000.24: git reset (sparse-v3) 0.93 0.30 -67.7%\n> 2000.25: git reset (sparse-v4) 0.94 0.29 -69.1%\n> 2000.26: git reset --hard (full-v3) 0.69 0.68 -1.4% \n> 2000.27: git reset --hard (full-v4) 0.75 0.68 -9.3% \n> 2000.28: git reset --hard (sparse-v3) 1.29 0.34 -73.6%\n> 2000.29: git reset --hard (sparse-v4) 1.31 0.34 -74.0%\n> 2000.30: git reset -- does-not-exist (full-v3) 0.54 0.51 -5.6% \n> 2000.31: git reset -- does-not-exist (full-v4) 0.54 0.52 -3.7% \n> 2000.32: git reset -- does-not-exist (sparse-v3) 1.02 0.31 -69.6%\n> 2000.33: git reset -- does-not-exist (sparse-v4) 1.07 0.30 -72.0%\n\nThis series looks like it really improves some cases, but at the cost of\nthat -70% improvement we've got a ~5% regression in 7/7 for the full-v3\n--does-not-exist cases. As noted in your 7/7 (which improves all other\ncases):\n\n (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n\nWhich b.t.w. I had to read a couple of times before realizig that its\nquoted:\n \n Test before after\n ------------------------------------------------------\n (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n (sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1%\n (sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3%\n\nIs just the does-not-exist part of this bigger table, are the other\ncases all ~0% changed, or ...?\n \nAnyway, until 7/7 the v3 had been sped up, but a ~10% increase landed us\nat ~+6%, and full-v4 had been ~0% but got ~6% worse?\n\nIs there a way we can get those improvements in performance without\nregressing on the full-* cases?\n\nAlso, these tests only check sparse performance, but isn't some of the\ncode being modified here general enough to not be used exclusively by\nthe sparse mode, full checkout cone or not?\n\nIt looks fairly easy to extend p2000-sparse-operations.sh to run the\nsame tests but just pretend that it's running in a \"full\" mode without\nactually setting up anyting sparse-specific (the meat of those tests\njust runs \"git status\" etc. How does that look with this series?\n\nSince only the CL and 7/7 quote numbers from p2000, and 7/7 is at least\na partial regression, it would be nice to have perf numbers on each\ncommit (if only as a one-off for ML consumption). Are there any more\nimprovements followed by regressions followed by improvements as we go\nalong? Would be useful to know...\n"},{"id":"437995","messageId":"xmqqfstftgk6.fsf@gitster.g","threadId":"56614","inReplyTo":"22c69bc60308fef13acd7c3aab4e11e175c89440.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-05T19:30:49Z","receivedAt":"2021-10-05T19:30:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Kevin Willford via GitGitGadget\" writes:\n\n> When using the sparse checkout feature, 'git reset' will add entries to\n> the index that will have the skip-worktree bit off but will leave the\n> working directory empty. File data is lost because the index version of\n> the files has been changed but there is nothing that is in the working\n> directory. This will cause the next 'git status' call to show either\n> deleted for files modified or deleting or nothing for files added. The\n> added files should be shown as untracked and modified files should be\n> shown as modified.\n\nI am on vacation today, so let me be brief.\n\nLet me see if I am understanding the situation correctly.\n\nWe have the index, with a path that records a blob, but the path is\nmarked with skip-wortree bit.\n\n $ rm -fr test && mkdir test && cd test\n $ git init .\n $ date >no-skip\n $ date >skip\n $ git add no-skip skip\n $ git commit -m initial\n 2 files changed, 2 insertions(+)\n create mode 100644 no-skip\n create mode 100644 skip\n $ date >no-skip\n $ date >skip\n $ git add no-skip skip\n $ git update-index --skip-wortree skip\n $ rm skip\n $ git commit -m second\n [master e9088ad] second\n 2 files changed, 2 insertions(+), 2 deletions(-)\n $ ls *skip\n no-skip\n $ git ls-files -t\n H no-skip\n S skip\n $ git status\n On branch master\n nothing to commit, working tree clean\n\nNote. There is no 'reset' done yet so far.\n\nThe user is happy with the state because\n\n (1) The user marked the path \"skip\" with skip-worktree bit, and\n thanks to that, even though \"skip\" is absent in the working\n tree, the \"git status\" does not complain.\n\n (2) The user marked the path \"skip\" with skip-worktree bit because\n the user did not want to see such a file in the working tree.\n And \"git commit -m second\", \"git ls-files -t\", or \"git status\"\n that were done to get here did not make it materialize in the\n working tree all of sudden.\n\nAnd then the user says \"git reset HEAD^\" to switch to a different\ncommit.\n\n $ git reset HEAD^\n $ ls *skip\n no-skip\n $ git ls-files -t\n M no-skip\n D skip\n $ git status -suno\n M no-skip\n D skip\n\nThe user is unhappy with the state because \"skip\" is shown as lost.\n\nDo I understand the situation you are trying to deal with correctly?\n\n> To fix this when the reset is running if there is not a file in the\n> working directory and if it will be missing with the new index entry or\n> was not missing in the previous version, we create the previous index\n> version of the file in the working directory so that status will report\n> correctly and the files will be availble for the user to deal with.\n\nAssuming I read the problem description correctly, I am highly\nskeptical that the above is a correct approach to keep the user\nhappy. Yes, if you created a working tree file with contents that\nmatch the blob recorded for the path in the initial commit when\n\"reset HEAD^\" is done, you may keep \"git status\" quiet, so (1) above\nwill be kept, but what about (2)? The user marked the path with\n\"skip\" but, because the path should not appear on the working tree.\nThe \"fix\" is countermanding that wish by the user, isn't it?\n\nWouldn't a fix to the situation be to \n\n * Add the blob for \"skip\" taken from the initial commit to the\n index, just like the entry for \"no-skip\" is updated;\n\n * But remember that \"skip\" was marked with \"skip-worktree\" bit\n immediately before \"git reset\" was asked to do its thing, and\n re-add the bit to the path in the index before \"git reset\" gives\n the control back to the usre;\n\n * And keep the working tree untouched, without writing anything out\n to \"skip\". If the user had a (possibly unrelated) file there, it\n will not be overwritten, and if the user left the path absent, it\n will still be absent.\n\nso that the last three diagnostic commands in the above sample\nsequence would instead read:\n\n $ ls *skip\n no-skip\n $ git ls-files -t\n M no-skip\n S skip\n $ git status -suno\n M no-skip\n\ni.e. skip gets updated in the index only, nothing changes in the\nworking tree for \"skip\" or \"no-skip\", and status reports that\n\"no-skip\" is different from the index but \"skip\" hasn't changed in\nthe working tree since the index (thanks to its skip-worktree bit).\n\nThen the user will be happy in the same way as the user was happy\nimmediately after the state marked with \"There is no 'reset' done\nyet so far.\" above, on both counts, not just for \"status does not\nreport something got changed\" part but also \"user didn't want to see\n'skip' in the working tree, and 'skip' did not materialize\" part.\n\nThanks.\n"},{"id":"438007","messageId":"8b9fe3f8-f0e3-4567-b20b-17c92bd1a5c5@github.com","threadId":"56614","inReplyTo":"878rz7lbm6.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/7] Sparse Index: integrate with reset","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-05T20:44:33Z","receivedAt":"2021-10-05T20:44:38Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Oct 05 2021, Victoria Dye via GitGitGadget wrote:\n> \n>> The p2000 tests demonstrate an overall ~70% execution time reduction across\n>> all tested usages of git reset using a sparse index:\n> \n> [...]\n> \n>> Test before after \n>> ------------------------------------------------------------------------\n>> 2000.22: git reset (full-v3) 0.48 0.51 +6.3% \n>> 2000.23: git reset (full-v4) 0.47 0.50 +6.4% \n>> 2000.24: git reset (sparse-v3) 0.93 0.30 -67.7%\n>> 2000.25: git reset (sparse-v4) 0.94 0.29 -69.1%\n>> 2000.26: git reset --hard (full-v3) 0.69 0.68 -1.4% \n>> 2000.27: git reset --hard (full-v4) 0.75 0.68 -9.3% \n>> 2000.28: git reset --hard (sparse-v3) 1.29 0.34 -73.6%\n>> 2000.29: git reset --hard (sparse-v4) 1.31 0.34 -74.0%\n>> 2000.30: git reset -- does-not-exist (full-v3) 0.54 0.51 -5.6% \n>> 2000.31: git reset -- does-not-exist (full-v4) 0.54 0.52 -3.7% \n>> 2000.32: git reset -- does-not-exist (sparse-v3) 1.02 0.31 -69.6%\n>> 2000.33: git reset -- does-not-exist (sparse-v4) 1.07 0.30 -72.0%\n> \n> This series looks like it really improves some cases, but at the cost of\n> that -70% improvement we've got a ~5% regression in 7/7 for the full-v3\n> --does-not-exist cases. As noted in your 7/7 (which improves all other\n> cases):\n> \n> (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n> (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n> \n\nNew performance numbers at the end - I think I have an explanation for this.\n\n> Which b.t.w. I had to read a couple of times before realizig that its\n> quoted:\n> \n> Test before after\n> ------------------------------------------------------\n> (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n> (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n> (sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1%\n> (sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3%\n> \n> Is just the does-not-exist part of this bigger table, are the other\n> cases all ~0% changed, or ...?\n> \n\nThese numbers were for the `git reset -- does-not-exist` case only. If I end\nup needing to send a V3, though, I'll probably remove the performance\nnumbers from 7/7 altogether - looking at them now, they make the commit\nmessage somewhat cluttered. That said, performance numbers *are* helpful for\nreviews on the mailing list, so I'd keep the information in the cover\nletter at the very least.\n\n> Anyway, until 7/7 the v3 had been sped up, but a ~10% increase landed us\n> at ~+6%, and full-v4 had been ~0% but got ~6% worse?\n> \n> Is there a way we can get those improvements in performance without\n> regressing on the full-* cases?\n> \n> Also, these tests only check sparse performance, but isn't some of the\n> code being modified here general enough to not be used exclusively by\n> the sparse mode, full checkout cone or not?\n> \n> It looks fairly easy to extend p2000-sparse-operations.sh to run the\n> same tests but just pretend that it's running in a \"full\" mode without\n> actually setting up anyting sparse-specific (the meat of those tests\n> just runs \"git status\" etc. How does that look with this series?\n> \n\nI updated `p2000` locally to do this but the setup was substantially slower\nfor the full checkout, to the point that it was infeasible to run the\ncomplete test for all relevant commits. Looking at the changes in this\nseries, nothing appears to affect the full checkout case differently than\nthe sparse checkout/full index case, so I'm fairly confident there won't be\na regression specific to full checkouts.\n\n> Since only the CL and 7/7 quote numbers from p2000, and 7/7 is at least\n> a partial regression, it would be nice to have perf numbers on each\n> commit (if only as a one-off for ML consumption). Are there any more\n> improvements followed by regressions followed by improvements as we go\n> along? Would be useful to know...\n> \n\nI don't think any of the apparent slowdowns seen in these results represent\nreal regressions. After re-running the performance tests, I saw variability\nof up to ~20% execution time across changes with commands that should see no\neffect on their execution time (e.g. sparse-v* from 1/7 to 4/7).\nAdditionally, I saw different increases & decreases each time for each\nend-to-end run of the tests. The most reliable, noticeable changes across\nthe test executions were:\n\n1. When each variant of `git reset` was integrated with sparse index, a\n 65-75% execution time reduction in relevant sparse-v* tests.\n2. `git reset -- does-not-exist` slower than `git reset` in 6/7,\n then matching its speed after 7/7.\n3. As of 7/7, full-v* to sparse-v* showing a 50% execution time reduction.\n\nMy guess is that the variability comes from general \"uncontrolled\" factors\nwhen running the tests (e.g., background processes on my system). The good\nnews is, when the tests are re-run with more trials (and the recent bugfix\nto `t/perf/perf-lib.sh` [1]), the execution times look a lot less worrisome \n(apologies for the table width, but I'd like to err on the side of providing\nmore complete information):\n\nTest base [1/7] [4/7] [5/7] [6/7] [7/7] \n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n2000.22: git reset (full-v3) 0.44(0.16+0.19) 0.44(0.17+0.18) +0.0% 0.44(0.17+0.19) +0.0% 0.45(0.17+0.18) +2.3% 0.44(0.17+0.19) +0.0% 0.45(0.17+0.18) +2.3% \n2000.23: git reset (full-v4) 0.43(0.16+0.18) 0.43(0.16+0.19) +0.0% 0.45(0.17+0.18) +4.7% 0.44(0.17+0.18) +2.3% 0.44(0.17+0.18) +2.3% 0.44(0.18+0.18) +2.3% \n2000.24: git reset (sparse-v3) 0.82(0.54+0.19) 0.84(0.56+0.19) +2.4% 0.81(0.54+0.19) -1.2% 0.88(0.60+0.19) +7.3% 0.27(0.03+0.45) -67.1% 0.27(0.03+0.47) -67.1%\n2000.25: git reset (sparse-v4) 0.82(0.55+0.18) 0.82(0.53+0.20) +0.0% 0.83(0.55+0.19) +1.2% 0.82(0.54+0.19) +0.0% 0.27(0.03+0.50) -67.1% 0.27(0.03+0.48) -67.1%\n2000.26: git reset --hard (full-v3) 0.71(0.38+0.24) 0.69(0.37+0.23) -2.8% 0.70(0.37+0.24) -1.4% 0.78(0.41+0.27) +9.9% 0.71(0.38+0.25) +0.0% 0.70(0.37+0.23) -1.4% \n2000.27: git reset --hard (full-v4) 0.71(0.38+0.23) 0.77(0.42+0.25) +8.5% 0.76(0.41+0.26) +7.0% 0.72(0.40+0.24) +1.4% 0.68(0.37+0.23) -4.2% 0.67(0.36+0.22) -5.6%\n2000.28: git reset --hard (sparse-v3) 1.29(0.93+0.26) 1.33(0.95+0.27) +3.1% 1.11(0.76+0.25) -14.0% 0.38(0.05+0.25) -70.5% 0.36(0.04+0.22) -72.1% 0.34(0.04+0.21) -73.6%\n2000.29: git reset --hard (sparse-v4) 1.17(0.84+0.24) 1.10(0.79+0.23) -6.0% 1.01(0.69+0.24) -13.7% 0.42(0.05+0.26) -64.1% 0.39(0.05+0.25) -66.7% 0.38(0.05+0.23) -67.5%\n2000.30: git reset -- does-not-exist (full-v3) 0.50(0.19+0.20) 0.50(0.19+0.20) +0.0% 0.53(0.21+0.22) +6.0% 0.47(0.18+0.19) -6.0% 0.45(0.18+0.18) -10.0% 0.45(0.18+0.19) -10.0%\n2000.31: git reset -- does-not-exist (full-v4) 0.45(0.18+0.18) 0.46(0.18+0.19) +2.2% 0.47(0.19+0.19) +4.4% 0.45(0.18+0.19) +0.0% 0.45(0.18+0.18) +0.0% 0.45(0.18+0.18) +0.0% \n2000.32: git reset -- does-not-exist (sparse-v3) 1.01(0.70+0.21) 0.91(0.62+0.20) -9.9% 0.93(0.64+0.20) -7.9% 0.89(0.61+0.20) -11.9% 0.48(0.23+0.46) -52.5% 0.27(0.03+0.49) -73.3%\n2000.33: git reset -- does-not-exist (sparse-v4) 0.99(0.67+0.21) 1.02(0.70+0.22) +3.0% 1.04(0.70+0.22) +5.1% 0.83(0.55+0.19) -16.2% 0.48(0.24+0.48) -51.5% 0.27(0.03+0.49) -72.7%\n\nNote that some commits in this series are not included because they don't\ntouch any code used by `git reset`.\n\n[1] https://lore.kernel.org/git/pull.1051.git.1633386543759.gitgitgadget@gmail.com/\n"},{"id":"438012","messageId":"9b99e856-24cc-03fd-7871-de92dc6e39b6@github.com","threadId":"56614","inReplyTo":"xmqqfstftgk6.fsf@gitster.g","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-05T21:59:28Z","receivedAt":"2021-10-05T22:00:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> Wouldn't a fix to the situation be to \n> \n> * Add the blob for \"skip\" taken from the initial commit to the\n> index, just like the entry for \"no-skip\" is updated;\n> \n> * But remember that \"skip\" was marked with \"skip-worktree\" bit\n> immediately before \"git reset\" was asked to do its thing, and\n> re-add the bit to the path in the index before \"git reset\" gives\n> the control back to the usre;\n> \n> * And keep the working tree untouched, without writing anything out\n> to \"skip\". If the user had a (possibly unrelated) file there, it\n> will not be overwritten, and if the user left the path absent, it\n> will still be absent.\n> \n> so that the last three diagnostic commands in the above sample\n> sequence would instead read:\n> \n> $ ls *skip\n> no-skip\n> $ git ls-files -t\n> M no-skip\n> S skip\n> $ git status -suno\n> M no-skip\n> \n> i.e. skip gets updated in the index only, nothing changes in the\n> working tree for \"skip\" or \"no-skip\", and status reports that\n> \"no-skip\" is different from the index but \"skip\" hasn't changed in\n> the working tree since the index (thanks to its skip-worktree bit).\n> \n> Then the user will be happy in the same way as the user was happy\n> immediately after the state marked with \"There is no 'reset' done\n> yet so far.\" above, on both counts, not just for \"status does not\n> report something got changed\" part but also \"user didn't want to see\n> 'skip' in the working tree, and 'skip' did not materialize\" part.\n> \n> Thanks.\n> \n\nThanks for the thorough explanation, I'm on-board with your approach (and\nwill re-roll the series with that implemented). A lot of my thought process\n(and confusion) came from a comment in e5ca291076 (t1092: document bad\nsparse-checkout behavior, 2021-07-14) suggesting that full and sparse\ncheckouts should have the same result in scenarios like the one you\noutlined above. The problem is, as noted earlier, it's impossible to tell\nwhether (using your example):\n\n1. the user deleted `skip` because they intentionally want to remove it from\n the worktree, and it should continue to be deleted after a reset.\n2. `skip` doesn't exist in the worktree because it's excluded from the\n sparse checkout definition and the user does not want its current state\n \"deleted\" after a reset.\n\nAs a result, there's no way `git reset --mixed` could be expected to behave\nthe same way in full checkouts as it does in sparse, and the most consistent\nsolution is that the worktree should remain untouched with `skip-worktree`\npreserved.\n"},{"id":"438016","messageId":"CABPp-BFFG=puwpB4Gz8nWU=CNMUrLbH7XPTkpSGqWedTK2NWLg@mail.gmail.com","threadId":"56614","inReplyTo":"22c69bc60308fef13acd7c3aab4e11e175c89440.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T01:46:43Z","receivedAt":"2021-10-06T01:46:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi!\n\nIt appears Junio has already commented on this patch and in more\ndetail, but since I already typed up some comments I'll send them\nalong in case they are useful.\n\nOn Tue, Oct 5, 2021 at 6:20 AM Kevin Willford via GitGitGadget\n wrote:\n>\n> From: Kevin Willford \n>\n> When using the sparse checkout feature, 'git reset' will add entries to\n> the index that will have the skip-worktree bit off but will leave the\n> working directory empty.\n\nYes, that seems like a problem.\n\n> File data is lost because the index version of\n> the files has been changed but there is nothing that is in the working\n> directory. This will cause the next 'git status' call to show either\n> deleted for files modified or deleting or nothing for files added. The\n> added files should be shown as untracked and modified files should be\n> shown as modified.\n\nWhy is the solution to add the files to the working tree rather than\nto make sure the files have the skip-worktree bit set? That's not at\nall what I would have expected.\n\n> To fix this when the reset is running if there is not a file in the\n> working directory and if it will be missing with the new index entry or\n> was not missing in the previous version, we create the previous index\n> version of the file in the working directory so that status will report\n> correctly and the files will be availble for the user to deal with.\n\ns/availble/available/\n\n>\n> This fixes a documented failure from t1092 that was created in 19a0acc\n> (t1092: test interesting sparse-checkout scenarios, 2021-01-23).\n>\n> Signed-off-by: Kevin Willford \n> Signed-off-by: Derrick Stolee \n> Signed-off-by: Victoria Dye \n> ---\n> builtin/reset.c | 24 ++++++++--\n> t/t1092-sparse-checkout-compatibility.sh | 4 +-\n> t/t7114-reset-sparse-checkout.sh | 61 ++++++++++++++++++++++++\n> 3 files changed, 83 insertions(+), 6 deletions(-)\n> create mode 100755 t/t7114-reset-sparse-checkout.sh\n>\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 51c9e2f43ff..3b75d3b2f20 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -25,6 +25,8 @@\n> #include \"cache-tree.h\"\n> #include \"submodule.h\"\n> #include \"submodule-config.h\"\n> +#include \"dir.h\"\n> +#include \"entry.h\"\n>\n> #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n>\n> @@ -130,11 +132,27 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> int intent_to_add = *(int *)data;\n>\n> for (i = 0; i < q->nr; i++) {\n> + int pos;\n> struct diff_filespec *one = q->queue[i]->one;\n> - int is_missing = !(one->mode && !is_null_oid(&one->oid));\n> + struct diff_filespec *two = q->queue[i]->two;\n> + int is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n\nIsn't !is_null_oid(&one->oid) redundant to checking one->mode? When\ndoes the diff machinery ever give you a non-zero mode with a null oid?\n\nAlso, is_in_reset_tree == !is_missing; I'll note that below.\n\n> struct cache_entry *ce;\n>\n> + /*\n> + * If the file being reset has `skip-worktree` enabled, we need\n> + * to check it out to prevent the file from being hard reset.\n\nI don't understand this comment. If the file wasn't originally in the\nindex (is_missing), and is being added to it, and is correctly marked\nas skip_worktree, and the file isn't in the working tree, then it\nsounds like everything is already in a good state. Files outside the\nsparse checkout are meant to have the skip_worktree bit set and be\nmissing from the working tree.\n\nAlso, I don't know what you mean by 'hard reset' here.\n\n> + */\n> + pos = cache_name_pos(two->path, strlen(two->path));\n> + if (pos >= 0 && ce_skip_worktree(active_cache[pos])) {\n> + struct checkout state = CHECKOUT_INIT;\n> + state.force = 1;\n> + state.refresh_cache = 1;\n> + state.istate = &the_index;\n> +\n> + checkout_entry(active_cache[pos], &state, NULL, NULL);\n\nDoes this introduce an error in the opposite direction from the one\nstated in the commit message? Namely we have two things that should\nbe in sync: the skip_worktree flag stating whether the file should be\npresent in the working directory (skip_worktree), and the question of\nwhether the file is actually in the working directory. In the commit\nmessage, you pointed out a case where the y were out of sync one way:\nthe skip_worktree flag was not set but the file was missing. Here you\nsay the skip_worktree flag is set, but you add it to the working tree\nanyway.\n\nOr am I misunderstanding the code?\n\n> + }\n> +\n\n[I did some slight editing to the diff to make the next two parts\nappear next to each other]\n\n> - if (is_missing && !intent_to_add) {\n> + if (!is_in_reset_tree && !intent_to_add) {\n\nI thought this was some subtle bugfix or something, and spent a while\ntrying to figure it out, before realizing that is_in_reset_tree was\nsimply defined as !is_missing (for some reason I was assuming it was\ndealing with two->mode while is_missing was looking at one->mode). So\nthis is a simple variable renaming, which I think is probably good,\nbut I'd prefer if this was separated into a different patch to make it\neasier to review.\n\n> remove_file_from_cache(one->path);\n> continue;\n> }\n> @@ -144,7 +162,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> if (!ce)\n> die(_(\"make_cache_entry failed for path '%s'\"),\n> one->path);\n> - if (is_missing) {\n> + if (!is_in_reset_tree) {\n\nsame note as above; the variable rename is good, but should be a separate patch.\n\n> ce->ce_flags |= CE_INTENT_TO_ADD;\n> set_object_name_for_intent_to_add_entry(ce);\n> }\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 886e78715fe..c5977152661 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -459,9 +459,7 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n> test_all_match git blame deep/deeper2/deepest/a\n> '\n>\n> -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> -# in this scenario, but it shouldn't.\n> -test_expect_failure 'checkout and reset (mixed)' '\n> +test_expect_success 'checkout and reset (mixed)' '\n> init_repos &&\n>\n> test_all_match git checkout -b reset-test update-deep &&\n> diff --git a/t/t7114-reset-sparse-checkout.sh b/t/t7114-reset-sparse-checkout.sh\n> new file mode 100755\n> index 00000000000..a8029707fb1\n> --- /dev/null\n> +++ b/t/t7114-reset-sparse-checkout.sh\n> @@ -0,0 +1,61 @@\n> +#!/bin/sh\n> +\n> +test_description='reset when using a sparse-checkout'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> + test_tick &&\n> + echo \"checkout file\" >c &&\n> + echo \"modify file\" >m &&\n> + echo \"delete file\" >d &&\n> + git add . &&\n> + git commit -m \"initial commit\" &&\n> + echo \"added file\" >a &&\n> + echo \"modification of a file\" >m &&\n> + git rm d &&\n> + git add . &&\n> + git commit -m \"second commit\" &&\n> + git checkout -b endCommit\n> +'\n> +\n> +test_expect_success 'reset when there is a sparse-checkout' '\n> + echo \"/c\" >.git/info/sparse-checkout &&\n> + test_config core.sparsecheckout true &&\n> + git checkout -B resetBranch &&\n> + test_path_is_missing m &&\n> + test_path_is_missing a &&\n> + test_path_is_missing d &&\n> + git reset HEAD~1 &&\n> + echo \"checkout file\" >expect &&\n> + test_cmp expect c &&\n> + echo \"added file\" >expect &&\n> + test_cmp expect a &&\n> + echo \"modification of a file\" >expect &&\n> + test_cmp expect m &&\n> + test_path_is_missing d\n> +'\n> +\n> +test_expect_success 'reset after deleting file without skip-worktree bit' '\n> + git checkout -f endCommit &&\n> + git clean -xdf &&\n> + cat >.git/info/sparse-checkout <<-\\EOF &&\n> + /c\n> + /m\n> + EOF\n> + test_config core.sparsecheckout true &&\n> + git checkout -B resetAfterDelete &&\n> + test_path_is_file m &&\n> + test_path_is_missing a &&\n> + test_path_is_missing d &&\n> + rm -f m &&\n> + git reset HEAD~1 &&\n> + echo \"checkout file\" >expect &&\n> + test_cmp expect c &&\n> + echo \"added file\" >expect &&\n> + test_cmp expect a &&\n> + test_path_is_missing m &&\n> + test_path_is_missing d\n> +'\n> +\n> +test_done\n> --\n> gitgitgadget\n>\n"},{"id":"438017","messageId":"CABPp-BHD3khX+q7crk4JnNbGd9fN-vJW0L1x=5JdzkFL6X1BBg@mail.gmail.com","threadId":"56614","inReplyTo":"f7cb9013d46731855c3ed42add62b021c0ad0c73.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T02:00:16Z","receivedAt":"2021-10-06T02:00:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 5, 2021 at 6:20 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add a new `--force-full-index` option to `git update-index`, which skips\n> explicitly setting `command_requires_full_index`. This lets `git\n> update-index --force-full-index` run as a command without sparse index\n> compatibility implemented, even after it receives sparse index compatibility\n> updates.\n>\n> By using `git update-index --force-full-index` in the `t1092` test\n> `sparse-index is expanded and converted back`, commands can continue to\n> integrate with the sparse index without the need to keep modifying the\n> command used in the test.\n\nSo...we're adding a permanent user-facing command line flag, whose\npurpose is just to help us with the transition work of implementing\nsparse indexes everywhere? Am I reading that right, or is that just\nthe reason for t1092 and there are more reasons for it elsewhere?\n\nAlso, I'm curious if update-index is the right place to add this. If\nyou don't want a sparse index anymore, wouldn't a user want to run\n git sparse-checkout disable\n? Or is the point that you do want to keep the sparse checkout, but\nyou just don't want the index to also be sparse? Still, even in that\ncase, it seems like adding a subcommand or flag to an existing\nsparse-checkout subcommand would feel more natural, since\nsparse-checkout is the command the user uses to request to get into a\nsparse-checkout and sparse index.\n\n\n> Signed-off-by: Victoria Dye \n> ---\n> Documentation/git-update-index.txt | 5 +++++\n> builtin/update-index.c | 11 +++++++++++\n> t/t1092-sparse-checkout-compatibility.sh | 2 +-\n> 3 files changed, 17 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\n> index 2853f168d97..06255e321a3 100644\n> --- a/Documentation/git-update-index.txt\n> +++ b/Documentation/git-update-index.txt\n> @@ -24,6 +24,7 @@ SYNOPSIS\n> [--[no-]fsmonitor]\n> [--really-refresh] [--unresolve] [--again | -g]\n> [--info-only] [--index-info]\n> + [--force-full-index]\n> [-z] [--stdin] [--index-version ]\n> [--verbose]\n> [--] [...]\n> @@ -170,6 +171,10 @@ time. Version 4 is relatively young (first released in 1.8.0 in\n> October 2012). Other Git implementations such as JGit and libgit2\n> may not support it yet.\n>\n> +--force-full-index::\n> + Force the command to operate on a full index, expanding a sparse\n> + index if necessary.\n> +\n> -z::\n> Only meaningful with `--stdin` or `--index-info`; paths are\n> separated with NUL character instead of LF.\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 187203e8bb5..32ada3ead77 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -964,6 +964,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> int split_index = -1;\n> int force_write = 0;\n> int fsmonitor = -1;\n> + int use_default_full_index = 0;\n> struct lock_file lock_file = LOCK_INIT;\n> struct parse_opt_ctx_t ctx;\n> strbuf_getline_fn getline_fn;\n> @@ -1069,6 +1070,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> {OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n> N_(\"clear fsmonitor valid bit\"),\n> PARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n> + OPT_SET_INT(0, \"force-full-index\", &use_default_full_index,\n> + N_(\"run with full index explicitly required\"), 1),\n> OPT_END()\n> };\n>\n> @@ -1082,6 +1085,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> if (newfd < 0)\n> lock_error = errno;\n>\n> + /*\n> + * If --force-full-index is set, the command should skip manually\n> + * setting `command_requires_full_index`.\n> + */\n> + prepare_repo_settings(r);\n> + if (!use_default_full_index)\n> + r->settings.command_requires_full_index = 1;\n> +\n> entries = read_cache();\n> if (entries < 0)\n> die(\"cache corrupted\");\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index c5977152661..b3c0d3b98ee 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> init_repos &&\n>\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> - git -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n> + git -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index &&\n> test_region index convert_to_sparse trace2.txt &&\n> test_region index ensure_full_index trace2.txt\n> '\n> --\n> gitgitgadget\n"},{"id":"438018","messageId":"CABPp-BFAiPsHZPjJtuuqfuxt950yo-pqpKD8TG9sNWDs-bEbEw@mail.gmail.com","threadId":"56614","inReplyTo":"c7e9d9f4e03fe0dbd8a112460f0ac0864f087c8f.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] reset: expand test coverage for sparse checkouts","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T02:04:37Z","receivedAt":"2021-10-06T02:04:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 5, 2021 at 6:21 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add new tests for `--merge` and `--keep` modes, as well as mixed reset with\n> pathspecs both inside and outside of the sparse checkout definition. New\n> performance test cases exercise various execution paths for `reset`.\n>\n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Derrick Stolee \n> Signed-off-by: Victoria Dye \n> ---\n> t/perf/p2000-sparse-operations.sh | 3 +\n> t/t1092-sparse-checkout-compatibility.sh | 107 +++++++++++++++++++++++\n> 2 files changed, 110 insertions(+)\n>\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index 597626276fb..bfd332120c8 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -110,5 +110,8 @@ test_perf_on_all git add -A\n> test_perf_on_all git add .\n> test_perf_on_all git commit -a -m A\n> test_perf_on_all git checkout -f -\n> +test_perf_on_all git reset\n> +test_perf_on_all git reset --hard\n> +test_perf_on_all git reset -- does-not-exist\n>\n> test_done\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index b3c0d3b98ee..f0723a6ac97 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -479,6 +479,113 @@ test_expect_success 'checkout and reset (mixed) [sparse]' '\n> test_sparse_match git reset update-folder2\n> '\n>\n> +# NEEDSWORK: with mixed reset, files with differences between HEAD and \n> +# will be added to the work tree even if outside the sparse checkout\n> +# definition, and even if the file is modified to a state of having no local\n> +# changes. The file is \"re-ignored\" if a hard reset is executed. We may want to\n> +# change this behavior in the future and enforce that files are not written\n> +# outside of the sparse checkout definition.\n\nYeah, I think this comment highlights some of the reasons that writing\nthe file to the working directory for those files isn't the way I'd\nprefer to resolve the inconsistency between the skip-worktree bit and\nthe presence of the file in the working directory.\n\n> +test_expect_success 'checkout and mixed reset file tracking [sparse]' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + test_all_match git reset update-folder1 &&\n> + test_all_match git reset update-deep &&\n> +\n> + # At this point, there are no changes in the working tree. However,\n> + # folder1/a now exists locally (even though it is outside of the sparse\n> + # paths).\n> + run_on_sparse test_path_exists folder1 &&\n> +\n> + run_on_all rm folder1/a &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset --hard update-deep &&\n> + run_on_sparse test_path_is_missing folder1 &&\n> + test_path_exists full-checkout/folder1\n> +'\n> +\n> +test_expect_success 'checkout and reset (merge)' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + run_on_all ../edit-contents a &&\n> + test_all_match git reset --merge deepest &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset --hard update-deep &&\n> + run_on_all ../edit-contents deep/a &&\n> + test_all_match test_must_fail git reset --merge deepest\n> +'\n> +\n> +test_expect_success 'checkout and reset (keep)' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + run_on_all ../edit-contents a &&\n> + test_all_match git reset --keep deepest &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset --hard update-deep &&\n> + run_on_all ../edit-contents deep/a &&\n> + test_all_match test_must_fail git reset --keep deepest\n> +'\n> +\n> +test_expect_success 'reset with pathspecs inside sparse definition' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + run_on_all ../edit-contents deep/a &&\n> +\n> + test_all_match git reset base -- deep/a &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset base -- nonexistent-file &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset deepest -- deep &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'reset with sparse directory pathspec outside definition' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + test_all_match git reset --hard update-folder1 &&\n> + test_all_match git reset base -- folder1 &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'reset with pathspec match in sparse directory' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + test_all_match git reset --hard update-folder1 &&\n> + test_all_match git reset base -- folder1/a &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'reset with wildcard pathspec' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b reset-test update-deep &&\n> + test_all_match git reset --hard update-folder1 &&\n> + test_all_match git reset base -- \\*/a &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> test_expect_success 'merge, cherry-pick, and rebase' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n>\n"},{"id":"438019","messageId":"CABPp-BHKjLVpL_U5viVA++hCZmGKBHxnubGCk=1YhTkpFc-E-Q@mail.gmail.com","threadId":"56614","inReplyTo":"49813c8d9ed94fd56f30eb204d346eb5a30473ca.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] reset: integrate with sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T02:15:55Z","receivedAt":"2021-10-06T02:16:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 5, 2021 at 6:21 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> `reset --soft` does not modify the index, so no compatibility changes are\n> needed for it to function without expanding the index. For all other reset\n> modes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is\n> explicitly expanded with `ensure_full_index` to maintain current behavior.\n\n\"to maintain current behavior\"? You are changing code here, which\nsuggests some kind of behavior is changing, but that description seems\nto be claiming the opposite. Is it some kind of preventative change\nto add ensure_full_index calls in an additional place, with a later\npatch in the series intending to remove the other one(s), so you're\nmaking sure that later changes won't cause unwanted behavioral\nchanges? Or was something else meant here?\n\nIf the above wasn't what you meant, but you're adding\nensure_full_index calls, does that suggest that we had some important\ncode paths that were not protected by such calls? I thought Stolee\nsaid we had them all covered (at least to the best of our knowledge),\nso I'm curious if we just discovered we missed some. If so, are there\nother codepaths like this one where we missed protective\nensure_full_index calls?\n\n> Additionally, the `read_cache()` check verifying an uncorrupted index is\n> moved after argument parsing and preparing the repo settings. The index is\n> not used by the preceding argument handling, but `read_cache()` does need to\n> be run after enabling sparse index for the command and before resetting.\n\nThis seems to be discussing what code changes are being made, but not\nwhy. I'm guessing at the reasoning, but is it something along the\nlines of:\n\n\"\"\"\nAlso, make sure to read_cache() after setting\ncommand_requires_full_index = 0, so that we don't unnecessarily expand\nthe index as part of our early index-corruption check.\n\"\"\"\n\n?\n\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/reset.c | 10 +++++++---\n> cache-tree.c | 1 +\n> 2 files changed, 8 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 3b75d3b2f20..e1f2a2bb2c4 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -184,6 +184,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n> opt.flags.override_submodule_config = 1;\n> opt.repo = the_repository;\n>\n> + ensure_full_index(&the_index);\n> if (do_diff_cache(tree_oid, &opt))\n> return 1;\n> diffcore_std(&opt);\n> @@ -261,9 +262,6 @@ static void parse_args(struct pathspec *pathspec,\n> }\n> *rev_ret = rev;\n>\n> - if (read_cache() < 0)\n> - die(_(\"index file corrupt\"));\n> -\n> parse_pathspec(pathspec, 0,\n> PATHSPEC_PREFER_FULL |\n> (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n> @@ -409,6 +407,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n> if (intent_to_add && reset_type != MIXED)\n> die(_(\"-N can only be used with --mixed\"));\n>\n> + prepare_repo_settings(the_repository);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> + if (read_cache() < 0)\n> + die(_(\"index file corrupt\"));\n> +\n> /* Soft reset does not touch the index file nor the working tree\n> * at all, but requires them in a good order. Other resets reset\n> * the index file to the tree object we are switching to. */\n> diff --git a/cache-tree.c b/cache-tree.c\n> index 90919f9e345..9be19c85b66 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n> cache_tree_free(&istate->cache_tree);\n> istate->cache_tree = cache_tree();\n>\n> + ensure_full_index(istate);\n> prime_cache_tree_rec(r, istate->cache_tree, tree);\n> istate->cache_changed |= CACHE_TREE_CHANGED;\n> trace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n> --\n> gitgitgadget\n>\n"},{"id":"438020","messageId":"CABPp-BFHNOMTerrV2Bhufb5d9d6JuM+CWW_t1jYg1Uu7F1dhhA@mail.gmail.com","threadId":"56614","inReplyTo":"78cd85d8dcc790251ce8235e649902cf6adf091a.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] reset: make sparse-aware (except --mixed)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T03:43:13Z","receivedAt":"2021-10-06T03:43:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 5, 2021 at 6:21 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> In order to accurately reconstruct the cache tree in `prime_cache_tree_rec`,\n> the function must determine whether the currently-processing directory in\n> the tree is sparse or not. If it is not sparse, the tree is parsed and\n> subtree recursively constructed. If it is sparse, no subtrees are added to\n> the tree and the entry count is set to 1 (representing the sparse directory\n> itself).\n>\n> Signed-off-by: Victoria Dye \n> ---\n> cache-tree.c | 44 +++++++++++++++++++++---\n> cache.h | 10 ++++++\n> read-cache.c | 22 ++++++++----\n> t/t1092-sparse-checkout-compatibility.sh | 15 ++++++--\n> 4 files changed, 78 insertions(+), 13 deletions(-)\n>\n> diff --git a/cache-tree.c b/cache-tree.c\n> index 9be19c85b66..9021669d682 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -740,15 +740,29 @@ out:\n> return ret;\n> }\n>\n> +static void prime_cache_tree_sparse_dir(struct repository *r,\n> + struct cache_tree *it,\n> + struct tree *tree,\n> + struct strbuf *tree_path)\n> +{\n> +\n> + oidcpy(&it->oid, &tree->object.oid);\n> + it->entry_count = 1;\n> + return;\n\nWhy are 'r' and 'tree_path' passed to this function?\n\n> +}\n> +\n> static void prime_cache_tree_rec(struct repository *r,\n> struct cache_tree *it,\n> - struct tree *tree)\n> + struct tree *tree,\n> + struct strbuf *tree_path)\n> {\n> + struct strbuf subtree_path = STRBUF_INIT;\n> struct tree_desc desc;\n> struct name_entry entry;\n> int cnt;\n>\n> oidcpy(&it->oid, &tree->object.oid);\n> +\n\nWhy the blank line addition here?\n\n> init_tree_desc(&desc, tree->buffer, tree->size);\n> cnt = 0;\n> while (tree_entry(&desc, &entry)) {\n> @@ -757,27 +771,49 @@ static void prime_cache_tree_rec(struct repository *r,\n> else {\n> struct cache_tree_sub *sub;\n> struct tree *subtree = lookup_tree(r, &entry.oid);\n> +\n> if (!subtree->object.parsed)\n> parse_tree(subtree);\n> sub = cache_tree_sub(it, entry.path);\n> sub->cache_tree = cache_tree();\n> - prime_cache_tree_rec(r, sub->cache_tree, subtree);\n\n> + strbuf_reset(&subtree_path);\n> + strbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1);\n> + strbuf_addbuf(&subtree_path, tree_path);\n> + strbuf_add(&subtree_path, entry.path, entry.pathlen);\n> + strbuf_addch(&subtree_path, '/');\n\nReconstructing the full path each time? And despite only being useful\nfor the sparse-index case?\n\nWould it be better to drop subtree_path from this function, then\nappend entry.path + '/' here to tree_path, and then after the if-block\nbelow, call strbuf_setlen to remove the part that this function call\nadded? That way, we don't need subtree_path, and don't have to copy\nthe leading path every time.\n\nAlso, maybe it'd be better to only do this strbuf manipulation if\nr->index->sparse_index, since it's not ever used otherwise?\n\n> +\n> + /*\n> + * If a sparse index is in use, the directory being processed may be\n> + * sparse. To confirm that, we can check whether an entry with that\n> + * exact name exists in the index. If it does, the created subtree\n> + * should be sparse. Otherwise, cache tree expansion should continue\n> + * as normal.\n> + */\n> + if (r->index->sparse_index &&\n> + index_entry_exists(r->index, subtree_path.buf, subtree_path.len))\n> + prime_cache_tree_sparse_dir(r, sub->cache_tree, subtree, &subtree_path);\n> + else\n> + prime_cache_tree_rec(r, sub->cache_tree, subtree, &subtree_path);\n> cnt += sub->cache_tree->entry_count;\n> }\n> }\n> it->entry_count = cnt;\n> +\n> + strbuf_release(&subtree_path);\n> }\n>\n> void prime_cache_tree(struct repository *r,\n> struct index_state *istate,\n> struct tree *tree)\n> {\n> + struct strbuf tree_path = STRBUF_INIT;\n> +\n> trace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n> cache_tree_free(&istate->cache_tree);\n> istate->cache_tree = cache_tree();\n>\n> - ensure_full_index(istate);\n> - prime_cache_tree_rec(r, istate->cache_tree, tree);\n> + prime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n> + strbuf_release(&tree_path);\n> istate->cache_changed |= CACHE_TREE_CHANGED;\n> trace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n> }\n> diff --git a/cache.h b/cache.h\n> index f6295f3b048..1d3e4665562 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n> */\n> int index_name_pos(struct index_state *, const char *name, int namelen);\n>\n> +/*\n> + * Determines whether an entry with the given name exists within the\n> + * given index. The return value is 1 if an exact match is found, otherwise\n> + * it is 0. Note that, unlike index_name_pos, this function does not expand\n> + * the index if it is sparse. If an item exists within the full index but it\n> + * is contained within a sparse directory (and not in the sparse index), 0 is\n> + * returned.\n> + */\n> +int index_entry_exists(struct index_state *, const char *name, int namelen);\n> +\n> /*\n> * Some functions return the negative complement of an insert position when a\n> * precise match was not found but a position was found where the entry would\n> diff --git a/read-cache.c b/read-cache.c\n> index f5d4385c408..ea1166895f8 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -551,7 +551,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n> return 0;\n> }\n>\n> -static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n> +static int index_name_stage_pos(struct index_state *istate,\n> + const char *name, int namelen,\n> + int stage,\n> + int search_sparse)\n\nIt'd be nicer to make search_sparse an enum defined within this file, so that...\n\n> {\n> int first, last;\n>\n> @@ -570,7 +573,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n> first = next+1;\n> }\n>\n> - if (istate->sparse_index &&\n> + if (search_sparse && istate->sparse_index &&\n> first > 0) {\n> /* Note: first <= istate->cache_nr */\n> struct cache_entry *ce = istate->cache[first - 1];\n> @@ -586,7 +589,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n> ce_namelen(ce) < namelen &&\n> !strncmp(name, ce->name, ce_namelen(ce))) {\n> ensure_full_index(istate);\n> - return index_name_stage_pos(istate, name, namelen, stage);\n> + return index_name_stage_pos(istate, name, namelen, stage, search_sparse);\n> }\n> }\n>\n> @@ -595,7 +598,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n>\n> int index_name_pos(struct index_state *istate, const char *name, int namelen)\n> {\n> - return index_name_stage_pos(istate, name, namelen, 0);\n> + return index_name_stage_pos(istate, name, namelen, 0, 1);\n\n...this could use SEARCH_SPARSE or some name like that which is more\nmeaningful than \"1\" here.\n\n> +}\n> +\n> +int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n> +{\n> + return index_name_stage_pos(istate, name, namelen, 0, 0) >= 0;\n\n...and likewise this spot could use SEARCH_FULL or some name like\nthat, which is more meaningful than the second \"0\".\n\nSimilarly for multiple call sites below...\n\n\n> }\n>\n> int remove_index_entry_at(struct index_state *istate, int pos)\n> @@ -1222,7 +1230,7 @@ static int has_dir_name(struct index_state *istate,\n> */\n> }\n>\n> - pos = index_name_stage_pos(istate, name, len, stage);\n> + pos = index_name_stage_pos(istate, name, len, stage, 1);\n> if (pos >= 0) {\n> /*\n> * Found one, but not so fast. This could\n> @@ -1322,7 +1330,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> strcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n> pos = index_pos_to_insert_pos(istate->cache_nr);\n> else\n> - pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n> + pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n>\n> /* existing match? Just replace it. */\n> if (pos >= 0) {\n> @@ -1357,7 +1365,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> if (!ok_to_replace)\n> return error(_(\"'%s' appears as both a file and as a directory\"),\n> ce->name);\n> - pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n> + pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n> pos = -pos-1;\n> }\n> return pos + 1;\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index f0723a6ac97..e301ef5633a 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -786,9 +786,9 @@ test_expect_success 'sparse-index is not expanded' '\n> ensure_not_expanded checkout - &&\n> ensure_not_expanded switch rename-out-to-out &&\n> ensure_not_expanded switch - &&\n> - git -C sparse-index reset --hard &&\n> + ensure_not_expanded reset --hard &&\n> ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n> - git -C sparse-index reset --hard &&\n> + ensure_not_expanded reset --hard &&\n> ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n>\n> echo >>sparse-index/README.md &&\n> @@ -798,6 +798,17 @@ test_expect_success 'sparse-index is not expanded' '\n> echo >>sparse-index/untracked.txt &&\n> ensure_not_expanded add . &&\n>\n> + for ref in update-deep update-folder1 update-folder2 update-deep\n> + do\n> + echo >>sparse-index/README.md &&\n> + ensure_not_expanded reset --hard $ref || return 1\n> + done &&\n> +\n> + ensure_not_expanded reset --hard update-deep &&\n> + ensure_not_expanded reset --keep base &&\n> + ensure_not_expanded reset --merge update-deep &&\n> + ensure_not_expanded reset --hard &&\n> +\n> ensure_not_expanded checkout -f update-deep &&\n> test_config -C sparse-index pull.twohead ort &&\n> (\n> --\n> gitgitgadget\n"},{"id":"438021","messageId":"CABPp-BHDog6U1bFKurNq+sKbNPOnmqbhB8W2wg0UeF=jb+ZScw@mail.gmail.com","threadId":"56614","inReplyTo":"5eaae0825af4cee84040b784c32190bb1de2f919.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/7] reset: make --mixed sparse-aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T04:43:11Z","receivedAt":"2021-10-06T04:43:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 5, 2021 at 6:21 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Sparse directory entries are \"diffed\" as trees in `diff_cache` (used\n> internally by `reset --mixed`), following a code path separate from\n> individual file handling. The use of `diff_tree_oid` there requires setting\n> explicit `change` and `add_remove` functions to process the internal\n> contents of a sparse directory.\n>\n> Additionally, the `recursive` diff option handles cases in which `reset\n> --mixed` must diff/merge files that are nested multiple levels deep in a\n> sparse directory.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/reset.c | 30 +++++++++++++++++++++++-\n> t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++-\n> 2 files changed, 41 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index e1f2a2bb2c4..ceb9b122897 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -175,6 +175,8 @@ static int read_from_tree(const struct pathspec *pathspec,\n> int intent_to_add)\n> {\n> struct diff_options opt;\n> + unsigned int i;\n> + char *skip_worktree_seen = NULL;\n>\n> memset(&opt, 0, sizeof(opt));\n> copy_pathspec(&opt.pathspec, pathspec);\n> @@ -182,9 +184,35 @@ static int read_from_tree(const struct pathspec *pathspec,\n> opt.format_callback = update_index_from_diff;\n> opt.format_callback_data = &intent_to_add;\n> opt.flags.override_submodule_config = 1;\n> + opt.flags.recursive = 1;\n> opt.repo = the_repository;\n> + opt.change = diff_change;\n> + opt.add_remove = diff_addremove;\n> +\n> + /*\n> + * When pathspec is given for resetting a cone-mode sparse checkout, it may\n> + * identify entries that are nested in sparse directories, in which case the\n> + * index should be expanded. For the sake of efficiency, this check is\n> + * overly-cautious: anything with a wildcard or a magic prefix requires\n> + * expansion, as well as literal paths that aren't in the sparse checkout\n> + * definition AND don't match any directory in the index.\n\ns/efficiency/efficiency of checking/ ? Being overly-cautious suggests\nyou'll expand to a full index more than is needed, and full indexes\nare more expensive. But perhaps the checking would be expensive too\nso you have a tradeoff?\n\nOr maybe s/efficiency/simplicity/?\n\n> + */\n> + if (pathspec->nr && the_index.sparse_index) {\n> + if (pathspec->magic || pathspec->has_wildcard) {\n> + ensure_full_index(&the_index);\n\ndir.c has the notion of matching the characters preceding the wildcard\ncharacters; look for \"no_wildcard_len\". If the pathspec doesn't match\na path up to no_wildcard_len, then the wildcard character(s) later in\nthe pathspec can't make the pathspec match that path.\n\nIt might at least be worth mentioning this as a possible future optimization.\n\n> + } else {\n> + for (i = 0; i < pathspec->nr; i++) {\n> + if (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) &&\n> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) {\n\nWhat if the pathspec corresponds to a sparse-directory in the index,\nbut possibly without the trailing '/' character? e.g.:\n\n git reset HEAD~1 -- sparse-directory\n\nOne should be able to reset that directory without recursing into\nit...does this code handle that? Does it handle it if we add the\ntrailing slash on the path for the reset command line?\n\n> + ensure_full_index(&the_index);\n> + break;\n> + }\n> + }\n> + }\n> + }\n> +\n> + free(skip_worktree_seen);\n>\n> - ensure_full_index(&the_index);\n> if (do_diff_cache(tree_oid, &opt))\n> return 1;\n> diffcore_std(&opt);\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index e301ef5633a..4afcbc2d673 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -804,11 +804,22 @@ test_expect_success 'sparse-index is not expanded' '\n> ensure_not_expanded reset --hard $ref || return 1\n> done &&\n>\n> + ensure_not_expanded reset --mixed base &&\n> ensure_not_expanded reset --hard update-deep &&\n> ensure_not_expanded reset --keep base &&\n> ensure_not_expanded reset --merge update-deep &&\n> - ensure_not_expanded reset --hard &&\n\nThis commit was only touching the --mixed case; why is it removing one\nof the tests for --hard?\n\n>\n> + ensure_not_expanded reset base -- deep/a &&\n> + ensure_not_expanded reset base -- nonexistent-file &&\n> + ensure_not_expanded reset deepest -- deep &&\n> +\n> + # Although folder1 is outside the sparse definition, it exists as a\n> + # directory entry in the index, so it will be reset without needing to\n> + # expand the full index.\n\nAh, I think this answers one of my earlier questions. Does it also\nwork with 'folder1/' as well as 'folder1'?\n\n> + ensure_not_expanded reset --hard update-folder1 &&\n\nWait...is update-folder1 a branch or a path? And if this commit is\nabout --mixed, why are --hard testcases being added?\n\n> + ensure_not_expanded reset base -- folder1 &&\n> +\n> + ensure_not_expanded reset --hard update-deep &&\n\nanother --hard testcase...was this an accidental squash by chance?\n\n\n\n> ensure_not_expanded checkout -f update-deep &&\n> test_config -C sparse-index pull.twohead ort &&\n> (\n> --\n> gitgitgadget\n>\n"},{"id":"438023","messageId":"CABPp-BE+rc9_V9AHrXGRCJZJ9nG8FzVxTqmB+-nA+GuNboonDg@mail.gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] Sparse Index: integrate with reset","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-06T05:46:00Z","receivedAt":"2021-10-06T05:46:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Victoria,\n\nOn Tue, Oct 5, 2021 at 6:20 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> This series integrates the sparse index with git reset and provides\n> miscellaneous fixes and improvements to the command in sparse checkouts.\n> This includes:\n>\n> 1. tests added to t1092 and p2000 to establish the baseline functionality\n> of the command\n> 2. repository settings to enable the sparse index with ensure_full_index\n> guarding any code paths that break tests without other compatibility\n> updates.\n> 3. modifications to remove or reduce the scope in which ensure_full_index\n> must be called.\n>\n> The sparse index updates are predicated on a fix originating from the\n> microsoft/git fork [1], correcting how git reset --mixed handles resetting\n> entries outside the sparse checkout definition. Additionally, a performance\n> \"bug\" in next_cache_entry with sparse index is corrected, preventing\n> repeatedly looping over already-searched entries.\n>\n> The p2000 tests demonstrate an overall ~70% execution time reduction across\n> all tested usages of git reset using a sparse index:\n>\n> Test before after\n> ------------------------------------------------------------------------\n> 2000.22: git reset (full-v3) 0.48 0.51 +6.3%\n> 2000.23: git reset (full-v4) 0.47 0.50 +6.4%\n> 2000.24: git reset (sparse-v3) 0.93 0.30 -67.7%\n> 2000.25: git reset (sparse-v4) 0.94 0.29 -69.1%\n> 2000.26: git reset --hard (full-v3) 0.69 0.68 -1.4%\n> 2000.27: git reset --hard (full-v4) 0.75 0.68 -9.3%\n> 2000.28: git reset --hard (sparse-v3) 1.29 0.34 -73.6%\n> 2000.29: git reset --hard (sparse-v4) 1.31 0.34 -74.0%\n> 2000.30: git reset -- does-not-exist (full-v3) 0.54 0.51 -5.6%\n> 2000.31: git reset -- does-not-exist (full-v4) 0.54 0.52 -3.7%\n> 2000.32: git reset -- does-not-exist (sparse-v3) 1.02 0.31 -69.6%\n> 2000.33: git reset -- does-not-exist (sparse-v4) 1.07 0.30 -72.0%\n>\n>\n>\n> Changes since V1\n> ================\n>\n> * Add --force-full-index option to update-index. The option is used\n> circumvent changing command_requires_full_index from its default value -\n> right now this is effectively a no-op, but will change once update-index\n> is integrated with sparse index. By using this option in the t1092\n> expand/collapse test, the command used to test will not need to be\n> updated with subsequent sparse index integrations.\n> * Update implementation of mixed reset for entries outside sparse checkout\n> definition. The condition in which a file should be checked out before\n> index reset is simplified to \"if it has skip-worktree enabled and a reset\n> would change the file, check it out\".\n> * After checking the behavior of update_index_from_diff with renames,\n> found that the diff used by reset does not produce diff queue entries\n> with different pathnames for one and two. Because of this, and that\n> nothing in the implementation seems to rely on identical path names, no\n> BUG check is added.\n> * Correct a bug in the sparse index is not expanded tests in t1092 where\n> failure of a git reset --mixed test was not being reported. Test now\n> verifies an appropriate scenario with corrected failure-checking.\n\nI read over the first six patches. I tried to read over the seventh,\nbut I've never figured out cache_bottom for some reason and I did\nnothing beyond spot checking when Stolee touched that area either.\n\nAnyway, I had lots of little comments, tweaks to the way to fix the\ninconsistency in patch 1, various questions, etc. It probably adds up\nto a lot, but it's all small fixable stuff; overall it looks like you\n(and Kevin) are making a solid contribution on the sparse-checkout\nstuff; I look forward to reading the next round.\n"},{"id":"438053","messageId":"f6ccec5f-1946-dc13-c19e-d6889f943bc2@gmail.com","threadId":"56614","inReplyTo":"22c69bc60308fef13acd7c3aab4e11e175c89440.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-06T10:31:12Z","receivedAt":"2021-10-06T10:31:25Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 05/10/21 20.20, Kevin Willford via GitGitGadget wrote:\n> When using the sparse checkout feature, 'git reset' will add entries to\n> the index that will have the skip-worktree bit off but will leave the\n> working directory empty. File data is lost because the index version of\n> the files has been changed but there is nothing that is in the working\n> directory. This will cause the next 'git status' call to show either\n> deleted for files modified or deleting or nothing for files added. The\n> added files should be shown as untracked and modified files should be\n> shown as modified.\n> \n\nBetter say `... but there is nothing in the working directory`.\n\n> To fix this when the reset is running if there is not a file in the\n> working directory and if it will be missing with the new index entry or\n> was not missing in the previous version, we create the previous index\n> version of the file in the working directory so that status will report\n> correctly and the files will be availble for the user to deal with.\n> \n\ns/availble/available\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"438055","messageId":"f30504f5-e4a3-1b93-08dc-fb9137d36f65@gmail.com","threadId":"56614","inReplyTo":"f7cb9013d46731855c3ed42add62b021c0ad0c73.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-06T10:33:16Z","receivedAt":"2021-10-06T10:33:22Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 05/10/21 20.20, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> Add a new `--force-full-index` option to `git update-index`, which skips\n> explicitly setting `command_requires_full_index`. This lets `git\n> update-index --force-full-index` run as a command without sparse index\n> compatibility implemented, even after it receives sparse index compatibility\n> updates.\n\n`... explicitly setting ...` or `... explicitly set ...`? I thought of \nthe latter.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"438056","messageId":"2d0c6f7c-ef7c-4d9a-4d57-d95404b6e20a@gmail.com","threadId":"56614","inReplyTo":"78cd85d8dcc790251ce8235e649902cf6adf091a.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] reset: make sparse-aware (except --mixed)","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-06T10:34:56Z","receivedAt":"2021-10-06T10:35:01Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 05/10/21 20.20, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> In order to accurately reconstruct the cache tree in `prime_cache_tree_rec`,\n> the function must determine whether the currently-processing directory in\n> the tree is sparse or not. If it is not sparse, the tree is parsed and\n> subtree recursively constructed. If it is sparse, no subtrees are added to\n> the tree and the entry count is set to 1 (representing the sparse directory\n> itself).\n> \n\nBetter say `If it is sparse, no subtrees ..., else the tree ...`\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"438057","messageId":"ca698f10-9870-021e-766c-d52c51447545@gmail.com","threadId":"56614","inReplyTo":"aa963eefae75d41983201e24398bc5692267b91b.1633440057.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 7/7] unpack-trees: improve performance of next_cache_entry","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-06T10:37:49Z","receivedAt":"2021-10-06T10:37:56Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 05/10/21 20.20, Victoria Dye via GitGitGadget wrote:\n> The `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n> (unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain\n> the benefit `cache_bottom` provides in non-sparse index cases, a separate\n> `hint` position indicates the first position `next_cache_entry` should\n> search, updated each execution with a new position. The performance of `git\n> reset -- does-not-exist` (testing the \"worst case\" in which all entries in\n> the index are unpacked with `next_cache_entry`) is significantly improved\n> for the sparse index case:\n\nDid you mean `a separate `hint` ... should be searched`?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"438064","messageId":"xmqq4k9utja9.fsf@gitster.g","threadId":"56614","inReplyTo":"9b99e856-24cc-03fd-7871-de92dc6e39b6@github.com","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-06T12:44:14Z","receivedAt":"2021-10-06T12:44:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Thanks for the thorough explanation, I'm on-board with your approach (and\n> will re-roll the series with that implemented). A lot of my thought process\n> (and confusion) came from a comment in e5ca291076 (t1092: document bad\n> sparse-checkout behavior, 2021-07-14) suggesting that full and sparse\n> checkouts should have the same result in scenarios like the one you\n> outlined above.\n\nThanks for bringing this up. I agree that it is crucial to clarify\nwhat use case we are aiming for. If the objective were to make a\nsparse checkout behave just like full checkout, the desired\nbehaviour would be very different from a system whose objective is\nto allow users to pretend as if the hidden parts of sparse checkout\ndo not even exist, which was the model my example was after. I\nagree with you that the \"comment\" in an earlier commit may have been\nunhelpful in that they stopped at \"should behave the same but they\nshouldn't\" without saying \"why they should behave the same\".\n\nIf the goal were to make sparse behave like full, continuing with\nthe previous example, after a\n\n $ git reset --mixed HEAD^\n\nthe user should be able to say\n\n $ git commit -a --amend\n\nto replace the original two-commit history with a single commit\nhistory that records the same resulting tree. If the path \"skip\"\nwere to be reset to the blob from the first commit, just like the\npath \"no-skip\" is, for such a \"commit -a --amend\" to work, we would\nneed to have a working tree file for \"skip\" magically materialized\nwith the contents from the *second* commit. After all, the whole\npoint of mixed (and soft) reset is that they do not (logically)\nchange the files in the working tree, so if you are resetting from\nthe second commit to the first, if you were to have a working tree\nfile, it should come from the second commit, so that both \"skip\"\nand \"no-skip\" should show \"changed in the working tree relative to\nthe index\", i.e.\n\n $ git reset --mixed HEAD^\n $ git ls-files -t\n M no-skip\n M skip\n\nWhile such a \"make sparse behave the same way as full\" can be made\ninternally consistent, however, as the above example shows, it would\nmake the resulting \"sparse checkout\" practically unusable.\n\nBy stepping back a bit and realizing that the reason why the user\nwanted to mark some path as \"skip-worktree\" was because the user had\nno intention to make any change to them, we can make it usable again,\nby not insisting that sparse should behave the same way as full.\n\nWhen we redesign these patches, I would like to see what we failed\nshort the last time gets improved. Instead of saying \"skip-worktree\nentries should stay so\" and stopping there, we should leave a note\nfor later readers to explain why they should.\n\nThanks.\n\n\n"},{"id":"438081","messageId":"xmqqo882rqm4.fsf@gitster.g","threadId":"56614","inReplyTo":"CABPp-BHKjLVpL_U5viVA++hCZmGKBHxnubGCk=1YhTkpFc-E-Q@mail.gmail.com","subject":"Re: [PATCH v2 4/7] reset: integrate with sparse index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-06T17:48:51Z","receivedAt":"2021-10-06T17:48:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Tue, Oct 5, 2021 at 6:21 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> `reset --soft` does not modify the index, so no compatibility changes are\n>> needed for it to function without expanding the index. For all other reset\n>> modes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is\n>> explicitly expanded with `ensure_full_index` to maintain current behavior.\n>\n> \"to maintain current behavior\"? You are changing code here, which\n> suggests some kind of behavior is changing, but that description seems\n> to be claiming the opposite. Is it some kind of preventative change\n> to add ensure_full_index calls in an additional place, with a later\n> patch in the series intending to remove the other one(s), so you're\n> making sure that later changes won't cause unwanted behavioral\n> changes? Or was something else meant here?\n>\n> If the above wasn't what you meant, but you're adding\n> ensure_full_index calls, does that suggest that we had some important\n> code paths that were not protected by such calls?\n\nThe original called read_cache() before we know which mode we\noperate in, near the end of parse_args(), which resulted in an\nunconditional call to ensure_full_index() in repo_read_index().\n\nThis patch delays the call to read_cache(). If parse_pathspec()\nand everything the original called after the point where it called\nread_cache() needed to have a populated in-core index, the change\ncan break things---I didn't check thoroughly, but I am guessing\nit is OK.\n\n>> Additionally, the `read_cache()` check verifying an uncorrupted index is\n>> moved after argument parsing and preparing the repo settings. The index is\n>> not used by the preceding argument handling, but `read_cache()` does need to\n>> be run after enabling sparse index for the command and before resetting.\n>\n> This seems to be discussing what code changes are being made, but not\n> why. I'm guessing at the reasoning, but is it something along the\n> lines of:\n>\n> \"\"\"\n> Also, make sure to read_cache() after setting\n> command_requires_full_index = 0, so that we don't unnecessarily expand\n> the index as part of our early index-corruption check.\n> \"\"\"\n\nI think it is more like \"we used to expand very early for all modes,\nbut with this change we move the read_cache() call to much later,\nand force it not to expand. The modes that call read_from_tree()\nneeds in-core index fully expanded, so we do so there, but the soft\nreset does not call it and would stop expanding.\"\n\n"},{"id":"438092","messageId":"24e500f9-cc63-2288-5be2-ec6ac2104025@github.com","threadId":"56614","inReplyTo":"CABPp-BFFG=puwpB4Gz8nWU=CNMUrLbH7XPTkpSGqWedTK2NWLg@mail.gmail.com","subject":"Re: [PATCH v2 1/7] reset: behave correctly with sparse-checkout","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-06T20:09:21Z","receivedAt":"2021-10-06T20:09:27Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n>> - int is_missing = !(one->mode && !is_null_oid(&one->oid));\n>> + struct diff_filespec *two = q->queue[i]->two;\n>> + int is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n> \n> Isn't !is_null_oid(&one->oid) redundant to checking one->mode? When\n> does the diff machinery ever give you a non-zero mode with a null oid?\n> \n\nIt looks like this originally only checked the mode, and the extra OID check\nwas introduced in ff00b682f2 (reset [] paths...: do not mishandle\nunmerged paths, 2011-07-13). I was able to remove `!is_null_oid(&one->oid)`\nfrom the condition and run the `t71*` tests without any failures, but I'm\nhesitant to remove it on the off chance that this handles a case I'm not\nthinking of.\n\n> Also, is_in_reset_tree == !is_missing; I'll note that below.\n> \n>> struct cache_entry *ce;\n>>\n>> + /*\n>> + * If the file being reset has `skip-worktree` enabled, we need\n>> + * to check it out to prevent the file from being hard reset.\n> \n> I don't understand this comment. If the file wasn't originally in the\n> index (is_missing), and is being added to it, and is correctly marked\n> as skip_worktree, and the file isn't in the working tree, then it\n> sounds like everything is already in a good state. Files outside the\n> sparse checkout are meant to have the skip_worktree bit set and be\n> missing from the working tree.\n> \n> Also, I don't know what you mean by 'hard reset' here.\n> \n>> + */\n>> + pos = cache_name_pos(two->path, strlen(two->path));\n>> + if (pos >= 0 && ce_skip_worktree(active_cache[pos])) {\n>> + struct checkout state = CHECKOUT_INIT;\n>> + state.force = 1;\n>> + state.refresh_cache = 1;\n>> + state.istate = &the_index;\n>> +\n>> + checkout_entry(active_cache[pos], &state, NULL, NULL);\n> \n> Does this introduce an error in the opposite direction from the one\n> stated in the commit message? Namely we have two things that should\n> be in sync: the skip_worktree flag stating whether the file should be\n> present in the working directory (skip_worktree), and the question of\n> whether the file is actually in the working directory. In the commit\n> message, you pointed out a case where the y were out of sync one way:\n> the skip_worktree flag was not set but the file was missing. Here you\n> say the skip_worktree flag is set, but you add it to the working tree\n> anyway.\n> \n> Or am I misunderstanding the code?\n> \n\nMost of this is addressed in [1], and you're right that what's in this \npatch isn't the right fix for the problem. This patch tried to solve the\nissue of \"skip-worktree is being ignored and reset files are showing up \ndeleted\" by continuing to ignore `skip-worktree`, but now checking out the\n`skip-worktree` files based on their pre-reset state in the index (unless\nthey, for some reason, were already present in the worktree). However, that\ncompletely disregards the reasoning for having `skip-worktree` in the first\nplace (the user wants the file *ignored* in the worktree) and violates the\npremise of `git reset --mixed` not modifying the worktree, so the better\nsolution is to set `skip-worktree` in the resulting index entry and not\ncheck out anything.\n\n[1] https://lore.kernel.org/git/9b99e856-24cc-03fd-7871-de92dc6e39b6@github.com/\n\n>> + }\n>> +\n> \n> [I did some slight editing to the diff to make the next two parts\n> appear next to each other]\n> \n>> - if (is_missing && !intent_to_add) {\n>> + if (!is_in_reset_tree && !intent_to_add) {\n> \n> I thought this was some subtle bugfix or something, and spent a while\n> trying to figure it out, before realizing that is_in_reset_tree was\n> simply defined as !is_missing (for some reason I was assuming it was\n> dealing with two->mode while is_missing was looking at one->mode). So\n> this is a simple variable renaming, which I think is probably good,\n> but I'd prefer if this was separated into a different patch to make it\n> easier to review.\n> \n\nGood call, I'll include this in V3.\n"},{"id":"438100","messageId":"3cfa0810-4449-1170-1f7b-cd029e5ce434@github.com","threadId":"56614","inReplyTo":"CABPp-BHD3khX+q7crk4JnNbGd9fN-vJW0L1x=5JdzkFL6X1BBg@mail.gmail.com","subject":"Re: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-06T20:40:38Z","receivedAt":"2021-10-06T20:40:45Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Tue, Oct 5, 2021 at 6:20 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Add a new `--force-full-index` option to `git update-index`, which skips\n>> explicitly setting `command_requires_full_index`. This lets `git\n>> update-index --force-full-index` run as a command without sparse index\n>> compatibility implemented, even after it receives sparse index compatibility\n>> updates.\n>>\n>> By using `git update-index --force-full-index` in the `t1092` test\n>> `sparse-index is expanded and converted back`, commands can continue to\n>> integrate with the sparse index without the need to keep modifying the\n>> command used in the test.\n> \n> So...we're adding a permanent user-facing command line flag, whose\n> purpose is just to help us with the transition work of implementing\n> sparse indexes everywhere? Am I reading that right, or is that just\n> the reason for t1092 and there are more reasons for it elsewhere?\n> \n> Also, I'm curious if update-index is the right place to add this. If\n> you don't want a sparse index anymore, wouldn't a user want to run\n> git sparse-checkout disable\n> ? Or is the point that you do want to keep the sparse checkout, but\n> you just don't want the index to also be sparse? Still, even in that\n> case, it seems like adding a subcommand or flag to an existing\n> sparse-checkout subcommand would feel more natural, since\n> sparse-checkout is the command the user uses to request to get into a\n> sparse-checkout and sparse index.\n> \n\nThis came out of a conversation [1] on an earlier version of this patch.\nBecause the `t1092 - sparse-index is expanded and converted back` test\nverifies sparse index compatibility (i.e., expand the index when reading,\ncollapse back to sparse when writing) on commands that don't have any sparse\nindex integration, it needed to be changed from `git reset` to something\nelse. However, as we keep integrating commands with sparse index we'd need\nto keep changing the command in the test, creating a bunch of patches doing\neffectively the same thing for no long-term benefit. \n\nThe `--force-full-index` flag isn't meant to be used externally or modify\nthe index in any \"new\" way - it's really just a \"test\" version of `git\nupdate-index` that we guarantee will accurately represent a command using\nthe default settings. Right now, it does exactly what `git update-index`\n(without the flag) does, and will only behave differently once `git\nupdate-index` is integrated with sparse index. Using `--force-full-index`,\nthe test won't need to be regularly updated and will continue to catch\nerrors like:\n\n1. Changing the default value of `command_requires_full_index` to 0\n2. Not expanding a sparse index to full when `command_requires_full_index`\n is 1\n3. Not collapsing the index back to sparse if sparse index is enabled\n\nI see the issue of introducing a test-only option (when sparse index is\nintegrated everywhere, shouldn't it be deprecated?). If there's a way to\nmake this more obviously internal/temporary, I'm happy to modify it. Or, if\nsemi-frequent updates of the command in the test aren't a huge issue, I can\nrevert to V1.\n\n[1] https://lore.kernel.org/git/xmqqr1d58v9x.fsf@gitster.g/\n"},{"id":"438105","messageId":"52237528-8ceb-d8d5-5590-5db3c8251717@github.com","threadId":"56614","inReplyTo":"CABPp-BFHNOMTerrV2Bhufb5d9d6JuM+CWW_t1jYg1Uu7F1dhhA@mail.gmail.com","subject":"Re: [PATCH v2 5/7] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-06T20:56:13Z","receivedAt":"2021-10-06T20:56:29Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n>> +static void prime_cache_tree_sparse_dir(struct repository *r,\n>> + struct cache_tree *it,\n>> + struct tree *tree,\n>> + struct strbuf *tree_path)\n>> +{\n>> +\n>> + oidcpy(&it->oid, &tree->object.oid);\n>> + it->entry_count = 1;\n>> + return;\n> \n> Why are 'r' and 'tree_path' passed to this function?\n> \n\nI mindlessly copied the function signature of `prime_cache_tree_rec` and\ndidn't notice those variables weren't needed (I'll remove them in V3).\n\n>> +}\n>> +\n>> static void prime_cache_tree_rec(struct repository *r,\n>> struct cache_tree *it,\n>> - struct tree *tree)\n>> + struct tree *tree,\n>> + struct strbuf *tree_path)\n>> {\n>> + struct strbuf subtree_path = STRBUF_INIT;\n>> struct tree_desc desc;\n>> struct name_entry entry;\n>> int cnt;\n>>\n>> oidcpy(&it->oid, &tree->object.oid);\n>> +\n> \n> Why the blank line addition here?\n> \n\nMy goal was to visually separate the parts of `prime_cache_tree_rec` that\nupdate the properties of the `tree` itself and the parts that deal with its\nentries. For me, it was helpful when reading and understanding what this\nfunction does and seemed like an good (minor) readability change.\n\n>> init_tree_desc(&desc, tree->buffer, tree->size);\n>> cnt = 0;\n>> while (tree_entry(&desc, &entry)) {\n>> @@ -757,27 +771,49 @@ static void prime_cache_tree_rec(struct repository *r,\n>> else {\n>> struct cache_tree_sub *sub;\n>> struct tree *subtree = lookup_tree(r, &entry.oid);\n>> +\n>> if (!subtree->object.parsed)\n>> parse_tree(subtree);\n>> sub = cache_tree_sub(it, entry.path);\n>> sub->cache_tree = cache_tree();\n>> - prime_cache_tree_rec(r, sub->cache_tree, subtree);\n> \n>> + strbuf_reset(&subtree_path);\n>> + strbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1);\n>> + strbuf_addbuf(&subtree_path, tree_path);\n>> + strbuf_add(&subtree_path, entry.path, entry.pathlen);\n>> + strbuf_addch(&subtree_path, '/');\n> \n> Reconstructing the full path each time? And despite only being useful\n> for the sparse-index case?\n> \n> Would it be better to drop subtree_path from this function, then\n> append entry.path + '/' here to tree_path, and then after the if-block\n> below, call strbuf_setlen to remove the part that this function call\n> added? That way, we don't need subtree_path, and don't have to copy\n> the leading path every time.\n> \n> Also, maybe it'd be better to only do this strbuf manipulation if\n> r->index->sparse_index, since it's not ever used otherwise?\n> \n\n[...]\n\n>> -static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n>> +static int index_name_stage_pos(struct index_state *istate,\n>> + const char *name, int namelen,\n>> + int stage,\n>> + int search_sparse)\n> \n> It'd be nicer to make search_sparse an enum defined within this file, so that...\n> \n>> {\n>> int first, last;\n>>\n>> @@ -570,7 +573,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n>> first = next+1;\n>> }\n>>\n>> - if (istate->sparse_index &&\n>> + if (search_sparse && istate->sparse_index &&\n>> first > 0) {\n>> /* Note: first <= istate->cache_nr */\n>> struct cache_entry *ce = istate->cache[first - 1];\n>> @@ -586,7 +589,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n>> ce_namelen(ce) < namelen &&\n>> !strncmp(name, ce->name, ce_namelen(ce))) {\n>> ensure_full_index(istate);\n>> - return index_name_stage_pos(istate, name, namelen, stage);\n>> + return index_name_stage_pos(istate, name, namelen, stage, search_sparse);\n>> }\n>> }\n>>\n>> @@ -595,7 +598,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n>>\n>> int index_name_pos(struct index_state *istate, const char *name, int namelen)\n>> {\n>> - return index_name_stage_pos(istate, name, namelen, 0);\n>> + return index_name_stage_pos(istate, name, namelen, 0, 1);\n> \n> ...this could use SEARCH_SPARSE or some name like that which is more\n> meaningful than \"1\" here.\n> \n>> +}\n>> +\n>> +int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n>> +{\n>> + return index_name_stage_pos(istate, name, namelen, 0, 0) >= 0;\n> \n> ...and likewise this spot could use SEARCH_FULL or some name like\n> that, which is more meaningful than the second \"0\".\n> \n> Similarly for multiple call sites below...\n> \n> \n\nI like all of these suggestions and will include them in the next version. Thanks!\n"},{"id":"438171","messageId":"8cc2ad6b-393c-9961-b911-be042bd2fb68@github.com","threadId":"56614","inReplyTo":"CABPp-BHDog6U1bFKurNq+sKbNPOnmqbhB8W2wg0UeF=jb+ZScw@mail.gmail.com","subject":"Re: [PATCH v2 6/7] reset: make --mixed sparse-aware","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-07T14:34:58Z","receivedAt":"2021-10-07T14:35:03Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n>> + /*\n>> + * When pathspec is given for resetting a cone-mode sparse checkout, it may\n>> + * identify entries that are nested in sparse directories, in which case the\n>> + * index should be expanded. For the sake of efficiency, this check is\n>> + * overly-cautious: anything with a wildcard or a magic prefix requires\n>> + * expansion, as well as literal paths that aren't in the sparse checkout\n>> + * definition AND don't match any directory in the index.\n> \n> s/efficiency/efficiency of checking/ ? Being overly-cautious suggests\n> you'll expand to a full index more than is needed, and full indexes\n> are more expensive. But perhaps the checking would be expensive too\n> so you have a tradeoff?\n> \n> Or maybe s/efficiency/simplicity/?\n> \n\n\"Simplicity\" is probably more appropriate, although the original intent was\n\"efficiency of checking\". I wanted to avoid repeated iteration over the\nindex (for example, matching the `no_wildcard_len` of each wildcard pathspec\nitem against each sparse directory in the index). However, to your point,\nexpanding the index is a more expensive operation anyway, so it's probably\nworth the more involved checks.\n\n>> + */\n>> + if (pathspec->nr && the_index.sparse_index) {\n>> + if (pathspec->magic || pathspec->has_wildcard) {\n>> + ensure_full_index(&the_index);\n> \n> dir.c has the notion of matching the characters preceding the wildcard\n> characters; look for \"no_wildcard_len\". If the pathspec doesn't match\n> a path up to no_wildcard_len, then the wildcard character(s) later in\n> the pathspec can't make the pathspec match that path.\n> \n> It might at least be worth mentioning this as a possible future optimization.\n> \n\nI'll incorporate a something like this into the next version.\n\n>> + } else {\n>> + for (i = 0; i < pathspec->nr; i++) {\n>> + if (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) &&\n>> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) {\n> \n> What if the pathspec corresponds to a sparse-directory in the index,\n> but possibly without the trailing '/' character? e.g.:\n> \n> git reset HEAD~1 -- sparse-directory\n> \n> One should be able to reset that directory without recursing into\n> it...does this code handle that? Does it handle it if we add the\n> trailing slash on the path for the reset command line?\n> \n\nIt handles both cases (with and without trailing slash), the former due to\n`!matches_skip_worktree(...)` and the latter due to\n`!path_in_cone_mode_sparse_checkout(...)`.\n\n>> + ensure_full_index(&the_index);\n>> + break;\n>> + }\n>> + }\n>> + }\n>> + }\n>> +\n>> + free(skip_worktree_seen);\n>>\n>> - ensure_full_index(&the_index);\n>> if (do_diff_cache(tree_oid, &opt))\n>> return 1;\n>> diffcore_std(&opt);\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index e301ef5633a..4afcbc2d673 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -804,11 +804,22 @@ test_expect_success 'sparse-index is not expanded' '\n>> ensure_not_expanded reset --hard $ref || return 1\n>> done &&\n>>\n>> + ensure_not_expanded reset --mixed base &&\n>> ensure_not_expanded reset --hard update-deep &&\n>> ensure_not_expanded reset --keep base &&\n>> ensure_not_expanded reset --merge update-deep &&\n>> - ensure_not_expanded reset --hard &&\n> \n> This commit was only touching the --mixed case; why is it removing one\n> of the tests for --hard?\n> \n\n[...]\n\n>> + ensure_not_expanded reset --hard update-folder1 &&\n> \n> Wait...is update-folder1 a branch or a path? And if this commit is\n> about --mixed, why are --hard testcases being added?\n> \n>> + ensure_not_expanded reset base -- folder1 &&\n>> +\n>> + ensure_not_expanded reset --hard update-deep &&\n> \n> another --hard testcase...was this an accidental squash by chance?\n> \n\nI included `git reset --hard` between the \"actual\" test cases so that the\n`git reset --mixed` tests would start in a \"clean\" state (clear out any\nmodified files), but it's unnecessary in most cases so I'll remove them in\nV3. To answer your other question, `update-folder1` is a branch.\n"},{"id":"438230","messageId":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v2.git.1633440057.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:30Z","receivedAt":"2021-10-07T21:15:44Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate a ~70% execution time reduction in git reset\nusing a sparse index, and no change (within expected variability [2]) using\na full index. Results summarized below [3, 4]:\n\nTest base [5/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.97(0.50+0.37) -3.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.96(0.50+0.36) -4.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 1.37(0.91+0.35) -18.5%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 1.41(0.94+0.35) -17.1%\n\nTest base [6/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.94(0.48+0.34) -6.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.95(0.51+0.34) -5.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 0.46(0.05+0.29) -72.6%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 0.46(0.06+0.29) -72.9%\n\nTest base [7/8]\n---------------------------------------------------------------------------\ngit reset (full-v3) 0.77(0.27+0.37) 0.72(0.26+0.32) -6.5%\ngit reset (full-v4) 0.75(0.27+0.34) 0.73(0.26+0.32) -2.7%\ngit reset (sparse-v3) 1.44(0.96+0.36) 0.43(0.04+0.96) -70.1%\ngit reset (sparse-v4) 1.46(0.97+0.36) 0.43(0.05+0.79) -70.5%\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.69(0.26+0.30) -4.2%\ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.71(0.27+0.32) -4.1% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.81(0.42+0.90) -44.1%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.79(0.42+0.76) -44.0%\n\nTest base [8/8] \n---------------------------------------------------------------------------\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.73(0.26+0.33) +1.4% \ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.74(0.27+0.32) +0.0% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.43(0.05+0.80) -70.3%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.44(0.05+0.76) -68.8%\n\n\n\nChanges since V1\n================\n\n * Add --force-full-index option to update-index. The option is used\n circumvent changing command_requires_full_index from its default value -\n right now this is effectively a no-op, but will change once update-index\n is integrated with sparse index. By using this option in the t1092\n expand/collapse test, the command used to test will not need to be\n updated with subsequent sparse index integrations.\n * Update implementation of mixed reset for entries outside sparse checkout\n definition. The condition in which a file should be checked out before\n index reset is simplified to \"if it has skip-worktree enabled and a reset\n would change the file, check it out\".\n * After checking the behavior of update_index_from_diff with renames,\n found that the diff used by reset does not produce diff queue entries\n with different pathnames for one and two. Because of this, and that\n nothing in the implementation seems to rely on identical path names, no\n BUG check is added.\n * Correct a bug in the sparse index is not expanded tests in t1092 where\n failure of a git reset --mixed test was not being reported. Test now\n verifies an appropriate scenario with corrected failure-checking.\n\n\nChanges since V2\n================\n\n * Replace patch adding checkouts for git reset --mixed with sparse checkout\n with preserving the skip-worktree flag (including a new test for git\n reset --mixed and update to t1092 - checkout and reset (mixed))\n * Move rename of is_missing into its own patch\n * Further extend t1092 tests and remove unnecessary commands/tests where\n possible\n * Refine logic determining which pathspecs require ensure_full_index in git\n reset --mixed, add related ensure_not_expanded tests\n * Add index_search_mode enum to index_name_stage_pos\n * Clean up variable usage & remove unnecessary subtree_path in\n prime_cache_tree_rec\n * Update cover letter performance data\n * More thoroughly explain changes in each commit message\n\nThanks! -Victoria\n\n[1] microsoft@6b8a074 [2]\nhttps://lore.kernel.org/git/8b9fe3f8-f0e3-4567-b20b-17c92bd1a5c5@github.com/\n[3] If a test and/or commit is not mentioned, there is no significant change\nto performance [4] Pathspec \"does-not-exist\" is changed to \"missing\" to save\nspace in performance report\n\nKevin Willford (1):\n reset: preserve skip-worktree bit in mixed reset\n\nVictoria Dye (7):\n reset: rename is_missing to !is_in_reset_tree\n update-index: add --force-full-index option for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n Documentation/git-update-index.txt | 5 +\n builtin/reset.c | 104 +++++++++++++++++-\n builtin/update-index.c | 11 ++\n cache-tree.c | 46 +++++++-\n cache.h | 10 ++\n read-cache.c | 27 +++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 133 ++++++++++++++++++++---\n t/t7102-reset.sh | 17 +++\n unpack-trees.c | 23 +++-\n 10 files changed, 342 insertions(+), 37 deletions(-)\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n\nRange-diff vs v2:\n\n -: ----------- > 1: ad7013a31aa reset: rename is_missing to !is_in_reset_tree\n 1: 22c69bc6030 ! 2: 1f6da84830b reset: behave correctly with sparse-checkout\n @@ Metadata\n Author: Kevin Willford \n \n ## Commit message ##\n - reset: behave correctly with sparse-checkout\n + reset: preserve skip-worktree bit in mixed reset\n \n - When using the sparse checkout feature, 'git reset' will add entries to\n - the index that will have the skip-worktree bit off but will leave the\n - working directory empty. File data is lost because the index version of\n - the files has been changed but there is nothing that is in the working\n - directory. This will cause the next 'git status' call to show either\n - deleted for files modified or deleting or nothing for files added. The\n - added files should be shown as untracked and modified files should be\n - shown as modified.\n + Change `update_index_from_diff` to set `skip-worktree` when applicable for\n + new index entries. When `git reset --mixed ` is run, entries in\n + the index with differences between the pre-reset HEAD and reset \n + are identified and handled with `update_index_from_diff`. For each file, a\n + new cache entry in inserted into the index, created from the side\n + of the reset (without changing the working tree). However, the newly-created\n + entry must have `skip-worktree` explicitly set in either of the following\n + scenarios:\n \n - To fix this when the reset is running if there is not a file in the\n - working directory and if it will be missing with the new index entry or\n - was not missing in the previous version, we create the previous index\n - version of the file in the working directory so that status will report\n - correctly and the files will be availble for the user to deal with.\n + 1. the file is in the current index and has `skip-worktree` set\n + 2. the file is not in the current index but is outside of a defined sparse\n + checkout definition\n \n - This fixes a documented failure from t1092 that was created in 19a0acc\n - (t1092: test interesting sparse-checkout scenarios, 2021-01-23).\n + Not setting the `skip-worktree` bit leads to likely-undesirable results for\n + a user. It causes `skip-worktree` settings to disappear on the\n + \"diff\"-containing files (but *only* the diff-containing files), leading to\n + those files now showing modifications in `git status`. For example, when\n + running `git reset --mixed` in a sparse checkout, some file entries outside\n + of sparse checkout could show up as deleted, despite the user never deleting\n + anything (and not wanting them on-disk anyway).\n \n - Signed-off-by: Kevin Willford \n - Signed-off-by: Derrick Stolee \n + Additionally, add a test to `t7102` to ensure `skip-worktree` is preserved\n + in a basic `git reset --mixed` scenario and update a failure-documenting\n + test from 19a0acc (t1092: test interesting sparse-checkout scenarios,\n + 2021-01-23) with new expected behavior.\n +\n + Helped-by: Junio C Hamano \n Signed-off-by: Victoria Dye \n \n ## builtin/reset.c ##\n @@ builtin/reset.c\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n +#include \"dir.h\"\n -+#include \"entry.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n \tfor (i = 0; i < q->nr; i++) {\n +\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n --\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n -+\t\tstruct diff_filespec *two = q->queue[i]->two;\n -+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n + \t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n +@@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n \n --\t\tif (is_missing && !intent_to_add) {\n + \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n + \t\t\t\t 0, 0);\n ++\n +\t\t/*\n -+\t\t * If the file being reset has `skip-worktree` enabled, we need\n -+\t\t * to check it out to prevent the file from being hard reset.\n ++\t\t * If the file 1) corresponds to an existing index entry with\n ++\t\t * skip-worktree set, or 2) does not exist in the index but is\n ++\t\t * outside the sparse checkout definition, add a skip-worktree bit\n ++\t\t * to the new index entry.\n +\t\t */\n -+\t\tpos = cache_name_pos(two->path, strlen(two->path));\n -+\t\tif (pos >= 0 && ce_skip_worktree(active_cache[pos])) {\n -+\t\t\tstruct checkout state = CHECKOUT_INIT;\n -+\t\t\tstate.force = 1;\n -+\t\t\tstate.refresh_cache = 1;\n -+\t\t\tstate.istate = &the_index;\n ++\t\tpos = cache_name_pos(one->path, strlen(one->path));\n ++\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n ++\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n ++\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n +\n -+\t\t\tcheckout_entry(active_cache[pos], &state, NULL, NULL);\n -+\t\t}\n -+\n -+\t\tif (!is_in_reset_tree && !intent_to_add) {\n - \t\t\tremove_file_from_cache(one->path);\n - \t\t\tcontinue;\n - \t\t}\n -@@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n --\t\tif (is_missing) {\n -+\t\tif (!is_in_reset_tree) {\n - \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n - \t\t\tset_object_name_for_intent_to_add_entry(ce);\n - \t\t}\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'blame with pathspec outside sparse definition' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'blame with pathsp\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\n + \ttest_all_match git reset deepest &&\n +-\ttest_all_match git reset update-folder1 &&\n +-\ttest_all_match git reset update-folder2\n +-'\n +-\n +-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n +-# in this scenario, but it shouldn't.\n +-test_expect_success 'checkout and reset (mixed) [sparse]' '\n +-\tinit_repos &&\n + \n +-\ttest_sparse_match git checkout -b reset-test update-deep &&\n +-\ttest_sparse_match git reset deepest &&\n ++\t# Because skip-worktree is preserved, resetting to update-folder1\n ++\t# will show worktree changes for full-checkout that are not present\n ++\t# in sparse-checkout or sparse-index.\n + \ttest_sparse_match git reset update-folder1 &&\n +-\ttest_sparse_match git reset update-folder2\n ++\trun_on_sparse test_path_is_missing folder1\n + '\n + \n + test_expect_success 'merge, cherry-pick, and rebase' '\n \n - ## t/t7114-reset-sparse-checkout.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description='reset when using a sparse-checkout'\n -+\n -+. ./test-lib.sh\n -+\n -+test_expect_success 'setup' '\n -+\ttest_tick &&\n -+\techo \"checkout file\" >c &&\n -+\techo \"modify file\" >m &&\n -+\techo \"delete file\" >d &&\n -+\tgit add . &&\n -+\tgit commit -m \"initial commit\" &&\n -+\techo \"added file\" >a &&\n -+\techo \"modification of a file\" >m &&\n -+\tgit rm d &&\n -+\tgit add . &&\n -+\tgit commit -m \"second commit\" &&\n -+\tgit checkout -b endCommit\n -+'\n -+\n -+test_expect_success 'reset when there is a sparse-checkout' '\n -+\techo \"/c\" >.git/info/sparse-checkout &&\n -+\ttest_config core.sparsecheckout true &&\n -+\tgit checkout -B resetBranch &&\n -+\ttest_path_is_missing m &&\n -+\ttest_path_is_missing a &&\n -+\ttest_path_is_missing d &&\n -+\tgit reset HEAD~1 &&\n -+\techo \"checkout file\" >expect &&\n -+\ttest_cmp expect c &&\n -+\techo \"added file\" >expect &&\n -+\ttest_cmp expect a &&\n -+\techo \"modification of a file\" >expect &&\n -+\ttest_cmp expect m &&\n -+\ttest_path_is_missing d\n -+'\n + ## t/t7102-reset.sh ##\n +@@ t/t7102-reset.sh: test_expect_success '--mixed refreshes the index' '\n + \ttest_cmp expect output\n + '\n + \n ++test_expect_success '--mixed preserves skip-worktree' '\n ++\techo 123 >>file2 &&\n ++\tgit add file2 &&\n ++\tgit update-index --skip-worktree file2 &&\n ++\tgit reset --mixed HEAD >output &&\n ++\ttest_must_be_empty output &&\n +\n -+test_expect_success 'reset after deleting file without skip-worktree bit' '\n -+\tgit checkout -f endCommit &&\n -+\tgit clean -xdf &&\n -+\tcat >.git/info/sparse-checkout <<-\\EOF &&\n -+\t/c\n -+\t/m\n ++\tcat >expect <<-\\EOF &&\n ++\tUnstaged changes after reset:\n ++\tM\tfile2\n +\tEOF\n -+\ttest_config core.sparsecheckout true &&\n -+\tgit checkout -B resetAfterDelete &&\n -+\ttest_path_is_file m &&\n -+\ttest_path_is_missing a &&\n -+\ttest_path_is_missing d &&\n -+\trm -f m &&\n -+\tgit reset HEAD~1 &&\n -+\techo \"checkout file\" >expect &&\n -+\ttest_cmp expect c &&\n -+\techo \"added file\" >expect &&\n -+\ttest_cmp expect a &&\n -+\ttest_path_is_missing m &&\n -+\ttest_path_is_missing d\n ++\tgit update-index --no-skip-worktree file2 &&\n ++\tgit add file2 &&\n ++\tgit reset --mixed HEAD >output &&\n ++\ttest_cmp expect output\n +'\n +\n -+test_done\n + test_expect_success 'resetting specific path that is unmerged' '\n + \tgit rm --cached file2 &&\n + \tF1=$(git rev-parse HEAD:file1) &&\n 2: f7cb9013d46 ! 3: 014a408ea5d update-index: add --force-full-index option for expand/collapse test\n @@ Commit message\n update-index: add --force-full-index option for expand/collapse test\n \n Add a new `--force-full-index` option to `git update-index`, which skips\n - explicitly setting `command_requires_full_index`. This lets `git\n - update-index --force-full-index` run as a command without sparse index\n - compatibility implemented, even after it receives sparse index compatibility\n - updates.\n + explicitly setting `command_requires_full_index`. This option, intended for\n + use in internal testing purposes only, lets `git update-index` run as a\n + command without sparse index compatibility implemented, even after it\n + receives updates to otherwise use the sparse index.\n \n - By using `git update-index --force-full-index` in the `t1092` test\n - `sparse-index is expanded and converted back`, commands can continue to\n - integrate with the sparse index without the need to keep modifying the\n - command used in the test.\n + The specific test `--force-full-index` is intended for - `t1092 -\n + sparse-index is expanded and converted back` - verifies index compatibility\n + in commands that do not change the default (enabled)\n + `command_requires_full_index` repo setting. In the past, the test used `git\n + reset`. However, as `reset` and other commands are integrated with the\n + sparse index, the command used in the test would need to keep changing.\n + Conversely, the `--force-full-index` option makes `git update-index` behave\n + like a not-yet-sparse-aware command, and can be used in the test\n + indefinitely without interfering with future sparse index integrations.\n \n + Helped-by: Junio C Hamano \n Signed-off-by: Victoria Dye \n \n ## Documentation/git-update-index.txt ##\n 3: c7e9d9f4e03 ! 4: 7f21cf53e9d reset: expand test coverage for sparse checkouts\n @@ Commit message\n reset: expand test coverage for sparse checkouts\n \n Add new tests for `--merge` and `--keep` modes, as well as mixed reset with\n - pathspecs both inside and outside of the sparse checkout definition. New\n - performance test cases exercise various execution paths for `reset`.\n + pathspecs. New performance test cases exercise various execution paths for\n + `reset`.\n \n Co-authored-by: Derrick Stolee \n Signed-off-by: Derrick Stolee \n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git add -A\n test_done\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout and reset (mixed) [sparse]' '\n - \ttest_sparse_match git reset update-folder2\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout and reset (mixed)' '\n + \trun_on_sparse test_path_is_missing folder1\n '\n \n -+# NEEDSWORK: with mixed reset, files with differences between HEAD and \n -+# will be added to the work tree even if outside the sparse checkout\n -+# definition, and even if the file is modified to a state of having no local\n -+# changes. The file is \"re-ignored\" if a hard reset is executed. We may want to\n -+# change this behavior in the future and enforce that files are not written\n -+# outside of the sparse checkout definition.\n -+test_expect_success 'checkout and mixed reset file tracking [sparse]' '\n -+\tinit_repos &&\n -+\n -+\ttest_all_match git checkout -b reset-test update-deep &&\n -+\ttest_all_match git reset update-folder1 &&\n -+\ttest_all_match git reset update-deep &&\n -+\n -+\t# At this point, there are no changes in the working tree. However,\n -+\t# folder1/a now exists locally (even though it is outside of the sparse\n -+\t# paths).\n -+\trun_on_sparse test_path_exists folder1 &&\n -+\n -+\trun_on_all rm folder1/a &&\n -+\ttest_all_match git status --porcelain=v2 &&\n -+\n -+\ttest_all_match git reset --hard update-deep &&\n -+\trun_on_sparse test_path_is_missing folder1 &&\n -+\ttest_path_exists full-checkout/folder1\n -+'\n -+\n +test_expect_success 'checkout and reset (merge)' '\n +\tinit_repos &&\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout and rese\n +\ttest_all_match git status --porcelain=v2\n +'\n +\n -+test_expect_success 'reset with sparse directory pathspec outside definition' '\n ++# Although the working tree differs between full and sparse checkouts after\n ++# reset, the state of the index is the same.\n ++test_expect_success 'reset with pathspecs outside sparse definition' '\n +\tinit_repos &&\n ++\ttest_all_match git checkout -b reset-test base &&\n +\n -+\ttest_all_match git checkout -b reset-test update-deep &&\n -+\ttest_all_match git reset --hard update-folder1 &&\n -+\ttest_all_match git reset base -- folder1 &&\n -+\ttest_all_match git status --porcelain=v2\n -+'\n ++\ttest_sparse_match git reset update-folder1 -- folder1 &&\n ++\tgit -C full-checkout reset update-folder1 -- folder1 &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n ++\ttest_all_match git rev-parse HEAD:folder1 &&\n +\n -+test_expect_success 'reset with pathspec match in sparse directory' '\n -+\tinit_repos &&\n -+\n -+\ttest_all_match git checkout -b reset-test update-deep &&\n -+\ttest_all_match git reset --hard update-folder1 &&\n -+\ttest_all_match git reset base -- folder1/a &&\n -+\ttest_all_match git status --porcelain=v2\n ++\ttest_sparse_match git reset update-folder2 -- folder2/a &&\n ++\tgit -C full-checkout reset update-folder2 -- folder2/a &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n ++\ttest_all_match git rev-parse HEAD:folder2/a\n +'\n +\n +test_expect_success 'reset with wildcard pathspec' '\n +\tinit_repos &&\n +\n +\ttest_all_match git checkout -b reset-test update-deep &&\n -+\ttest_all_match git reset --hard update-folder1 &&\n +\ttest_all_match git reset base -- \\*/a &&\n -+\ttest_all_match git status --porcelain=v2\n ++\ttest_all_match git status --porcelain=v2 &&\n ++\ttest_all_match git rev-parse HEAD:folder1/a &&\n ++\n ++\ttest_all_match git reset base -- folder\\* &&\n ++\ttest_all_match git status --porcelain=v2 &&\n ++\ttest_all_match git rev-parse HEAD:folder2\n +'\n +\n test_expect_success 'merge, cherry-pick, and rebase' '\n 4: 49813c8d9ed ! 5: a2d6212e287 reset: integrate with sparse index\n @@ Metadata\n ## Commit message ##\n reset: integrate with sparse index\n \n - `reset --soft` does not modify the index, so no compatibility changes are\n - needed for it to function without expanding the index. For all other reset\n - modes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is\n - explicitly expanded with `ensure_full_index` to maintain current behavior.\n + Disable `command_requires_full_index` repo setting and add\n + `ensure_full_index` guards around code paths that cannot yet use sparse\n + directory index entries. `reset --soft` does not modify the index, so no\n + compatibility changes are needed for it to function without expanding the\n + index. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`),\n + the full index is expanded to prevent cache tree corruption and invalid\n + variable accesses.\n \n Additionally, the `read_cache()` check verifying an uncorrupted index is\n moved after argument parsing and preparing the repo settings. The index is\n - not used by the preceding argument handling, but `read_cache()` does need to\n - be run after enabling sparse index for the command and before resetting.\n + not used by the preceding argument handling, but `read_cache()` must be run\n + *after* enabling sparse index for the command (so that the index is not\n + expanded unnecessarily) and *before* using the index for reset (so that it\n + is verified as uncorrupted).\n \n Signed-off-by: Victoria Dye \n \n 5: 78cd85d8dcc ! 6: 330e0c09774 reset: make sparse-aware (except --mixed)\n @@ Metadata\n ## Commit message ##\n reset: make sparse-aware (except --mixed)\n \n - In order to accurately reconstruct the cache tree in `prime_cache_tree_rec`,\n - the function must determine whether the currently-processing directory in\n - the tree is sparse or not. If it is not sparse, the tree is parsed and\n - subtree recursively constructed. If it is sparse, no subtrees are added to\n - the tree and the entry count is set to 1 (representing the sparse directory\n - itself).\n + Remove `ensure_full_index` guard on `prime_cache_tree` and update\n + `prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\n + the cache tree. While processing a tree's entries, `prime_cache_tree_rec`\n + must determine whether a directory entry is sparse or not by searching for\n + it in the index (*without* expanding the index). If a matching sparse\n + directory index entry is found, no subtrees are added to the cache tree\n + entry and the entry count is set to 1 (representing the sparse directory\n + itself). Otherwise, the tree is assumed to not be sparse and its subtrees\n + are recursively added to the cache tree.\n \n + Helped-by: Elijah Newren \n Signed-off-by: Victoria Dye \n \n ## cache-tree.c ##\n @@ cache-tree.c: out:\n \treturn ret;\n }\n \n -+static void prime_cache_tree_sparse_dir(struct repository *r,\n -+\t\t\t\t\tstruct cache_tree *it,\n -+\t\t\t\t\tstruct tree *tree,\n -+\t\t\t\t\tstruct strbuf *tree_path)\n ++static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n ++\t\t\t\t\tstruct tree *tree)\n +{\n +\n +\toidcpy(&it->oid, &tree->object.oid);\n +\tit->entry_count = 1;\n -+\treturn;\n +}\n +\n static void prime_cache_tree_rec(struct repository *r,\n @@ cache-tree.c: out:\n +\t\t\t\t struct tree *tree,\n +\t\t\t\t struct strbuf *tree_path)\n {\n -+\tstruct strbuf subtree_path = STRBUF_INIT;\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n ++\tint base_path_len = tree_path->len;\n \n \toidcpy(&it->oid, &tree->object.oid);\n +\n @@ cache-tree.c: static void prime_cache_tree_rec(struct repository *r,\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n -\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n -+\t\t\tstrbuf_reset(&subtree_path);\n -+\t\t\tstrbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1);\n -+\t\t\tstrbuf_addbuf(&subtree_path, tree_path);\n -+\t\t\tstrbuf_add(&subtree_path, entry.path, entry.pathlen);\n -+\t\t\tstrbuf_addch(&subtree_path, '/');\n ++\n ++\t\t\t/*\n ++\t\t\t * Recursively-constructed subtree path is only needed when working\n ++\t\t\t * in a sparse index (where it's used to determine whether the\n ++\t\t\t * subtree is a sparse directory in the index).\n ++\t\t\t */\n ++\t\t\tif (r->index->sparse_index) {\n ++\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n ++\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n ++\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n ++\t\t\t\tstrbuf_addch(tree_path, '/');\n ++\t\t\t}\n +\n +\t\t\t/*\n +\t\t\t * If a sparse index is in use, the directory being processed may be\n @@ cache-tree.c: static void prime_cache_tree_rec(struct repository *r,\n +\t\t\t * as normal.\n +\t\t\t */\n +\t\t\tif (r->index->sparse_index &&\n -+\t\t\t index_entry_exists(r->index, subtree_path.buf, subtree_path.len))\n -+\t\t\t\tprime_cache_tree_sparse_dir(r, sub->cache_tree, subtree, &subtree_path);\n ++\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n ++\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n +\t\t\telse\n -+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, &subtree_path);\n ++\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n - \tit->entry_count = cnt;\n +\n -+\tstrbuf_release(&subtree_path);\n + \tit->entry_count = cnt;\n }\n \n - void prime_cache_tree(struct repository *r,\n +@@ cache-tree.c: void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n @@ cache.h: struct cache_entry *index_file_exists(struct index_state *istate, const\n * precise match was not found but a position was found where the entry would\n \n ## read-cache.c ##\n +@@\n + */\n + #define CACHE_ENTRY_PATH_LENGTH 80\n + \n ++enum index_search_mode {\n ++\tNO_EXPAND_SPARSE = 0,\n ++\tEXPAND_SPARSE = 1\n ++};\n ++\n + static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n + {\n + \tstruct cache_entry *ce;\n @@ read-cache.c: int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n @@ read-cache.c: int cache_name_stage_compare(const char *name1, int len1, int stag\n +static int index_name_stage_pos(struct index_state *istate,\n +\t\t\t\tconst char *name, int namelen,\n +\t\t\t\tint stage,\n -+\t\t\t\tint search_sparse)\n ++\t\t\t\tenum index_search_mode search_mode)\n {\n \tint first, last;\n \n @@ read-cache.c: static int index_name_stage_pos(struct index_state *istate, const\n \t}\n \n -\tif (istate->sparse_index &&\n -+\tif (search_sparse && istate->sparse_index &&\n ++\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n @@ read-cache.c: static int index_name_stage_pos(struct index_state *istate, const\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n -\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n -+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_sparse);\n ++\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n \t\t}\n \t}\n \n @@ read-cache.c: static int index_name_stage_pos(struct index_state *istate, const\n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n -\treturn index_name_stage_pos(istate, name, namelen, 0);\n -+\treturn index_name_stage_pos(istate, name, namelen, 0, 1);\n ++\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n +}\n +\n +int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n +{\n -+\treturn index_name_stage_pos(istate, name, namelen, 0, 0) >= 0;\n ++\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n @@ read-cache.c: static int has_dir_name(struct index_state *istate,\n \t\t}\n \n -\t\tpos = index_name_stage_pos(istate, name, len, stage);\n -+\t\tpos = index_name_stage_pos(istate, name, len, stage, 1);\n ++\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n @@ read-cache.c: static int add_index_entry_with_check(struct index_state *istate,\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n -\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n -+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n ++\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n @@ read-cache.c: static int add_index_entry_with_check(struct index_state *istate,\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n -\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n -+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1);\n ++\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\n 6: 5eaae0825af ! 7: 6ef8e4e31d3 reset: make --mixed sparse-aware\n @@ Metadata\n ## Commit message ##\n reset: make --mixed sparse-aware\n \n - Sparse directory entries are \"diffed\" as trees in `diff_cache` (used\n - internally by `reset --mixed`), following a code path separate from\n - individual file handling. The use of `diff_tree_oid` there requires setting\n - explicit `change` and `add_remove` functions to process the internal\n - contents of a sparse directory.\n + Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n + reset --mixed` to ensure it can use sparse directory index entries wherever\n + possible. Sparse directory entries are reset use `diff_tree_oid`, which\n + requires `change` and `add_remove` functions to process the internal\n + contents of the sparse directory. The `recursive` diff option handles cases\n + in which `reset --mixed` must diff/merge files that are nested multiple\n + levels deep in a sparse directory.\n \n - Additionally, the `recursive` diff option handles cases in which `reset\n - --mixed` must diff/merge files that are nested multiple levels deep in a\n - sparse directory.\n + The use of pathspecs with `git reset --mixed` introduces scenarios in which\n + internal contents of sparse directories may be matched by the pathspec. In\n + order to reset *all* files in the repo that may match the pathspec, the\n + following conditions on the pathspec require index expansion before\n + performing the reset:\n \n + * \"magic\" pathspecs\n + * wildcard pathspecs that do not match only in-cone files or entire sparse\n + directories\n + * literal pathspecs matching something outside the sparse checkout\n + definition\n +\n + Helped-by: Elijah Newren \n Signed-off-by: Victoria Dye \n \n ## builtin/reset.c ##\n -@@ builtin/reset.c: static int read_from_tree(const struct pathspec *pathspec,\n - \t\t\t int intent_to_add)\n - {\n - \tstruct diff_options opt;\n -+\tunsigned int i;\n -+\tchar *skip_worktree_seen = NULL;\n +@@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n + \t\t * If the file 1) corresponds to an existing index entry with\n + \t\t * skip-worktree set, or 2) does not exist in the index but is\n + \t\t * outside the sparse checkout definition, add a skip-worktree bit\n +-\t\t * to the new index entry.\n ++\t\t * to the new index entry. Note that a sparse index will be expanded\n ++\t\t * if this entry is outside the sparse cone - this is necessary\n ++\t\t * to properly construct the reset sparse directory.\n + \t\t */\n + \t\tpos = cache_name_pos(one->path, strlen(one->path));\n + \t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n +@@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n + \t}\n + }\n \n - \tmemset(&opt, 0, sizeof(opt));\n - \tcopy_pathspec(&opt.pathspec, pathspec);\n -@@ builtin/reset.c: static int read_from_tree(const struct pathspec *pathspec,\n - \topt.format_callback = update_index_from_diff;\n - \topt.format_callback_data = &intent_to_add;\n - \topt.flags.override_submodule_config = 1;\n -+\topt.flags.recursive = 1;\n - \topt.repo = the_repository;\n -+\topt.change = diff_change;\n -+\topt.add_remove = diff_addremove;\n ++static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n ++{\n ++\tunsigned int i, pos;\n ++\tint res = 0;\n ++\tchar *skip_worktree_seen = NULL;\n +\n +\t/*\n -+\t * When pathspec is given for resetting a cone-mode sparse checkout, it may\n -+\t * identify entries that are nested in sparse directories, in which case the\n -+\t * index should be expanded. For the sake of efficiency, this check is\n -+\t * overly-cautious: anything with a wildcard or a magic prefix requires\n -+\t * expansion, as well as literal paths that aren't in the sparse checkout\n -+\t * definition AND don't match any directory in the index.\n ++\t * When using a magic pathspec, assume for the sake of simplicity that\n ++\t * the index needs to be expanded to match all matchable files.\n +\t */\n -+\tif (pathspec->nr && the_index.sparse_index) {\n -+\t\tif (pathspec->magic || pathspec->has_wildcard) {\n -+\t\t\tensure_full_index(&the_index);\n -+\t\t} else {\n -+\t\t\tfor (i = 0; i < pathspec->nr; i++) {\n -+\t\t\t\tif (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) &&\n -+\t\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) {\n -+\t\t\t\t\tensure_full_index(&the_index);\n ++\tif (pathspec->magic)\n ++\t\treturn 1;\n ++\n ++\tfor (i = 0; i < pathspec->nr; i++) {\n ++\t\tstruct pathspec_item item = pathspec->items[i];\n ++\n ++\t\t/*\n ++\t\t * If the pathspec item has a wildcard, the index should be expanded\n ++\t\t * if the pathspec has the possibility of matching a subset of entries inside\n ++\t\t * of a sparse directory (but not the entire directory).\n ++\t\t *\n ++\t\t * If the pathspec item is a literal path, the index only needs to be expanded\n ++\t\t * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n ++\t\t * expand for in-cone files) and b) it doesn't match any sparse directories\n ++\t\t * (since we can reset whole sparse directories without expanding them).\n ++\t\t */\n ++\t\tif (item.nowildcard_len < item.len) {\n ++\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n ++\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n ++\n ++\t\t\t\tif (!S_ISSPARSEDIR(ce->ce_mode))\n ++\t\t\t\t\tcontinue;\n ++\n ++\t\t\t\t/*\n ++\t\t\t\t * If the pre-wildcard length is longer than the sparse\n ++\t\t\t\t * directory name and the sparse directory is the first\n ++\t\t\t\t * component of the pathspec, need to expand the index.\n ++\t\t\t\t */\n ++\t\t\t\tif (item.nowildcard_len > ce_namelen(ce) &&\n ++\t\t\t\t !strncmp(item.original, ce->name, ce_namelen(ce))) {\n ++\t\t\t\t\tres = 1;\n ++\t\t\t\t\tbreak;\n ++\t\t\t\t}\n ++\n ++\t\t\t\t/*\n ++\t\t\t\t * If the pre-wildcard length is shorter than the sparse\n ++\t\t\t\t * directory and the pathspec does not match the whole\n ++\t\t\t\t * directory, need to expand the index.\n ++\t\t\t\t */\n ++\t\t\t\tif (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n ++\t\t\t\t wildmatch(item.original, ce->name, 0)) {\n ++\t\t\t\t\tres = 1;\n +\t\t\t\t\tbreak;\n +\t\t\t\t}\n +\t\t\t}\n -+\t\t}\n ++\t\t} else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n ++\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n ++\t\t\tres = 1;\n ++\n ++\t\tif (res > 0)\n ++\t\t\tbreak;\n +\t}\n +\n +\tfree(skip_worktree_seen);\n ++\treturn res;\n ++}\n ++\n + static int read_from_tree(const struct pathspec *pathspec,\n + \t\t\t struct object_id *tree_oid,\n + \t\t\t int intent_to_add)\n +@@ builtin/reset.c: static int read_from_tree(const struct pathspec *pathspec,\n + \topt.format_callback = update_index_from_diff;\n + \topt.format_callback_data = &intent_to_add;\n + \topt.flags.override_submodule_config = 1;\n ++\topt.flags.recursive = 1;\n + \topt.repo = the_repository;\n ++\topt.change = diff_change;\n ++\topt.add_remove = diff_addremove;\n ++\n ++\tif (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n ++\t\tensure_full_index(&the_index);\n \n -\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is n\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n --\tensure_not_expanded reset --hard &&\n + \tensure_not_expanded reset --hard &&\n \n +\tensure_not_expanded reset base -- deep/a &&\n +\tensure_not_expanded reset base -- nonexistent-file &&\n +\tensure_not_expanded reset deepest -- deep &&\n +\n +\t# Although folder1 is outside the sparse definition, it exists as a\n -+\t# directory entry in the index, so it will be reset without needing to\n -+\t# expand the full index.\n -+\tensure_not_expanded reset --hard update-folder1 &&\n -+\tensure_not_expanded reset base -- folder1 &&\n ++\t# directory entry in the index, so the pathspec will not force the\n ++\t# index to be expanded.\n ++\tensure_not_expanded reset deepest -- folder1 &&\n ++\tensure_not_expanded reset deepest -- folder1/ &&\n ++\n ++\t# Wildcard identifies only in-cone files, no index expansion\n ++\tensure_not_expanded reset deepest -- deep/\\* &&\n ++\n ++\t# Wildcard identifies only full sparse directories, no index expansion\n ++\tensure_not_expanded reset deepest -- folder\\* &&\n +\n -+\tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n 7: aa963eefae7 ! 8: c7145e039f3 unpack-trees: improve performance of next_cache_entry\n @@ Commit message\n beginning of the index each time it is called.\n \n The `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n - (unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain\n - the benefit `cache_bottom` provides in non-sparse index cases, a separate\n - `hint` position indicates the first position `next_cache_entry` should\n - search, updated each execution with a new position. The performance of `git\n - reset -- does-not-exist` (testing the \"worst case\" in which all entries in\n - the index are unpacked with `next_cache_entry`) is significantly improved\n - for the sparse index case:\n -\n - Test before after\n - ------------------------------------------------------\n - (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2%\n - (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2%\n - (sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1%\n - (sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3%\n + (unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the\n + benefit `cache_bottom` provides in non-sparse index cases, a separate `hint`\n + position indicates the first position `next_cache_entry` should search,\n + updated each execution with a new position.\n \n Signed-off-by: Victoria Dye \n \n\n-- \ngitgitgadget\n"},{"id":"438229","messageId":"ad7013a31aa5688747c1606d8c38aa96365c3711.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] reset: rename is_missing to !is_in_reset_tree","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:31Z","receivedAt":"2021-10-07T21:15:45Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRename and invert value of `is_missing` to `is_in_reset_tree` to make the\nvariable more descriptive of what it represents.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..d3695ce43c4 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -131,10 +131,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n \n-\t\tif (is_missing && !intent_to_add) {\n+\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n \t\t\tcontinue;\n \t\t}\n@@ -144,7 +144,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n-\t\tif (is_missing) {\n+\t\tif (!is_in_reset_tree) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n \t\t\tset_object_name_for_intent_to_add_entry(ce);\n \t\t}\n-- \ngitgitgadget\n\n"},{"id":"438231","messageId":"1f6da84830b00b7d33ac8ae4c6a46a90606c901f.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Kevin Willford via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:32Z","receivedAt":"2021-10-07T21:15:46Z","isPatch":true,"sender":{"key":"kewillf@microsoft.com","avatar":null},"body":"From: Kevin Willford \n\nChange `update_index_from_diff` to set `skip-worktree` when applicable for\nnew index entries. When `git reset --mixed ` is run, entries in\nthe index with differences between the pre-reset HEAD and reset \nare identified and handled with `update_index_from_diff`. For each file, a\nnew cache entry in inserted into the index, created from the side\nof the reset (without changing the working tree). However, the newly-created\nentry must have `skip-worktree` explicitly set in either of the following\nscenarios:\n\n1. the file is in the current index and has `skip-worktree` set\n2. the file is not in the current index but is outside of a defined sparse\n checkout definition\n\nNot setting the `skip-worktree` bit leads to likely-undesirable results for\na user. It causes `skip-worktree` settings to disappear on the\n\"diff\"-containing files (but *only* the diff-containing files), leading to\nthose files now showing modifications in `git status`. For example, when\nrunning `git reset --mixed` in a sparse checkout, some file entries outside\nof sparse checkout could show up as deleted, despite the user never deleting\nanything (and not wanting them on-disk anyway).\n\nAdditionally, add a test to `t7102` to ensure `skip-worktree` is preserved\nin a basic `git reset --mixed` scenario and update a failure-documenting\ntest from 19a0acc (t1092: test interesting sparse-checkout scenarios,\n2021-01-23) with new expected behavior.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 14 ++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 19 +++++--------------\n t/t7102-reset.sh | 17 +++++++++++++++++\n 3 files changed, 36 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex d3695ce43c4..e441b6601b9 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,7 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -130,6 +131,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n+\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n \t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n@@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n+\n+\t\t/*\n+\t\t * If the file 1) corresponds to an existing index entry with\n+\t\t * skip-worktree set, or 2) does not exist in the index but is\n+\t\t * outside the sparse checkout definition, add a skip-worktree bit\n+\t\t * to the new index entry.\n+\t\t */\n+\t\tpos = cache_name_pos(one->path, strlen(one->path));\n+\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n+\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n+\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..889079f55b8 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,26 +459,17 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\n \ttest_all_match git reset deepest &&\n-\ttest_all_match git reset update-folder1 &&\n-\ttest_all_match git reset update-folder2\n-'\n-\n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_success 'checkout and reset (mixed) [sparse]' '\n-\tinit_repos &&\n \n-\ttest_sparse_match git checkout -b reset-test update-deep &&\n-\ttest_sparse_match git reset deepest &&\n+\t# Because skip-worktree is preserved, resetting to update-folder1\n+\t# will show worktree changes for full-checkout that are not present\n+\t# in sparse-checkout or sparse-index.\n \ttest_sparse_match git reset update-folder1 &&\n-\ttest_sparse_match git reset update-folder2\n+\trun_on_sparse test_path_is_missing folder1\n '\n \n test_expect_success 'merge, cherry-pick, and rebase' '\ndiff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\nindex 601b2bf97f0..d05426062ec 100755\n--- a/t/t7102-reset.sh\n+++ b/t/t7102-reset.sh\n@@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n \ttest_cmp expect output\n '\n \n+test_expect_success '--mixed preserves skip-worktree' '\n+\techo 123 >>file2 &&\n+\tgit add file2 &&\n+\tgit update-index --skip-worktree file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_must_be_empty output &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tUnstaged changes after reset:\n+\tM\tfile2\n+\tEOF\n+\tgit update-index --no-skip-worktree file2 &&\n+\tgit add file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_cmp expect output\n+'\n+\n test_expect_success 'resetting specific path that is unmerged' '\n \tgit rm --cached file2 &&\n \tF1=$(git rev-parse HEAD:file1) &&\n-- \ngitgitgadget\n\n"},{"id":"438232","messageId":"014a408ea5d9894197c60f8d712749ea3cc39c9d.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:33Z","receivedAt":"2021-10-07T21:15:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd a new `--force-full-index` option to `git update-index`, which skips\nexplicitly setting `command_requires_full_index`. This option, intended for\nuse in internal testing purposes only, lets `git update-index` run as a\ncommand without sparse index compatibility implemented, even after it\nreceives updates to otherwise use the sparse index.\n\nThe specific test `--force-full-index` is intended for - `t1092 -\nsparse-index is expanded and converted back` - verifies index compatibility\nin commands that do not change the default (enabled)\n`command_requires_full_index` repo setting. In the past, the test used `git\nreset`. However, as `reset` and other commands are integrated with the\nsparse index, the command used in the test would need to keep changing.\nConversely, the `--force-full-index` option makes `git update-index` behave\nlike a not-yet-sparse-aware command, and can be used in the test\nindefinitely without interfering with future sparse index integrations.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n Documentation/git-update-index.txt | 5 +++++\n builtin/update-index.c | 11 +++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 2 +-\n 3 files changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..06255e321a3 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -24,6 +24,7 @@ SYNOPSIS\n \t [--[no-]fsmonitor]\n \t [--really-refresh] [--unresolve] [--again | -g]\n \t [--info-only] [--index-info]\n+\t [--force-full-index]\n \t [-z] [--stdin] [--index-version ]\n \t [--verbose]\n \t [--] [...]\n@@ -170,6 +171,10 @@ time. Version 4 is relatively young (first released in 1.8.0 in\n October 2012). Other Git implementations such as JGit and libgit2\n may not support it yet.\n \n+--force-full-index::\n+\tForce the command to operate on a full index, expanding a sparse\n+\tindex if necessary.\n+\n -z::\n \tOnly meaningful with `--stdin` or `--index-info`; paths are\n \tseparated with NUL character instead of LF.\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..32ada3ead77 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -964,6 +964,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tint split_index = -1;\n \tint force_write = 0;\n \tint fsmonitor = -1;\n+\tint use_default_full_index = 0;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct parse_opt_ctx_t ctx;\n \tstrbuf_getline_fn getline_fn;\n@@ -1069,6 +1070,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t{OPTION_SET_INT, 0, \"no-fsmonitor-valid\", &mark_fsmonitor_only, NULL,\n \t\t\tN_(\"clear fsmonitor valid bit\"),\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG},\n+\t\tOPT_SET_INT(0, \"force-full-index\", &use_default_full_index,\n+\t\t\tN_(\"run with full index explicitly required\"), 1),\n \t\tOPT_END()\n \t};\n \n@@ -1082,6 +1085,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \tif (newfd < 0)\n \t\tlock_error = errno;\n \n+\t/*\n+\t * If --force-full-index is set, the command should skip manually\n+\t * setting `command_requires_full_index`.\n+\t */\n+\tprepare_repo_settings(r);\n+\tif (!use_default_full_index)\n+\t\tr->settings.command_requires_full_index = 1;\n+\n \tentries = read_cache();\n \tif (entries < 0)\n \t\tdie(\"cache corrupted\");\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 889079f55b8..4aa4fef7b4f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -635,7 +635,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"438233","messageId":"7f21cf53e9d71de8c6351ae0f21ed531b7d13c42.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:34Z","receivedAt":"2021-10-07T21:15:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs. New performance test cases exercise various execution paths for\n`reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 84 ++++++++++++++++++++++++\n 2 files changed, 87 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4aa4fef7b4f..875cdcb0495 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -472,6 +472,90 @@ test_expect_success 'checkout and reset (mixed)' '\n \trun_on_sparse test_path_is_missing folder1\n '\n \n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+# Although the working tree differs between full and sparse checkouts after\n+# reset, the state of the index is the same.\n+test_expect_success 'reset with pathspecs outside sparse definition' '\n+\tinit_repos &&\n+\ttest_all_match git checkout -b reset-test base &&\n+\n+\ttest_sparse_match git reset update-folder1 -- folder1 &&\n+\tgit -C full-checkout reset update-folder1 -- folder1 &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1 &&\n+\n+\ttest_sparse_match git reset update-folder2 -- folder2/a &&\n+\tgit -C full-checkout reset update-folder2 -- folder2/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2/a\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset base -- \\*/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1/a &&\n+\n+\ttest_all_match git reset base -- folder\\* &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"438234","messageId":"a2d6212e287136b52b7eb904682523ed246f07e2.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:35Z","receivedAt":"2021-10-07T21:15:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nDisable `command_requires_full_index` repo setting and add\n`ensure_full_index` guards around code paths that cannot yet use sparse\ndirectory index entries. `reset --soft` does not modify the index, so no\ncompatibility changes are needed for it to function without expanding the\nindex. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`),\nthe full index is expanded to prevent cache tree corruption and invalid\nvariable accesses.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` must be run\n*after* enabling sparse index for the command (so that the index is not\nexpanded unnecessarily) and *before* using the index for reset (so that it\nis verified as uncorrupted).\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex e441b6601b9..0ac0de7dc97 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -180,6 +180,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -257,9 +258,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -405,6 +403,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"438235","messageId":"330e0c0977480d0506801854fcaa6c9f2b014569.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:36Z","receivedAt":"2021-10-07T21:15:51Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove `ensure_full_index` guard on `prime_cache_tree` and update\n`prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\nthe cache tree. While processing a tree's entries, `prime_cache_tree_rec`\nmust determine whether a directory entry is sparse or not by searching for\nit in the index (*without* expanding the index). If a matching sparse\ndirectory index entry is found, no subtrees are added to the cache tree\nentry and the entry count is set to 1 (representing the sparse directory\nitself). Otherwise, the tree is assumed to not be sparse and its subtrees\nare recursively added to the cache tree.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 47 ++++++++++++++++++++++--\n cache.h | 10 +++++\n read-cache.c | 27 ++++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 +++++++-\n 4 files changed, 86 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..2866101052c 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,26 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n+\tint base_path_len = tree_path->len;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,14 +768,40 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\n+\t\t\t/*\n+\t\t\t * Recursively-constructed subtree path is only needed when working\n+\t\t\t * in a sparse index (where it's used to determine whether the\n+\t\t\t * subtree is a sparse directory in the index).\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index) {\n+\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n+\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n+\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n+\t\t\t\tstrbuf_addch(tree_path, '/');\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n+\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n+\n \tit->entry_count = cnt;\n }\n \n@@ -772,12 +809,14 @@ void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..c079ece981a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -68,6 +68,11 @@\n */\n #define CACHE_ENTRY_PATH_LENGTH 80\n \n+enum index_search_mode {\n+\tNO_EXPAND_SPARSE = 0,\n+\tEXPAND_SPARSE = 1\n+};\n+\n static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct cache_entry *ce;\n@@ -551,7 +556,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tenum index_search_mode search_mode)\n {\n \tint first, last;\n \n@@ -570,7 +578,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +594,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n \t\t}\n \t}\n \n@@ -595,7 +603,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1235,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1335,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1370,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 875cdcb0495..4ac93874cb2 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -756,9 +756,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -768,6 +768,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"438236","messageId":"6ef8e4e31d39386e3a6fa1e6c24acd2c005fbd54.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:37Z","receivedAt":"2021-10-07T21:15:53Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove the `ensure_full_index` guard on `read_from_tree` and update `git\nreset --mixed` to ensure it can use sparse directory index entries wherever\npossible. Sparse directory entries are reset use `diff_tree_oid`, which\nrequires `change` and `add_remove` functions to process the internal\ncontents of the sparse directory. The `recursive` diff option handles cases\nin which `reset --mixed` must diff/merge files that are nested multiple\nlevels deep in a sparse directory.\n\nThe use of pathspecs with `git reset --mixed` introduces scenarios in which\ninternal contents of sparse directories may be matched by the pathspec. In\norder to reset *all* files in the repo that may match the pathspec, the\nfollowing conditions on the pathspec require index expansion before\nperforming the reset:\n\n* \"magic\" pathspecs\n* wildcard pathspecs that do not match only in-cone files or entire sparse\n directories\n* literal pathspecs matching something outside the sparse checkout\n definition\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 78 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n 2 files changed, 93 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 0ac0de7dc97..60517e7e1d6 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t * If the file 1) corresponds to an existing index entry with\n \t\t * skip-worktree set, or 2) does not exist in the index but is\n \t\t * outside the sparse checkout definition, add a skip-worktree bit\n-\t\t * to the new index entry.\n+\t\t * to the new index entry. Note that a sparse index will be expanded\n+\t\t * if this entry is outside the sparse cone - this is necessary\n+\t\t * to properly construct the reset sparse directory.\n \t\t */\n \t\tpos = cache_name_pos(one->path, strlen(one->path));\n \t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n@@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t}\n }\n \n+static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n+{\n+\tunsigned int i, pos;\n+\tint res = 0;\n+\tchar *skip_worktree_seen = NULL;\n+\n+\t/*\n+\t * When using a magic pathspec, assume for the sake of simplicity that\n+\t * the index needs to be expanded to match all matchable files.\n+\t */\n+\tif (pathspec->magic)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\tstruct pathspec_item item = pathspec->items[i];\n+\n+\t\t/*\n+\t\t * If the pathspec item has a wildcard, the index should be expanded\n+\t\t * if the pathspec has the possibility of matching a subset of entries inside\n+\t\t * of a sparse directory (but not the entire directory).\n+\t\t *\n+\t\t * If the pathspec item is a literal path, the index only needs to be expanded\n+\t\t * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n+\t\t * expand for in-cone files) and b) it doesn't match any sparse directories\n+\t\t * (since we can reset whole sparse directories without expanding them).\n+\t\t */\n+\t\tif (item.nowildcard_len < item.len) {\n+\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n+\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n+\n+\t\t\t\tif (!S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is longer than the sparse\n+\t\t\t\t * directory name and the sparse directory is the first\n+\t\t\t\t * component of the pathspec, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (item.nowildcard_len > ce_namelen(ce) &&\n+\t\t\t\t !strncmp(item.original, ce->name, ce_namelen(ce))) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is shorter than the sparse\n+\t\t\t\t * directory and the pathspec does not match the whole\n+\t\t\t\t * directory, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n+\t\t\t\t wildmatch(item.original, ce->name, 0)) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n+\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n+\t\t\tres = 1;\n+\n+\t\tif (res > 0)\n+\t\t\tbreak;\n+\t}\n+\n+\tfree(skip_worktree_seen);\n+\treturn res;\n+}\n+\n static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t struct object_id *tree_oid,\n \t\t\t int intent_to_add)\n@@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\tif (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n+\t\tensure_full_index(&the_index);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4ac93874cb2..c9343ff5b9c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -774,11 +774,28 @@ test_expect_success 'sparse-index is not expanded' '\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n+\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n \tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so the pathspec will not force the\n+\t# index to be expanded.\n+\tensure_not_expanded reset deepest -- folder1 &&\n+\tensure_not_expanded reset deepest -- folder1/ &&\n+\n+\t# Wildcard identifies only in-cone files, no index expansion\n+\tensure_not_expanded reset deepest -- deep/\\* &&\n+\n+\t# Wildcard identifies only full sparse directories, no index expansion\n+\tensure_not_expanded reset deepest -- folder\\* &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"438237","messageId":"c7145e039f3644339ba47296f4bad15aafc6b17a.1633641339.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-07T21:15:38Z","receivedAt":"2021-10-07T21:15:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the\nbenefit `cache_bottom` provides in non-sparse index cases, a separate `hint`\nposition indicates the first position `next_cache_entry` should search,\nupdated each execution with a new position.\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"438254","messageId":"45a1e0aa-4857-6647-cc60-bc47f32ae0c0@gmail.com","threadId":"56614","inReplyTo":"014a408ea5d9894197c60f8d712749ea3cc39c9d.1633641339.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-10-08T02:50:03Z","receivedAt":"2021-10-08T02:50:11Z","isPatch":true,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 08/10/21 04.15, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> Add a new `--force-full-index` option to `git update-index`, which skips\n> explicitly setting `command_requires_full_index`. This option, intended for\n> use in internal testing purposes only, lets `git update-index` run as a\n> command without sparse index compatibility implemented, even after it\n> receives updates to otherwise use the sparse index.\n> \n> The specific test `--force-full-index` is intended for - `t1092 -\n> sparse-index is expanded and converted back` - verifies index compatibility\n> in commands that do not change the default (enabled)\n> `command_requires_full_index` repo setting. In the past, the test used `git\n> reset`. However, as `reset` and other commands are integrated with the\n> sparse index, the command used in the test would need to keep changing.\n> Conversely, the `--force-full-index` option makes `git update-index` behave\n> like a not-yet-sparse-aware command, and can be used in the test\n> indefinitely without interfering with future sparse index integrations.\n> \n> Helped-by: Junio C Hamano \n> Signed-off-by: Victoria Dye \n\nGrammar looks OK.\n\nReviewed-by: Bagas Sanjaya \n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"438257","messageId":"CABPp-BF+bEUcyE0N79uRCkpCayJx_NMqOpnMSHHrpJM5a9hAWw@mail.gmail.com","threadId":"56614","inReplyTo":"3cfa0810-4449-1170-1f7b-cd029e5ce434@github.com","subject":"Re: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-10-08T03:42:21Z","receivedAt":"2021-10-08T03:42:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 6, 2021 at 1:40 PM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Tue, Oct 5, 2021 at 6:20 AM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n> >> Add a new `--force-full-index` option to `git update-index`, which skips\n> >> explicitly setting `command_requires_full_index`. This lets `git\n> >> update-index --force-full-index` run as a command without sparse index\n> >> compatibility implemented, even after it receives sparse index compatibility\n> >> updates.\n> >>\n> >> By using `git update-index --force-full-index` in the `t1092` test\n> >> `sparse-index is expanded and converted back`, commands can continue to\n> >> integrate with the sparse index without the need to keep modifying the\n> >> command used in the test.\n> >\n> > So...we're adding a permanent user-facing command line flag, whose\n> > purpose is just to help us with the transition work of implementing\n> > sparse indexes everywhere? Am I reading that right, or is that just\n> > the reason for t1092 and there are more reasons for it elsewhere?\n> >\n> > Also, I'm curious if update-index is the right place to add this. If\n> > you don't want a sparse index anymore, wouldn't a user want to run\n> > git sparse-checkout disable\n> > ? Or is the point that you do want to keep the sparse checkout, but\n> > you just don't want the index to also be sparse? Still, even in that\n> > case, it seems like adding a subcommand or flag to an existing\n> > sparse-checkout subcommand would feel more natural, since\n> > sparse-checkout is the command the user uses to request to get into a\n> > sparse-checkout and sparse index.\n> >\n>\n> This came out of a conversation [1] on an earlier version of this patch.\n> Because the `t1092 - sparse-index is expanded and converted back` test\n> verifies sparse index compatibility (i.e., expand the index when reading,\n> collapse back to sparse when writing) on commands that don't have any sparse\n> index integration, it needed to be changed from `git reset` to something\n> else. However, as we keep integrating commands with sparse index we'd need\n> to keep changing the command in the test, creating a bunch of patches doing\n> effectively the same thing for no long-term benefit.\n>\n> The `--force-full-index` flag isn't meant to be used externally or modify\n> the index in any \"new\" way - it's really just a \"test\" version of `git\n> update-index` that we guarantee will accurately represent a command using\n> the default settings. Right now, it does exactly what `git update-index`\n> (without the flag) does, and will only behave differently once `git\n> update-index` is integrated with sparse index. Using `--force-full-index`,\n> the test won't need to be regularly updated and will continue to catch\n> errors like:\n>\n> 1. Changing the default value of `command_requires_full_index` to 0\n> 2. Not expanding a sparse index to full when `command_requires_full_index`\n> is 1\n> 3. Not collapsing the index back to sparse if sparse index is enabled\n>\n> I see the issue of introducing a test-only option (when sparse index is\n> integrated everywhere, shouldn't it be deprecated?). If there's a way to\n> make this more obviously internal/temporary, I'm happy to modify it. Or, if\n> semi-frequent updates of the command in the test aren't a huge issue, I can\n> revert to V1.\n\nIf it's a test-only capability you need, I'd say add it under\nt/helpers/ somewhere, either a new flag for an existing subcommand of\ntest-tool, or a new subcommand for test-tool.\n"},{"id":"438271","messageId":"xmqq35pcm6m9.fsf@gitster.g","threadId":"56614","inReplyTo":"014a408ea5d9894197c60f8d712749ea3cc39c9d.1633641339.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T05:24:14Z","receivedAt":"2021-10-08T05:24:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> +\t/*\n> +\t * If --force-full-index is set, the command should skip manually\n> +\t * setting `command_requires_full_index`.\n> +\t */\n\nHmph, doesn't that feel unnaturally backwards, though?\n\nThe settings.command_requires_full_index bit forces read-cache to\ncall ensure_full_index() immediately after the in-core index is read\nfrom the disk. If we are forcing operating on the full index, I'd\nimagine that we'd be making sure that ensure_full_index() to be\ncalled.\n\nI do not see anything in the code that ensures active_cache_changed\nto be flipped on. So the new test that says\n\n git -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index\n\nmay not call ensure_full_index(), but because nothing marks\nthe_index as changed, I think we won't call write_locked_index() at\nthe end of cmd_update_index(). IOW, what we have in the test patch\nmay be an expensive noop, no?\n\nOr perhaps I am reading the patch completely incorrectly. I dunno.\n\n> +\tprepare_repo_settings(r);\n> +\tif (!use_default_full_index)\n> +\t\tr->settings.command_requires_full_index = 1;\n> +\n> \tentries = read_cache();\n> \tif (entries < 0)\n> \t\tdie(\"cache corrupted\");\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 889079f55b8..4aa4fef7b4f 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -635,7 +635,7 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> \tinit_repos &&\n> \n> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> -\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n> +\t\tgit -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index &&\n> \ttest_region index convert_to_sparse trace2.txt &&\n> \ttest_region index ensure_full_index trace2.txt\n> '\n"},{"id":"438283","messageId":"xmqqr1cvlwfi.fsf@gitster.g","threadId":"56614","inReplyTo":"1f6da84830b00b7d33ac8ae4c6a46a90606c901f.1633641339.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T09:04:17Z","receivedAt":"2021-10-08T09:04:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Kevin Willford via GitGitGadget\" writes:\n\n> @@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> \n> \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n> \t\t\t\t 0, 0);\n> +\n> +\t\t/*\n> +\t\t * If the file 1) corresponds to an existing index entry with\n> +\t\t * skip-worktree set, or 2) does not exist in the index but is\n> +\t\t * outside the sparse checkout definition, add a skip-worktree bit\n> +\t\t * to the new index entry.\n> +\t\t */\n> +\t\tpos = cache_name_pos(one->path, strlen(one->path));\n> +\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n> +\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n> +\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n\nOK. Nicely explained.\n"},{"id":"438291","messageId":"c1dd1fef-2d48-cc18-5786-10e174b487a7@gmail.com","threadId":"56614","inReplyTo":"330e0c0977480d0506801854fcaa6c9f2b014569.1633641339.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-08T11:09:30Z","receivedAt":"2021-10-08T11:09:40Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Victoria\n\nOn 07/10/2021 22:15, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> Remove `ensure_full_index` guard on `prime_cache_tree` and update\n> `prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\n> the cache tree. While processing a tree's entries, `prime_cache_tree_rec`\n> must determine whether a directory entry is sparse or not by searching for\n> it in the index (*without* expanding the index). If a matching sparse\n> directory index entry is found, no subtrees are added to the cache tree\n> entry and the entry count is set to 1 (representing the sparse directory\n> itself). Otherwise, the tree is assumed to not be sparse and its subtrees\n> are recursively added to the cache tree.\n\nI was looking at the callers to prime_cache_tree() this morning and \nwould like to suggest an alternative approach - just delete \nprime_cache_tree() and all of its callers! As far as I can see it is \nonly ever called after a successful call to unpack_trees() and since \n52fca2184d (\"unpack-trees: populate cache-tree on successful merge\", \n2015-07-28) unpack_trees() updates the cache tree for the caller. All \nthe call sites are pretty obvious apart from the one in \nt/help/test-fast-rebase.c where unpack_trees() is called by \nmerge_switch_to_result()\n\nBest Wishes\n\nPhillip\n\n> Helped-by: Elijah Newren \n> Signed-off-by: Victoria Dye \n> ---\n> cache-tree.c | 47 ++++++++++++++++++++++--\n> cache.h | 10 +++++\n> read-cache.c | 27 ++++++++++----\n> t/t1092-sparse-checkout-compatibility.sh | 15 +++++++-\n> 4 files changed, 86 insertions(+), 13 deletions(-)\n> \n> diff --git a/cache-tree.c b/cache-tree.c\n> index 9be19c85b66..2866101052c 100644\n> --- a/cache-tree.c\n> +++ b/cache-tree.c\n> @@ -740,15 +740,26 @@ out:\n> \treturn ret;\n> }\n> \n> +static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n> +\t\t\t\t\tstruct tree *tree)\n> +{\n> +\n> +\toidcpy(&it->oid, &tree->object.oid);\n> +\tit->entry_count = 1;\n> +}\n> +\n> static void prime_cache_tree_rec(struct repository *r,\n> \t\t\t\t struct cache_tree *it,\n> -\t\t\t\t struct tree *tree)\n> +\t\t\t\t struct tree *tree,\n> +\t\t\t\t struct strbuf *tree_path)\n> {\n> \tstruct tree_desc desc;\n> \tstruct name_entry entry;\n> \tint cnt;\n> +\tint base_path_len = tree_path->len;\n> \n> \toidcpy(&it->oid, &tree->object.oid);\n> +\n> \tinit_tree_desc(&desc, tree->buffer, tree->size);\n> \tcnt = 0;\n> \twhile (tree_entry(&desc, &entry)) {\n> @@ -757,14 +768,40 @@ static void prime_cache_tree_rec(struct repository *r,\n> \t\telse {\n> \t\t\tstruct cache_tree_sub *sub;\n> \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n> +\n> \t\t\tif (!subtree->object.parsed)\n> \t\t\t\tparse_tree(subtree);\n> \t\t\tsub = cache_tree_sub(it, entry.path);\n> \t\t\tsub->cache_tree = cache_tree();\n> -\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n> +\n> +\t\t\t/*\n> +\t\t\t * Recursively-constructed subtree path is only needed when working\n> +\t\t\t * in a sparse index (where it's used to determine whether the\n> +\t\t\t * subtree is a sparse directory in the index).\n> +\t\t\t */\n> +\t\t\tif (r->index->sparse_index) {\n> +\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n> +\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n> +\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n> +\t\t\t\tstrbuf_addch(tree_path, '/');\n> +\t\t\t}\n> +\n> +\t\t\t/*\n> +\t\t\t * If a sparse index is in use, the directory being processed may be\n> +\t\t\t * sparse. To confirm that, we can check whether an entry with that\n> +\t\t\t * exact name exists in the index. If it does, the created subtree\n> +\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n> +\t\t\t * as normal.\n> +\t\t\t */\n> +\t\t\tif (r->index->sparse_index &&\n> +\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n> +\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n> +\t\t\telse\n> +\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n> \t\t\tcnt += sub->cache_tree->entry_count;\n> \t\t}\n> \t}\n> +\n> \tit->entry_count = cnt;\n> }\n> \n> @@ -772,12 +809,14 @@ void prime_cache_tree(struct repository *r,\n> \t\t struct index_state *istate,\n> \t\t struct tree *tree)\n> {\n> +\tstruct strbuf tree_path = STRBUF_INIT;\n> +\n> \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n> \tcache_tree_free(&istate->cache_tree);\n> \tistate->cache_tree = cache_tree();\n> \n> -\tensure_full_index(istate);\n> -\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n> +\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n> +\tstrbuf_release(&tree_path);\n> \tistate->cache_changed |= CACHE_TREE_CHANGED;\n> \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n> }\n> diff --git a/cache.h b/cache.h\n> index f6295f3b048..1d3e4665562 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n> */\n> int index_name_pos(struct index_state *, const char *name, int namelen);\n> \n> +/*\n> + * Determines whether an entry with the given name exists within the\n> + * given index. The return value is 1 if an exact match is found, otherwise\n> + * it is 0. Note that, unlike index_name_pos, this function does not expand\n> + * the index if it is sparse. If an item exists within the full index but it\n> + * is contained within a sparse directory (and not in the sparse index), 0 is\n> + * returned.\n> + */\n> +int index_entry_exists(struct index_state *, const char *name, int namelen);\n> +\n> /*\n> * Some functions return the negative complement of an insert position when a\n> * precise match was not found but a position was found where the entry would\n> diff --git a/read-cache.c b/read-cache.c\n> index f5d4385c408..c079ece981a 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -68,6 +68,11 @@\n> */\n> #define CACHE_ENTRY_PATH_LENGTH 80\n> \n> +enum index_search_mode {\n> +\tNO_EXPAND_SPARSE = 0,\n> +\tEXPAND_SPARSE = 1\n> +};\n> +\n> static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n> {\n> \tstruct cache_entry *ce;\n> @@ -551,7 +556,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n> \treturn 0;\n> }\n> \n> -static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n> +static int index_name_stage_pos(struct index_state *istate,\n> +\t\t\t\tconst char *name, int namelen,\n> +\t\t\t\tint stage,\n> +\t\t\t\tenum index_search_mode search_mode)\n> {\n> \tint first, last;\n> \n> @@ -570,7 +578,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n> \t\tfirst = next+1;\n> \t}\n> \n> -\tif (istate->sparse_index &&\n> +\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n> \t first > 0) {\n> \t\t/* Note: first <= istate->cache_nr */\n> \t\tstruct cache_entry *ce = istate->cache[first - 1];\n> @@ -586,7 +594,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n> \t\t ce_namelen(ce) < namelen &&\n> \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n> \t\t\tensure_full_index(istate);\n> -\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n> +\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n> \t\t}\n> \t}\n> \n> @@ -595,7 +603,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n> \n> int index_name_pos(struct index_state *istate, const char *name, int namelen)\n> {\n> -\treturn index_name_stage_pos(istate, name, namelen, 0);\n> +\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n> +}\n> +\n> +int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n> +{\n> +\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n> }\n> \n> int remove_index_entry_at(struct index_state *istate, int pos)\n> @@ -1222,7 +1235,7 @@ static int has_dir_name(struct index_state *istate,\n> \t\t\t */\n> \t\t}\n> \n> -\t\tpos = index_name_stage_pos(istate, name, len, stage);\n> +\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n> \t\tif (pos >= 0) {\n> \t\t\t/*\n> \t\t\t * Found one, but not so fast. This could\n> @@ -1322,7 +1335,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n> \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n> \telse\n> -\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n> +\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n> \n> \t/* existing match? Just replace it. */\n> \tif (pos >= 0) {\n> @@ -1357,7 +1370,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> \t\tif (!ok_to_replace)\n> \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n> \t\t\t\t ce->name);\n> -\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n> +\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n> \t\tpos = -pos-1;\n> \t}\n> \treturn pos + 1;\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 875cdcb0495..4ac93874cb2 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -756,9 +756,9 @@ test_expect_success 'sparse-index is not expanded' '\n> \tensure_not_expanded checkout - &&\n> \tensure_not_expanded switch rename-out-to-out &&\n> \tensure_not_expanded switch - &&\n> -\tgit -C sparse-index reset --hard &&\n> +\tensure_not_expanded reset --hard &&\n> \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n> -\tgit -C sparse-index reset --hard &&\n> +\tensure_not_expanded reset --hard &&\n> \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n> \n> \techo >>sparse-index/README.md &&\n> @@ -768,6 +768,17 @@ test_expect_success 'sparse-index is not expanded' '\n> \techo >>sparse-index/untracked.txt &&\n> \tensure_not_expanded add . &&\n> \n> +\tfor ref in update-deep update-folder1 update-folder2 update-deep\n> +\tdo\n> +\t\techo >>sparse-index/README.md &&\n> +\t\tensure_not_expanded reset --hard $ref || return 1\n> +\tdone &&\n> +\n> +\tensure_not_expanded reset --hard update-deep &&\n> +\tensure_not_expanded reset --keep base &&\n> +\tensure_not_expanded reset --merge update-deep &&\n> +\tensure_not_expanded reset --hard &&\n> +\n> \tensure_not_expanded checkout -f update-deep &&\n> \ttest_config -C sparse-index pull.twohead ort &&\n> \t(\n> \n\n"},{"id":"438293","messageId":"72f4df1e-54ef-6bdb-ef71-1f115e92c9a5@github.com","threadId":"56614","inReplyTo":"xmqq35pcm6m9.fsf@gitster.g","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-08T15:47:45Z","receivedAt":"2021-10-08T15:47:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> +\t/*\n>> +\t * If --force-full-index is set, the command should skip manually\n>> +\t * setting `command_requires_full_index`.\n>> +\t */\n> \n> Hmph, doesn't that feel unnaturally backwards, though?\n> \n> The settings.command_requires_full_index bit forces read-cache to\n> call ensure_full_index() immediately after the in-core index is read\n> from the disk. If we are forcing operating on the full index, I'd\n> imagine that we'd be making sure that ensure_full_index() to be\n> called.\n> \n\nI tried coming up with a user-facing name that wasn't too focused on the\ninternal implementation, but it ends up being misleading. The intention was\nto have this be a variation of `git update-index` that uses the default\nsetting for `command_requires_full_index` but then proceeds to read and\nwrite the index as `update-index` normally would. Something like\n`--use-default-index-sparsity` might have been more accurate?\n\n> I do not see anything in the code that ensures active_cache_changed\n> to be flipped on. So the new test that says\n> \n> git -C sparse-index -c core.fsmonitor=\"\" update-index --force-full-index\n> \n> may not call ensure_full_index(), but because nothing marks\n> the_index as changed, I think we won't call write_locked_index() at\n> the end of cmd_update_index(). IOW, what we have in the test patch\n> may be an expensive noop, no?\n> \n\nIn the test's use-case, `active_cache_changed` ends up set to\n`CACHE_TREE_CHANGED`, which forces writing the index. It is still\neffectively a no-op, but it serves the needs of the test.\n\nIn any case, Elijah suggested using a `test-tool` subcommand for this\npurpose [1], which I think is more appropriate overall. Something like\n`test-tool read-write-cache` can be implemented to make no mention of\n`command_requires_full_index` (therefore using its default value) and force\na basic read & write of the index. It also eliminates the issue of having a\nuser-facing name at all, and can easily be removed once all sparse index\nintegrations are done.\n\n[1] https://lore.kernel.org/git/CABPp-BF+bEUcyE0N79uRCkpCayJx_NMqOpnMSHHrpJM5a9hAWw@mail.gmail.com/\n"},{"id":"438297","messageId":"xmqqmtnjl9w0.fsf@gitster.g","threadId":"56614","inReplyTo":"CABPp-BF+bEUcyE0N79uRCkpCayJx_NMqOpnMSHHrpJM5a9hAWw@mail.gmail.com","subject":"Re: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T17:11:11Z","receivedAt":"2021-10-08T17:11:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> I see the issue of introducing a test-only option (when sparse index is\n>> integrated everywhere, shouldn't it be deprecated?). If there's a way to\n>> make this more obviously internal/temporary, I'm happy to modify it. Or, if\n>> semi-frequent updates of the command in the test aren't a huge issue, I can\n>> revert to V1.\n>\n> If it's a test-only capability you need, I'd say add it under\n> t/helpers/ somewhere, either a new flag for an existing subcommand of\n> test-tool, or a new subcommand for test-tool.\n\nIs the ability to force expanding to full index completely useless\nin the field? For diagnosing breakage the end-users may see in the\nwild, or perhaps in a specialist usecase for whatever reason working\non full index is preferable and the user may want to force it once\nto correct an earlier mistake to enable sparse-index before toggling\nthe configuration off, or something?\n\nIf we do not foresee any such reason, I'd agree it is good to move\nthat to t/helpers/; otherwise, I think update-index is as good as\nany other place, and the option will sit well next to other options\nlike \"--[no-]skip-worktree\", \"--[no-]assume-unchanged\". It would\nmost likely need to be used together with \"--force-write-index\" (or\nbe made to imply the latter) to be useful, I suspect.\n\nThanks.\n"},{"id":"438298","messageId":"52768318-ef8a-b26f-d4bc-d5c91779ccdb@github.com","threadId":"56614","inReplyTo":"c1dd1fef-2d48-cc18-5786-10e174b487a7@gmail.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-08T17:14:13Z","receivedAt":"2021-10-08T17:14:19Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Phillip Wood wrote:\n> Hi Victoria\n> \n> On 07/10/2021 22:15, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye \n>>\n>> Remove `ensure_full_index` guard on `prime_cache_tree` and update\n>> `prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\n>> the cache tree. While processing a tree's entries, `prime_cache_tree_rec`\n>> must determine whether a directory entry is sparse or not by searching for\n>> it in the index (*without* expanding the index). If a matching sparse\n>> directory index entry is found, no subtrees are added to the cache tree\n>> entry and the entry count is set to 1 (representing the sparse directory\n>> itself). Otherwise, the tree is assumed to not be sparse and its subtrees\n>> are recursively added to the cache tree.\n> \n> I was looking at the callers to prime_cache_tree() this morning and would like to suggest an alternative approach - just delete prime_cache_tree() and all of its callers! As far as I can see it is only ever called after a successful call to unpack_trees() and since 52fca2184d (\"unpack-trees: populate cache-tree on successful merge\", 2015-07-28) unpack_trees() updates the cache tree for the caller. All the call sites are pretty obvious apart from the one in t/help/test-fast-rebase.c where unpack_trees() is called by merge_switch_to_result()\n> \n\nIt looks like `prime_cache_tree` can be removed mostly without issue, but\nit causes the two last tests in `t4058-diff-duplicates.sh` to fail. Those\ntests document failure cases when dealing with duplicate tree entries [1],\nand it looks like `prime_cache_tree` was creating the appearance of a\nfully-reset index but was still leaving it in a state where subsequent\noperations could fail.\n\nI'm inclined to say the solution here would be to update the tests to\ndocument the \"new\" failure behavior and proceed with removing\n`prime_cache_tree`, because:\n\n* the test using `git reset --hard` disables `GIT_TEST_CHECK_CACHE_TREE`,\n indicating that `prime_cache_tree` already wasn't behaving correctly\n* attempting to fix the overarching issues with duplicate tree entries will\n substantially delay this patch series\n* a duplicate entry fix is largely unrelated to the intended scope of the\n series\n\nAnother option would be to leave `prime_cache_tree` as it is, but with it\nbeing apparently useless outside of mostly-broken use cases in `t4058`, it\nseems like a waste to keep it around.\n\n[1] ac14de13b2 (t4058: explore duplicate tree entry handling in a bit more detail, 2020-12-11)\n"},{"id":"438299","messageId":"xmqqily7l9i5.fsf@gitster.g","threadId":"56614","inReplyTo":"72f4df1e-54ef-6bdb-ef71-1f115e92c9a5@github.com","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T17:19:30Z","receivedAt":"2021-10-08T17:19:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> I tried coming up with a user-facing name that wasn't too focused on the\n> internal implementation, but it ends up being misleading. The intention was\n> to have this be a variation of `git update-index` that uses the default\n> setting for `command_requires_full_index` but then proceeds to read and\n> write the index as `update-index` normally would. Something like\n> `--use-default-index-sparsity` might have been more accurate?\n\nThe option name in the reviewed patch does imply \"we force expanding\nto full\" and not \"use the default\", so it probably needs renaming,\nif we want the \"use the default\" semantics. But is that useful in\nthe context of the test you are using it in place of \"reset\" or \"mv\"?\nEven if the default is somehow flipped to use sparse always, wouldn't\nthe particular test want the index expanded? I dunno.\n\n> In the test's use-case, `active_cache_changed` ends up set to\n> `CACHE_TREE_CHANGED`, which forces writing the index. It is still\n> effectively a no-op, but it serves the needs of the test.\n\nAh, cache-tree is updated, then it's OK.\n\nAs to test-tool vs end-user-accessible-command, I do not have a\nstrong opinion, but use your imagination and ask Derrick or somebody\nelse for their imagination to see if such a \"force expand\" feature\nmay be something the end-users might need an access to in order to\ndig themselves out of a hole (in which case, it may be better to\nmake it end-user-accessible) or not (in which case, test-tool is\nmore appropriate).\n\nThanks.\n\n\n"},{"id":"438307","messageId":"xmqq1r4vl65w.fsf@gitster.g","threadId":"56614","inReplyTo":"52768318-ef8a-b26f-d4bc-d5c91779ccdb@github.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-08T18:31:39Z","receivedAt":"2021-10-08T18:31:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Phillip Wood wrote:\n\n>> I was looking at the callers to prime_cache_tree() this morning\n>> and would like to suggest an alternative approach - just delete\n>> prime_cache_tree() and all of its callers!\n\nDo you mean the calls added by new patches without understanding\nwhat they are doing, or all calls to it?\n\nEvery time you update a path in the index from the working tree\n(e.g. \"git add\") and other sources, the directory in the cache-tree\nthat includes the path is invalidated, and the surviving subtrees of\ncache-tree is used to speed up writing the index as a tree object,\ndoing \"diff-index --cached\" (hence \"git status\"), etc. So over\ntime, the cache-tree \"degrades\" as you muck with the index entries.\n\nWhen you write out the index as a tree, we by definition have to\nknow the object names of all the tree objects that correspond to\neach directory in the index. A fully valid cache-tree is saved when\nit happens, so the above process can start over.\n\nThere are cases other than \"git write-tree\" that we can cheaply\nlearn the object names of all the tree objects that correspond to\neach directory in the index. When we read the index from an\nexisting tree object, we know which tree (and its subtrees) we\npopulated the index from, so we can salvage a degraded cache-tree.\n\n\"reset --hard\" and \"reset --mixed\" may be good opportunities, so is\n\"checkout \" that starts from a clean index. And cache tree\npriming is a mechanism to take advantage of such an opportunity.\n\nThe cache-tree does not have to be primed and all you lose is\nperformance, so priming can be removed mostly \"without an issue\", if\nyou are not paying attention to cache-tree degradation. Priming\nwith incorrect data, however, would leave permanent damage by\nwriting a wrong tree via \"git write-tree\" (hence \"git commit\") and\nshowing a wrong diff via \"git diff-index [--cached]\" (hence \"git\nstatus\" and probably \"git add -- \"), so not priming is\nsafer than priming incorrectly.\n\nHTH.\n"},{"id":"438403","messageId":"47d4c810-0b56-45b0-944c-72c4d047f9b6@gmail.com","threadId":"56614","inReplyTo":"xmqq1r4vl65w.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-09T11:18:03Z","receivedAt":"2021-10-09T11:18:10Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 08/10/2021 19:31, Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n>> Phillip Wood wrote:\n> \n>>> I was looking at the callers to prime_cache_tree() this morning\n>>> and would like to suggest an alternative approach - just delete\n>>> prime_cache_tree() and all of its callers!\n> \n> Do you mean the calls added by new patches without understanding\n> what they are doing, or all calls to it?\n\nI mean all calls to prime_cache_tree() after having understood (or at \nleast thinking that I understand) what they are doing. As I tried to \nexplain in the part of my message that you have cut\n\n(a) a successful call to unpack_trees() updates the cache tree\n\n(b) all the existing calls to prime_cache_tree() follow a successful \ncall to unpack_trees() and nothing touches in index in between the call \nto unpack_trees() and prime_cache_tree().\n\nMaybe I've misunderstood something but that leads me believe those calls \ncan be removed without degrading performance.\n\nBest Wishes\n\nPhillip\n\n> Every time you update a path in the index from the working tree\n> (e.g. \"git add\") and other sources, the directory in the cache-tree\n> that includes the path is invalidated, and the surviving subtrees of\n> cache-tree is used to speed up writing the index as a tree object,\n> doing \"diff-index --cached\" (hence \"git status\"), etc. So over\n> time, the cache-tree \"degrades\" as you muck with the index entries.\n> \n> When you write out the index as a tree, we by definition have to\n> know the object names of all the tree objects that correspond to\n> each directory in the index. A fully valid cache-tree is saved when\n> it happens, so the above process can start over.\n> \n> There are cases other than \"git write-tree\" that we can cheaply\n> learn the object names of all the tree objects that correspond to\n> each directory in the index. When we read the index from an\n> existing tree object, we know which tree (and its subtrees) we\n> populated the index from, so we can salvage a degraded cache-tree.\n> \n> \"reset --hard\" and \"reset --mixed\" may be good opportunities, so is\n> \"checkout \" that starts from a clean index. And cache tree\n> priming is a mechanism to take advantage of such an opportunity.\n> \n> The cache-tree does not have to be primed and all you lose is\n> performance, so priming can be removed mostly \"without an issue\", if\n> you are not paying attention to cache-tree degradation. Priming\n> with incorrect data, however, would leave permanent damage by\n> writing a wrong tree via \"git write-tree\" (hence \"git commit\") and\n> showing a wrong diff via \"git diff-index [--cached]\" (hence \"git\n> status\" and probably \"git add -- \"), so not priming is\n> safer than priming incorrectly.\n> \n> HTH.\n> \n\n"},{"id":"438433","messageId":"xmqqlf30edvf.fsf@gitster.g","threadId":"56614","inReplyTo":"47d4c810-0b56-45b0-944c-72c4d047f9b6@gmail.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-10T22:03:48Z","receivedAt":"2021-10-10T22:03:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> On 08/10/2021 19:31, Junio C Hamano wrote:\n>> Victoria Dye writes:\n>> \n>>> Phillip Wood wrote:\n>> \n>>>> I was looking at the callers to prime_cache_tree() this morning\n>>>> and would like to suggest an alternative approach - just delete\n>>>> prime_cache_tree() and all of its callers!\n>> Do you mean the calls added by new patches without understanding\n>> what they are doing, or all calls to it?\n>\n> I mean all calls to prime_cache_tree() after having understood (or at\n> least thinking that I understand) what they are doing.\n\nSorry, my statement was confusingly written. I meant \"calls added\nby new patches, written by those who do not understand what\nprime_cache_tree() calls are doing\", but after re-reading it, I\nthink it could be taken to be referring to \"you may be commenting\nwithout understanding what prime_cache_tree() calls are doing\",\nwhich wasn't my intention.\n\n> (a) a successful call to unpack_trees() updates the cache tree\n>\n> (b) all the existing calls to prime_cache_tree() follow a successful\n> call to unpack_trees() and nothing touches in index in between the\n> call to unpack_trees() and prime_cache_tree().\n\nAhh, OK.\n\nI think we originally avoided calling cache_tree_update() lightly\n(because it is essentially a \"write-tree\", a fairly heavy-weight\noperation, without I/O) and instead relied on prime_cache_tree() to\nget degraded cache-tree back into freshness.\n\nWhat I forgot was that 52fca218 (unpack-trees: populate cache-tree\non successful merge, 2015-07-28) added cache_tree_update() there at\nthe end of unpack_trees(). The commit covers quite a wide range of\noperations---the log message says \"merge\", but in fact anything that\nuses unpack_trees() including branch switching and the resetting of\nthe index are affected, and they cause a full reconstruction of the\ncache tree by calling cache_tree_update().\n\nFor most callers of prime_cache_tree(), like the ones in \"git\nread-tree\" and \"git reset\", it is immediately obvious that we just\nread from the same tree, and we should have everything from the tree\nand nothing else in the resulting index, so it is clear that the\nprime_cache_tree() call is recreating the same cache-tree\ninformation that we already should have computed ourselves, and\nthese calls can go (or if \"prime\" is still cheaper than \"update\",\nthese callers can pass an option to tell unpack_trees() to skip the\ncache_tree_update() call, because they will call \"prime\" immediately\nafter).\n\nFor other callers it is not immediately obvious, but I trust you are\ncorrectly reading the code ;-)\n\nThanks.\n\n\n\n"},{"id":"438464","messageId":"cc60c6e7-ecef-ae22-8ec7-ab290ff2b830@gmail.com","threadId":"56614","inReplyTo":"xmqqily7l9i5.fsf@gitster.g","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-10-11T14:12:24Z","receivedAt":"2021-10-11T14:30:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/8/21 1:19 PM, Junio C Hamano wrote:\n> Victoria Dye writes:\n> \n>> I tried coming up with a user-facing name that wasn't too focused on the\n>> internal implementation, but it ends up being misleading. The intention was\n>> to have this be a variation of `git update-index` that uses the default\n>> setting for `command_requires_full_index` but then proceeds to read and\n>> write the index as `update-index` normally would. Something like\n>> `--use-default-index-sparsity` might have been more accurate?\n> \n> The option name in the reviewed patch does imply \"we force expanding\n> to full\" and not \"use the default\", so it probably needs renaming,\n> if we want the \"use the default\" semantics. But is that useful in\n> the context of the test you are using it in place of \"reset\" or \"mv\"?\n> Even if the default is somehow flipped to use sparse always, wouldn't\n> the particular test want the index expanded? I dunno.\n> \n>> In the test's use-case, `active_cache_changed` ends up set to\n>> `CACHE_TREE_CHANGED`, which forces writing the index. It is still\n>> effectively a no-op, but it serves the needs of the test.\n> \n> Ah, cache-tree is updated, then it's OK.\n> \n> As to test-tool vs end-user-accessible-command, I do not have a\n> strong opinion, but use your imagination and ask Derrick or somebody\n> else for their imagination to see if such a \"force expand\" feature\n> may be something the end-users might need an access to in order to\n> dig themselves out of a hole (in which case, it may be better to\n> make it end-user-accessible) or not (in which case, test-tool is\n> more appropriate).\n\nI think there is something to be said about the name being confusing,\nbecause the current implementation focuses on \"expand a sparse index\nupon read\" but it also allows the index to be written as sparse.\n\nConversely, if the user runs\n\n git -c index.sparse=false update-index ...\n\nthen the index.sparse config setting forbids conversion from full to\nsparse, but does not say anything about expanding to full.\n\nPerhaps this should be corrected: the index.sparse=false setting\nshould expand a sparse index to a full one, then prevent it from\nbeing converted to a sparse one on write.\n\nThis diff should do it:\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 564283c7e7e..04df1051e18 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2376,7 +2376,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \tprepare_repo_settings(istate->repo);\n-\tif (istate->repo->settings.command_requires_full_index)\n+\tif (!istate->repo->settings.sparse_index ||\n+\t istate->repo->settings.command_requires_full_index)\n \t\tensure_full_index(istate);\n \n \treturn istate->cache_nr;\n\nVictoria, what are your thoughts about including such a change?\n\nJunio, would it be better to change the config setting, and then\nupdate this test to use the config setting over a command-line flag?\nThis would allow us to punt on the --force-full-index flag until we\nhave time to focus on the 'git update-index' command and interactions\nlike this.\n\nThanks,\n-Stolee\n"},{"id":"438466","messageId":"b7227617-0c99-3899-a234-037fe9c23558@github.com","threadId":"56614","inReplyTo":"cc60c6e7-ecef-ae22-8ec7-ab290ff2b830@gmail.com","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-11T15:05:58Z","receivedAt":"2021-10-11T15:06:03Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee wrote:\n> On 10/8/21 1:19 PM, Junio C Hamano wrote:\n>> Victoria Dye writes:\n>>\n>>> I tried coming up with a user-facing name that wasn't too focused on the\n>>> internal implementation, but it ends up being misleading. The intention was\n>>> to have this be a variation of `git update-index` that uses the default\n>>> setting for `command_requires_full_index` but then proceeds to read and\n>>> write the index as `update-index` normally would. Something like\n>>> `--use-default-index-sparsity` might have been more accurate?\n>>\n>> The option name in the reviewed patch does imply \"we force expanding\n>> to full\" and not \"use the default\", so it probably needs renaming,\n>> if we want the \"use the default\" semantics. But is that useful in\n>> the context of the test you are using it in place of \"reset\" or \"mv\"?\n>> Even if the default is somehow flipped to use sparse always, wouldn't\n>> the particular test want the index expanded? I dunno.\n>>\n>>> In the test's use-case, `active_cache_changed` ends up set to\n>>> `CACHE_TREE_CHANGED`, which forces writing the index. It is still\n>>> effectively a no-op, but it serves the needs of the test.\n>>\n>> Ah, cache-tree is updated, then it's OK.\n>>\n>> As to test-tool vs end-user-accessible-command, I do not have a\n>> strong opinion, but use your imagination and ask Derrick or somebody\n>> else for their imagination to see if such a \"force expand\" feature\n>> may be something the end-users might need an access to in order to\n>> dig themselves out of a hole (in which case, it may be better to\n>> make it end-user-accessible) or not (in which case, test-tool is\n>> more appropriate).\n> \n> I think there is something to be said about the name being confusing,\n> because the current implementation focuses on \"expand a sparse index\n> upon read\" but it also allows the index to be written as sparse.\n> \n\nThis helps clarify what I was misinterpreting in the test. It isn't looking\nfor \"default\" behavior, it's verifying whether trace2 logs capture index\nexpansion and collapse when those operations are expected to happen,\nregardless of whether that's because `command_requires_full_index` is 1 or\nbecause the command needs to use entries inside of sparse directories. With\nthat interpretation, I can replace the command with `git reset\nupdate-folder1 -- folder1/a` and get the same result (without needing to\nchange the test in the future *or* add a new `git` command option /\n`test-tool` subcommand).\n\n> Conversely, if the user runs\n> \n> git -c index.sparse=false update-index ...\n> \n> then the index.sparse config setting forbids conversion from full to\n> sparse, but does not say anything about expanding to full.\n> \n> Perhaps this should be corrected: the index.sparse=false setting\n> should expand a sparse index to a full one, then prevent it from\n> being converted to a sparse one on write.\n> \n> This diff should do it:\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index 564283c7e7e..04df1051e18 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -2376,7 +2376,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> \tif (!istate->repo)\n> \t\tistate->repo = the_repository;\n> \tprepare_repo_settings(istate->repo);\n> -\tif (istate->repo->settings.command_requires_full_index)\n> +\tif (!istate->repo->settings.sparse_index ||\n> +\t istate->repo->settings.command_requires_full_index)\n> \t\tensure_full_index(istate);\n> \n> \treturn istate->cache_nr;\n> \n> Victoria, what are your thoughts about including such a change?\n> \n\nI think this is a worthwhile change, but I'd prefer submitting it separately\n(either in an upcoming sparse index integration or on its own). It's not\ndirectly needed by anything in this series, and I'd like to avoid adding\nfeatures to the scope if possible.\n"},{"id":"438467","messageId":"xmqq5yu3eg8w.fsf@gitster.g","threadId":"56614","inReplyTo":"cc60c6e7-ecef-ae22-8ec7-ab290ff2b830@gmail.com","subject":"Re: [PATCH v3 3/8] update-index: add --force-full-index option for expand/collapse test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-11T15:24:47Z","receivedAt":"2021-10-11T15:24:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Junio, would it be better to change the config setting, and then\n> update this test to use the config setting over a command-line flag?\n> This would allow us to punt on the --force-full-index flag until we\n> have time to focus on the 'git update-index' command and interactions\n> like this.\n\nI do not have a strong opinion on where we add the feature; as long\nas we have a way to let us avoid having to unnecessarily change this\nparticular test, that's perfectly fine, and if we can reuse it as a\nway for end-users to help those who are debugging their issues, that\nwould be an added bonus.\n\nThanks.\n\n"},{"id":"438471","messageId":"4a002488-09ba-efed-ac70-c0ec16d7dbe2@github.com","threadId":"56614","inReplyTo":"xmqqlf30edvf.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-11T15:55:00Z","receivedAt":"2021-10-11T15:55:07Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Junio C Hamano wrote:\n> For most callers of prime_cache_tree(), like the ones in \"git\n> read-tree\" and \"git reset\", it is immediately obvious that we just\n> read from the same tree, and we should have everything from the tree\n> and nothing else in the resulting index, so it is clear that the\n> prime_cache_tree() call is recreating the same cache-tree\n> information that we already should have computed ourselves, and\n> these calls can go (or if \"prime\" is still cheaper than \"update\",\n> these callers can pass an option to tell unpack_trees() to skip the\n> cache_tree_update() call, because they will call \"prime\" immediately\n> after).\n> \n\nAfter some basic performance testing of `git reset [--hard]`, it's not clear\nwhether `cache_tree_update` is definitively faster or slower than\n`prime_cache_tree`; more conclusive results would indicate which of the two\ncould be skipped. I'd like to defer this to a future patch (tracking it with\nan internal issue so I don't forget) where I can perform a more thorough\nanalysis across all of the commands currently using `prime_cache_tree` and\nupdate its usage accordingly.\n"},{"id":"438475","messageId":"xmqqh7dnczb3.fsf@gitster.g","threadId":"56614","inReplyTo":"4a002488-09ba-efed-ac70-c0ec16d7dbe2@github.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-11T16:16:00Z","receivedAt":"2021-10-11T16:16:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> After some basic performance testing of `git reset [--hard]`, it's not clear\n> whether `cache_tree_update` is definitively faster or slower than\n> `prime_cache_tree`; more conclusive results would indicate which of the two\n> could be skipped. I'd like to defer this to a future patch (tracking it with\n> an internal issue so I don't forget) where I can perform a more thorough\n> analysis across all of the commands currently using `prime_cache_tree` and\n> update its usage accordingly.\n\nYup. That sounds sensible. Concentrating on correctness first is a\ngood direction to go.\n"},{"id":"438492","messageId":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v3.git.1633641339.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:14Z","receivedAt":"2021-10-11T20:30:27Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate a ~70% execution time reduction in git reset\nusing a sparse index, and no change (within expected variability [2]) using\na full index. Results summarized below [3, 4]:\n\nTest base [5/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.97(0.50+0.37) -3.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.96(0.50+0.36) -4.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 1.37(0.91+0.35) -18.5%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 1.41(0.94+0.35) -17.1%\n\nTest base [6/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.94(0.48+0.34) -6.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.95(0.51+0.34) -5.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 0.46(0.05+0.29) -72.6%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 0.46(0.06+0.29) -72.9%\n\nTest base [7/8]\n---------------------------------------------------------------------------\ngit reset (full-v3) 0.77(0.27+0.37) 0.72(0.26+0.32) -6.5%\ngit reset (full-v4) 0.75(0.27+0.34) 0.73(0.26+0.32) -2.7%\ngit reset (sparse-v3) 1.44(0.96+0.36) 0.43(0.04+0.96) -70.1%\ngit reset (sparse-v4) 1.46(0.97+0.36) 0.43(0.05+0.79) -70.5%\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.69(0.26+0.30) -4.2%\ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.71(0.27+0.32) -4.1% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.81(0.42+0.90) -44.1%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.79(0.42+0.76) -44.0%\n\nTest base [8/8] \n---------------------------------------------------------------------------\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.73(0.26+0.33) +1.4% \ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.74(0.27+0.32) +0.0% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.43(0.05+0.80) -70.3%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.44(0.05+0.76) -68.8%\n\n\n\nChanges since V1\n================\n\n * Add --force-full-index option to update-index. The option is used\n circumvent changing command_requires_full_index from its default value -\n right now this is effectively a no-op, but will change once update-index\n is integrated with sparse index. By using this option in the t1092\n expand/collapse test, the command used to test will not need to be\n updated with subsequent sparse index integrations.\n * Update implementation of mixed reset for entries outside sparse checkout\n definition. The condition in which a file should be checked out before\n index reset is simplified to \"if it has skip-worktree enabled and a reset\n would change the file, check it out\".\n * After checking the behavior of update_index_from_diff with renames,\n found that the diff used by reset does not produce diff queue entries\n with different pathnames for one and two. Because of this, and that\n nothing in the implementation seems to rely on identical path names, no\n BUG check is added.\n * Correct a bug in the sparse index is not expanded tests in t1092 where\n failure of a git reset --mixed test was not being reported. Test now\n verifies an appropriate scenario with corrected failure-checking.\n\n\nChanges since V2\n================\n\n * Replace patch adding checkouts for git reset --mixed with sparse checkout\n with preserving the skip-worktree flag (including a new test for git\n reset --mixed and update to t1092 - checkout and reset (mixed))\n * Move rename of is_missing into its own patch\n * Further extend t1092 tests and remove unnecessary commands/tests where\n possible\n * Refine logic determining which pathspecs require ensure_full_index in git\n reset --mixed, add related ensure_not_expanded tests\n * Add index_search_mode enum to index_name_stage_pos\n * Clean up variable usage & remove unnecessary subtree_path in\n prime_cache_tree_rec\n * Update cover letter performance data\n * More thoroughly explain changes in each commit message\n\n\nChanges since V3\n================\n\n * Replace git update-index --force-full-index with git reset update-folder1\n -- folder1/a, remove introduction of new --force-full-index option\n entirely, and add comment clarifying the intent of sparse-index is\n expanded and converted back test\n * Fix authorship on reset: preserve skip-worktree bit in mixed reset\n (current patch fully replaces original patch, but metadata of the\n original wasn't properly replaced)\n\nThanks! -Victoria\n\n[1] microsoft@6b8a074 [2]\nhttps://lore.kernel.org/git/8b9fe3f8-f0e3-4567-b20b-17c92bd1a5c5@github.com/\n[3] If a test and/or commit is not mentioned, there is no significant change\nto performance [4] Pathspec \"does-not-exist\" is changed to \"missing\" to save\nspace in performance report\n\nVictoria Dye (8):\n reset: rename is_missing to !is_in_reset_tree\n reset: preserve skip-worktree bit in mixed reset\n sparse-index: update command for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n builtin/reset.c | 104 ++++++++++++++++-\n cache-tree.c | 46 +++++++-\n cache.h | 10 ++\n read-cache.c | 27 +++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 137 ++++++++++++++++++++---\n t/t7102-reset.sh | 17 +++\n unpack-trees.c | 23 +++-\n 8 files changed, 330 insertions(+), 37 deletions(-)\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n\nRange-diff vs v3:\n\n 1: ad7013a31aa = 1: ad7013a31aa reset: rename is_missing to !is_in_reset_tree\n 2: 1f6da84830b ! 2: bd72bd175da reset: preserve skip-worktree bit in mixed reset\n @@\n ## Metadata ##\n -Author: Kevin Willford \n +Author: Victoria Dye \n \n ## Commit message ##\n reset: preserve skip-worktree bit in mixed reset\n 3: 014a408ea5d < -: ----------- update-index: add --force-full-index option for expand/collapse test\n -: ----------- > 3: c4df0d6b136 sparse-index: update command for expand/collapse test\n 4: 7f21cf53e9d = 4: cfbb23e9fe2 reset: expand test coverage for sparse checkouts\n 5: a2d6212e287 = 5: 62fdbf2ad26 reset: integrate with sparse index\n 6: 330e0c09774 = 6: b0d437207e7 reset: make sparse-aware (except --mixed)\n 7: 6ef8e4e31d3 = 7: 00d14fb60bd reset: make --mixed sparse-aware\n 8: c7145e039f3 = 8: e523dadb8bf unpack-trees: improve performance of next_cache_entry\n\n-- \ngitgitgadget\n"},{"id":"438493","messageId":"ad7013a31aa5688747c1606d8c38aa96365c3711.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] reset: rename is_missing to !is_in_reset_tree","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:15Z","receivedAt":"2021-10-11T20:30:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRename and invert value of `is_missing` to `is_in_reset_tree` to make the\nvariable more descriptive of what it represents.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..d3695ce43c4 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -131,10 +131,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n \n-\t\tif (is_missing && !intent_to_add) {\n+\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n \t\t\tcontinue;\n \t\t}\n@@ -144,7 +144,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n-\t\tif (is_missing) {\n+\t\tif (!is_in_reset_tree) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n \t\t\tset_object_name_for_intent_to_add_entry(ce);\n \t\t}\n-- \ngitgitgadget\n\n"},{"id":"438494","messageId":"bd72bd175da52a167c27bfba1310e050d9e33a50.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:16Z","receivedAt":"2021-10-11T20:30:30Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nChange `update_index_from_diff` to set `skip-worktree` when applicable for\nnew index entries. When `git reset --mixed ` is run, entries in\nthe index with differences between the pre-reset HEAD and reset \nare identified and handled with `update_index_from_diff`. For each file, a\nnew cache entry in inserted into the index, created from the side\nof the reset (without changing the working tree). However, the newly-created\nentry must have `skip-worktree` explicitly set in either of the following\nscenarios:\n\n1. the file is in the current index and has `skip-worktree` set\n2. the file is not in the current index but is outside of a defined sparse\n checkout definition\n\nNot setting the `skip-worktree` bit leads to likely-undesirable results for\na user. It causes `skip-worktree` settings to disappear on the\n\"diff\"-containing files (but *only* the diff-containing files), leading to\nthose files now showing modifications in `git status`. For example, when\nrunning `git reset --mixed` in a sparse checkout, some file entries outside\nof sparse checkout could show up as deleted, despite the user never deleting\nanything (and not wanting them on-disk anyway).\n\nAdditionally, add a test to `t7102` to ensure `skip-worktree` is preserved\nin a basic `git reset --mixed` scenario and update a failure-documenting\ntest from 19a0acc (t1092: test interesting sparse-checkout scenarios,\n2021-01-23) with new expected behavior.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 14 ++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 19 +++++--------------\n t/t7102-reset.sh | 17 +++++++++++++++++\n 3 files changed, 36 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex d3695ce43c4..e441b6601b9 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,7 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -130,6 +131,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n+\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n \t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n@@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n+\n+\t\t/*\n+\t\t * If the file 1) corresponds to an existing index entry with\n+\t\t * skip-worktree set, or 2) does not exist in the index but is\n+\t\t * outside the sparse checkout definition, add a skip-worktree bit\n+\t\t * to the new index entry.\n+\t\t */\n+\t\tpos = cache_name_pos(one->path, strlen(one->path));\n+\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n+\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n+\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..889079f55b8 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,26 +459,17 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\n \ttest_all_match git reset deepest &&\n-\ttest_all_match git reset update-folder1 &&\n-\ttest_all_match git reset update-folder2\n-'\n-\n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_success 'checkout and reset (mixed) [sparse]' '\n-\tinit_repos &&\n \n-\ttest_sparse_match git checkout -b reset-test update-deep &&\n-\ttest_sparse_match git reset deepest &&\n+\t# Because skip-worktree is preserved, resetting to update-folder1\n+\t# will show worktree changes for full-checkout that are not present\n+\t# in sparse-checkout or sparse-index.\n \ttest_sparse_match git reset update-folder1 &&\n-\ttest_sparse_match git reset update-folder2\n+\trun_on_sparse test_path_is_missing folder1\n '\n \n test_expect_success 'merge, cherry-pick, and rebase' '\ndiff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\nindex 601b2bf97f0..d05426062ec 100755\n--- a/t/t7102-reset.sh\n+++ b/t/t7102-reset.sh\n@@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n \ttest_cmp expect output\n '\n \n+test_expect_success '--mixed preserves skip-worktree' '\n+\techo 123 >>file2 &&\n+\tgit add file2 &&\n+\tgit update-index --skip-worktree file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_must_be_empty output &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tUnstaged changes after reset:\n+\tM\tfile2\n+\tEOF\n+\tgit update-index --no-skip-worktree file2 &&\n+\tgit add file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_cmp expect output\n+'\n+\n test_expect_success 'resetting specific path that is unmerged' '\n \tgit rm --cached file2 &&\n \tF1=$(git rev-parse HEAD:file1) &&\n-- \ngitgitgadget\n\n"},{"id":"438495","messageId":"c4df0d6b1368a534491e21f2aaf1f5dc3e448821.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:17Z","receivedAt":"2021-10-11T20:30:30Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn anticipation of `git reset --hard` being able to use the sparse index\nwithout expanding it, replace the command in `sparse-index is expanded and\nconverted back` with `git reset -- folder1/a`. This command will need to\nexpand the index to work properly, even after integrating the rest of\n`reset` with sparse index.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 889079f55b8..e1422797013 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -631,11 +631,15 @@ test_expect_success 'submodule handling' '\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n '\n \n+# When working with a sparse index, some commands will need to expand the\n+# index to operate properly. If those commands also write the index back\n+# to disk, they need to convert the index to sparse before writing.\n+# This test verifies that both of these events are logged in trace2 logs.\n test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"438496","messageId":"cfbb23e9fe2d25140ae0db65479dcd3798428e1c.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 4/8] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:18Z","receivedAt":"2021-10-11T20:30:32Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs. New performance test cases exercise various execution paths for\n`reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 84 ++++++++++++++++++++++++\n 2 files changed, 87 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e1422797013..535686a2954 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -472,6 +472,90 @@ test_expect_success 'checkout and reset (mixed)' '\n \trun_on_sparse test_path_is_missing folder1\n '\n \n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+# Although the working tree differs between full and sparse checkouts after\n+# reset, the state of the index is the same.\n+test_expect_success 'reset with pathspecs outside sparse definition' '\n+\tinit_repos &&\n+\ttest_all_match git checkout -b reset-test base &&\n+\n+\ttest_sparse_match git reset update-folder1 -- folder1 &&\n+\tgit -C full-checkout reset update-folder1 -- folder1 &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1 &&\n+\n+\ttest_sparse_match git reset update-folder2 -- folder2/a &&\n+\tgit -C full-checkout reset update-folder2 -- folder2/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2/a\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset base -- \\*/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1/a &&\n+\n+\ttest_all_match git reset base -- folder\\* &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"438497","messageId":"62fdbf2ad261e5820d42590100d38972c6d5e490.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:19Z","receivedAt":"2021-10-11T20:30:34Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nDisable `command_requires_full_index` repo setting and add\n`ensure_full_index` guards around code paths that cannot yet use sparse\ndirectory index entries. `reset --soft` does not modify the index, so no\ncompatibility changes are needed for it to function without expanding the\nindex. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`),\nthe full index is expanded to prevent cache tree corruption and invalid\nvariable accesses.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` must be run\n*after* enabling sparse index for the command (so that the index is not\nexpanded unnecessarily) and *before* using the index for reset (so that it\nis verified as uncorrupted).\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex e441b6601b9..0ac0de7dc97 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -180,6 +180,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -257,9 +258,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -405,6 +403,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"438498","messageId":"00d14fb60bd952ac541627e0a1c56370036e3ca0.1633984223.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:21Z","receivedAt":"2021-10-11T20:30:36Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove the `ensure_full_index` guard on `read_from_tree` and update `git\nreset --mixed` to ensure it can use sparse directory index entries wherever\npossible. Sparse directory entries are reset use `diff_tree_oid`, which\nrequires `change` and `add_remove` functions to process the internal\ncontents of the sparse directory. The `recursive` diff option handles cases\nin which `reset --mixed` must diff/merge files that are nested multiple\nlevels deep in a sparse directory.\n\nThe use of pathspecs with `git reset --mixed` introduces scenarios in which\ninternal contents of sparse directories may be matched by the pathspec. In\norder to reset *all* files in the repo that may match the pathspec, the\nfollowing conditions on the pathspec require index expansion before\nperforming the reset:\n\n* \"magic\" pathspecs\n* wildcard pathspecs that do not match only in-cone files or entire sparse\n directories\n* literal pathspecs matching something outside the sparse checkout\n definition\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 78 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n 2 files changed, 93 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 0ac0de7dc97..60517e7e1d6 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t * If the file 1) corresponds to an existing index entry with\n \t\t * skip-worktree set, or 2) does not exist in the index but is\n \t\t * outside the sparse checkout definition, add a skip-worktree bit\n-\t\t * to the new index entry.\n+\t\t * to the new index entry. Note that a sparse index will be expanded\n+\t\t * if this entry is outside the sparse cone - this is necessary\n+\t\t * to properly construct the reset sparse directory.\n \t\t */\n \t\tpos = cache_name_pos(one->path, strlen(one->path));\n \t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n@@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t}\n }\n \n+static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n+{\n+\tunsigned int i, pos;\n+\tint res = 0;\n+\tchar *skip_worktree_seen = NULL;\n+\n+\t/*\n+\t * When using a magic pathspec, assume for the sake of simplicity that\n+\t * the index needs to be expanded to match all matchable files.\n+\t */\n+\tif (pathspec->magic)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\tstruct pathspec_item item = pathspec->items[i];\n+\n+\t\t/*\n+\t\t * If the pathspec item has a wildcard, the index should be expanded\n+\t\t * if the pathspec has the possibility of matching a subset of entries inside\n+\t\t * of a sparse directory (but not the entire directory).\n+\t\t *\n+\t\t * If the pathspec item is a literal path, the index only needs to be expanded\n+\t\t * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n+\t\t * expand for in-cone files) and b) it doesn't match any sparse directories\n+\t\t * (since we can reset whole sparse directories without expanding them).\n+\t\t */\n+\t\tif (item.nowildcard_len < item.len) {\n+\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n+\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n+\n+\t\t\t\tif (!S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is longer than the sparse\n+\t\t\t\t * directory name and the sparse directory is the first\n+\t\t\t\t * component of the pathspec, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (item.nowildcard_len > ce_namelen(ce) &&\n+\t\t\t\t !strncmp(item.original, ce->name, ce_namelen(ce))) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is shorter than the sparse\n+\t\t\t\t * directory and the pathspec does not match the whole\n+\t\t\t\t * directory, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n+\t\t\t\t wildmatch(item.original, ce->name, 0)) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n+\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n+\t\t\tres = 1;\n+\n+\t\tif (res > 0)\n+\t\t\tbreak;\n+\t}\n+\n+\tfree(skip_worktree_seen);\n+\treturn res;\n+}\n+\n static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t struct object_id *tree_oid,\n \t\t\t int intent_to_add)\n@@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\tif (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n+\t\tensure_full_index(&the_index);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 78476de18ea..f19c1b3e2eb 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -778,11 +778,28 @@ test_expect_success 'sparse-index is not expanded' '\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n+\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n \tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so the pathspec will not force the\n+\t# index to be expanded.\n+\tensure_not_expanded reset deepest -- folder1 &&\n+\tensure_not_expanded reset deepest -- folder1/ &&\n+\n+\t# Wildcard identifies only in-cone files, no index expansion\n+\tensure_not_expanded reset deepest -- deep/\\* &&\n+\n+\t# Wildcard identifies only full sparse directories, no index expansion\n+\tensure_not_expanded reset deepest -- folder\\* &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"438499","messageId":"b0d437207e774d297848ba90e1726cae6614b255.1633984222.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:20Z","receivedAt":"2021-10-11T20:30:36Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove `ensure_full_index` guard on `prime_cache_tree` and update\n`prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\nthe cache tree. While processing a tree's entries, `prime_cache_tree_rec`\nmust determine whether a directory entry is sparse or not by searching for\nit in the index (*without* expanding the index). If a matching sparse\ndirectory index entry is found, no subtrees are added to the cache tree\nentry and the entry count is set to 1 (representing the sparse directory\nitself). Otherwise, the tree is assumed to not be sparse and its subtrees\nare recursively added to the cache tree.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 47 ++++++++++++++++++++++--\n cache.h | 10 +++++\n read-cache.c | 27 ++++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 +++++++-\n 4 files changed, 86 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..2866101052c 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,26 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n+\tint base_path_len = tree_path->len;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,14 +768,40 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\n+\t\t\t/*\n+\t\t\t * Recursively-constructed subtree path is only needed when working\n+\t\t\t * in a sparse index (where it's used to determine whether the\n+\t\t\t * subtree is a sparse directory in the index).\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index) {\n+\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n+\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n+\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n+\t\t\t\tstrbuf_addch(tree_path, '/');\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n+\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n+\n \tit->entry_count = cnt;\n }\n \n@@ -772,12 +809,14 @@ void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..c079ece981a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -68,6 +68,11 @@\n */\n #define CACHE_ENTRY_PATH_LENGTH 80\n \n+enum index_search_mode {\n+\tNO_EXPAND_SPARSE = 0,\n+\tEXPAND_SPARSE = 1\n+};\n+\n static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct cache_entry *ce;\n@@ -551,7 +556,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tenum index_search_mode search_mode)\n {\n \tint first, last;\n \n@@ -570,7 +578,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +594,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n \t\t}\n \t}\n \n@@ -595,7 +603,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1235,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1335,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1370,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 535686a2954..78476de18ea 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -760,9 +760,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -772,6 +772,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"438500","messageId":"e523dadb8bfced1874fe40f36c0873a52f0f0594.1633984223.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-11T20:30:22Z","receivedAt":"2021-10-11T20:30:37Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the\nbenefit `cache_bottom` provides in non-sparse index cases, a separate `hint`\nposition indicates the first position `next_cache_entry` should search,\nupdated each execution with a new position.\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"438529","messageId":"cd62bf13-fdf1-af00-cc25-ef20abcfe152@gmail.com","threadId":"56614","inReplyTo":"xmqqlf30edvf.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-12T10:16:09Z","receivedAt":"2021-10-12T10:16:15Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 10/10/2021 23:03, Junio C Hamano wrote:\n> Phillip Wood writes:\n> \n>> On 08/10/2021 19:31, Junio C Hamano wrote:\n>>> Victoria Dye writes:\n>>>\n>>>> Phillip Wood wrote:\n>>>\n>>>>> I was looking at the callers to prime_cache_tree() this morning\n>>>>> and would like to suggest an alternative approach - just delete\n>>>>> prime_cache_tree() and all of its callers!\n>>> Do you mean the calls added by new patches without understanding\n>>> what they are doing, or all calls to it?\n>>\n>> I mean all calls to prime_cache_tree() after having understood (or at\n>> least thinking that I understand) what they are doing.\n> \n> Sorry, my statement was confusingly written. I meant \"calls added\n> by new patches, written by those who do not understand what\n> prime_cache_tree() calls are doing\", but after re-reading it, I\n> think it could be taken to be referring to \"you may be commenting\n> without understanding what prime_cache_tree() calls are doing\",\n> which wasn't my intention.\n\nThanks for clarifying that, I had misunderstood what you had written.\n\n>> (a) a successful call to unpack_trees() updates the cache tree\n>>\n>> (b) all the existing calls to prime_cache_tree() follow a successful\n>> call to unpack_trees() and nothing touches in index in between the\n>> call to unpack_trees() and prime_cache_tree().\n> \n> Ahh, OK.\n> \n> I think we originally avoided calling cache_tree_update() lightly\n> (because it is essentially a \"write-tree\", a fairly heavy-weight\n> operation, without I/O) and instead relied on prime_cache_tree() to\n> get degraded cache-tree back into freshness.\n> \n> What I forgot was that 52fca218 (unpack-trees: populate cache-tree\n> on successful merge, 2015-07-28) added cache_tree_update() there at\n> the end of unpack_trees(). The commit covers quite a wide range of\n> operations---the log message says \"merge\", but in fact anything that\n> uses unpack_trees() including branch switching and the resetting of\n> the index are affected, and they cause a full reconstruction of the\n> cache tree by calling cache_tree_update().\n> \n> For most callers of prime_cache_tree(), like the ones in \"git\n> read-tree\" and \"git reset\", it is immediately obvious that we just\n> read from the same tree, and we should have everything from the tree\n> and nothing else in the resulting index, so it is clear that the\n> prime_cache_tree() call is recreating the same cache-tree\n> information that we already should have computed ourselves, and\n> these calls can go (or if \"prime\" is still cheaper than \"update\",\n> these callers can pass an option to tell unpack_trees() to skip the\n> cache_tree_update() call, because they will call \"prime\" immediately\n> after).\n\nI haven't really thought this through but could we teach unpack_trees() \nto call prime_cache_tree() rather than cache_tree_update() when that \nwould be safe? For callers that use oneway_merge() merge it should \nalways be safe I think and it might be possible to modify twoway_merge() \nto signal if the final tree in the index matches the second one passed \nto it. We could have a more general mechanism for the callback to signal \nif it is safe to prime the tree but I suspect the callers that are using \ncustom callbacks are not updating the whole tree.\n\nBest Wishes\n\nPhillip\n\n> For other callers it is not immediately obvious, but I trust you are\n> correctly reading the code ;-)\n> \n> Thanks.\n> \n> \n> \n\n"},{"id":"438530","messageId":"600e5da3-6b30-4ce6-d8d2-984199d2aa35@gmail.com","threadId":"56614","inReplyTo":"52768318-ef8a-b26f-d4bc-d5c91779ccdb@github.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2021-10-12T10:17:30Z","receivedAt":"2021-10-12T10:17:36Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 08/10/2021 18:14, Victoria Dye wrote:\n> Phillip Wood wrote:\n>> Hi Victoria\n>>\n>> On 07/10/2021 22:15, Victoria Dye via GitGitGadget wrote:\n>>> From: Victoria Dye \n>>>\n>>> Remove `ensure_full_index` guard on `prime_cache_tree` and update\n>>> `prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\n>>> the cache tree. While processing a tree's entries, `prime_cache_tree_rec`\n>>> must determine whether a directory entry is sparse or not by searching for\n>>> it in the index (*without* expanding the index). If a matching sparse\n>>> directory index entry is found, no subtrees are added to the cache tree\n>>> entry and the entry count is set to 1 (representing the sparse directory\n>>> itself). Otherwise, the tree is assumed to not be sparse and its subtrees\n>>> are recursively added to the cache tree.\n>>\n>> I was looking at the callers to prime_cache_tree() this morning and would like to suggest an alternative approach - just delete prime_cache_tree() and all of its callers! As far as I can see it is only ever called after a successful call to unpack_trees() and since 52fca2184d (\"unpack-trees: populate cache-tree on successful merge\", 2015-07-28) unpack_trees() updates the cache tree for the caller. All the call sites are pretty obvious apart from the one in t/help/test-fast-rebase.c where unpack_trees() is called by merge_switch_to_result()\n>>\n> \n> It looks like `prime_cache_tree` can be removed mostly without issue, but\n> it causes the two last tests in `t4058-diff-duplicates.sh` to fail. Those\n> tests document failure cases when dealing with duplicate tree entries [1],\n> and it looks like `prime_cache_tree` was creating the appearance of a\n> fully-reset index but was still leaving it in a state where subsequent\n> operations could fail.\n> \n> I'm inclined to say the solution here would be to update the tests to\n> document the \"new\" failure behavior and proceed with removing\n> `prime_cache_tree`, because:\n> \n> * the test using `git reset --hard` disables `GIT_TEST_CHECK_CACHE_TREE`,\n> indicating that `prime_cache_tree` already wasn't behaving correctly\n> * attempting to fix the overarching issues with duplicate tree entries will\n> substantially delay this patch series\n> * a duplicate entry fix is largely unrelated to the intended scope of the\n> series\n\nThat sounds like a good way forward\n\nBest Wishes\n\nPhillip\n\n> Another option would be to leave `prime_cache_tree` as it is, but with it\n> being apparently useless outside of mostly-broken use cases in `t4058`, it\n> seems like a waste to keep it around.\n> \n> [1] ac14de13b2 (t4058: explore duplicate tree entry handling in a bit more detail, 2020-12-11)\n> \n\n"},{"id":"438586","messageId":"xmqqv9222gwh.fsf@gitster.g","threadId":"56614","inReplyTo":"cd62bf13-fdf1-af00-cc25-ef20abcfe152@gmail.com","subject":"Re: [PATCH v3 6/8] reset: make sparse-aware (except --mixed)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-12T19:15:58Z","receivedAt":"2021-10-12T19:16:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood writes:\n\n> I haven't really thought this through but could we teach\n> unpack_trees() to call prime_cache_tree() rather than\n> cache_tree_update() when that would be safe? For callers that use\n> oneway_merge() merge it should always be safe I think and it might be\n> possible to modify twoway_merge() to signal if the final tree in the\n> index matches the second one passed to it. We could have a more\n> general mechanism for the callback to signal if it is safe to prime\n> the tree but I suspect the callers that are using custom callbacks are\n> not updating the whole tree.\n\nBefore going in any direction, other than doing nothing ;-), we'd\nneed to see how expensive \"prime\" and \"update\" are. \n\nHaving said that.\n\n * Your idea is quite beneficial for callers of unpack_trees() as\n they no longer have to decide whether they want to make a\n separate call to \"prime\".\n\n * Right now we do not seem to have a codepath that\n\n (1) populates the index entries from existing trees (not\n necessarily making the index in complete sync with the trees)\n without unpack_trees() and\n\n (2) does \"prime\" to fix the cache tree\n\n but such a codepath may want to do either \"prime\" or \"update\", or\n neither. When it knows that it damages cache-tree so badly, and\n that it is often expected that the user would make many other\n changes to the index before writing it out as a tree, it may\n choose not to do either.\n\nThanks.\n"},{"id":"439336","messageId":"YXI75uFLdk2mnP7g@google.com","threadId":"56614","inReplyTo":"bd72bd175da52a167c27bfba1310e050d9e33a50.1633984222.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-10-22T04:19:50Z","receivedAt":"2021-10-22T04:19:58Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Mon, Oct 11, 2021 at 08:30:16PM +0000, Victoria Dye via GitGitGadget wrote:\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index d3695ce43c4..e441b6601b9 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -25,6 +25,7 @@\n> #include \"cache-tree.h\"\n> #include \"submodule.h\"\n> #include \"submodule-config.h\"\n> +#include \"dir.h\"\n> \n> #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n> \n> @@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> \n> \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n> \t\t\t\t 0, 0);\n> +\n> +\t\t/*\n> +\t\t * If the file 1) corresponds to an existing index entry with\n> +\t\t * skip-worktree set, or 2) does not exist in the index but is\n> +\t\t * outside the sparse checkout definition, add a skip-worktree bit\n> +\t\t * to the new index entry.\n> +\t\t */\n> +\t\tpos = cache_name_pos(one->path, strlen(one->path));\n> +\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n> +\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n> +\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n\nTo put it another way and check my understanding (because I'm not\nfamiliar with the sparse-index yet): if the file exists in the index but\nwe didn't care about the worktree anyway, then skip it; if the file\ndoesn't exist in the index but it also isn't in the sparse-checkout\ncone, then also skip it, because we don't care about the file anyway.\n\nI was going to ask if we could check ce_skip_worktree() without checking\npos first, but I suppose a negative pos would make the array deref\npretty unhappy. Ok.\n\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 886e78715fe..889079f55b8 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -459,26 +459,17 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n> \ttest_all_match git blame deep/deeper2/deepest/a\n> '\n> \n> -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> -# in this scenario, but it shouldn't.\n> -test_expect_failure 'checkout and reset (mixed)' '\n> +test_expect_success 'checkout and reset (mixed)' '\n\nOoh ooh, we can start using these tests :) Always exciting.\n\n> \tinit_repos &&\n> \n> \ttest_all_match git checkout -b reset-test update-deep &&\n> \ttest_all_match git reset deepest &&\n> -\ttest_all_match git reset update-folder1 &&\n> -\ttest_all_match git reset update-folder2\n> -'\n> -\n> -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> -# in this scenario, but it shouldn't.\n> -test_expect_success 'checkout and reset (mixed) [sparse]' '\n> -\tinit_repos &&\n> \n> -\ttest_sparse_match git checkout -b reset-test update-deep &&\n> -\ttest_sparse_match git reset deepest &&\n> +\t# Because skip-worktree is preserved, resetting to update-folder1\n> +\t# will show worktree changes for full-checkout that are not present\n> +\t# in sparse-checkout or sparse-index.\n\nThis doesn't really have anything to do with your patch. But I'm having\na very hard time understanding what each branch you're switching between\nand basing on is for; this entire test suite is a little miserly with\ncomments. I *think* your comment is saying that you're not bothering to\ncheck test_all_match because you know that the full-checkout tree won't\nmatch? But I also don't see that being asserted; test_sparse_match looks\nto compare sparse-checkout and sparse-index trees but doesn't say\nanything at all about the full-checkout tree, right?\n\n> \ttest_sparse_match git reset update-folder1 &&\n> -\ttest_sparse_match git reset update-folder2\n> +\trun_on_sparse test_path_is_missing folder1\n> '\n> \n> test_expect_success 'merge, cherry-pick, and rebase' '\n> diff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\n> index 601b2bf97f0..d05426062ec 100755\n> --- a/t/t7102-reset.sh\n> +++ b/t/t7102-reset.sh\n> @@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success '--mixed preserves skip-worktree' '\n> +\techo 123 >>file2 &&\n\nfile2 is just in the worktree...\n\n> +\tgit add file2 &&\n\n...and now it's in the index...\n\n> +\tgit update-index --skip-worktree file2 &&\n\n...and now we're asking Git to ignore worktree changes to file2...\n\n> +\tgit reset --mixed HEAD >output &&\n\nBut now I'm a little confused, maybe because of 'git reset' syntax. I'd\nexpect this to say \"ah yes, the index is different from HEAD, it's got\nthis file2 thingie\" and still reset the index; I'm surprised that\n--skip-worktree, which sounds like it's saying only \"don't consider\nwhat's going on in the worktree\". So I would expect this to still delete\nfile2 from the index. But instead I guess it is keeping file2 in the\nindex with the \"who cares what happened in the wt\" marker?\n\n> +\ttest_must_be_empty output &&\n> +\n> +\tcat >expect <<-\\EOF &&\n> +\tUnstaged changes after reset:\n> +\tM\tfile2\n> +\tEOF\n> +\tgit update-index --no-skip-worktree file2 &&\n> +\tgit add file2 &&\n> +\tgit reset --mixed HEAD >output &&\n> +\ttest_cmp expect output\n> +'\n> +\n> test_expect_success 'resetting specific path that is unmerged' '\n> \tgit rm --cached file2 &&\n> \tF1=$(git rev-parse HEAD:file1) &&\n> -- \n> gitgitgadget\n> \n"},{"id":"439551","messageId":"16fbc2dc-6fdd-ed0d-ebc6-3b0566142879@github.com","threadId":"56614","inReplyTo":"YXI75uFLdk2mnP7g@google.com","subject":"Re: [PATCH v4 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-10-25T15:59:19Z","receivedAt":"2021-10-25T15:59:23Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Emily Shaffer wrote:\n> On Mon, Oct 11, 2021 at 08:30:16PM +0000, Victoria Dye via GitGitGadget wrote:\n>> diff --git a/builtin/reset.c b/builtin/reset.c\n>> index d3695ce43c4..e441b6601b9 100644\n>> --- a/builtin/reset.c\n>> +++ b/builtin/reset.c\n>> @@ -25,6 +25,7 @@\n>> #include \"cache-tree.h\"\n>> #include \"submodule.h\"\n>> #include \"submodule-config.h\"\n>> +#include \"dir.h\"\n>> \n>> #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n>> \n>> @@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> \n>> \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n>> \t\t\t\t 0, 0);\n>> +\n>> +\t\t/*\n>> +\t\t * If the file 1) corresponds to an existing index entry with\n>> +\t\t * skip-worktree set, or 2) does not exist in the index but is\n>> +\t\t * outside the sparse checkout definition, add a skip-worktree bit\n>> +\t\t * to the new index entry.\n>> +\t\t */\n>> +\t\tpos = cache_name_pos(one->path, strlen(one->path));\n>> +\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n>> +\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n>> +\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n> \n> To put it another way and check my understanding (because I'm not\n> familiar with the sparse-index yet): if the file exists in the index but\n> we didn't care about the worktree anyway, then skip it; if the file\n> doesn't exist in the index but it also isn't in the sparse-checkout\n> cone, then also skip it, because we don't care about the file anyway.\n> \n> I was going to ask if we could check ce_skip_worktree() without checking\n> pos first, but I suppose a negative pos would make the array deref\n> pretty unhappy. Ok.\n> \n\nExactly! Generally the current skip-worktree flag is the \"source of truth\"\non whether to add the flag to the new entry, but if the file isn't in the\nindex pre-reset, the sparse checkout patterns are used to determine if it\nshould have skip-worktree applied.\n\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 886e78715fe..889079f55b8 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -459,26 +459,17 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n>> \ttest_all_match git blame deep/deeper2/deepest/a\n>> '\n>> \n>> -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n>> -# in this scenario, but it shouldn't.\n>> -test_expect_failure 'checkout and reset (mixed)' '\n>> +test_expect_success 'checkout and reset (mixed)' '\n> \n> Ooh ooh, we can start using these tests :) Always exciting.\n> \n>> \tinit_repos &&\n>> \n>> \ttest_all_match git checkout -b reset-test update-deep &&\n>> \ttest_all_match git reset deepest &&\n>> -\ttest_all_match git reset update-folder1 &&\n>> -\ttest_all_match git reset update-folder2\n>> -'\n>> -\n>> -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n>> -# in this scenario, but it shouldn't.\n>> -test_expect_success 'checkout and reset (mixed) [sparse]' '\n>> -\tinit_repos &&\n>> \n>> -\ttest_sparse_match git checkout -b reset-test update-deep &&\n>> -\ttest_sparse_match git reset deepest &&\n>> +\t# Because skip-worktree is preserved, resetting to update-folder1\n>> +\t# will show worktree changes for full-checkout that are not present\n>> +\t# in sparse-checkout or sparse-index.\n> \n> This doesn't really have anything to do with your patch. But I'm having\n> a very hard time understanding what each branch you're switching between\n> and basing on is for; this entire test suite is a little miserly with\n> comments. \n\nThe branches used in this test are:\n\n* `base` is the initial branch; all branches in the list here contain one\n commit on top of `base`\n* `update-deep` modifies a file inside the sparse checkout cone (`deep/a`) \n* `deepest` modifies a file deep inside the sparse checkout cone\n (`deep/deeper1/deepest/a`)\n* `update-folder1` and `update-folder2` each modify a file outside the\n sparse checkout cone (`folder1/a` and `folder2/a`, respectively)\n\nThere are other branches used throughout the file, but they deal with more\ncomplicated conflict scenarios not used in this particular test case. \n\n> I *think* your comment is saying that you're not bothering to\n> check test_all_match because you know that the full-checkout tree won't\n> match? But I also don't see that being asserted; test_sparse_match looks\n> to compare sparse-checkout and sparse-index trees but doesn't say\n> anything at all about the full-checkout tree, right?\n> \n\nYour understanding is correct (it's attempting to explain why we're not\nusing `test_all_match`, unlike earlier assertions in the test). That said,\nit probably _should_ assert on the differences from `full-checkout`, namely\nthat \"M\tfolder1/a\" would appear in `full-checkout` but not in\n`sparse-checkout`/`sparse-index`. I can add that in my next version.\n\n>> \ttest_sparse_match git reset update-folder1 &&\n>> -\ttest_sparse_match git reset update-folder2\n>> +\trun_on_sparse test_path_is_missing folder1\n>> '\n>> \n>> test_expect_success 'merge, cherry-pick, and rebase' '\n>> diff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\n>> index 601b2bf97f0..d05426062ec 100755\n>> --- a/t/t7102-reset.sh\n>> +++ b/t/t7102-reset.sh\n>> @@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n>> \ttest_cmp expect output\n>> '\n>> \n>> +test_expect_success '--mixed preserves skip-worktree' '\n>> +\techo 123 >>file2 &&\n> \n> file2 is just in the worktree...\n> \n>> +\tgit add file2 &&\n> \n> ...and now it's in the index...\n> \n>> +\tgit update-index --skip-worktree file2 &&\n> \n> ...and now we're asking Git to ignore worktree changes to file2...\n> \n>> +\tgit reset --mixed HEAD >output &&\n> \n> But now I'm a little confused, maybe because of 'git reset' syntax. I'd\n> expect this to say \"ah yes, the index is different from HEAD, it's got\n> this file2 thingie\" and still reset the index; I'm surprised that\n> --skip-worktree, which sounds like it's saying only \"don't consider\n> what's going on in the worktree\". So I would expect this to still delete\n> file2 from the index. But instead I guess it is keeping file2 in the\n> index with the \"who cares what happened in the wt\" marker?\n> \n\nYes - `git update-index --skip-worktree` sets the skip-worktree flag on the\nindex entry of the specified file(s) (in this case, `file2`). So, because\n`file2` is in the index but ignored in the worktree, the file isn't\nidentified as \"modified\" after `git reset --mixed HEAD`. Once the\nskip-worktree flag is removed (with `git update-index --no-skip-worktree`),\nthe reset results in it showing up as \"modified\".\n\n>> +\ttest_must_be_empty output &&\n>> +\n>> +\tcat >expect <<-\\EOF &&\n>> +\tUnstaged changes after reset:\n>> +\tM\tfile2\n>> +\tEOF\n>> +\tgit update-index --no-skip-worktree file2 &&\n>> +\tgit add file2 &&\n>> +\tgit reset --mixed HEAD >output &&\n>> +\ttest_cmp expect output\n>> +'\n>> +\n>> test_expect_success 'resetting specific path that is unmerged' '\n>> \tgit rm --cached file2 &&\n>> \tF1=$(git rev-parse HEAD:file1) &&\n>> -- \n>> gitgitgadget\n>>\n\n"},{"id":"439779","messageId":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v4.git.1633984222.gitgitgadget@gmail.com","subject":"[PATCH v5 0/8] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:15Z","receivedAt":"2021-10-27T14:39:27Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate a ~70% execution time reduction in git reset\nusing a sparse index, and no change (within expected variability [2]) using\na full index. Results summarized below [3, 4]:\n\nTest base [5/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.97(0.50+0.37) -3.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.96(0.50+0.36) -4.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 1.37(0.91+0.35) -18.5%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 1.41(0.94+0.35) -17.1%\n\nTest base [6/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.94(0.48+0.34) -6.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.95(0.51+0.34) -5.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 0.46(0.05+0.29) -72.6%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 0.46(0.06+0.29) -72.9%\n\nTest base [7/8]\n---------------------------------------------------------------------------\ngit reset (full-v3) 0.77(0.27+0.37) 0.72(0.26+0.32) -6.5%\ngit reset (full-v4) 0.75(0.27+0.34) 0.73(0.26+0.32) -2.7%\ngit reset (sparse-v3) 1.44(0.96+0.36) 0.43(0.04+0.96) -70.1%\ngit reset (sparse-v4) 1.46(0.97+0.36) 0.43(0.05+0.79) -70.5%\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.69(0.26+0.30) -4.2%\ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.71(0.27+0.32) -4.1% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.81(0.42+0.90) -44.1%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.79(0.42+0.76) -44.0%\n\nTest base [8/8] \n---------------------------------------------------------------------------\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.73(0.26+0.33) +1.4% \ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.74(0.27+0.32) +0.0% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.43(0.05+0.80) -70.3%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.44(0.05+0.76) -68.8%\n\n\n\nChanges since V1\n================\n\n * Add --force-full-index option to update-index. The option is used\n circumvent changing command_requires_full_index from its default value -\n right now this is effectively a no-op, but will change once update-index\n is integrated with sparse index. By using this option in the t1092\n expand/collapse test, the command used to test will not need to be\n updated with subsequent sparse index integrations.\n * Update implementation of mixed reset for entries outside sparse checkout\n definition. The condition in which a file should be checked out before\n index reset is simplified to \"if it has skip-worktree enabled and a reset\n would change the file, check it out\".\n * After checking the behavior of update_index_from_diff with renames,\n found that the diff used by reset does not produce diff queue entries\n with different pathnames for one and two. Because of this, and that\n nothing in the implementation seems to rely on identical path names, no\n BUG check is added.\n * Correct a bug in the sparse index is not expanded tests in t1092 where\n failure of a git reset --mixed test was not being reported. Test now\n verifies an appropriate scenario with corrected failure-checking.\n\n\nChanges since V2\n================\n\n * Replace patch adding checkouts for git reset --mixed with sparse checkout\n with preserving the skip-worktree flag (including a new test for git\n reset --mixed and update to t1092 - checkout and reset (mixed))\n * Move rename of is_missing into its own patch\n * Further extend t1092 tests and remove unnecessary commands/tests where\n possible\n * Refine logic determining which pathspecs require ensure_full_index in git\n reset --mixed, add related ensure_not_expanded tests\n * Add index_search_mode enum to index_name_stage_pos\n * Clean up variable usage & remove unnecessary subtree_path in\n prime_cache_tree_rec\n * Update cover letter performance data\n * More thoroughly explain changes in each commit message\n\n\nChanges since V3\n================\n\n * Replace git update-index --force-full-index with git reset update-folder1\n -- folder1/a, remove introduction of new --force-full-index option\n entirely, and add comment clarifying the intent of sparse-index is\n expanded and converted back test\n * Fix authorship on reset: preserve skip-worktree bit in mixed reset\n (current patch fully replaces original patch, but metadata of the\n original wasn't properly replaced)\n\n\nChanges since V4\n================\n\n * Update t1092 test 'checkout and reset (mixed)' to explicitly verify\n differences between sparse and full checkouts\n\nThanks! -Victoria\n\n[1] microsoft@6b8a074 [2]\nhttps://lore.kernel.org/git/8b9fe3f8-f0e3-4567-b20b-17c92bd1a5c5@github.com/\n[3] If a test and/or commit is not mentioned, there is no significant change\nto performance [4] Pathspec \"does-not-exist\" is changed to \"missing\" to save\nspace in performance report\n\nVictoria Dye (8):\n reset: rename is_missing to !is_in_reset_tree\n reset: preserve skip-worktree bit in mixed reset\n sparse-index: update command for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n builtin/reset.c | 104 ++++++++++++++++-\n cache-tree.c | 46 +++++++-\n cache.h | 10 ++\n read-cache.c | 27 +++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 140 ++++++++++++++++++++---\n t/t7102-reset.sh | 17 +++\n unpack-trees.c | 23 +++-\n 8 files changed, 333 insertions(+), 37 deletions(-)\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n\nRange-diff vs v4:\n\n 1: ad7013a31aa = 1: ad7013a31aa reset: rename is_missing to !is_in_reset_tree\n 2: bd72bd175da ! 2: b221b00b7e0 reset: preserve skip-worktree bit in mixed reset\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'blame with pathsp\n -\ttest_sparse_match git checkout -b reset-test update-deep &&\n -\ttest_sparse_match git reset deepest &&\n +\t# Because skip-worktree is preserved, resetting to update-folder1\n -+\t# will show worktree changes for full-checkout that are not present\n ++\t# will show worktree changes for folder1/a in full-checkout, but not\n +\t# in sparse-checkout or sparse-index.\n ++\tgit -C full-checkout reset update-folder1 >full-checkout-out &&\n \ttest_sparse_match git reset update-folder1 &&\n -\ttest_sparse_match git reset update-folder2\n ++\tgrep \"M\tfolder1/a\" full-checkout-out &&\n ++\t! grep \"M\tfolder1/a\" sparse-checkout-out &&\n +\trun_on_sparse test_path_is_missing folder1\n '\n \n 3: c4df0d6b136 = 3: 1bb2ca92c60 sparse-index: update command for expand/collapse test\n 4: cfbb23e9fe2 = 4: cc76c694647 reset: expand test coverage for sparse checkouts\n 5: 62fdbf2ad26 = 5: 217ae445418 reset: integrate with sparse index\n 6: b0d437207e7 = 6: a3e2fd59867 reset: make sparse-aware (except --mixed)\n 7: 00d14fb60bd = 7: a9135a5ed64 reset: make --mixed sparse-aware\n 8: e523dadb8bf = 8: f91d1dcf024 unpack-trees: improve performance of next_cache_entry\n\n-- \ngitgitgadget\n"},{"id":"439780","messageId":"ad7013a31aa5688747c1606d8c38aa96365c3711.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 1/8] reset: rename is_missing to !is_in_reset_tree","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:16Z","receivedAt":"2021-10-27T14:39:29Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRename and invert value of `is_missing` to `is_in_reset_tree` to make the\nvariable more descriptive of what it represents.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..d3695ce43c4 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -131,10 +131,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n \n-\t\tif (is_missing && !intent_to_add) {\n+\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n \t\t\tcontinue;\n \t\t}\n@@ -144,7 +144,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n-\t\tif (is_missing) {\n+\t\tif (!is_in_reset_tree) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n \t\t\tset_object_name_for_intent_to_add_entry(ce);\n \t\t}\n-- \ngitgitgadget\n\n"},{"id":"439781","messageId":"b221b00b7e06a3b135b9f68ce87cffaa7d782581.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:17Z","receivedAt":"2021-10-27T14:39:32Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nChange `update_index_from_diff` to set `skip-worktree` when applicable for\nnew index entries. When `git reset --mixed ` is run, entries in\nthe index with differences between the pre-reset HEAD and reset \nare identified and handled with `update_index_from_diff`. For each file, a\nnew cache entry in inserted into the index, created from the side\nof the reset (without changing the working tree). However, the newly-created\nentry must have `skip-worktree` explicitly set in either of the following\nscenarios:\n\n1. the file is in the current index and has `skip-worktree` set\n2. the file is not in the current index but is outside of a defined sparse\n checkout definition\n\nNot setting the `skip-worktree` bit leads to likely-undesirable results for\na user. It causes `skip-worktree` settings to disappear on the\n\"diff\"-containing files (but *only* the diff-containing files), leading to\nthose files now showing modifications in `git status`. For example, when\nrunning `git reset --mixed` in a sparse checkout, some file entries outside\nof sparse checkout could show up as deleted, despite the user never deleting\nanything (and not wanting them on-disk anyway).\n\nAdditionally, add a test to `t7102` to ensure `skip-worktree` is preserved\nin a basic `git reset --mixed` scenario and update a failure-documenting\ntest from 19a0acc (t1092: test interesting sparse-checkout scenarios,\n2021-01-23) with new expected behavior.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 14 ++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++--------------\n t/t7102-reset.sh | 17 +++++++++++++++++\n 3 files changed, 39 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex d3695ce43c4..e441b6601b9 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,7 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -130,6 +131,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n+\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n \t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n@@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n+\n+\t\t/*\n+\t\t * If the file 1) corresponds to an existing index entry with\n+\t\t * skip-worktree set, or 2) does not exist in the index but is\n+\t\t * outside the sparse checkout definition, add a skip-worktree bit\n+\t\t * to the new index entry.\n+\t\t */\n+\t\tpos = cache_name_pos(one->path, strlen(one->path));\n+\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n+\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n+\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..c7449afe965 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,26 +459,20 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\n \ttest_all_match git reset deepest &&\n-\ttest_all_match git reset update-folder1 &&\n-\ttest_all_match git reset update-folder2\n-'\n-\n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_success 'checkout and reset (mixed) [sparse]' '\n-\tinit_repos &&\n \n-\ttest_sparse_match git checkout -b reset-test update-deep &&\n-\ttest_sparse_match git reset deepest &&\n+\t# Because skip-worktree is preserved, resetting to update-folder1\n+\t# will show worktree changes for folder1/a in full-checkout, but not\n+\t# in sparse-checkout or sparse-index.\n+\tgit -C full-checkout reset update-folder1 >full-checkout-out &&\n \ttest_sparse_match git reset update-folder1 &&\n-\ttest_sparse_match git reset update-folder2\n+\tgrep \"M\tfolder1/a\" full-checkout-out &&\n+\t! grep \"M\tfolder1/a\" sparse-checkout-out &&\n+\trun_on_sparse test_path_is_missing folder1\n '\n \n test_expect_success 'merge, cherry-pick, and rebase' '\ndiff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\nindex 601b2bf97f0..d05426062ec 100755\n--- a/t/t7102-reset.sh\n+++ b/t/t7102-reset.sh\n@@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n \ttest_cmp expect output\n '\n \n+test_expect_success '--mixed preserves skip-worktree' '\n+\techo 123 >>file2 &&\n+\tgit add file2 &&\n+\tgit update-index --skip-worktree file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_must_be_empty output &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tUnstaged changes after reset:\n+\tM\tfile2\n+\tEOF\n+\tgit update-index --no-skip-worktree file2 &&\n+\tgit add file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_cmp expect output\n+'\n+\n test_expect_success 'resetting specific path that is unmerged' '\n \tgit rm --cached file2 &&\n \tF1=$(git rev-parse HEAD:file1) &&\n-- \ngitgitgadget\n\n"},{"id":"439782","messageId":"1bb2ca92c60be453d7b151e0a1fb5d471a2e5090.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 3/8] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:18Z","receivedAt":"2021-10-27T14:39:34Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn anticipation of `git reset --hard` being able to use the sparse index\nwithout expanding it, replace the command in `sparse-index is expanded and\nconverted back` with `git reset -- folder1/a`. This command will need to\nexpand the index to work properly, even after integrating the rest of\n`reset` with sparse index.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c7449afe965..cab6340a9d0 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -634,11 +634,15 @@ test_expect_success 'submodule handling' '\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n '\n \n+# When working with a sparse index, some commands will need to expand the\n+# index to operate properly. If those commands also write the index back\n+# to disk, they need to convert the index to sparse before writing.\n+# This test verifies that both of these events are logged in trace2 logs.\n test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"439783","messageId":"cc76c6946476d51a269da3ad91aaae014e2a48d8.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 4/8] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:19Z","receivedAt":"2021-10-27T14:39:35Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs. New performance test cases exercise various execution paths for\n`reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 84 ++++++++++++++++++++++++\n 2 files changed, 87 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex cab6340a9d0..a8583030b38 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -475,6 +475,90 @@ test_expect_success 'checkout and reset (mixed)' '\n \trun_on_sparse test_path_is_missing folder1\n '\n \n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+# Although the working tree differs between full and sparse checkouts after\n+# reset, the state of the index is the same.\n+test_expect_success 'reset with pathspecs outside sparse definition' '\n+\tinit_repos &&\n+\ttest_all_match git checkout -b reset-test base &&\n+\n+\ttest_sparse_match git reset update-folder1 -- folder1 &&\n+\tgit -C full-checkout reset update-folder1 -- folder1 &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1 &&\n+\n+\ttest_sparse_match git reset update-folder2 -- folder2/a &&\n+\tgit -C full-checkout reset update-folder2 -- folder2/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2/a\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\ttest_all_match git reset base -- \\*/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1/a &&\n+\n+\ttest_all_match git reset base -- folder\\* &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"439784","messageId":"217ae44541878b712b45db9525a6de5b854f27bb.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 5/8] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:20Z","receivedAt":"2021-10-27T14:39:38Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nDisable `command_requires_full_index` repo setting and add\n`ensure_full_index` guards around code paths that cannot yet use sparse\ndirectory index entries. `reset --soft` does not modify the index, so no\ncompatibility changes are needed for it to function without expanding the\nindex. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`),\nthe full index is expanded to prevent cache tree corruption and invalid\nvariable accesses.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` must be run\n*after* enabling sparse index for the command (so that the index is not\nexpanded unnecessarily) and *before* using the index for reset (so that it\nis verified as uncorrupted).\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex e441b6601b9..0ac0de7dc97 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -180,6 +180,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -257,9 +258,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -405,6 +403,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"439785","messageId":"a3e2fd59867115c2b2f275eab5edf961773fca6b.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:21Z","receivedAt":"2021-10-27T14:39:39Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove `ensure_full_index` guard on `prime_cache_tree` and update\n`prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\nthe cache tree. While processing a tree's entries, `prime_cache_tree_rec`\nmust determine whether a directory entry is sparse or not by searching for\nit in the index (*without* expanding the index). If a matching sparse\ndirectory index entry is found, no subtrees are added to the cache tree\nentry and the entry count is set to 1 (representing the sparse directory\nitself). Otherwise, the tree is assumed to not be sparse and its subtrees\nare recursively added to the cache tree.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 47 ++++++++++++++++++++++--\n cache.h | 10 +++++\n read-cache.c | 27 ++++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 +++++++-\n 4 files changed, 86 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..2866101052c 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,26 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n+\tint base_path_len = tree_path->len;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,14 +768,40 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\n+\t\t\t/*\n+\t\t\t * Recursively-constructed subtree path is only needed when working\n+\t\t\t * in a sparse index (where it's used to determine whether the\n+\t\t\t * subtree is a sparse directory in the index).\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index) {\n+\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n+\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n+\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n+\t\t\t\tstrbuf_addch(tree_path, '/');\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n+\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n+\n \tit->entry_count = cnt;\n }\n \n@@ -772,12 +809,14 @@ void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..c079ece981a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -68,6 +68,11 @@\n */\n #define CACHE_ENTRY_PATH_LENGTH 80\n \n+enum index_search_mode {\n+\tNO_EXPAND_SPARSE = 0,\n+\tEXPAND_SPARSE = 1\n+};\n+\n static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct cache_entry *ce;\n@@ -551,7 +556,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tenum index_search_mode search_mode)\n {\n \tint first, last;\n \n@@ -570,7 +578,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +594,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n \t\t}\n \t}\n \n@@ -595,7 +603,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1235,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1335,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1370,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a8583030b38..5664ff8f039 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -763,9 +763,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -775,6 +775,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"439786","messageId":"a9135a5ed64f031fa2b46baabf179611dde60e71.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:22Z","receivedAt":"2021-10-27T14:39:40Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove the `ensure_full_index` guard on `read_from_tree` and update `git\nreset --mixed` to ensure it can use sparse directory index entries wherever\npossible. Sparse directory entries are reset use `diff_tree_oid`, which\nrequires `change` and `add_remove` functions to process the internal\ncontents of the sparse directory. The `recursive` diff option handles cases\nin which `reset --mixed` must diff/merge files that are nested multiple\nlevels deep in a sparse directory.\n\nThe use of pathspecs with `git reset --mixed` introduces scenarios in which\ninternal contents of sparse directories may be matched by the pathspec. In\norder to reset *all* files in the repo that may match the pathspec, the\nfollowing conditions on the pathspec require index expansion before\nperforming the reset:\n\n* \"magic\" pathspecs\n* wildcard pathspecs that do not match only in-cone files or entire sparse\n directories\n* literal pathspecs matching something outside the sparse checkout\n definition\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 78 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n 2 files changed, 93 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 0ac0de7dc97..60517e7e1d6 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t * If the file 1) corresponds to an existing index entry with\n \t\t * skip-worktree set, or 2) does not exist in the index but is\n \t\t * outside the sparse checkout definition, add a skip-worktree bit\n-\t\t * to the new index entry.\n+\t\t * to the new index entry. Note that a sparse index will be expanded\n+\t\t * if this entry is outside the sparse cone - this is necessary\n+\t\t * to properly construct the reset sparse directory.\n \t\t */\n \t\tpos = cache_name_pos(one->path, strlen(one->path));\n \t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n@@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t}\n }\n \n+static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n+{\n+\tunsigned int i, pos;\n+\tint res = 0;\n+\tchar *skip_worktree_seen = NULL;\n+\n+\t/*\n+\t * When using a magic pathspec, assume for the sake of simplicity that\n+\t * the index needs to be expanded to match all matchable files.\n+\t */\n+\tif (pathspec->magic)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\tstruct pathspec_item item = pathspec->items[i];\n+\n+\t\t/*\n+\t\t * If the pathspec item has a wildcard, the index should be expanded\n+\t\t * if the pathspec has the possibility of matching a subset of entries inside\n+\t\t * of a sparse directory (but not the entire directory).\n+\t\t *\n+\t\t * If the pathspec item is a literal path, the index only needs to be expanded\n+\t\t * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n+\t\t * expand for in-cone files) and b) it doesn't match any sparse directories\n+\t\t * (since we can reset whole sparse directories without expanding them).\n+\t\t */\n+\t\tif (item.nowildcard_len < item.len) {\n+\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n+\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n+\n+\t\t\t\tif (!S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is longer than the sparse\n+\t\t\t\t * directory name and the sparse directory is the first\n+\t\t\t\t * component of the pathspec, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (item.nowildcard_len > ce_namelen(ce) &&\n+\t\t\t\t !strncmp(item.original, ce->name, ce_namelen(ce))) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is shorter than the sparse\n+\t\t\t\t * directory and the pathspec does not match the whole\n+\t\t\t\t * directory, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n+\t\t\t\t wildmatch(item.original, ce->name, 0)) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n+\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n+\t\t\tres = 1;\n+\n+\t\tif (res > 0)\n+\t\t\tbreak;\n+\t}\n+\n+\tfree(skip_worktree_seen);\n+\treturn res;\n+}\n+\n static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t struct object_id *tree_oid,\n \t\t\t int intent_to_add)\n@@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\tif (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n+\t\tensure_full_index(&the_index);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 5664ff8f039..44d5e11c762 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -781,11 +781,28 @@ test_expect_success 'sparse-index is not expanded' '\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n+\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n \tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so the pathspec will not force the\n+\t# index to be expanded.\n+\tensure_not_expanded reset deepest -- folder1 &&\n+\tensure_not_expanded reset deepest -- folder1/ &&\n+\n+\t# Wildcard identifies only in-cone files, no index expansion\n+\tensure_not_expanded reset deepest -- deep/\\* &&\n+\n+\t# Wildcard identifies only full sparse directories, no index expansion\n+\tensure_not_expanded reset deepest -- folder\\* &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"439787","messageId":"f91d1dcf024bfc5639db40d113985cb3094eddad.1635345563.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v5 8/8] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-10-27T14:39:23Z","receivedAt":"2021-10-27T14:39:40Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the\nbenefit `cache_bottom` provides in non-sparse index cases, a separate `hint`\nposition indicates the first position `next_cache_entry` should search,\nupdated each execution with a new position.\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"441861","messageId":"CABPp-BEcWutcLJWJvO-DC5B59EO4focgCJG1OWYg080j8ti44Q@mail.gmail.com","threadId":"56614","inReplyTo":"6ef8e4e31d39386e3a6fa1e6c24acd2c005fbd54.1633641339.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/8] reset: make --mixed sparse-aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-20T22:02:30Z","receivedAt":"2021-11-20T22:02:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Oct 7, 2021 at 2:15 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n> reset --mixed` to ensure it can use sparse directory index entries wherever\n> possible. Sparse directory entries are reset use `diff_tree_oid`, which\n\nI am having trouble parsing this second sentence. Was this meant to\nbe 'Sparse directory entries _which_ are reset use...'?\n\n> requires `change` and `add_remove` functions to process the internal\n> contents of the sparse directory. The `recursive` diff option handles cases\n> in which `reset --mixed` must diff/merge files that are nested multiple\n> levels deep in a sparse directory.\n>\n> The use of pathspecs with `git reset --mixed` introduces scenarios in which\n> internal contents of sparse directories may be matched by the pathspec. In\n> order to reset *all* files in the repo that may match the pathspec, the\n> following conditions on the pathspec require index expansion before\n> performing the reset:\n>\n> * \"magic\" pathspecs\n> * wildcard pathspecs that do not match only in-cone files or entire sparse\n> directories\n> * literal pathspecs matching something outside the sparse checkout\n> definition\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Victoria Dye \n> ---\n> builtin/reset.c | 78 +++++++++++++++++++++++-\n> t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n> 2 files changed, 93 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 0ac0de7dc97..60517e7e1d6 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> * If the file 1) corresponds to an existing index entry with\n> * skip-worktree set, or 2) does not exist in the index but is\n> * outside the sparse checkout definition, add a skip-worktree bit\n> - * to the new index entry.\n> + * to the new index entry. Note that a sparse index will be expanded\n> + * if this entry is outside the sparse cone - this is necessary\n> + * to properly construct the reset sparse directory.\n> */\n> pos = cache_name_pos(one->path, strlen(one->path));\n> if ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n> @@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> }\n> }\n>\n> +static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n> +{\n> + unsigned int i, pos;\n> + int res = 0;\n> + char *skip_worktree_seen = NULL;\n> +\n> + /*\n> + * When using a magic pathspec, assume for the sake of simplicity that\n> + * the index needs to be expanded to match all matchable files.\n> + */\n> + if (pathspec->magic)\n> + return 1;\n> +\n> + for (i = 0; i < pathspec->nr; i++) {\n> + struct pathspec_item item = pathspec->items[i];\n> +\n> + /*\n> + * If the pathspec item has a wildcard, the index should be expanded\n> + * if the pathspec has the possibility of matching a subset of entries inside\n> + * of a sparse directory (but not the entire directory).\n> + *\n> + * If the pathspec item is a literal path, the index only needs to be expanded\n> + * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n> + * expand for in-cone files) and b) it doesn't match any sparse directories\n> + * (since we can reset whole sparse directories without expanding them).\n> + */\n> + if (item.nowildcard_len < item.len) {\n> + for (pos = 0; pos < active_nr; pos++) {\n> + struct cache_entry *ce = active_cache[pos];\n> +\n> + if (!S_ISSPARSEDIR(ce->ce_mode))\n> + continue;\n\nThis double loop over all pathspecs and over all index entries reminds\nme of the original non-cone mode sparsity patterns. Stolee introduced\ncone mode patterns specifically to avoid the expensiveness of such\ndouble loops (cf.\nhttps://lore.kernel.org/git/19d664a5dada87a9a8dcf18d7548582275593f10.1566313865.git.gitgitgadget@gmail.com/).\nCan one of the functions he added allow us to avoid this double loop,\nor are there complications that don't allow this (e.g. the actually\nSKIP_WORKTREE paths don't quite match the requested sparsity paths in\nsome cases, or here we are faced with just a leading path of multiple\nindex entries)?\n\n> + /*\n> + * If the pre-wildcard length is longer than the sparse\n> + * directory name and the sparse directory is the first\n> + * component of the pathspec, need to expand the index.\n> + */\n> + if (item.nowildcard_len > ce_namelen(ce) &&\n> + !strncmp(item.original, ce->name, ce_namelen(ce))) {\n> + res = 1;\n> + break;\n> + }\n> +\n> + /*\n> + * If the pre-wildcard length is shorter than the sparse\n> + * directory and the pathspec does not match the whole\n> + * directory, need to expand the index.\n> + */\n> + if (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n> + wildmatch(item.original, ce->name, 0)) {\n> + res = 1;\n> + break;\n> + }\n> + }\n> + } else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n\nOh, so you can at least generally avoid the double loop. That's good.\nSo is this just a case of wildcards are special and there isn't a way,\neven in cone-mode, to avoid the double loop?\n\n(Given that I'm so tardy in reviewing this, even if the answer is that\nthe double loop is avoidable, or if we just don't know, I'd be totally\nfine with a 'TODO: consider whether this double loop could be avoided\nin cone mode using some kind of variant of\npath_in_cone_mode_sparse_checkout()')\n\n> + res = 1;\n> +\n> + if (res > 0)\n> + break;\n> + }\n> +\n> + free(skip_worktree_seen);\n> + return res;\n> +}\n> +\n> static int read_from_tree(const struct pathspec *pathspec,\n> struct object_id *tree_oid,\n> int intent_to_add)\n> @@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n> opt.format_callback = update_index_from_diff;\n> opt.format_callback_data = &intent_to_add;\n> opt.flags.override_submodule_config = 1;\n> + opt.flags.recursive = 1;\n> opt.repo = the_repository;\n> + opt.change = diff_change;\n> + opt.add_remove = diff_addremove;\n> +\n> + if (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n> + ensure_full_index(&the_index);\n>\n> - ensure_full_index(&the_index);\n> if (do_diff_cache(tree_oid, &opt))\n> return 1;\n> diffcore_std(&opt);\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 4ac93874cb2..c9343ff5b9c 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -774,11 +774,28 @@ test_expect_success 'sparse-index is not expanded' '\n> ensure_not_expanded reset --hard $ref || return 1\n> done &&\n>\n> + ensure_not_expanded reset --mixed base &&\n> ensure_not_expanded reset --hard update-deep &&\n> ensure_not_expanded reset --keep base &&\n> ensure_not_expanded reset --merge update-deep &&\n> ensure_not_expanded reset --hard &&\n>\n> + ensure_not_expanded reset base -- deep/a &&\n> + ensure_not_expanded reset base -- nonexistent-file &&\n> + ensure_not_expanded reset deepest -- deep &&\n> +\n> + # Although folder1 is outside the sparse definition, it exists as a\n> + # directory entry in the index, so the pathspec will not force the\n> + # index to be expanded.\n> + ensure_not_expanded reset deepest -- folder1 &&\n> + ensure_not_expanded reset deepest -- folder1/ &&\n> +\n> + # Wildcard identifies only in-cone files, no index expansion\n> + ensure_not_expanded reset deepest -- deep/\\* &&\n> +\n> + # Wildcard identifies only full sparse directories, no index expansion\n> + ensure_not_expanded reset deepest -- folder\\* &&\n> +\n> ensure_not_expanded checkout -f update-deep &&\n> test_config -C sparse-index pull.twohead ort &&\n> (\n> --\n> gitgitgadget\n"},{"id":"441862","messageId":"CABPp-BH9qXZObVkEyuLOzoOvw_uPfC_n9QSR=by2+-GVgAGgSw@mail.gmail.com","threadId":"56614","inReplyTo":"a9135a5ed64f031fa2b46baabf179611dde60e71.1635345563.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 7/8] reset: make --mixed sparse-aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-20T22:05:55Z","receivedAt":"2021-11-20T22:06:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry, one more thing...\n\nOn Wed, Oct 27, 2021 at 7:39 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n> reset --mixed` to ensure it can use sparse directory index entries wherever\n> possible. Sparse directory entries are reset use `diff_tree_oid`, which\n> requires `change` and `add_remove` functions to process the internal\n> contents of the sparse directory. The `recursive` diff option handles cases\n> in which `reset --mixed` must diff/merge files that are nested multiple\n> levels deep in a sparse directory.\n>\n> The use of pathspecs with `git reset --mixed` introduces scenarios in which\n> internal contents of sparse directories may be matched by the pathspec. In\n> order to reset *all* files in the repo that may match the pathspec, the\n> following conditions on the pathspec require index expansion before\n> performing the reset:\n>\n> * \"magic\" pathspecs\n> * wildcard pathspecs that do not match only in-cone files or entire sparse\n> directories\n> * literal pathspecs matching something outside the sparse checkout\n> definition\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Victoria Dye \n> ---\n> builtin/reset.c | 78 +++++++++++++++++++++++-\n> t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n> 2 files changed, 93 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/reset.c b/builtin/reset.c\n> index 0ac0de7dc97..60517e7e1d6 100644\n> --- a/builtin/reset.c\n> +++ b/builtin/reset.c\n> @@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> * If the file 1) corresponds to an existing index entry with\n> * skip-worktree set, or 2) does not exist in the index but is\n> * outside the sparse checkout definition, add a skip-worktree bit\n> - * to the new index entry.\n> + * to the new index entry. Note that a sparse index will be expanded\n> + * if this entry is outside the sparse cone - this is necessary\n> + * to properly construct the reset sparse directory.\n> */\n> pos = cache_name_pos(one->path, strlen(one->path));\n> if ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n> @@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n> }\n> }\n>\n> +static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n> +{\n> + unsigned int i, pos;\n> + int res = 0;\n> + char *skip_worktree_seen = NULL;\n> +\n> + /*\n> + * When using a magic pathspec, assume for the sake of simplicity that\n> + * the index needs to be expanded to match all matchable files.\n> + */\n> + if (pathspec->magic)\n> + return 1;\n> +\n> + for (i = 0; i < pathspec->nr; i++) {\n> + struct pathspec_item item = pathspec->items[i];\n> +\n> + /*\n> + * If the pathspec item has a wildcard, the index should be expanded\n> + * if the pathspec has the possibility of matching a subset of entries inside\n> + * of a sparse directory (but not the entire directory).\n> + *\n> + * If the pathspec item is a literal path, the index only needs to be expanded\n> + * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n> + * expand for in-cone files) and b) it doesn't match any sparse directories\n> + * (since we can reset whole sparse directories without expanding them).\n> + */\n> + if (item.nowildcard_len < item.len) {\n> + for (pos = 0; pos < active_nr; pos++) {\n> + struct cache_entry *ce = active_cache[pos];\n> +\n> + if (!S_ISSPARSEDIR(ce->ce_mode))\n> + continue;\n> +\n> + /*\n> + * If the pre-wildcard length is longer than the sparse\n> + * directory name and the sparse directory is the first\n> + * component of the pathspec, need to expand the index.\n> + */\n> + if (item.nowildcard_len > ce_namelen(ce) &&\n> + !strncmp(item.original, ce->name, ce_namelen(ce))) {\n> + res = 1;\n> + break;\n> + }\n> +\n> + /*\n> + * If the pre-wildcard length is shorter than the sparse\n> + * directory and the pathspec does not match the whole\n> + * directory, need to expand the index.\n> + */\n> + if (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n> + wildmatch(item.original, ce->name, 0)) {\n> + res = 1;\n> + break;\n> + }\n> + }\n> + } else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n> + res = 1;\n> +\n> + if (res > 0)\n> + break;\n> + }\n> +\n> + free(skip_worktree_seen);\n> + return res;\n> +}\n> +\n> static int read_from_tree(const struct pathspec *pathspec,\n> struct object_id *tree_oid,\n> int intent_to_add)\n> @@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n> opt.format_callback = update_index_from_diff;\n> opt.format_callback_data = &intent_to_add;\n> opt.flags.override_submodule_config = 1;\n> + opt.flags.recursive = 1;\n> opt.repo = the_repository;\n> + opt.change = diff_change;\n> + opt.add_remove = diff_addremove;\n> +\n> + if (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n> + ensure_full_index(&the_index);\n>\n> - ensure_full_index(&the_index);\n> if (do_diff_cache(tree_oid, &opt))\n> return 1;\n> diffcore_std(&opt);\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 5664ff8f039..44d5e11c762 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -781,11 +781,28 @@ test_expect_success 'sparse-index is not expanded' '\n> ensure_not_expanded reset --hard $ref || return 1\n> done &&\n>\n> + ensure_not_expanded reset --mixed base &&\n> ensure_not_expanded reset --hard update-deep &&\n> ensure_not_expanded reset --keep base &&\n> ensure_not_expanded reset --merge update-deep &&\n> ensure_not_expanded reset --hard &&\n>\n> + ensure_not_expanded reset base -- deep/a &&\n> + ensure_not_expanded reset base -- nonexistent-file &&\n> + ensure_not_expanded reset deepest -- deep &&\n> +\n> + # Although folder1 is outside the sparse definition, it exists as a\n> + # directory entry in the index, so the pathspec will not force the\n> + # index to be expanded.\n> + ensure_not_expanded reset deepest -- folder1 &&\n> + ensure_not_expanded reset deepest -- folder1/ &&\n> +\n> + # Wildcard identifies only in-cone files, no index expansion\n> + ensure_not_expanded reset deepest -- deep/\\* &&\n> +\n> + # Wildcard identifies only full sparse directories, no index expansion\n> + ensure_not_expanded reset deepest -- folder\\* &&\n> +\n\nYou've added two testcases where a wildcard results in no index\nexpansion; should there also be a test where a wildcard results in\nindex expansion for completeness?\n"},{"id":"441863","messageId":"CABPp-BEKEC74qx_KrZTLDdt_bsgAQhR9MhO9v9pTtVvFyjaX6A@mail.gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 0/8] Sparse Index: integrate with reset","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-20T22:13:48Z","receivedAt":"2021-11-20T22:14:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi!\n\nOn Wed, Oct 27, 2021 at 7:39 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> This series integrates the sparse index with git reset and provides\n> miscellaneous fixes and improvements to the command in sparse checkouts.\n...\n> Changes since V3\n> ================\n>\n> * Replace git update-index --force-full-index with git reset update-folder1\n> -- folder1/a, remove introduction of new --force-full-index option\n> entirely, and add comment clarifying the intent of sparse-index is\n> expanded and converted back test\n> * Fix authorship on reset: preserve skip-worktree bit in mixed reset\n> (current patch fully replaces original patch, but metadata of the\n> original wasn't properly replaced)\n>\n>\n> Changes since V4\n> ================\n>\n> * Update t1092 test 'checkout and reset (mixed)' to explicitly verify\n> differences between sparse and full checkouts\n\nI apologize for my tardiness in reviewing your updated series. You\nhave addressed all my feedback from v2 and things look really good. I\nhad a couple small questions on patch 7.\n\nAs with my previous review, I kinda skipped over the last patch\nbecause I never figured out the cache_bottom stuff. I read it to see\nif there were any obvious mistakes to someone unfamiliar with that\nmechanism (i.e. me) but didn't see anything. I read over the earlier\npatches much more carefully.\n\nAnyway, other than patch 7 -- where I only had a minor nit on the\ncommit message plus two questions (which might result in no changes),\nthe series looks good.\n"},{"id":"441966","messageId":"c97e4252-b17a-c8c1-3bde-cbfe22a6e4d5@github.com","threadId":"56614","inReplyTo":"CABPp-BEcWutcLJWJvO-DC5B59EO4focgCJG1OWYg080j8ti44Q@mail.gmail.com","subject":"Re: [PATCH v3 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-11-22T16:46:02Z","receivedAt":"2021-11-22T16:46:06Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Thu, Oct 7, 2021 at 2:15 PM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n>> reset --mixed` to ensure it can use sparse directory index entries wherever\n>> possible. Sparse directory entries are reset use `diff_tree_oid`, which\n> \n> I am having trouble parsing this second sentence. Was this meant to\n> be 'Sparse directory entries _which_ are reset use...'?\n> \n\nIt should be \"Sparse directory entries are reset _using_\n`diff_tree_oid`...\".\n\n>> requires `change` and `add_remove` functions to process the internal\n>> contents of the sparse directory. The `recursive` diff option handles cases\n>> in which `reset --mixed` must diff/merge files that are nested multiple\n>> levels deep in a sparse directory.\n>>\n>> The use of pathspecs with `git reset --mixed` introduces scenarios in which\n>> internal contents of sparse directories may be matched by the pathspec. In\n>> order to reset *all* files in the repo that may match the pathspec, the\n>> following conditions on the pathspec require index expansion before\n>> performing the reset:\n>>\n>> * \"magic\" pathspecs\n>> * wildcard pathspecs that do not match only in-cone files or entire sparse\n>> directories\n>> * literal pathspecs matching something outside the sparse checkout\n>> definition\n>>\n>> Helped-by: Elijah Newren \n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/reset.c | 78 +++++++++++++++++++++++-\n>> t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n>> 2 files changed, 93 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/reset.c b/builtin/reset.c\n>> index 0ac0de7dc97..60517e7e1d6 100644\n>> --- a/builtin/reset.c\n>> +++ b/builtin/reset.c\n>> @@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> * If the file 1) corresponds to an existing index entry with\n>> * skip-worktree set, or 2) does not exist in the index but is\n>> * outside the sparse checkout definition, add a skip-worktree bit\n>> - * to the new index entry.\n>> + * to the new index entry. Note that a sparse index will be expanded\n>> + * if this entry is outside the sparse cone - this is necessary\n>> + * to properly construct the reset sparse directory.\n>> */\n>> pos = cache_name_pos(one->path, strlen(one->path));\n>> if ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n>> @@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> }\n>> }\n>>\n>> +static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n>> +{\n>> + unsigned int i, pos;\n>> + int res = 0;\n>> + char *skip_worktree_seen = NULL;\n>> +\n>> + /*\n>> + * When using a magic pathspec, assume for the sake of simplicity that\n>> + * the index needs to be expanded to match all matchable files.\n>> + */\n>> + if (pathspec->magic)\n>> + return 1;\n>> +\n>> + for (i = 0; i < pathspec->nr; i++) {\n>> + struct pathspec_item item = pathspec->items[i];\n>> +\n>> + /*\n>> + * If the pathspec item has a wildcard, the index should be expanded\n>> + * if the pathspec has the possibility of matching a subset of entries inside\n>> + * of a sparse directory (but not the entire directory).\n>> + *\n>> + * If the pathspec item is a literal path, the index only needs to be expanded\n>> + * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n>> + * expand for in-cone files) and b) it doesn't match any sparse directories\n>> + * (since we can reset whole sparse directories without expanding them).\n>> + */\n>> + if (item.nowildcard_len < item.len) {\n>> + for (pos = 0; pos < active_nr; pos++) {\n>> + struct cache_entry *ce = active_cache[pos];\n>> +\n>> + if (!S_ISSPARSEDIR(ce->ce_mode))\n>> + continue;\n> \n> This double loop over all pathspecs and over all index entries reminds\n> me of the original non-cone mode sparsity patterns. Stolee introduced\n> cone mode patterns specifically to avoid the expensiveness of such\n> double loops (cf.\n> https://lore.kernel.org/git/19d664a5dada87a9a8dcf18d7548582275593f10.1566313865.git.gitgitgadget@gmail.com/).\n> Can one of the functions he added allow us to avoid this double loop,\n> or are there complications that don't allow this (e.g. the actually\n> SKIP_WORKTREE paths don't quite match the requested sparsity paths in\n> some cases, or here we are faced with just a leading path of multiple\n> index entries)?\n> \n>> + /*\n>> + * If the pre-wildcard length is longer than the sparse\n>> + * directory name and the sparse directory is the first\n>> + * component of the pathspec, need to expand the index.\n>> + */\n>> + if (item.nowildcard_len > ce_namelen(ce) &&\n>> + !strncmp(item.original, ce->name, ce_namelen(ce))) {\n>> + res = 1;\n>> + break;\n>> + }\n>> +\n>> + /*\n>> + * If the pre-wildcard length is shorter than the sparse\n>> + * directory and the pathspec does not match the whole\n>> + * directory, need to expand the index.\n>> + */\n>> + if (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n>> + wildmatch(item.original, ce->name, 0)) {\n>> + res = 1;\n>> + break;\n>> + }\n>> + }\n>> + } else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n>> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n> \n> Oh, so you can at least generally avoid the double loop. That's good.\n> So is this just a case of wildcards are special and there isn't a way,\n> even in cone-mode, to avoid the double loop?\n> \n\nThe wildcard pathspecs are difficult to handle as \"cleanly\" as the\nnon-wildcard pathspecs, since the condition for expanding the index is\nwhether the pathspec has the potential to match some, but not all, contents\nof any given sparse directory. Luckily, the double loop is constrained to\nonly sparse directories in the index and wildcard pathspecs, and exits as\nsoon as there is any indication that the index needs to be expanded.\n\n> (Given that I'm so tardy in reviewing this, even if the answer is that\n> the double loop is avoidable, or if we just don't know, I'd be totally\n> fine with a 'TODO: consider whether this double loop could be avoided\n> in cone mode using some kind of variant of\n> path_in_cone_mode_sparse_checkout()')\n> \n\nI had originally tried something like this, but even if a wildcard pathspec\nis in-cone, it could match something outside of it. For example, '*.c' is\nin-cone, but has the potential to match only some of the files in a sparse\ndirectory containing both *.c and *.h files. \n\nLooking at it again, though, if you constrain\n`path_in_cone_mode_sparse_checkout()` pathspecs further and require that the\nonly post-`nowildcard_len` characters are '*', there's no risk of matching\npartial subsets of files in sparse directories. I'll add that in a re-roll.\n\n>> + res = 1;\n>> +\n>> + if (res > 0)\n>> + break;\n>> + }\n>> +\n>> + free(skip_worktree_seen);\n>> + return res;\n>> +}\n>> +\n>> static int read_from_tree(const struct pathspec *pathspec,\n>> struct object_id *tree_oid,\n>> int intent_to_add)\n>> @@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n>> opt.format_callback = update_index_from_diff;\n>> opt.format_callback_data = &intent_to_add;\n>> opt.flags.override_submodule_config = 1;\n>> + opt.flags.recursive = 1;\n>> opt.repo = the_repository;\n>> + opt.change = diff_change;\n>> + opt.add_remove = diff_addremove;\n>> +\n>> + if (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n>> + ensure_full_index(&the_index);\n>>\n>> - ensure_full_index(&the_index);\n>> if (do_diff_cache(tree_oid, &opt))\n>> return 1;\n>> diffcore_std(&opt);\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 4ac93874cb2..c9343ff5b9c 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -774,11 +774,28 @@ test_expect_success 'sparse-index is not expanded' '\n>> ensure_not_expanded reset --hard $ref || return 1\n>> done &&\n>>\n>> + ensure_not_expanded reset --mixed base &&\n>> ensure_not_expanded reset --hard update-deep &&\n>> ensure_not_expanded reset --keep base &&\n>> ensure_not_expanded reset --merge update-deep &&\n>> ensure_not_expanded reset --hard &&\n>>\n>> + ensure_not_expanded reset base -- deep/a &&\n>> + ensure_not_expanded reset base -- nonexistent-file &&\n>> + ensure_not_expanded reset deepest -- deep &&\n>> +\n>> + # Although folder1 is outside the sparse definition, it exists as a\n>> + # directory entry in the index, so the pathspec will not force the\n>> + # index to be expanded.\n>> + ensure_not_expanded reset deepest -- folder1 &&\n>> + ensure_not_expanded reset deepest -- folder1/ &&\n>> +\n>> + # Wildcard identifies only in-cone files, no index expansion\n>> + ensure_not_expanded reset deepest -- deep/\\* &&\n>> +\n>> + # Wildcard identifies only full sparse directories, no index expansion\n>> + ensure_not_expanded reset deepest -- folder\\* &&\n>> +\n>> ensure_not_expanded checkout -f update-deep &&\n>> test_config -C sparse-index pull.twohead ort &&\n>> (\n>> --\n>> gitgitgadget\n\n"},{"id":"441969","messageId":"b3f33d40-d418-f285-4a32-1db7a2c4c465@github.com","threadId":"56614","inReplyTo":"CABPp-BH9qXZObVkEyuLOzoOvw_uPfC_n9QSR=by2+-GVgAGgSw@mail.gmail.com","subject":"Re: [PATCH v5 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-11-22T16:54:04Z","receivedAt":"2021-11-22T16:54:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> Sorry, one more thing...\n> \n> On Wed, Oct 27, 2021 at 7:39 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n>> reset --mixed` to ensure it can use sparse directory index entries wherever\n>> possible. Sparse directory entries are reset use `diff_tree_oid`, which\n>> requires `change` and `add_remove` functions to process the internal\n>> contents of the sparse directory. The `recursive` diff option handles cases\n>> in which `reset --mixed` must diff/merge files that are nested multiple\n>> levels deep in a sparse directory.\n>>\n>> The use of pathspecs with `git reset --mixed` introduces scenarios in which\n>> internal contents of sparse directories may be matched by the pathspec. In\n>> order to reset *all* files in the repo that may match the pathspec, the\n>> following conditions on the pathspec require index expansion before\n>> performing the reset:\n>>\n>> * \"magic\" pathspecs\n>> * wildcard pathspecs that do not match only in-cone files or entire sparse\n>> directories\n>> * literal pathspecs matching something outside the sparse checkout\n>> definition\n>>\n>> Helped-by: Elijah Newren \n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/reset.c | 78 +++++++++++++++++++++++-\n>> t/t1092-sparse-checkout-compatibility.sh | 17 ++++++\n>> 2 files changed, 93 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/reset.c b/builtin/reset.c\n>> index 0ac0de7dc97..60517e7e1d6 100644\n>> --- a/builtin/reset.c\n>> +++ b/builtin/reset.c\n>> @@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> * If the file 1) corresponds to an existing index entry with\n>> * skip-worktree set, or 2) does not exist in the index but is\n>> * outside the sparse checkout definition, add a skip-worktree bit\n>> - * to the new index entry.\n>> + * to the new index entry. Note that a sparse index will be expanded\n>> + * if this entry is outside the sparse cone - this is necessary\n>> + * to properly construct the reset sparse directory.\n>> */\n>> pos = cache_name_pos(one->path, strlen(one->path));\n>> if ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n>> @@ -166,6 +168,73 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n>> }\n>> }\n>>\n>> +static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n>> +{\n>> + unsigned int i, pos;\n>> + int res = 0;\n>> + char *skip_worktree_seen = NULL;\n>> +\n>> + /*\n>> + * When using a magic pathspec, assume for the sake of simplicity that\n>> + * the index needs to be expanded to match all matchable files.\n>> + */\n>> + if (pathspec->magic)\n>> + return 1;\n>> +\n>> + for (i = 0; i < pathspec->nr; i++) {\n>> + struct pathspec_item item = pathspec->items[i];\n>> +\n>> + /*\n>> + * If the pathspec item has a wildcard, the index should be expanded\n>> + * if the pathspec has the possibility of matching a subset of entries inside\n>> + * of a sparse directory (but not the entire directory).\n>> + *\n>> + * If the pathspec item is a literal path, the index only needs to be expanded\n>> + * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n>> + * expand for in-cone files) and b) it doesn't match any sparse directories\n>> + * (since we can reset whole sparse directories without expanding them).\n>> + */\n>> + if (item.nowildcard_len < item.len) {\n>> + for (pos = 0; pos < active_nr; pos++) {\n>> + struct cache_entry *ce = active_cache[pos];\n>> +\n>> + if (!S_ISSPARSEDIR(ce->ce_mode))\n>> + continue;\n>> +\n>> + /*\n>> + * If the pre-wildcard length is longer than the sparse\n>> + * directory name and the sparse directory is the first\n>> + * component of the pathspec, need to expand the index.\n>> + */\n>> + if (item.nowildcard_len > ce_namelen(ce) &&\n>> + !strncmp(item.original, ce->name, ce_namelen(ce))) {\n>> + res = 1;\n>> + break;\n>> + }\n>> +\n>> + /*\n>> + * If the pre-wildcard length is shorter than the sparse\n>> + * directory and the pathspec does not match the whole\n>> + * directory, need to expand the index.\n>> + */\n>> + if (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n>> + wildmatch(item.original, ce->name, 0)) {\n>> + res = 1;\n>> + break;\n>> + }\n>> + }\n>> + } else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n>> + !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n>> + res = 1;\n>> +\n>> + if (res > 0)\n>> + break;\n>> + }\n>> +\n>> + free(skip_worktree_seen);\n>> + return res;\n>> +}\n>> +\n>> static int read_from_tree(const struct pathspec *pathspec,\n>> struct object_id *tree_oid,\n>> int intent_to_add)\n>> @@ -178,9 +247,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n>> opt.format_callback = update_index_from_diff;\n>> opt.format_callback_data = &intent_to_add;\n>> opt.flags.override_submodule_config = 1;\n>> + opt.flags.recursive = 1;\n>> opt.repo = the_repository;\n>> + opt.change = diff_change;\n>> + opt.add_remove = diff_addremove;\n>> +\n>> + if (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n>> + ensure_full_index(&the_index);\n>>\n>> - ensure_full_index(&the_index);\n>> if (do_diff_cache(tree_oid, &opt))\n>> return 1;\n>> diffcore_std(&opt);\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 5664ff8f039..44d5e11c762 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -781,11 +781,28 @@ test_expect_success 'sparse-index is not expanded' '\n>> ensure_not_expanded reset --hard $ref || return 1\n>> done &&\n>>\n>> + ensure_not_expanded reset --mixed base &&\n>> ensure_not_expanded reset --hard update-deep &&\n>> ensure_not_expanded reset --keep base &&\n>> ensure_not_expanded reset --merge update-deep &&\n>> ensure_not_expanded reset --hard &&\n>>\n>> + ensure_not_expanded reset base -- deep/a &&\n>> + ensure_not_expanded reset base -- nonexistent-file &&\n>> + ensure_not_expanded reset deepest -- deep &&\n>> +\n>> + # Although folder1 is outside the sparse definition, it exists as a\n>> + # directory entry in the index, so the pathspec will not force the\n>> + # index to be expanded.\n>> + ensure_not_expanded reset deepest -- folder1 &&\n>> + ensure_not_expanded reset deepest -- folder1/ &&\n>> +\n>> + # Wildcard identifies only in-cone files, no index expansion\n>> + ensure_not_expanded reset deepest -- deep/\\* &&\n>> +\n>> + # Wildcard identifies only full sparse directories, no index expansion\n>> + ensure_not_expanded reset deepest -- folder\\* &&\n>> +\n> \n> You've added two testcases where a wildcard results in no index\n> expansion; should there also be a test where a wildcard results in\n> index expansion for completeness?\n> \n\nThe tests haven't verified when the index *is* expanded for any of the\ncommands implemented so far (save for the one ensuring that, when the index\nis expanded, the expansion is logged via trace2). I see the value in it\n(e.g. using the tests to demonstrate what can trigger index expansion).\nConversely, if the tests are intended to confirm \"successful\" sparse index\nsupport (where index expansion is equivalent to \"the sparse index is not\nsupported\"), then verifying index expansion doesn't necessarily fit that\npurpose. \n\nI'm not sure which interpretation is \"correct\", but if it does make sense to\ntest expansion cases I'm happy to add them here (or, if not in this series,\nadd a TODO to include them in the future).\n\nFor what it's worth, the test 'reset with wildcard pathspec' in [4/8] is\nintended to cover a broader set of wildcard scenarios (verifying\ncorrectness, rather than index expansion). Given the other updates I intend\nto make to wildcard handling, I'm planning on adding cases to that test in \nmy next re-roll.\n"},{"id":"442474","messageId":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v5.git.1635345563.gitgitgadget@gmail.com","subject":"[PATCH v6 0/8] Sparse Index: integrate with reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:35Z","receivedAt":"2021-11-29T19:35:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series integrates the sparse index with git reset and provides\nmiscellaneous fixes and improvements to the command in sparse checkouts.\nThis includes:\n\n 1. tests added to t1092 and p2000 to establish the baseline functionality\n of the command\n 2. repository settings to enable the sparse index with ensure_full_index\n guarding any code paths that break tests without other compatibility\n updates.\n 3. modifications to remove or reduce the scope in which ensure_full_index\n must be called.\n\nThe sparse index updates are predicated on a fix originating from the\nmicrosoft/git fork [1], correcting how git reset --mixed handles resetting\nentries outside the sparse checkout definition. Additionally, a performance\n\"bug\" in next_cache_entry with sparse index is corrected, preventing\nrepeatedly looping over already-searched entries.\n\nThe p2000 tests demonstrate a ~70% execution time reduction in git reset\nusing a sparse index, and no change (within expected variability [2]) using\na full index. Results summarized below [3, 4]:\n\nTest base [5/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.97(0.50+0.37) -3.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.96(0.50+0.36) -4.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 1.37(0.91+0.35) -18.5%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 1.41(0.94+0.35) -17.1%\n\nTest base [6/8] \n-----------------------------------------------------------------------\ngit reset --hard (full-v3) 1.00(0.50+0.39) 0.94(0.48+0.34) -6.0% \ngit reset --hard (full-v4) 1.00(0.51+0.38) 0.95(0.51+0.34) -5.0% \ngit reset --hard (sparse-v3) 1.68(1.17+0.39) 0.46(0.05+0.29) -72.6%\ngit reset --hard (sparse-v4) 1.70(1.18+0.40) 0.46(0.06+0.29) -72.9%\n\nTest base [7/8]\n---------------------------------------------------------------------------\ngit reset (full-v3) 0.77(0.27+0.37) 0.72(0.26+0.32) -6.5%\ngit reset (full-v4) 0.75(0.27+0.34) 0.73(0.26+0.32) -2.7%\ngit reset (sparse-v3) 1.44(0.96+0.36) 0.43(0.04+0.96) -70.1%\ngit reset (sparse-v4) 1.46(0.97+0.36) 0.43(0.05+0.79) -70.5%\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.69(0.26+0.30) -4.2%\ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.71(0.27+0.32) -4.1% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.81(0.42+0.90) -44.1%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.79(0.42+0.76) -44.0%\n\nTest base [8/8] \n---------------------------------------------------------------------------\ngit reset -- missing (full-v3) 0.72(0.26+0.32) 0.73(0.26+0.33) +1.4% \ngit reset -- missing (full-v4) 0.74(0.28+0.33) 0.74(0.27+0.32) +0.0% \ngit reset -- missing (sparse-v3) 1.45(0.97+0.35) 0.43(0.05+0.80) -70.3%\ngit reset -- missing (sparse-v4) 1.41(0.94+0.34) 0.44(0.05+0.76) -68.8%\n\n\n\nChanges since V1\n================\n\n * Add --force-full-index option to update-index. The option is used\n circumvent changing command_requires_full_index from its default value -\n right now this is effectively a no-op, but will change once update-index\n is integrated with sparse index. By using this option in the t1092\n expand/collapse test, the command used to test will not need to be\n updated with subsequent sparse index integrations.\n * Update implementation of mixed reset for entries outside sparse checkout\n definition. The condition in which a file should be checked out before\n index reset is simplified to \"if it has skip-worktree enabled and a reset\n would change the file, check it out\".\n * After checking the behavior of update_index_from_diff with renames,\n found that the diff used by reset does not produce diff queue entries\n with different pathnames for one and two. Because of this, and that\n nothing in the implementation seems to rely on identical path names, no\n BUG check is added.\n * Correct a bug in the sparse index is not expanded tests in t1092 where\n failure of a git reset --mixed test was not being reported. Test now\n verifies an appropriate scenario with corrected failure-checking.\n\n\nChanges since V2\n================\n\n * Replace patch adding checkouts for git reset --mixed with sparse checkout\n with preserving the skip-worktree flag (including a new test for git\n reset --mixed and update to t1092 - checkout and reset (mixed))\n * Move rename of is_missing into its own patch\n * Further extend t1092 tests and remove unnecessary commands/tests where\n possible\n * Refine logic determining which pathspecs require ensure_full_index in git\n reset --mixed, add related ensure_not_expanded tests\n * Add index_search_mode enum to index_name_stage_pos\n * Clean up variable usage & remove unnecessary subtree_path in\n prime_cache_tree_rec\n * Update cover letter performance data\n * More thoroughly explain changes in each commit message\n\n\nChanges since V3\n================\n\n * Replace git update-index --force-full-index with git reset update-folder1\n -- folder1/a, remove introduction of new --force-full-index option\n entirely, and add comment clarifying the intent of sparse-index is\n expanded and converted back test\n * Fix authorship on reset: preserve skip-worktree bit in mixed reset\n (current patch fully replaces original patch, but metadata of the\n original wasn't properly replaced)\n\n\nChanges since V4\n================\n\n * Update t1092 test 'checkout and reset (mixed)' to explicitly verify\n differences between sparse and full checkouts\n\n\nChanges since V5\n================\n\n * Update t1092 test 'reset with wildcard pathspec' with more cases and\n better checks\n * Add \"special case\" wildcard pathspec check when determining whether to\n expand the index (avoids double-loop over pathspecs & index entries)\n\nThanks! -Victoria\n\n[1] microsoft@6b8a074 [2]\nhttps://lore.kernel.org/git/8b9fe3f8-f0e3-4567-b20b-17c92bd1a5c5@github.com/\n[3] If a test and/or commit is not mentioned, there is no significant change\nto performance [4] Pathspec \"does-not-exist\" is changed to \"missing\" to save\nspace in performance report\n\nVictoria Dye (8):\n reset: rename is_missing to !is_in_reset_tree\n reset: preserve skip-worktree bit in mixed reset\n sparse-index: update command for expand/collapse test\n reset: expand test coverage for sparse checkouts\n reset: integrate with sparse index\n reset: make sparse-aware (except --mixed)\n reset: make --mixed sparse-aware\n unpack-trees: improve performance of next_cache_entry\n\n builtin/reset.c | 113 ++++++++++++++++-\n cache-tree.c | 46 ++++++-\n cache.h | 10 ++\n read-cache.c | 27 ++--\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 154 ++++++++++++++++++++---\n t/t7102-reset.sh | 17 +++\n unpack-trees.c | 23 +++-\n 8 files changed, 356 insertions(+), 37 deletions(-)\n\n\nbase-commit: cefe983a320c03d7843ac78e73bd513a27806845\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1048%2Fvdye%2Fvdye%2Fsparse-index-part1-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1048/vdye/vdye/sparse-index-part1-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1048\n\nRange-diff vs v5:\n\n 1: ad7013a31aa = 1: ad7013a31aa reset: rename is_missing to !is_in_reset_tree\n 2: b221b00b7e0 = 2: b221b00b7e0 reset: preserve skip-worktree bit in mixed reset\n 3: 1bb2ca92c60 = 3: 1bb2ca92c60 sparse-index: update command for expand/collapse test\n 4: cc76c694647 ! 4: 741a2c9ffaa reset: expand test coverage for sparse checkouts\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout and rese\n +test_expect_success 'reset with wildcard pathspec' '\n +\tinit_repos &&\n +\n -+\ttest_all_match git checkout -b reset-test update-deep &&\n -+\ttest_all_match git reset base -- \\*/a &&\n -+\ttest_all_match git status --porcelain=v2 &&\n -+\ttest_all_match git rev-parse HEAD:folder1/a &&\n ++\ttest_all_match git reset update-deep -- deep\\* &&\n ++\ttest_all_match git ls-files -s -- deep &&\n +\n -+\ttest_all_match git reset base -- folder\\* &&\n -+\ttest_all_match git status --porcelain=v2 &&\n -+\ttest_all_match git rev-parse HEAD:folder2\n ++\ttest_all_match git reset deepest -- deep\\*\\*\\* &&\n ++\ttest_all_match git ls-files -s -- deep &&\n ++\n ++\t# The following `git reset`s result in updating the index on files with\n ++\t# `skip-worktree` enabled. To avoid failing due to discrepencies in reported\n ++\t# \"modified\" files, `test_sparse_match` reset is performed separately from\n ++\t# \"full-checkout\" reset, then the index contents of all repos are verified.\n ++\n ++\ttest_sparse_match git reset update-folder1 -- \\*/a &&\n ++\tgit -C full-checkout reset update-folder1 -- \\*/a &&\n ++\ttest_all_match git ls-files -s -- deep/a folder1/a &&\n ++\n ++\ttest_sparse_match git reset update-folder2 -- folder\\* &&\n ++\tgit -C full-checkout reset update-folder2 -- folder\\* &&\n ++\ttest_all_match git ls-files -s -- folder10 folder1 folder2 &&\n ++\n ++\ttest_sparse_match git reset base -- folder1/\\* &&\n ++\tgit -C full-checkout reset base -- folder1/\\* &&\n ++\ttest_all_match git ls-files -s -- folder1\n +'\n +\n test_expect_success 'merge, cherry-pick, and rebase' '\n 5: 217ae445418 = 5: 65b0eafd27c reset: integrate with sparse index\n 6: a3e2fd59867 = 6: 908c84005b9 reset: make sparse-aware (except --mixed)\n 7: a9135a5ed64 ! 7: 822d7344587 reset: make --mixed sparse-aware\n @@ Commit message\n \n Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n reset --mixed` to ensure it can use sparse directory index entries wherever\n - possible. Sparse directory entries are reset use `diff_tree_oid`, which\n + possible. Sparse directory entries are reset using `diff_tree_oid`, which\n requires `change` and `add_remove` functions to process the internal\n contents of the sparse directory. The `recursive` diff option handles cases\n in which `reset --mixed` must diff/merge files that are nested multiple\n @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n +\t\t * (since we can reset whole sparse directories without expanding them).\n +\t\t */\n +\t\tif (item.nowildcard_len < item.len) {\n ++\t\t\t/*\n ++\t\t\t * Special case: if the pattern is a path inside the cone\n ++\t\t\t * followed by only wildcards, the pattern cannot match\n ++\t\t\t * partial sparse directories, so we don't expand the index.\n ++\t\t\t */\n ++\t\t\tif (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n ++\t\t\t strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n ++\t\t\t\tcontinue;\n ++\n +\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n +\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n +\n 8: f91d1dcf024 = 8: ddd97fb2837 unpack-trees: improve performance of next_cache_entry\n\n-- \ngitgitgadget\n"},{"id":"442475","messageId":"ad7013a31aa5688747c1606d8c38aa96365c3711.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 1/8] reset: rename is_missing to !is_in_reset_tree","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:36Z","receivedAt":"2021-11-29T19:35:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRename and invert value of `is_missing` to `is_in_reset_tree` to make the\nvariable more descriptive of what it represents.\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 51c9e2f43ff..d3695ce43c4 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -131,10 +131,10 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n-\t\tint is_missing = !(one->mode && !is_null_oid(&one->oid));\n+\t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n \n-\t\tif (is_missing && !intent_to_add) {\n+\t\tif (!is_in_reset_tree && !intent_to_add) {\n \t\t\tremove_file_from_cache(one->path);\n \t\t\tcontinue;\n \t\t}\n@@ -144,7 +144,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\n-\t\tif (is_missing) {\n+\t\tif (!is_in_reset_tree) {\n \t\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n \t\t\tset_object_name_for_intent_to_add_entry(ce);\n \t\t}\n-- \ngitgitgadget\n\n"},{"id":"442476","messageId":"b221b00b7e06a3b135b9f68ce87cffaa7d782581.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 2/8] reset: preserve skip-worktree bit in mixed reset","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:37Z","receivedAt":"2021-11-29T19:35:42Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nChange `update_index_from_diff` to set `skip-worktree` when applicable for\nnew index entries. When `git reset --mixed ` is run, entries in\nthe index with differences between the pre-reset HEAD and reset \nare identified and handled with `update_index_from_diff`. For each file, a\nnew cache entry in inserted into the index, created from the side\nof the reset (without changing the working tree). However, the newly-created\nentry must have `skip-worktree` explicitly set in either of the following\nscenarios:\n\n1. the file is in the current index and has `skip-worktree` set\n2. the file is not in the current index but is outside of a defined sparse\n checkout definition\n\nNot setting the `skip-worktree` bit leads to likely-undesirable results for\na user. It causes `skip-worktree` settings to disappear on the\n\"diff\"-containing files (but *only* the diff-containing files), leading to\nthose files now showing modifications in `git status`. For example, when\nrunning `git reset --mixed` in a sparse checkout, some file entries outside\nof sparse checkout could show up as deleted, despite the user never deleting\nanything (and not wanting them on-disk anyway).\n\nAdditionally, add a test to `t7102` to ensure `skip-worktree` is preserved\nin a basic `git reset --mixed` scenario and update a failure-documenting\ntest from 19a0acc (t1092: test interesting sparse-checkout scenarios,\n2021-01-23) with new expected behavior.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 14 ++++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++--------------\n t/t7102-reset.sh | 17 +++++++++++++++++\n 3 files changed, 39 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex d3695ce43c4..e441b6601b9 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -25,6 +25,7 @@\n #include \"cache-tree.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"dir.h\"\n \n #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000)\n \n@@ -130,6 +131,7 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \tint intent_to_add = *(int *)data;\n \n \tfor (i = 0; i < q->nr; i++) {\n+\t\tint pos;\n \t\tstruct diff_filespec *one = q->queue[i]->one;\n \t\tint is_in_reset_tree = one->mode && !is_null_oid(&one->oid);\n \t\tstruct cache_entry *ce;\n@@ -141,6 +143,18 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \n \t\tce = make_cache_entry(&the_index, one->mode, &one->oid, one->path,\n \t\t\t\t 0, 0);\n+\n+\t\t/*\n+\t\t * If the file 1) corresponds to an existing index entry with\n+\t\t * skip-worktree set, or 2) does not exist in the index but is\n+\t\t * outside the sparse checkout definition, add a skip-worktree bit\n+\t\t * to the new index entry.\n+\t\t */\n+\t\tpos = cache_name_pos(one->path, strlen(one->path));\n+\t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n+\t\t (pos < 0 && !path_in_sparse_checkout(one->path, &the_index)))\n+\t\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\n \t\tif (!ce)\n \t\t\tdie(_(\"make_cache_entry failed for path '%s'\"),\n \t\t\t one->path);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..c7449afe965 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -459,26 +459,20 @@ test_expect_failure 'blame with pathspec outside sparse definition' '\n \ttest_all_match git blame deep/deeper2/deepest/a\n '\n \n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_failure 'checkout and reset (mixed)' '\n+test_expect_success 'checkout and reset (mixed)' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b reset-test update-deep &&\n \ttest_all_match git reset deepest &&\n-\ttest_all_match git reset update-folder1 &&\n-\ttest_all_match git reset update-folder2\n-'\n-\n-# NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-# in this scenario, but it shouldn't.\n-test_expect_success 'checkout and reset (mixed) [sparse]' '\n-\tinit_repos &&\n \n-\ttest_sparse_match git checkout -b reset-test update-deep &&\n-\ttest_sparse_match git reset deepest &&\n+\t# Because skip-worktree is preserved, resetting to update-folder1\n+\t# will show worktree changes for folder1/a in full-checkout, but not\n+\t# in sparse-checkout or sparse-index.\n+\tgit -C full-checkout reset update-folder1 >full-checkout-out &&\n \ttest_sparse_match git reset update-folder1 &&\n-\ttest_sparse_match git reset update-folder2\n+\tgrep \"M\tfolder1/a\" full-checkout-out &&\n+\t! grep \"M\tfolder1/a\" sparse-checkout-out &&\n+\trun_on_sparse test_path_is_missing folder1\n '\n \n test_expect_success 'merge, cherry-pick, and rebase' '\ndiff --git a/t/t7102-reset.sh b/t/t7102-reset.sh\nindex 601b2bf97f0..d05426062ec 100755\n--- a/t/t7102-reset.sh\n+++ b/t/t7102-reset.sh\n@@ -472,6 +472,23 @@ test_expect_success '--mixed refreshes the index' '\n \ttest_cmp expect output\n '\n \n+test_expect_success '--mixed preserves skip-worktree' '\n+\techo 123 >>file2 &&\n+\tgit add file2 &&\n+\tgit update-index --skip-worktree file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_must_be_empty output &&\n+\n+\tcat >expect <<-\\EOF &&\n+\tUnstaged changes after reset:\n+\tM\tfile2\n+\tEOF\n+\tgit update-index --no-skip-worktree file2 &&\n+\tgit add file2 &&\n+\tgit reset --mixed HEAD >output &&\n+\ttest_cmp expect output\n+'\n+\n test_expect_success 'resetting specific path that is unmerged' '\n \tgit rm --cached file2 &&\n \tF1=$(git rev-parse HEAD:file1) &&\n-- \ngitgitgadget\n\n"},{"id":"442477","messageId":"1bb2ca92c60be453d7b151e0a1fb5d471a2e5090.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 3/8] sparse-index: update command for expand/collapse test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:38Z","receivedAt":"2021-11-29T19:35:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIn anticipation of `git reset --hard` being able to use the sparse index\nwithout expanding it, replace the command in `sparse-index is expanded and\nconverted back` with `git reset -- folder1/a`. This command will need to\nexpand the index to work properly, even after integrating the rest of\n`reset` with sparse index.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c7449afe965..cab6340a9d0 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -634,11 +634,15 @@ test_expect_success 'submodule handling' '\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n '\n \n+# When working with a sparse index, some commands will need to expand the\n+# index to operate properly. If those commands also write the index back\n+# to disk, they need to convert the index to sparse before writing.\n+# This test verifies that both of these events are logged in trace2 logs.\n test_expect_success 'sparse-index is expanded and converted back' '\n \tinit_repos &&\n \n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit -C sparse-index -c core.fsmonitor=\"\" reset --hard &&\n+\t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n \ttest_region index ensure_full_index trace2.txt\n '\n-- \ngitgitgadget\n\n"},{"id":"442478","messageId":"741a2c9ffaa8f3ea5e7fc7d3d44cf0ed3654cff2.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 4/8] reset: expand test coverage for sparse checkouts","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:39Z","receivedAt":"2021-11-29T19:35:47Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd new tests for `--merge` and `--keep` modes, as well as mixed reset with\npathspecs. New performance test cases exercise various execution paths for\n`reset`.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 3 +\n t/t1092-sparse-checkout-compatibility.sh | 98 ++++++++++++++++++++++++\n 2 files changed, 101 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 597626276fb..bfd332120c8 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -110,5 +110,8 @@ test_perf_on_all git add -A\n test_perf_on_all git add .\n test_perf_on_all git commit -a -m A\n test_perf_on_all git checkout -f -\n+test_perf_on_all git reset\n+test_perf_on_all git reset --hard\n+test_perf_on_all git reset -- does-not-exist\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex cab6340a9d0..4125525ab86 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -475,6 +475,104 @@ test_expect_success 'checkout and reset (mixed)' '\n \trun_on_sparse test_path_is_missing folder1\n '\n \n+test_expect_success 'checkout and reset (merge)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --merge deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --merge deepest\n+'\n+\n+test_expect_success 'checkout and reset (keep)' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents a &&\n+\ttest_all_match git reset --keep deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\ttest_all_match test_must_fail git reset --keep deepest\n+'\n+\n+test_expect_success 'reset with pathspecs inside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b reset-test update-deep &&\n+\trun_on_all ../edit-contents deep/a &&\n+\n+\ttest_all_match git reset base -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset base -- nonexistent-file &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset deepest -- deep &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+# Although the working tree differs between full and sparse checkouts after\n+# reset, the state of the index is the same.\n+test_expect_success 'reset with pathspecs outside sparse definition' '\n+\tinit_repos &&\n+\ttest_all_match git checkout -b reset-test base &&\n+\n+\ttest_sparse_match git reset update-folder1 -- folder1 &&\n+\tgit -C full-checkout reset update-folder1 -- folder1 &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder1 &&\n+\n+\ttest_sparse_match git reset update-folder2 -- folder2/a &&\n+\tgit -C full-checkout reset update-folder2 -- folder2/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_all_match git rev-parse HEAD:folder2/a\n+'\n+\n+test_expect_success 'reset with wildcard pathspec' '\n+\tinit_repos &&\n+\n+\ttest_all_match git reset update-deep -- deep\\* &&\n+\ttest_all_match git ls-files -s -- deep &&\n+\n+\ttest_all_match git reset deepest -- deep\\*\\*\\* &&\n+\ttest_all_match git ls-files -s -- deep &&\n+\n+\t# The following `git reset`s result in updating the index on files with\n+\t# `skip-worktree` enabled. To avoid failing due to discrepencies in reported\n+\t# \"modified\" files, `test_sparse_match` reset is performed separately from\n+\t# \"full-checkout\" reset, then the index contents of all repos are verified.\n+\n+\ttest_sparse_match git reset update-folder1 -- \\*/a &&\n+\tgit -C full-checkout reset update-folder1 -- \\*/a &&\n+\ttest_all_match git ls-files -s -- deep/a folder1/a &&\n+\n+\ttest_sparse_match git reset update-folder2 -- folder\\* &&\n+\tgit -C full-checkout reset update-folder2 -- folder\\* &&\n+\ttest_all_match git ls-files -s -- folder10 folder1 folder2 &&\n+\n+\ttest_sparse_match git reset base -- folder1/\\* &&\n+\tgit -C full-checkout reset base -- folder1/\\* &&\n+\ttest_all_match git ls-files -s -- folder1\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"442479","messageId":"65b0eafd27c76a8b4ced38a5521413c1c23fc2bf.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 5/8] reset: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:40Z","receivedAt":"2021-11-29T19:35:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nDisable `command_requires_full_index` repo setting and add\n`ensure_full_index` guards around code paths that cannot yet use sparse\ndirectory index entries. `reset --soft` does not modify the index, so no\ncompatibility changes are needed for it to function without expanding the\nindex. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`),\nthe full index is expanded to prevent cache tree corruption and invalid\nvariable accesses.\n\nAdditionally, the `read_cache()` check verifying an uncorrupted index is\nmoved after argument parsing and preparing the repo settings. The index is\nnot used by the preceding argument handling, but `read_cache()` must be run\n*after* enabling sparse index for the command (so that the index is not\nexpanded unnecessarily) and *before* using the index for reset (so that it\nis verified as uncorrupted).\n\nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 10 +++++++---\n cache-tree.c | 1 +\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex e441b6601b9..0ac0de7dc97 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -180,6 +180,7 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.flags.override_submodule_config = 1;\n \topt.repo = the_repository;\n \n+\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\n@@ -257,9 +258,6 @@ static void parse_args(struct pathspec *pathspec,\n \t}\n \t*rev_ret = rev;\n \n-\tif (read_cache() < 0)\n-\t\tdie(_(\"index file corrupt\"));\n-\n \tparse_pathspec(pathspec, 0,\n \t\t PATHSPEC_PREFER_FULL |\n \t\t (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0),\n@@ -405,6 +403,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix)\n \tif (intent_to_add && reset_type != MIXED)\n \t\tdie(_(\"-N can only be used with --mixed\"));\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\tif (read_cache() < 0)\n+\t\tdie(_(\"index file corrupt\"));\n+\n \t/* Soft reset does not touch the index file nor the working tree\n \t * at all, but requires them in a good order. Other resets reset\n \t * the index file to the tree object we are switching to. */\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 90919f9e345..9be19c85b66 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r,\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n+\tensure_full_index(istate);\n \tprime_cache_tree_rec(r, istate->cache_tree, tree);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n-- \ngitgitgadget\n\n"},{"id":"442480","messageId":"908c84005b9b0961a0a5b3b3c34b841a884c0ada.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 6/8] reset: make sparse-aware (except --mixed)","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:41Z","receivedAt":"2021-11-29T19:35:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove `ensure_full_index` guard on `prime_cache_tree` and update\n`prime_cache_tree_rec` to correctly reconstruct sparse directory entries in\nthe cache tree. While processing a tree's entries, `prime_cache_tree_rec`\nmust determine whether a directory entry is sparse or not by searching for\nit in the index (*without* expanding the index). If a matching sparse\ndirectory index entry is found, no subtrees are added to the cache tree\nentry and the entry count is set to 1 (representing the sparse directory\nitself). Otherwise, the tree is assumed to not be sparse and its subtrees\nare recursively added to the cache tree.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n cache-tree.c | 47 ++++++++++++++++++++++--\n cache.h | 10 +++++\n read-cache.c | 27 ++++++++++----\n t/t1092-sparse-checkout-compatibility.sh | 15 +++++++-\n 4 files changed, 86 insertions(+), 13 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 9be19c85b66..2866101052c 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -740,15 +740,26 @@ out:\n \treturn ret;\n }\n \n+static void prime_cache_tree_sparse_dir(struct cache_tree *it,\n+\t\t\t\t\tstruct tree *tree)\n+{\n+\n+\toidcpy(&it->oid, &tree->object.oid);\n+\tit->entry_count = 1;\n+}\n+\n static void prime_cache_tree_rec(struct repository *r,\n \t\t\t\t struct cache_tree *it,\n-\t\t\t\t struct tree *tree)\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct strbuf *tree_path)\n {\n \tstruct tree_desc desc;\n \tstruct name_entry entry;\n \tint cnt;\n+\tint base_path_len = tree_path->len;\n \n \toidcpy(&it->oid, &tree->object.oid);\n+\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \tcnt = 0;\n \twhile (tree_entry(&desc, &entry)) {\n@@ -757,14 +768,40 @@ static void prime_cache_tree_rec(struct repository *r,\n \t\telse {\n \t\t\tstruct cache_tree_sub *sub;\n \t\t\tstruct tree *subtree = lookup_tree(r, &entry.oid);\n+\n \t\t\tif (!subtree->object.parsed)\n \t\t\t\tparse_tree(subtree);\n \t\t\tsub = cache_tree_sub(it, entry.path);\n \t\t\tsub->cache_tree = cache_tree();\n-\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree);\n+\n+\t\t\t/*\n+\t\t\t * Recursively-constructed subtree path is only needed when working\n+\t\t\t * in a sparse index (where it's used to determine whether the\n+\t\t\t * subtree is a sparse directory in the index).\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index) {\n+\t\t\t\tstrbuf_setlen(tree_path, base_path_len);\n+\t\t\t\tstrbuf_grow(tree_path, base_path_len + entry.pathlen + 1);\n+\t\t\t\tstrbuf_add(tree_path, entry.path, entry.pathlen);\n+\t\t\t\tstrbuf_addch(tree_path, '/');\n+\t\t\t}\n+\n+\t\t\t/*\n+\t\t\t * If a sparse index is in use, the directory being processed may be\n+\t\t\t * sparse. To confirm that, we can check whether an entry with that\n+\t\t\t * exact name exists in the index. If it does, the created subtree\n+\t\t\t * should be sparse. Otherwise, cache tree expansion should continue\n+\t\t\t * as normal.\n+\t\t\t */\n+\t\t\tif (r->index->sparse_index &&\n+\t\t\t index_entry_exists(r->index, tree_path->buf, tree_path->len))\n+\t\t\t\tprime_cache_tree_sparse_dir(sub->cache_tree, subtree);\n+\t\t\telse\n+\t\t\t\tprime_cache_tree_rec(r, sub->cache_tree, subtree, tree_path);\n \t\t\tcnt += sub->cache_tree->entry_count;\n \t\t}\n \t}\n+\n \tit->entry_count = cnt;\n }\n \n@@ -772,12 +809,14 @@ void prime_cache_tree(struct repository *r,\n \t\t struct index_state *istate,\n \t\t struct tree *tree)\n {\n+\tstruct strbuf tree_path = STRBUF_INIT;\n+\n \ttrace2_region_enter(\"cache-tree\", \"prime_cache_tree\", the_repository);\n \tcache_tree_free(&istate->cache_tree);\n \tistate->cache_tree = cache_tree();\n \n-\tensure_full_index(istate);\n-\tprime_cache_tree_rec(r, istate->cache_tree, tree);\n+\tprime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path);\n+\tstrbuf_release(&tree_path);\n \tistate->cache_changed |= CACHE_TREE_CHANGED;\n \ttrace2_region_leave(\"cache-tree\", \"prime_cache_tree\", the_repository);\n }\ndiff --git a/cache.h b/cache.h\nindex f6295f3b048..1d3e4665562 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n */\n int index_name_pos(struct index_state *, const char *name, int namelen);\n \n+/*\n+ * Determines whether an entry with the given name exists within the\n+ * given index. The return value is 1 if an exact match is found, otherwise\n+ * it is 0. Note that, unlike index_name_pos, this function does not expand\n+ * the index if it is sparse. If an item exists within the full index but it\n+ * is contained within a sparse directory (and not in the sparse index), 0 is\n+ * returned.\n+ */\n+int index_entry_exists(struct index_state *, const char *name, int namelen);\n+\n /*\n * Some functions return the negative complement of an insert position when a\n * precise match was not found but a position was found where the entry would\ndiff --git a/read-cache.c b/read-cache.c\nindex f5d4385c408..c079ece981a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -68,6 +68,11 @@\n */\n #define CACHE_ENTRY_PATH_LENGTH 80\n \n+enum index_search_mode {\n+\tNO_EXPAND_SPARSE = 0,\n+\tEXPAND_SPARSE = 1\n+};\n+\n static inline struct cache_entry *mem_pool__ce_alloc(struct mem_pool *mem_pool, size_t len)\n {\n \tstruct cache_entry *ce;\n@@ -551,7 +556,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate,\n+\t\t\t\tconst char *name, int namelen,\n+\t\t\t\tint stage,\n+\t\t\t\tenum index_search_mode search_mode)\n {\n \tint first, last;\n \n@@ -570,7 +578,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\tfirst = next+1;\n \t}\n \n-\tif (istate->sparse_index &&\n+\tif (search_mode == EXPAND_SPARSE && istate->sparse_index &&\n \t first > 0) {\n \t\t/* Note: first <= istate->cache_nr */\n \t\tstruct cache_entry *ce = istate->cache[first - 1];\n@@ -586,7 +594,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t ce_namelen(ce) < namelen &&\n \t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n \t\t\tensure_full_index(istate);\n-\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage, search_mode);\n \t\t}\n \t}\n \n@@ -595,7 +603,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \n int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n-\treturn index_name_stage_pos(istate, name, namelen, 0);\n+\treturn index_name_stage_pos(istate, name, namelen, 0, EXPAND_SPARSE);\n+}\n+\n+int index_entry_exists(struct index_state *istate, const char *name, int namelen)\n+{\n+\treturn index_name_stage_pos(istate, name, namelen, 0, NO_EXPAND_SPARSE) >= 0;\n }\n \n int remove_index_entry_at(struct index_state *istate, int pos)\n@@ -1222,7 +1235,7 @@ static int has_dir_name(struct index_state *istate,\n \t\t\t */\n \t\t}\n \n-\t\tpos = index_name_stage_pos(istate, name, len, stage);\n+\t\tpos = index_name_stage_pos(istate, name, len, stage, EXPAND_SPARSE);\n \t\tif (pos >= 0) {\n \t\t\t/*\n \t\t\t * Found one, but not so fast. This could\n@@ -1322,7 +1335,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n \t\tpos = index_pos_to_insert_pos(istate->cache_nr);\n \telse\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n@@ -1357,7 +1370,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \t\tif (!ok_to_replace)\n \t\t\treturn error(_(\"'%s' appears as both a file and as a directory\"),\n \t\t\t\t ce->name);\n-\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \t\tpos = -pos-1;\n \t}\n \treturn pos + 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 4125525ab86..871cc3fcb8d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -777,9 +777,9 @@ test_expect_success 'sparse-index is not expanded' '\n \tensure_not_expanded checkout - &&\n \tensure_not_expanded switch rename-out-to-out &&\n \tensure_not_expanded switch - &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded checkout rename-out-to-out -- deep/deeper1 &&\n-\tgit -C sparse-index reset --hard &&\n+\tensure_not_expanded reset --hard &&\n \tensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 &&\n \n \techo >>sparse-index/README.md &&\n@@ -789,6 +789,17 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tfor ref in update-deep update-folder1 update-folder2 update-deep\n+\tdo\n+\t\techo >>sparse-index/README.md &&\n+\t\tensure_not_expanded reset --hard $ref || return 1\n+\tdone &&\n+\n+\tensure_not_expanded reset --hard update-deep &&\n+\tensure_not_expanded reset --keep base &&\n+\tensure_not_expanded reset --merge update-deep &&\n+\tensure_not_expanded reset --hard &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"442481","messageId":"822d7344587f698e73abba1ca726c3a905f7b403.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 7/8] reset: make --mixed sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:42Z","receivedAt":"2021-11-29T19:35:52Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove the `ensure_full_index` guard on `read_from_tree` and update `git\nreset --mixed` to ensure it can use sparse directory index entries wherever\npossible. Sparse directory entries are reset using `diff_tree_oid`, which\nrequires `change` and `add_remove` functions to process the internal\ncontents of the sparse directory. The `recursive` diff option handles cases\nin which `reset --mixed` must diff/merge files that are nested multiple\nlevels deep in a sparse directory.\n\nThe use of pathspecs with `git reset --mixed` introduces scenarios in which\ninternal contents of sparse directories may be matched by the pathspec. In\norder to reset *all* files in the repo that may match the pathspec, the\nfollowing conditions on the pathspec require index expansion before\nperforming the reset:\n\n* \"magic\" pathspecs\n* wildcard pathspecs that do not match only in-cone files or entire sparse\n directories\n* literal pathspecs matching something outside the sparse checkout\n definition\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 87 +++++++++++++++++++++++-\n t/t1092-sparse-checkout-compatibility.sh | 17 +++++\n 2 files changed, 102 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex 0ac0de7dc97..dcb79fb43a3 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -148,7 +148,9 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t\t * If the file 1) corresponds to an existing index entry with\n \t\t * skip-worktree set, or 2) does not exist in the index but is\n \t\t * outside the sparse checkout definition, add a skip-worktree bit\n-\t\t * to the new index entry.\n+\t\t * to the new index entry. Note that a sparse index will be expanded\n+\t\t * if this entry is outside the sparse cone - this is necessary\n+\t\t * to properly construct the reset sparse directory.\n \t\t */\n \t\tpos = cache_name_pos(one->path, strlen(one->path));\n \t\tif ((pos >= 0 && ce_skip_worktree(active_cache[pos])) ||\n@@ -166,6 +168,82 @@ static void update_index_from_diff(struct diff_queue_struct *q,\n \t}\n }\n \n+static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n+{\n+\tunsigned int i, pos;\n+\tint res = 0;\n+\tchar *skip_worktree_seen = NULL;\n+\n+\t/*\n+\t * When using a magic pathspec, assume for the sake of simplicity that\n+\t * the index needs to be expanded to match all matchable files.\n+\t */\n+\tif (pathspec->magic)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < pathspec->nr; i++) {\n+\t\tstruct pathspec_item item = pathspec->items[i];\n+\n+\t\t/*\n+\t\t * If the pathspec item has a wildcard, the index should be expanded\n+\t\t * if the pathspec has the possibility of matching a subset of entries inside\n+\t\t * of a sparse directory (but not the entire directory).\n+\t\t *\n+\t\t * If the pathspec item is a literal path, the index only needs to be expanded\n+\t\t * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't\n+\t\t * expand for in-cone files) and b) it doesn't match any sparse directories\n+\t\t * (since we can reset whole sparse directories without expanding them).\n+\t\t */\n+\t\tif (item.nowildcard_len < item.len) {\n+\t\t\t/*\n+\t\t\t * Special case: if the pattern is a path inside the cone\n+\t\t\t * followed by only wildcards, the pattern cannot match\n+\t\t\t * partial sparse directories, so we don't expand the index.\n+\t\t\t */\n+\t\t\tif (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n+\t\t\t strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n+\t\t\t\tcontinue;\n+\n+\t\t\tfor (pos = 0; pos < active_nr; pos++) {\n+\t\t\t\tstruct cache_entry *ce = active_cache[pos];\n+\n+\t\t\t\tif (!S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\t\t\tcontinue;\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is longer than the sparse\n+\t\t\t\t * directory name and the sparse directory is the first\n+\t\t\t\t * component of the pathspec, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (item.nowildcard_len > ce_namelen(ce) &&\n+\t\t\t\t !strncmp(item.original, ce->name, ce_namelen(ce))) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\n+\t\t\t\t/*\n+\t\t\t\t * If the pre-wildcard length is shorter than the sparse\n+\t\t\t\t * directory and the pathspec does not match the whole\n+\t\t\t\t * directory, need to expand the index.\n+\t\t\t\t */\n+\t\t\t\tif (!strncmp(item.original, ce->name, item.nowildcard_len) &&\n+\t\t\t\t wildmatch(item.original, ce->name, 0)) {\n+\t\t\t\t\tres = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t} else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n+\t\t\t !matches_skip_worktree(pathspec, i, &skip_worktree_seen))\n+\t\t\tres = 1;\n+\n+\t\tif (res > 0)\n+\t\t\tbreak;\n+\t}\n+\n+\tfree(skip_worktree_seen);\n+\treturn res;\n+}\n+\n static int read_from_tree(const struct pathspec *pathspec,\n \t\t\t struct object_id *tree_oid,\n \t\t\t int intent_to_add)\n@@ -178,9 +256,14 @@ static int read_from_tree(const struct pathspec *pathspec,\n \topt.format_callback = update_index_from_diff;\n \topt.format_callback_data = &intent_to_add;\n \topt.flags.override_submodule_config = 1;\n+\topt.flags.recursive = 1;\n \topt.repo = the_repository;\n+\topt.change = diff_change;\n+\topt.add_remove = diff_addremove;\n+\n+\tif (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec))\n+\t\tensure_full_index(&the_index);\n \n-\tensure_full_index(&the_index);\n \tif (do_diff_cache(tree_oid, &opt))\n \t\treturn 1;\n \tdiffcore_std(&opt);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 871cc3fcb8d..77e302a0ef3 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -795,11 +795,28 @@ test_expect_success 'sparse-index is not expanded' '\n \t\tensure_not_expanded reset --hard $ref || return 1\n \tdone &&\n \n+\tensure_not_expanded reset --mixed base &&\n \tensure_not_expanded reset --hard update-deep &&\n \tensure_not_expanded reset --keep base &&\n \tensure_not_expanded reset --merge update-deep &&\n \tensure_not_expanded reset --hard &&\n \n+\tensure_not_expanded reset base -- deep/a &&\n+\tensure_not_expanded reset base -- nonexistent-file &&\n+\tensure_not_expanded reset deepest -- deep &&\n+\n+\t# Although folder1 is outside the sparse definition, it exists as a\n+\t# directory entry in the index, so the pathspec will not force the\n+\t# index to be expanded.\n+\tensure_not_expanded reset deepest -- folder1 &&\n+\tensure_not_expanded reset deepest -- folder1/ &&\n+\n+\t# Wildcard identifies only in-cone files, no index expansion\n+\tensure_not_expanded reset deepest -- deep/\\* &&\n+\n+\t# Wildcard identifies only full sparse directories, no index expansion\n+\tensure_not_expanded reset deepest -- folder\\* &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"442482","messageId":"ddd97fb2837135b49ea3c681bcdc06908cfc47c4.1638201164.git.gitgitgadget@gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"[PATCH v6 8/8] unpack-trees: improve performance of next_cache_entry","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-29T15:52:43Z","receivedAt":"2021-11-29T19:35:53Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nTo find the first non-unpacked cache entry, `next_cache_entry` iterates\nthrough index, starting at `cache_bottom`. The performance of this in full\nindexes is helped by `cache_bottom` advancing with each invocation of\n`mark_ce_used` (called by `unpack_index_entry`). However, the presence of\nsparse directories can prevent the `cache_bottom` from advancing in a sparse\nindex case, effectively forcing `next_cache_entry` to search from the\nbeginning of the index each time it is called.\n\nThe `cache_bottom` must be preserved for the sparse index (see 17a1bb570b\n(unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the\nbenefit `cache_bottom` provides in non-sparse index cases, a separate `hint`\nposition indicates the first position `next_cache_entry` should search,\nupdated each execution with a new position.\n\nSigned-off-by: Victoria Dye \n---\n unpack-trees.c | 23 +++++++++++++++++------\n 1 file changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8ea0a542da8..b94733de6be 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce,\n \t}\n }\n \n-static struct cache_entry *next_cache_entry(struct unpack_trees_options *o)\n+static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint)\n {\n \tconst struct index_state *index = o->src_index;\n \tint pos = o->cache_bottom;\n \n+\tif (*hint > pos)\n+\t\tpos = *hint;\n+\n \twhile (pos < index->cache_nr) {\n \t\tstruct cache_entry *ce = index->cache[pos];\n-\t\tif (!(ce->ce_flags & CE_UNPACKED))\n+\t\tif (!(ce->ce_flags & CE_UNPACKED)) {\n+\t\t\t*hint = pos + 1;\n \t\t\treturn ce;\n+\t\t}\n \t\tpos++;\n \t}\n+\n+\t*hint = pos;\n \treturn NULL;\n }\n \n@@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str\n \n \t/* Are we supposed to look at the index too? */\n \tif (o->merge) {\n+\t\tint hint = -1;\n \t\twhile (1) {\n \t\t\tint cmp;\n \t\t\tstruct cache_entry *ce;\n \n \t\t\tif (o->diff_index_cached)\n-\t\t\t\tce = next_cache_entry(o);\n+\t\t\t\tce = next_cache_entry(o, &hint);\n \t\t\telse\n \t\t\t\tce = find_cache_entry(info, p);\n \n@@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *,\n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n {\n \tstruct repository *repo = the_repository;\n-\tint i, ret;\n+\tint i, hint, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n \tint free_pattern_list = 0;\n@@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tinfo.pathspec = o->pathspec;\n \n \t\tif (o->prefix) {\n+\t\t\thint = -1;\n+\n \t\t\t/*\n \t\t\t * Unpack existing index entries that sort before the\n \t\t\t * prefix the tree is spliced into. Note that o->merge\n \t\t\t * is always true in this case.\n \t\t\t */\n \t\t\twhile (1) {\n-\t\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\t\tif (!ce)\n \t\t\t\t\tbreak;\n \t\t\t\tif (ce_in_traverse_path(ce, &info))\n@@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t/* Any left-over entries in the index? */\n \tif (o->merge) {\n+\t\thint = -1;\n \t\twhile (1) {\n-\t\t\tstruct cache_entry *ce = next_cache_entry(o);\n+\t\t\tstruct cache_entry *ce = next_cache_entry(o, &hint);\n \t\t\tif (!ce)\n \t\t\t\tbreak;\n \t\t\tif (unpack_index_entry(ce, o) < 0)\n-- \ngitgitgadget\n"},{"id":"442541","messageId":"CABPp-BG0iDHf268UAnRyA=0y0T69YTc+bLMdxCmSbrL8s=9ziA@mail.gmail.com","threadId":"56614","inReplyTo":"pull.1048.v6.git.1638201164.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 0/8] Sparse Index: integrate with reset","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-29T18:37:19Z","receivedAt":"2021-11-29T22:35:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Mon, Nov 29, 2021 at 7:52 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> Changes since V5\n> ================\n>\n> * Update t1092 test 'reset with wildcard pathspec' with more cases and\n> better checks\n> * Add \"special case\" wildcard pathspec check when determining whether to\n> expand the index (avoids double-loop over pathspecs & index entries)\n\nLooks pretty good. However, I'm worried this special case you added\nat my prodding might be problematic, and that I may have been wrong to\nprod you into it...\n\n> Thanks! -Victoria\n>\n> Range-diff vs v5:\n>\n> 7: a9135a5ed64 ! 7: 822d7344587 reset: make --mixed sparse-aware\n> @@ Commit message\n>\n> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n> reset --mixed` to ensure it can use sparse directory index entries wherever\n> - possible. Sparse directory entries are reset use `diff_tree_oid`, which\n> + possible. Sparse directory entries are reset using `diff_tree_oid`, which\n> requires `change` and `add_remove` functions to process the internal\n> contents of the sparse directory. The `recursive` diff option handles cases\n> in which `reset --mixed` must diff/merge files that are nested multiple\n> @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n> + * (since we can reset whole sparse directories without expanding them).\n> + */\n> + if (item.nowildcard_len < item.len) {\n> ++ /*\n> ++ * Special case: if the pattern is a path inside the cone\n> ++ * followed by only wildcards, the pattern cannot match\n> ++ * partial sparse directories, so we don't expand the index.\n> ++ */\n> ++ if (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n> ++ strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n\nI usually expect in an &&-chain to see the cheaper function call first\n(because that ordering often avoids the need to call the second\nfunction), and I would presume that strspn() would be the cheaper of\nthe two. Did you switch the order because you expect the strspn call\nto nearly always return true, though?\n\nCould the strspn() call be replaced by a `item.len ==\nitem.nowildcard_len + 1`? I mean, sure, folks could list multiple\nasterisks in a row in their pathspec, but that seems super unlikely\nand even if it does happen the code will just fall back to the slower\ncodepath and still give them the right answer. And the simpler check\nfeels a lot easier to parse for human readers.\n\nBut I'm worried there's a deeper issue here:\n\n\nIs the wildcard character (or characters) in path treated as a literal\nby path_in_cone_mode_sparse_checkout()? I think it is...and I'm\nworried that may be incorrect. For example, if the path is\n\n foo/*\n\nand the user has done a\n\n git sparse-checkout set foo/bar/\n\nThen 'foo/baz/file' is not in the sparse checkout. However, 'foo/*'\nshould match 'foo/baz/file' and yet 'foo/*' when treated as a literal\npath would be considered in the sparse checkout by\npath_in_cone_mode_sparse_checkout. Does this result in the code\nreturning an incorrect answer? (Or did I misunderstand something so\nfar?)\n\nI'm wondering if I misled you earlier in my musings about whether we\ncould avoid the slow codepath for pathspecs with wildcard characters.\nMaybe there's no safe optimization here and wildcard characters should\nalways go through the slower codepath.\n\n> ++ continue;\n> ++\n> + for (pos = 0; pos < active_nr; pos++) {\n> + struct cache_entry *ce = active_cache[pos];\n> +\n> 8: f91d1dcf024 = 8: ddd97fb2837 unpack-trees: improve performance of next_cache_entry\n>\n> --\n> gitgitgadget\n"},{"id":"442559","messageId":"9c5b7067-b0e3-0153-5cd3-042bae92f91e@github.com","threadId":"56614","inReplyTo":"CABPp-BG0iDHf268UAnRyA=0y0T69YTc+bLMdxCmSbrL8s=9ziA@mail.gmail.com","subject":"Re: [PATCH v6 0/8] Sparse Index: integrate with reset","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2021-11-29T19:44:10Z","receivedAt":"2021-11-29T23:07:27Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> Hi,\n> \n> On Mon, Nov 29, 2021 at 7:52 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> Changes since V5\n>> ================\n>>\n>> * Update t1092 test 'reset with wildcard pathspec' with more cases and\n>> better checks\n>> * Add \"special case\" wildcard pathspec check when determining whether to\n>> expand the index (avoids double-loop over pathspecs & index entries)\n> \n> Looks pretty good. However, I'm worried this special case you added\n> at my prodding might be problematic, and that I may have been wrong to\n> prod you into it...\n> \n>> Thanks! -Victoria\n>>\n>> Range-diff vs v5:\n>>\n>> 7: a9135a5ed64 ! 7: 822d7344587 reset: make --mixed sparse-aware\n>> @@ Commit message\n>>\n>> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n>> reset --mixed` to ensure it can use sparse directory index entries wherever\n>> - possible. Sparse directory entries are reset use `diff_tree_oid`, which\n>> + possible. Sparse directory entries are reset using `diff_tree_oid`, which\n>> requires `change` and `add_remove` functions to process the internal\n>> contents of the sparse directory. The `recursive` diff option handles cases\n>> in which `reset --mixed` must diff/merge files that are nested multiple\n>> @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n>> + * (since we can reset whole sparse directories without expanding them).\n>> + */\n>> + if (item.nowildcard_len < item.len) {\n>> ++ /*\n>> ++ * Special case: if the pattern is a path inside the cone\n>> ++ * followed by only wildcards, the pattern cannot match\n>> ++ * partial sparse directories, so we don't expand the index.\n>> ++ */\n>> ++ if (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n>> ++ strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n> \n> I usually expect in an &&-chain to see the cheaper function call first\n> (because that ordering often avoids the need to call the second\n> function), and I would presume that strspn() would be the cheaper of\n> the two. Did you switch the order because you expect the strspn call\n> to nearly always return true, though?\n> \n\nThis is a miss on my part, the `strspn()` check is probably less expensive\nand should be first.\n\n> Could the strspn() call be replaced by a `item.len ==\n> item.nowildcard_len + 1`? I mean, sure, folks could list multiple\n> asterisks in a row in their pathspec, but that seems super unlikely\n> and even if it does happen the code will just fall back to the slower\n> codepath and still give them the right answer. And the simpler check\n> feels a lot easier to parse for human readers.\n> \n\nAgreed on wanting better readability - if the multiple-wildcard case is\nunlikely, the `PATHSPEC_ONESTAR` flag would indicate whether the pathspec\nends in a single wildcard character. If that flag is still too obscure,\nthough, I can stick with the length comparison.\n\n> But I'm worried there's a deeper issue here:\n> \n> \n> Is the wildcard character (or characters) in path treated as a literal\n> by path_in_cone_mode_sparse_checkout()? I think it is...and I'm\n> worried that may be incorrect. For example, if the path is\n> \n> foo/*\n> \n> and the user has done a\n> \n> git sparse-checkout set foo/bar/\n> \n> Then 'foo/baz/file' is not in the sparse checkout. However, 'foo/*'\n> should match 'foo/baz/file' and yet 'foo/*' when treated as a literal\n> path would be considered in the sparse checkout by\n> path_in_cone_mode_sparse_checkout. Does this result in the code\n> returning an incorrect answer? (Or did I misunderstand something so\n> far?)\n> \n\nCorrect: `path_in_cone_mode_sparse_checkout` interprets the wildcard\nliterally, and the checks here take that into account. The goal of\n`pathspec_needs_expanded_index` is to determine if the pathspec *may* match\nonly partial contents of a sparse directory (like '*.c', or 'f*le'). For a\n`git reset --mixed`, only this scenario requires expansion; if an entire\nsparse directory is matched by a pathspec, the entire sparse directory is\nreset. \n\nUsing your example, 'foo/*' does match 'foo/baz/file', but it also matches\n'foo/' itself; as a result, the `foo/` sparse directory index entry is reset\n(rather than some individual files contained within it). The same goes for a\npatchspec like 'fo*' (\"in-cone\" and ending in a wildcard). Conversely, a\npathspec like 'foo/ba*' would _not_ work (it wouldn't match something like\n'foo/test-file'), and neither would 'f*o' (it would match all of 'foo', but\nwould only match files ending in \"o\" in a directory 'f/').\n\nHope that helps!\n\n> I'm wondering if I misled you earlier in my musings about whether we\n> could avoid the slow codepath for pathspecs with wildcard characters.\n> Maybe there's no safe optimization here and wildcard characters should\n> always go through the slower codepath.\n> \n>> ++ continue;\n>> ++\n>> + for (pos = 0; pos < active_nr; pos++) {\n>> + struct cache_entry *ce = active_cache[pos];\n>> +\n>> 8: f91d1dcf024 = 8: ddd97fb2837 unpack-trees: improve performance of next_cache_entry\n>>\n>> --\n>> gitgitgadget\n\n"},{"id":"442597","messageId":"CABPp-BGtc9Q5hD_e3xh5VXQeCpD-_ADx9OTnPv4VoVsfaQvUiA@mail.gmail.com","threadId":"56614","inReplyTo":"9c5b7067-b0e3-0153-5cd3-042bae92f91e@github.com","subject":"Re: [PATCH v6 0/8] Sparse Index: integrate with reset","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-30T03:59:45Z","receivedAt":"2021-11-30T04:00:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Nov 29, 2021 at 11:44 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > Hi,\n> >\n> > On Mon, Nov 29, 2021 at 7:52 AM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> Changes since V5\n> >> ================\n> >>\n> >> * Update t1092 test 'reset with wildcard pathspec' with more cases and\n> >> better checks\n> >> * Add \"special case\" wildcard pathspec check when determining whether to\n> >> expand the index (avoids double-loop over pathspecs & index entries)\n> >\n> > Looks pretty good. However, I'm worried this special case you added\n> > at my prodding might be problematic, and that I may have been wrong to\n> > prod you into it...\n> >\n> >> Thanks! -Victoria\n> >>\n> >> Range-diff vs v5:\n> >>\n> >> 7: a9135a5ed64 ! 7: 822d7344587 reset: make --mixed sparse-aware\n> >> @@ Commit message\n> >>\n> >> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n> >> reset --mixed` to ensure it can use sparse directory index entries wherever\n> >> - possible. Sparse directory entries are reset use `diff_tree_oid`, which\n> >> + possible. Sparse directory entries are reset using `diff_tree_oid`, which\n> >> requires `change` and `add_remove` functions to process the internal\n> >> contents of the sparse directory. The `recursive` diff option handles cases\n> >> in which `reset --mixed` must diff/merge files that are nested multiple\n> >> @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n> >> + * (since we can reset whole sparse directories without expanding them).\n> >> + */\n> >> + if (item.nowildcard_len < item.len) {\n> >> ++ /*\n> >> ++ * Special case: if the pattern is a path inside the cone\n> >> ++ * followed by only wildcards, the pattern cannot match\n> >> ++ * partial sparse directories, so we don't expand the index.\n> >> ++ */\n> >> ++ if (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n> >> ++ strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n> >\n> > I usually expect in an &&-chain to see the cheaper function call first\n> > (because that ordering often avoids the need to call the second\n> > function), and I would presume that strspn() would be the cheaper of\n> > the two. Did you switch the order because you expect the strspn call\n> > to nearly always return true, though?\n> >\n>\n> This is a miss on my part, the `strspn()` check is probably less expensive\n> and should be first.\n>\n> > Could the strspn() call be replaced by a `item.len ==\n> > item.nowildcard_len + 1`? I mean, sure, folks could list multiple\n> > asterisks in a row in their pathspec, but that seems super unlikely\n> > and even if it does happen the code will just fall back to the slower\n> > codepath and still give them the right answer. And the simpler check\n> > feels a lot easier to parse for human readers.\n> >\n>\n> Agreed on wanting better readability - if the multiple-wildcard case is\n> unlikely, the `PATHSPEC_ONESTAR` flag would indicate whether the pathspec\n> ends in a single wildcard character. If that flag is still too obscure,\n> though, I can stick with the length comparison.\n>\n> > But I'm worried there's a deeper issue here:\n> >\n> >\n> > Is the wildcard character (or characters) in path treated as a literal\n> > by path_in_cone_mode_sparse_checkout()? I think it is...and I'm\n> > worried that may be incorrect. For example, if the path is\n> >\n> > foo/*\n> >\n> > and the user has done a\n> >\n> > git sparse-checkout set foo/bar/\n> >\n> > Then 'foo/baz/file' is not in the sparse checkout. However, 'foo/*'\n> > should match 'foo/baz/file' and yet 'foo/*' when treated as a literal\n> > path would be considered in the sparse checkout by\n> > path_in_cone_mode_sparse_checkout. Does this result in the code\n> > returning an incorrect answer? (Or did I misunderstand something so\n> > far?)\n> >\n>\n> Correct: `path_in_cone_mode_sparse_checkout` interprets the wildcard\n> literally, and the checks here take that into account. The goal of\n> `pathspec_needs_expanded_index` is to determine if the pathspec *may* match\n> only partial contents of a sparse directory (like '*.c', or 'f*le'). For a\n> `git reset --mixed`, only this scenario requires expansion; if an entire\n> sparse directory is matched by a pathspec, the entire sparse directory is\n> reset.\n>\n> Using your example, 'foo/*' does match 'foo/baz/file', but it also matches\n> 'foo/' itself; as a result, the `foo/` sparse directory index entry is reset\n> (rather than some individual files contained within it). The same goes for a\n> patchspec like 'fo*' (\"in-cone\" and ending in a wildcard). Conversely, a\n> pathspec like 'foo/ba*' would _not_ work (it wouldn't match something like\n> 'foo/test-file'), and neither would 'f*o' (it would match all of 'foo', but\n> would only match files ending in \"o\" in a directory 'f/').\n>\n> Hope that helps!\n\nAh, yes, thanks for the explanation. :-)\n\n> > I'm wondering if I misled you earlier in my musings about whether we\n> > could avoid the slow codepath for pathspecs with wildcard characters.\n> > Maybe there's no safe optimization here and wildcard characters should\n> > always go through the slower codepath.\n> >\n> >> ++ continue;\n> >> ++\n> >> + for (pos = 0; pos < active_nr; pos++) {\n> >> + struct cache_entry *ce = active_cache[pos];\n> >> +\n> >> 8: f91d1dcf024 = 8: ddd97fb2837 unpack-trees: improve performance of next_cache_entry\n> >>\n> >> --\n> >> gitgitgadget\n>\n"},{"id":"442800","messageId":"211201.86v9089i8m.gmgdl@evledraar.gmail.com","threadId":"56614","inReplyTo":"9c5b7067-b0e3-0153-5cd3-042bae92f91e@github.com","subject":"Re: [PATCH v6 0/8] Sparse Index: integrate with reset","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-01T20:26:12Z","receivedAt":"2021-12-01T20:35:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 29 2021, Victoria Dye wrote:\n\n> Elijah Newren wrote:\n>> Hi,\n>> \n>> On Mon, Nov 29, 2021 at 7:52 AM Victoria Dye via GitGitGadget\n>> wrote:\n>>>\n>>> Changes since V5\n>>> ================\n>>>\n>>> * Update t1092 test 'reset with wildcard pathspec' with more cases and\n>>> better checks\n>>> * Add \"special case\" wildcard pathspec check when determining whether to\n>>> expand the index (avoids double-loop over pathspecs & index entries)\n>> \n>> Looks pretty good. However, I'm worried this special case you added\n>> at my prodding might be problematic, and that I may have been wrong to\n>> prod you into it...\n>> \n>>> Thanks! -Victoria\n>>>\n>>> Range-diff vs v5:\n>>>\n>>> 7: a9135a5ed64 ! 7: 822d7344587 reset: make --mixed sparse-aware\n>>> @@ Commit message\n>>>\n>>> Remove the `ensure_full_index` guard on `read_from_tree` and update `git\n>>> reset --mixed` to ensure it can use sparse directory index entries wherever\n>>> - possible. Sparse directory entries are reset use `diff_tree_oid`, which\n>>> + possible. Sparse directory entries are reset using `diff_tree_oid`, which\n>>> requires `change` and `add_remove` functions to process the internal\n>>> contents of the sparse directory. The `recursive` diff option handles cases\n>>> in which `reset --mixed` must diff/merge files that are nested multiple\n>>> @@ builtin/reset.c: static void update_index_from_diff(struct diff_queue_struct *q,\n>>> + * (since we can reset whole sparse directories without expanding them).\n>>> + */\n>>> + if (item.nowildcard_len < item.len) {\n>>> ++ /*\n>>> ++ * Special case: if the pattern is a path inside the cone\n>>> ++ * followed by only wildcards, the pattern cannot match\n>>> ++ * partial sparse directories, so we don't expand the index.\n>>> ++ */\n>>> ++ if (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n>>> ++ strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n>> \n>> I usually expect in an &&-chain to see the cheaper function call first\n>> (because that ordering often avoids the need to call the second\n>> function), and I would presume that strspn() would be the cheaper of\n>> the two. Did you switch the order because you expect the strspn call\n>> to nearly always return true, though?\n>> \n>\n> This is a miss on my part, the `strspn()` check is probably less expensive\n> and should be first.\n\nI doubt it matters either way, and I didn't look into this to any degree\nof carefulness.\n\nBut having followed the breadcrumb trail from the \"What's Cooking\"\ndiscussion & looked at the code one thing that stuck out for me was that\npath_in_cone_mode_sparse_checkout() appears returns 1 inconditionally in\nsome cases based on global state:\n\n\t/*\n\t * We default to accepting a path if there are no patterns or\n\t * they are of the wrong type.\n\t */\n\tif (init_sparse_checkout_patterns(istate) ||\n\t (require_cone_mode &&\n\t !istate->sparse_checkout_patterns->use_cone_patterns))\n\t\treturn 1;\n\nSo moreso than the nano-optimization of strspn()\nv.s. path_in_cone_mode_sparse_checkout() I found it a bit odd that we're\ncalling something in a loop where presumably we can punt out a lot\nearlier, and at least make that \"continue\" a \"break\" or \"return\" in that\ncase.\n\nI.e. something in this direction (this patch obviously doesn't even\ncompile, but should clarify what I'm blathering about :); but again, I\nreally haven't looked at this properly, so just food for thought:\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex b1ff699b43a..cefdabb09c2 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -187,6 +187,9 @@ static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n \tif (pathspec->magic)\n \t\treturn 1;\n \n+\tif (cant_possibly_have_path_in_cone_mode_blah_blah(&the_index))\n+\t\treturn 1;\n+\n \tfor (i = 0; i < pathspec->nr; i++) {\n \t\tstruct pathspec_item item = pathspec->items[i];\n \ndiff --git a/dir.c b/dir.c\nindex 5aa6fbad0b7..19f2d989dd3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1456,14 +1456,8 @@ int init_sparse_checkout_patterns(struct index_state *istate)\n \treturn 0;\n }\n \n-static int path_in_sparse_checkout_1(const char *path,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t int require_cone_mode)\n+int cant_possibly_have_path_in_cone_mode_blah_blah(...)\n {\n-\tint dtype = DT_REG;\n-\tenum pattern_match_result match = UNDECIDED;\n-\tconst char *end, *slash;\n-\n \t/*\n \t * We default to accepting a path if there are no patterns or\n \t * they are of the wrong type.\n@@ -1472,6 +1466,16 @@ static int path_in_sparse_checkout_1(const char *path,\n \t (require_cone_mode &&\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n \t\treturn 1;\n+}\n+\n+\n+static int path_in_sparse_checkout_1(const char *path,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t int require_cone_mode)\n+{\n+\tint dtype = DT_REG;\n+\tenum pattern_match_result match = UNDECIDED;\n+\tconst char *end, *slash;\n \n \t/*\n \t * If UNDECIDED, use the match from the parent dir (recursively), or\n"}]}