{"thread":{"id":"57180","subject":"[PATCH 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","startedAt":"2022-01-04T17:37:05Z","lastAt":"2022-01-27T20:04:33Z","messageCount":39,"participants":["Victoria Dye via GitGitGadget","Elijah Newren","Victoria Dye","Derrick Stolee","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"445431","messageId":"pull.1109.git.1641317820.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":null,"subject":"[PATCH 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:51Z","receivedAt":"2022-01-04T17:37:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series continues the work to integrate commands with the sparse index,\nadding integrations with 'git clean', 'git checkout-index', and 'git\nupdate-index'. These three commands, while useful in their own right, are\nupdated mainly because they're used in 'git stash'. A future series will\nintegrate sparse index with 'stash' directly, but its subcommands must be\nintegrated to avoid the performance cost of each one expanding and\ncollapsing the index.\n\nThe series is broken up into 4 parts:\n\n * Patches 1-2 are minor fixups to the 'git reset' sparse index integration\n in response to discussion [1] that came after the series was ready for\n merge to 'next'.\n * Patch 3 integrates 'git clean' with the sparse index.\n * Patches 4-6 integrate 'git checkout-index' with the sparse index and\n introduce a new '--ignore-skip-worktree-bits' option for use with 'git\n checkout-index --all'.\n * This involves changing the behavior of '--all' to respect\n 'skip-worktree' by default (i.e., it won't check out 'skip-worktree'\n files). The '--ignore-skip-worktree-bits' option can be specified to\n force checkout of 'skip-worktree' files, if desired.\n * Patches 7-9 integrate 'git update-index' with the sparse index.\n * Note that, although this integrates the sparse index with\n '--cacheinfo', sparse directories still cannot be updated using that\n option (see the prior discussion [2] for more details on why)\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/CABPp-BG0iDHf268UAnRyA=0y0T69YTc+bLMdxCmSbrL8s=9ziA@mail.gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/a075091c-d0d4-db5d-fa21-c9d6c90c343e@gmail.com/\n\nVictoria Dye (9):\n reset: fix validation in sparse index test\n reset: reorder wildcard pathspec conditions\n clean: integrate with sparse index\n checkout-index: expand sparse checkout compatibility tests\n checkout-index: add --ignore-skip-worktree-bits option\n checkout-index: integrate with sparse index\n update-index: add tests for sparse-checkout compatibility\n update-index: integrate with sparse index\n update-index: reduce scope of index expansion in do_reupdate\n\n Documentation/git-checkout-index.txt | 11 +-\n builtin/checkout-index.c | 40 +++-\n builtin/clean.c | 3 +\n builtin/reset.c | 12 +-\n builtin/update-index.c | 17 +-\n read-cache.c | 10 +-\n t/perf/p2000-sparse-operations.sh | 2 +\n t/t1092-sparse-checkout-compatibility.sh | 230 ++++++++++++++++++++++-\n 8 files changed, 306 insertions(+), 19 deletions(-)\n\n\nbase-commit: dcc0cd074f0c639a0df20461a301af6d45bd582e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1109%2Fvdye%2Fsparse%2Fupdate-index_checkout-index_clean-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1109/vdye/sparse/update-index_checkout-index_clean-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1109\n-- \ngitgitgadget\n"},{"id":"445432","messageId":"eefb6ab4c61460d21dca3877b83a0e6c2852c43c.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 1/9] reset: fix validation in sparse index test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:52Z","receivedAt":"2022-01-04T17:37:08Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate t1092 test 'reset with pathspecs outside sparse definition' to verify\nindex contents. The use of `rev-parse` verifies the contents of HEAD, not\nthe index, providing no real validation of the reset results. Conversely,\n`ls-files` reports the contents of the index (OIDs, flags, filenames), which\nare then compared across checkouts to ensure compatible index states.\n\nFixes 741a2c9ffa (reset: expand test coverage for sparse checkouts,\n2021-09-27).\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 49f70a65692..d5167e7ed69 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -596,13 +596,11 @@ test_expect_success 'reset with pathspecs outside sparse definition' '\n \n \ttest_sparse_match git reset update-folder1 -- folder1 &&\n \tgit -C full-checkout reset update-folder1 -- folder1 &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_all_match git rev-parse HEAD:folder1 &&\n+\ttest_all_match git ls-files -s -- folder1 &&\n \n \ttest_sparse_match git reset update-folder2 -- folder2/a &&\n \tgit -C full-checkout reset update-folder2 -- folder2/a &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_all_match git rev-parse HEAD:folder2/a\n+\ttest_all_match git ls-files -s -- folder2/a\n '\n \n test_expect_success 'reset with wildcard pathspec' '\n-- \ngitgitgadget\n\n"},{"id":"445433","messageId":"0194d894c2f9c5ae4bbaac4cd6fca89933a04789.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 2/9] reset: reorder wildcard pathspec conditions","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:53Z","receivedAt":"2022-01-04T17:37:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRearrange conditions in method determining whether index expansion is\nnecessary when a pathspec is specified for `git reset`, placing less\nexpensive condition first. Additionally, add details & examples to related\ncode comments to help with readability.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex b1ff699b43a..79b40385b99 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -204,10 +204,16 @@ static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n \t\t\t/*\n \t\t\t * Special case: if the pattern is a path inside the cone\n \t\t\t * followed by only wildcards, the pattern cannot match\n-\t\t\t * partial sparse directories, so we don't expand the index.\n+\t\t\t * partial sparse directories, so we know we don't need to\n+\t\t\t * expand the index.\n+\t\t\t *\n+\t\t\t * Examples:\n+\t\t\t * - in-cone/foo***: doesn't need expanded index\n+\t\t\t * - not-in-cone/bar*: may need expanded index\n+\t\t\t * - **.c: may need expanded index\n \t\t\t */\n-\t\t\tif (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n-\t\t\t strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n+\t\t\tif (strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len &&\n+\t\t\t path_in_cone_mode_sparse_checkout(item.original, &the_index))\n \t\t\t\tcontinue;\n \n \t\t\tfor (pos = 0; pos < active_nr; pos++) {\n-- \ngitgitgadget\n\n"},{"id":"445434","messageId":"52aec13d18e339024e350733ee9d0b5c6b9008e6.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 3/9] clean: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:54Z","receivedAt":"2022-01-04T17:37:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove full index requirement for `git clean` and test to ensure the index\nis not expanded in `git clean`. Add to existing test for `git clean` to\nverify cleanup of untracked files in sparse directories is consistent\nbetween sparse index and non-sparse index checkouts.\n\nSigned-off-by: Victoria Dye \n---\n builtin/clean.c | 3 +++\n t/t1092-sparse-checkout-compatibility.sh | 21 +++++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex 98a2860409b..5628fc7103e 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -983,6 +983,9 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tdir.flags |= DIR_KEEP_UNTRACKED_CONTENTS;\n \t}\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tif (read_cache() < 0)\n \t\tdie(_(\"index file corrupt\"));\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d5167e7ed69..05587361452 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -764,23 +764,42 @@ test_expect_success 'clean' '\n \ttest_all_match git commit -m \"ignore bogus files\" &&\n \n \trun_on_sparse mkdir folder1 &&\n+\trun_on_all mkdir -p deep/untracked-deep &&\n \trun_on_all touch folder1/bogus &&\n+\trun_on_all touch folder1/untracked &&\n+\trun_on_all touch deep/untracked-deep/bogus &&\n+\trun_on_all touch deep/untracked-deep/untracked &&\n \n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git clean -f &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_exists folder1/bogus &&\n+\trun_on_all test_path_is_missing folder1/untracked &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/untracked &&\n+\n+\ttest_all_match git clean -fd &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_sparse_match ls &&\n+\ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_exists folder1/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n+\trun_on_all test_path_is_missing deep/untracked-deep/untracked &&\n \n \ttest_all_match git clean -xf &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_is_missing folder1/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n \n \ttest_all_match git clean -xdf &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_is_missing deep/untracked-deep/bogus &&\n \n \ttest_sparse_match test_path_is_dir folder1\n '\n@@ -920,6 +939,8 @@ test_expect_success 'sparse-index is not expanded' '\n \t# Wildcard identifies only full sparse directories, no index expansion\n \tensure_not_expanded reset deepest -- folder\\* &&\n \n+\tensure_not_expanded clean -fd &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"445435","messageId":"e6a8671f6be5afad9a495be66c0735f6b741c844.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 4/9] checkout-index: expand sparse checkout compatibility tests","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:55Z","receivedAt":"2022-01-04T17:37:14Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd tests to cover `checkout-index`, with a focus on cases interesting in a\nsparse checkout (e.g., files specified outside sparse checkout definition).\nNew tests are intended to serve as a baseline for expected behavior and\nperformance when integrating `checkout-index` with the sparse index.\n\nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 54 ++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex cb777c74a24..54f8602f3c1 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -117,5 +117,6 @@ test_perf_on_all git diff\n test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n+test_perf_on_all git checkout-index -f --all\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 05587361452..db7ad41109b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -755,6 +755,60 @@ test_expect_success 'cherry-pick with conflicts' '\n \ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n '\n \n+test_expect_success 'checkout-index inside sparse definition' '\n+\tinit_repos &&\n+\n+\trun_on_all rm -f deep/a &&\n+\ttest_all_match git checkout-index -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\techo test >>new-a &&\n+\trun_on_all cp ../new-a a &&\n+\ttest_all_match test_must_fail git checkout-index -- a &&\n+\ttest_all_match git checkout-index -f -- a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'checkout-index outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# File does not exist on disk yet for sparse checkouts, so checkout-index\n+\t# succeeds without -f\n+\ttest_sparse_match git checkout-index -- folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n+\n+\trun_on_sparse rm -rf folder1 &&\n+\techo test >new-a &&\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_all cp ../new-a folder1/a &&\n+\n+\ttest_all_match test_must_fail git checkout-index -- folder1/a &&\n+\ttest_all_match git checkout-index -f -- folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n+'\n+\n+test_expect_success 'checkout-index with folders' '\n+\tinit_repos &&\n+\n+\t# Inside checkout definition\n+\ttest_all_match test_must_fail git checkout-index -f -- deep/ &&\n+\n+\t# Outside checkout definition\n+\ttest_all_match test_must_fail git checkout-index -f -- folder1/\n+'\n+\n+# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n+# files (even those outside the sparse definition) on disk. However, these files\n+# don't appear in the percentage of tracked files in git status.\n+test_expect_failure 'checkout-index --all' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout-index --all &&\n+\ttest_sparse_match test_path_is_missing folder1\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"445436","messageId":"ec9a751e8dcde99cfaa5c0a855f063de9284057e.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 5/9] checkout-index: add --ignore-skip-worktree-bits option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:56Z","receivedAt":"2022-01-04T17:37:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate `checkout-index --all` to no longer refresh files that have the\n`skip-worktree` bit set. The newly-added `--ignore-skip-worktree-bits`\noption, when used with `--all`, maintains the old behavior and checks out\nall files regardless of `skip-worktree`.\n\nThe ability to toggle whether files should be checked-out based on\n`skip-worktree` already exists in `git checkout` and `git restore` (both of\nwhich have an `--ignore-skip-worktree-bits` option). Adding the option to\n`checkout-index` (and changing the corresponding default behavior to respect\nthe `skip-worktree` bit) is especially helpful for sparse-checkout: it\nprevents inadvertent creation of *all* files outside the sparse definition\non disk and eliminates the need to expand a sparse index by default when\nusing the `--all` option.\n\nInternal usage of `checkout-index` in `git stash` and `git filter-branch` do\nnot make explicit use of files with `skip-worktree` enabled, so\n`--ignore-skip-worktree-bits` is not added to them.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/git-checkout-index.txt | 11 +++++++++--\n builtin/checkout-index.c | 12 ++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 10 +++++-----\n 3 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-checkout-index.txt b/Documentation/git-checkout-index.txt\nindex 4d33e7be0f5..2815f3d4b19 100644\n--- a/Documentation/git-checkout-index.txt\n+++ b/Documentation/git-checkout-index.txt\n@@ -12,6 +12,7 @@ SYNOPSIS\n 'git checkout-index' [-u] [-q] [-a] [-f] [-n] [--prefix=]\n \t\t [--stage=|all]\n \t\t [--temp]\n+\t\t [--ignore-skip-worktree-bits]\n \t\t [-z] [--stdin]\n \t\t [--] [...]\n \n@@ -37,8 +38,9 @@ OPTIONS\n \n -a::\n --all::\n-\tchecks out all files in the index. Cannot be used\n-\ttogether with explicit filenames.\n+\tchecks out all files in the index except for those with the\n+\tskip-worktree bit set (see `--ignore-skip-worktree-bits`).\n+\tCannot be used together with explicit filenames.\n \n -n::\n --no-create::\n@@ -59,6 +61,11 @@ OPTIONS\n \twrite the content to temporary files. The temporary name\n \tassociations will be written to stdout.\n \n+--ignore-skip-worktree-bits::\n+\tCheck out all files, including those with the skip-worktree bit\n+\tset. Note: may only be used with `--all`; skip-worktree is\n+\tignored when explicit filenames are specified.\n+\n --stdin::\n \tInstead of taking list of paths from the command line,\n \tread list of paths from the standard input. Paths are\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex e21620d964e..2053a80103a 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -7,6 +7,7 @@\n #define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"quote.h\"\n #include \"cache-tree.h\"\n@@ -116,7 +117,7 @@ static int checkout_file(const char *name, const char *prefix)\n \treturn -1;\n }\n \n-static int checkout_all(const char *prefix, int prefix_length)\n+static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n {\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n@@ -125,6 +126,8 @@ static int checkout_all(const char *prefix, int prefix_length)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n+\t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n+\t\t\tcontinue;\n \t\tif (ce_stage(ce) != checkout_stage\n \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n \t\t\tcontinue;\n@@ -176,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tint i;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tint all = 0;\n+\tint ignore_skip_worktree = 0;\n \tint read_from_stdin = 0;\n \tint prefix_length;\n \tint force = 0, quiet = 0, not_new = 0;\n@@ -185,6 +189,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_checkout_index_options[] = {\n \t\tOPT_BOOL('a', \"all\", &all,\n \t\t\tN_(\"check out all files in the index\")),\n+\t\tOPT_BOOL(0, \"ignore-skip-worktree-bits\", &ignore_skip_worktree,\n+\t\t\tN_(\"do not skip files with skip-worktree set\")),\n \t\tOPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n \t\tOPT__QUIET(&quiet,\n \t\t\tN_(\"no warning for existing files and files not in index\")),\n@@ -247,6 +253,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \n \t\tif (all)\n \t\t\tdie(\"git checkout-index: don't mix '--all' and explicit filenames\");\n+\t\tif (ignore_skip_worktree)\n+\t\t\tdie(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n \t\tif (read_from_stdin)\n \t\t\tdie(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n \t\tp = prefix_path(prefix, prefix_length, arg);\n@@ -280,7 +288,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (all)\n-\t\terr |= checkout_all(prefix, prefix_length);\n+\t\terr |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n \n \tif (pc_workers > 1)\n \t\terr |= run_parallel_checkout(&state, pc_workers, pc_threshold,\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex db7ad41109b..fad61d96107 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -799,14 +799,14 @@ test_expect_success 'checkout-index with folders' '\n \ttest_all_match test_must_fail git checkout-index -f -- folder1/\n '\n \n-# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n-# files (even those outside the sparse definition) on disk. However, these files\n-# don't appear in the percentage of tracked files in git status.\n-test_expect_failure 'checkout-index --all' '\n+test_expect_success 'checkout-index --all' '\n \tinit_repos &&\n \n \ttest_all_match git checkout-index --all &&\n-\ttest_sparse_match test_path_is_missing folder1\n+\ttest_sparse_match test_path_is_missing folder1 &&\n+\n+\ttest_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n+\ttest_all_match test_path_exists folder1\n '\n \n test_expect_success 'clean' '\n-- \ngitgitgadget\n\n"},{"id":"445437","messageId":"18c00fc9dd373bd5cfb527cb7d672a5a1b3b0588.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 6/9] checkout-index: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:57Z","receivedAt":"2022-01-04T17:37:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd repository settings to allow usage of the sparse index.\n\nWhen using the `--all` option, sparse directories are ignored by default due\nto the `skip-worktree` flag, so there is no need to expand the index. If\n`--ignore-skip-worktree-bits` is specified, the index is expanded in order\nto check out all files.\n\nWhen checking out individual files, existing behavior in a full index is to\nexit with an error if a directory is specified (as the directory name will\nnot match an index entry). However, it is possible in a sparse index to\nmatch a directory name to a sparse directory index entry, but checking out\nthat sparse directory still results in an error on checkout. To reduce some\npotential confusion for users, `checkout_file(...)` explicitly exits with an\ninformative error if provided with a sparse directory name. The test\ncorresponding to this scenario verifies the error message, which now differs\nbetween sparse index and non-sparse index checkouts.\n\nSigned-off-by: Victoria Dye \n---\n builtin/checkout-index.c | 28 ++++++++++++++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++-\n 2 files changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 2053a80103a..9c5657ccf22 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -66,6 +66,7 @@ static int checkout_file(const char *name, const char *prefix)\n \tint namelen = strlen(name);\n \tint pos = cache_name_pos(name, namelen);\n \tint has_same_name = 0;\n+\tint is_file = 0;\n \tint did_checkout = 0;\n \tint errs = 0;\n \n@@ -79,6 +80,9 @@ static int checkout_file(const char *name, const char *prefix)\n \t\t\tbreak;\n \t\thas_same_name = 1;\n \t\tpos++;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tbreak;\n+\t\tis_file = 1;\n \t\tif (ce_stage(ce) != checkout_stage\n \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n \t\t\tcontinue;\n@@ -107,6 +111,8 @@ static int checkout_file(const char *name, const char *prefix)\n \t\tfprintf(stderr, \"git checkout-index: %s \", name);\n \t\tif (!has_same_name)\n \t\t\tfprintf(stderr, \"is not in the cache\");\n+\t\telse if (!is_file)\n+\t\t\tfprintf(stderr, \"is a sparse directory\");\n \t\telse if (checkout_stage)\n \t\t\tfprintf(stderr, \"does not exist at stage %d\",\n \t\t\t\tcheckout_stage);\n@@ -122,10 +128,25 @@ static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_w\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tif (!ce_skip_worktree(ce))\n+\t\t\t\tBUG(\"sparse directory '%s' does not have skip-worktree set\", ce->name);\n+\n+\t\t\t/*\n+\t\t\t * If the current entry is a sparse directory and skip-worktree\n+\t\t\t * entries are being checked out, expand the index and continue\n+\t\t\t * the loop on the current index position (now pointing to the\n+\t\t\t * first entry inside the expanded sparse directory).\n+\t\t\t */\n+\t\t\tif (ignore_skip_worktree) {\n+\t\t\t\tensure_full_index(&the_index);\n+\t\t\t\tce = active_cache[i];\n+\t\t\t}\n+\t\t}\n+\n \t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (ce_stage(ce) != checkout_stage\n@@ -218,6 +239,9 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tgit_config(git_default_config, NULL);\n \tprefix_length = prefix ? strlen(prefix) : 0;\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tif (read_cache() < 0) {\n \t\tdie(\"invalid cache\");\n \t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex fad61d96107..6ecf1f2bf8e 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -796,7 +796,14 @@ test_expect_success 'checkout-index with folders' '\n \ttest_all_match test_must_fail git checkout-index -f -- deep/ &&\n \n \t# Outside checkout definition\n-\ttest_all_match test_must_fail git checkout-index -f -- folder1/\n+\t# Note: although all tests fail (as expected), the messaging differs. For\n+\t# non-sparse index checkouts, the error is that the \"file\" does not appear\n+\t# in the index; for sparse checkouts, the error is explicitly that the\n+\t# entry is a sparse directory.\n+\trun_on_all test_must_fail git checkout-index -f -- folder1/ &&\n+\ttest_cmp full-checkout-err sparse-checkout-err &&\n+\t! test_cmp full-checkout-err sparse-index-err &&\n+\tgrep \"is a sparse directory\" sparse-index-err\n '\n \n test_expect_success 'checkout-index --all' '\n@@ -965,6 +972,8 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tensure_not_expanded checkout-index -f a &&\n+\tensure_not_expanded checkout-index -f --all &&\n \tfor ref in update-deep update-folder1 update-folder2 update-deep\n \tdo\n \t\techo >>sparse-index/README.md &&\n-- \ngitgitgadget\n\n"},{"id":"445438","messageId":"3b734f89c0f2f473d71a5d3c85faec6d48a0e350.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:58Z","receivedAt":"2022-01-04T17:37:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIntroduce tests for a variety of `git update-index` use cases, including\nperformance scenarios. Tests for `update-index add/remove` are specifically\nfocused on how `git stash` uses `git update-index` as a subcommand to\nprepare for sparse index integration with `stash` in a future series.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 125 +++++++++++++++++++++++\n 2 files changed, 126 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 54f8602f3c1..7dbed330160 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n test_perf_on_all git checkout-index -f --all\n+test_perf_on_all git update-index --add --remove\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 6ecf1f2bf8e..6804ab23a27 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -630,6 +630,131 @@ test_expect_success 'reset with wildcard pathspec' '\n \ttest_all_match git ls-files -s -- folder1\n '\n \n+test_expect_success 'update-index modify outside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\t# Create & modify folder1/a\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n+\trun_on_all ../edit-contents folder1/a &&\n+\n+\t# If file has skip-worktree enabled, update-index does not modify the\n+\t# index entry\n+\ttest_sparse_match git update-index folder1/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\n+\t# When skip-worktree is disabled (even on files outside sparse cone), file\n+\t# is updated in the index\n+\ttest_sparse_match git update-index --no-skip-worktree folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git update-index folder1/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --add outside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\t# Create folder1, add new file\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_all ../edit-contents folder1/b &&\n+\n+\t# Similar to `git add`, the untracked out-of-cone file is added to the index\n+\t# identically across sparse and non-sparse checkouts\n+\ttest_all_match git update-index --add folder1/b &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --remove outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n+\t# not removed from the index if they do not exist on disk\n+\ttest_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# When the flag is _not_ specified, out-of-cone, not-on-disk files are\n+\t# removed from the index\n+\trm full-checkout/folder1/a &&\n+\ttest_all_match git update-index --remove folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n+\t# a skip-worktree file from the index (and disk) when both are specified\n+\ttest_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index with directories' '\n+\tinit_repos &&\n+\n+\t# update-index will exit silently when provided with a directory name\n+\t# containing a trailing slash\n+\ttest_all_match git update-index deep/ folder1/ &&\n+\tgrep \"Ignoring path deep/\" sparse-checkout-err &&\n+\tgrep \"Ignoring path folder1/\" sparse-checkout-err &&\n+\n+\t# When update-index is given a directory name WITHOUT a trailing slash, it will\n+\t# behave in different ways depending on the status of the directory on disk:\n+\t# * if it exists, the command exits with an error (\"add individual files instead\")\n+\t# * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n+\t# file and either triggers an error (\"does not exist and --remove not passed\")\n+\t# or is ignored completely (when using --remove)\n+\ttest_all_match test_must_fail git update-index deep &&\n+\trun_on_all test_must_fail git update-indexe folder1 &&\n+\ttest_must_fail git -C full-checkout update-index --remove folder1 &&\n+\ttest_sparse_match git update-index --remove folder1 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --again file outside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\ttest_all_match git checkout -b test-reupdate &&\n+\n+\t# Update HEAD without modifying the index to introduce a difference in\n+\t# folder1/a\n+\ttest_sparse_match git reset --soft update-folder1 &&\n+\n+\t# Because folder1/a differs in the index vs HEAD,\n+\t# `git update-index --remove --again` will effectively perform\n+\t# `git update-index --remove folder1/a` and remove the folder1/a\n+\ttest_sparse_match git update-index --remove --again &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --cacheinfo' '\n+\tinit_repos &&\n+\n+\tdeep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n+\tfolder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n+\tfolder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n+\n+\ttest_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# Cannot add sparse directory, even in sparse index case\n+\ttest_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n+\n+\t# Sparse match only - because folder1/a is outside the sparse checkout\n+\t# definition (and thus not on-disk), it will appear \"deleted\" in\n+\t# unstaged changes.\n+\ttest_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n+\ttest_sparse_match git status --porcelain=v2\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"445439","messageId":"c5b98e36516b8f832576816c202be7581f261909.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 8/9] update-index: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:36:59Z","receivedAt":"2022-01-04T17:37:19Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable usage of the sparse index with `update-index`. Most variations of\n`update-index` work without explicitly expanding the index or making any\nother updates in or outside of `update-index.c`.\n\nThe one usage requiring additional changes is `--cacheinfo`; if a file\ninside a sparse directory was specified, the index would not be expanded\nuntil after the cache tree is invalidated, leading to a mismatch between the\nindex and cache tree. This scenario is handled by rearranging\n`add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the\nindex *before* attempting to invalidate the relevant cache tree path,\navoiding cache tree/index corruption.\n\nSigned-off-by: Victoria Dye \n---\n builtin/update-index.c | 3 +++\n read-cache.c | 10 +++++++---\n t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n 3 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..605cc693bbd 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1077,6 +1077,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\tprepare_repo_settings(r);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \t/* we will diagnose later if it turns out that we need to update it */\n \tnewfd = hold_locked_index(&lock_file, 0);\n \tif (newfd < 0)\ndiff --git a/read-cache.c b/read-cache.c\nindex cbe73f14e5e..b4600e954b6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1339,9 +1339,6 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \tint skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n \tint new_only = option & ADD_CACHE_NEW_ONLY;\n \n-\tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n-\t\tcache_tree_invalidate_path(istate, ce->name);\n-\n \t/*\n \t * If this entry's path sorts after the last entry in the index,\n \t * we can avoid searching for it.\n@@ -1352,6 +1349,13 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \telse\n \t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n+\t/*\n+\t * Cache tree path should be invalidated only after index_name_stage_pos,\n+\t * in case it expands a sparse index.\n+\t */\n+\tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n+\t\tcache_tree_invalidate_path(istate, ce->name);\n+\n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n \t\tif (!new_only)\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 6804ab23a27..bc0741c970d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1216,6 +1216,18 @@ test_expect_success 'sparse index is not expanded: blame' '\n \tdone\n '\n \n+test_expect_success 'sparse index is not expanded: update-index' '\n+\tinit_repos &&\n+\n+\techo \"test\" >sparse-index/README.md &&\n+\techo \"test2\" >sparse-index/a &&\n+\trm -f sparse-index/deep/a &&\n+\n+\tensure_not_expanded update-index --add README.md &&\n+\tensure_not_expanded update-index a &&\n+\tensure_not_expanded update-index --remove deep/a\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"445440","messageId":"de7fc14356231a60c8cac9aa6f92a7fec1560b6a.1641317820.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH 9/9] update-index: reduce scope of index expansion in do_reupdate","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-04T17:37:00Z","receivedAt":"2022-01-04T17:37:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nExpand the full index (and redo reupdate operation) only if a sparse\ndirectory in the index differs from HEAD. Only the index entries that differ\nbetween the index and HEAD are updated when performing `git update-index\n--again`, so unmodified sparse directories are safely skipped. The index\ndoes need to be expanded when sparse directories contain changes, though,\nbecause `update_one(...)` will not operate on sparse directory index\nentries.\n\nBecause the index should only be expanded if a sparse directory is modified,\nadd a test ensuring the index is not expanded when differences only exist\nwithin the sparse cone.\n\nSigned-off-by: Victoria Dye \n---\n builtin/update-index.c | 14 +++++++++++---\n t/t1092-sparse-checkout-compatibility.sh | 5 ++++-\n 2 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 605cc693bbd..52ecc714d99 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -606,7 +606,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tif (mode == S_IFDIR) {\n+\tif (!the_index.sparse_index && mode == S_IFDIR) {\n \t\tif (which)\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n@@ -743,8 +743,6 @@ static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n \t\tstruct cache_entry *old = NULL;\n@@ -761,6 +759,16 @@ static int do_reupdate(int ac, const char **av,\n \t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n+\n+\t\t/* At this point, we know the contents of the sparse directory are\n+\t\t * modified with respect to HEAD, so we expand the index and restart\n+\t\t * to process each path individually\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tensure_full_index(&the_index);\n+\t\t\tgoto redo;\n+\t\t}\n+\n \t\t/* Be careful. The working tree may not have the\n \t\t * path anymore, in which case, under 'allow_remove',\n \t\t * or worse yet 'allow_replace', active_nr may decrease.\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex bc0741c970d..0863c9747c4 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1225,7 +1225,10 @@ test_expect_success 'sparse index is not expanded: update-index' '\n \n \tensure_not_expanded update-index --add README.md &&\n \tensure_not_expanded update-index a &&\n-\tensure_not_expanded update-index --remove deep/a\n+\tensure_not_expanded update-index --remove deep/a &&\n+\n+\tensure_not_expanded reset --soft update-deep &&\n+\tensure_not_expanded update-index --add --remove --again\n '\n \n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n-- \ngitgitgadget\n"},{"id":"445581","messageId":"CABPp-BFqTqK3grMLv_odKGwDhUaU8p1epNzv825kTqLAgMrK9g@mail.gmail.com","threadId":"57180","inReplyTo":"e6a8671f6be5afad9a495be66c0735f6b741c844.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/9] checkout-index: expand sparse checkout compatibility tests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-05T21:04:43Z","receivedAt":"2022-01-05T21:05:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add tests to cover `checkout-index`, with a focus on cases interesting in a\n> sparse checkout (e.g., files specified outside sparse checkout definition).\n> New tests are intended to serve as a baseline for expected behavior and\n> performance when integrating `checkout-index` with the sparse index.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1092-sparse-checkout-compatibility.sh | 54 ++++++++++++++++++++++++\n> 2 files changed, 55 insertions(+)\n>\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index cb777c74a24..54f8602f3c1 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -117,5 +117,6 @@ test_perf_on_all git diff\n> test_perf_on_all git diff --cached\n> test_perf_on_all git blame $SPARSE_CONE/a\n> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> +test_perf_on_all git checkout-index -f --all\n>\n> test_done\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 05587361452..db7ad41109b 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -755,6 +755,60 @@ test_expect_success 'cherry-pick with conflicts' '\n> test_all_match test_must_fail git cherry-pick to-cherry-pick\n> '\n>\n> +test_expect_success 'checkout-index inside sparse definition' '\n> + init_repos &&\n> +\n> + run_on_all rm -f deep/a &&\n> + test_all_match git checkout-index -- deep/a &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + echo test >>new-a &&\n> + run_on_all cp ../new-a a &&\n> + test_all_match test_must_fail git checkout-index -- a &&\n> + test_all_match git checkout-index -f -- a &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'checkout-index outside sparse definition' '\n> + init_repos &&\n> +\n> + # File does not exist on disk yet for sparse checkouts, so checkout-index\n> + # succeeds without -f\n> + test_sparse_match git checkout-index -- folder1/a &&\n\nWhoa, um, what is `git checkout-index -- folder1/a` even supposed to\nmean with the sparse-index and cone mode? That's an individual file,\nso should folder1 and everything under it have the SKIP_WORKTREE bit\ncleared? Should folder1 just stop being a sparse directory, with all\nsiblings of folder1/a retaining the SKIP_WORKTREE bit (so folder1/0\ncould become a sparse directory) while folder1/a loses the\nSKIP_WORKTREE bit, kind of violating cone mode but perhaps just until\nthe next sparse-checkout reapply?\n\nThe test sadly doesn't look at the SKIP_WORKTREE bits at all, which\nfeels like an important oversight (unless you meant to test this\nlater?). Especially since my manual testing shows that this test\nleaves the files as SKIP_WORKTREE despite writing the files to the\nworking tree. I think the present-despite-skip-worktree problem I've\ndiscussed elsewhere is growing in importance rather than shrinking;\nyou don't need to address that problem with this series but I raise it\nmostly so we're aware to at least test it. I'll keep reading to see\nif you mention this or fix it up in subsequent patches.\n\n> + test_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n> + test_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n> +\n> + run_on_sparse rm -rf folder1 &&\n> + echo test >new-a &&\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_all cp ../new-a folder1/a &&\n> +\n> + test_all_match test_must_fail git checkout-index -- folder1/a &&\n> + test_all_match git checkout-index -f -- folder1/a &&\n> + test_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n> + test_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n> +'\n> +\n> +test_expect_success 'checkout-index with folders' '\n> + init_repos &&\n> +\n> + # Inside checkout definition\n> + test_all_match test_must_fail git checkout-index -f -- deep/ &&\n> +\n> + # Outside checkout definition\n> + test_all_match test_must_fail git checkout-index -f -- folder1/\n\nPassing a directory to checkout-index? I'm glad they all immediately\nfail, but is the plan just to make sure that this continues? I'm not\nsure what could make sense otherwise. You've piqued my curiosity;\nI'll keep reading the series to see.\n\n> +'\n> +\n> +# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n> +# files (even those outside the sparse definition) on disk. However, these files\n> +# don't appear in the percentage of tracked files in git status.\n\nAh, so you _are_ noticing the present-despite-SKIP_WORKTREE files. I\nthink it might be nice to test for these a bit earlier, but it's nice\nto see some test here for them.\n\nI think that present-despite-SKIP_WORKTREE files is an erroneous\ncondition, one we should avoid triggering, and one we should help\nusers clean up from. For checkout-index, the fact that it currently\nmakes more of these files is buggy. It should either (1) clear the\nSKIP_WORKTREE bit when it writes the files to the working tree, or (2)\navoid writing the files to the working tree.\n\nAnd if we choose (1), there's already a --no-create option we could\npiggy-back on to allow folks to not write the SKIP_WORKTREE files.\n\n> +test_expect_failure 'checkout-index --all' '\n> + init_repos &&\n> +\n> + test_all_match git checkout-index --all &&\n> + test_sparse_match test_path_is_missing folder1\n\nAh, it looks like you're choosing (2). That may be fine, but an\ninteresting anecdote:\n\nWhile attempting to adopt sparse checkouts at $DAYJOB (and\nparticularly using cone mode), we found the code structure just didn't\nquite work. We needed some directories to be ignored for sparse\ncheckouts to be meaningful at all, but we had some files that were\nsiblings to those directories that were needed for builds to function.\nWe came up with a hack to \"add a few files back\", using \"git\ncheckout-index -- $FILENAME\". We expected that hack to write the\nlisted file(s) to the working tree -- though I think we also had logic\nto then run the stuff we needed and then delete these temporary files.\nWe used this hack starting in Feb 2020, and eventually restructured\nthe code to not need this hack in Feb 2021. I'll have to mull over\nwhether your choice of option (2) might cause us some problems if\nsomeone (a) uses a new git version to (b) access an old version of our\ncode and (c) really wants to work with a sparse checkout (since the\n\"checkout-index\" stuff was part of the build logic checked into the\ncode). I think your change here is fine (because not using sparse\ncheckouts is an option, we told folks it was experimental, and those\nare old versions that are only getting security fixes in special\ncircumstances), but let me think about it for a bit...\n\n> +'\n> +\n> test_expect_success 'clean' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n"},{"id":"445603","messageId":"CABPp-BGCCzgNJEaZ9fgHCkW52ws-ef+00GBhy+LZ5Tx+XkhqDw@mail.gmail.com","threadId":"57180","inReplyTo":"ec9a751e8dcde99cfaa5c0a855f063de9284057e.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/9] checkout-index: add --ignore-skip-worktree-bits option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-06T01:52:20Z","receivedAt":"2022-01-06T01:52:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Update `checkout-index --all` to no longer refresh files that have the\n> `skip-worktree` bit set. The newly-added `--ignore-skip-worktree-bits`\n> option, when used with `--all`, maintains the old behavior and checks out\n> all files regardless of `skip-worktree`.\n>\n> The ability to toggle whether files should be checked-out based on\n> `skip-worktree` already exists in `git checkout` and `git restore` (both of\n> which have an `--ignore-skip-worktree-bits` option).\n\nI learned something new.\n\nAnd ick, what a name. Why not --ignore-sparsity or something? Oh well...\n\n> Adding the option to\n> `checkout-index` (and changing the corresponding default behavior to respect\n> the `skip-worktree` bit) is especially helpful for sparse-checkout: it\n> prevents inadvertent creation of *all* files outside the sparse definition\n> on disk and eliminates the need to expand a sparse index by default when\n> using the `--all` option.\n>\n> Internal usage of `checkout-index` in `git stash` and `git filter-branch` do\n> not make explicit use of files with `skip-worktree` enabled, so\n> `--ignore-skip-worktree-bits` is not added to them.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> Documentation/git-checkout-index.txt | 11 +++++++++--\n> builtin/checkout-index.c | 12 ++++++++++--\n> t/t1092-sparse-checkout-compatibility.sh | 10 +++++-----\n> 3 files changed, 24 insertions(+), 9 deletions(-)\n>\n> diff --git a/Documentation/git-checkout-index.txt b/Documentation/git-checkout-index.txt\n> index 4d33e7be0f5..2815f3d4b19 100644\n> --- a/Documentation/git-checkout-index.txt\n> +++ b/Documentation/git-checkout-index.txt\n> @@ -12,6 +12,7 @@ SYNOPSIS\n> 'git checkout-index' [-u] [-q] [-a] [-f] [-n] [--prefix=]\n> [--stage=|all]\n> [--temp]\n> + [--ignore-skip-worktree-bits]\n> [-z] [--stdin]\n> [--] [...]\n>\n> @@ -37,8 +38,9 @@ OPTIONS\n>\n> -a::\n> --all::\n> - checks out all files in the index. Cannot be used\n> - together with explicit filenames.\n> + checks out all files in the index except for those with the\n> + skip-worktree bit set (see `--ignore-skip-worktree-bits`).\n> + Cannot be used together with explicit filenames.\n>\n> -n::\n> --no-create::\n> @@ -59,6 +61,11 @@ OPTIONS\n> write the content to temporary files. The temporary name\n> associations will be written to stdout.\n>\n> +--ignore-skip-worktree-bits::\n> + Check out all files, including those with the skip-worktree bit\n> + set. Note: may only be used with `--all`; skip-worktree is\n> + ignored when explicit filenames are specified.\n\nWhy this restriction? What if the user ran\n git checkout-index -- '*.c'\nThat's not an explicit filename, but a glob.\n\n> +\n> --stdin::\n> Instead of taking list of paths from the command line,\n> read list of paths from the standard input. Paths are\n> diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\n> index e21620d964e..2053a80103a 100644\n> --- a/builtin/checkout-index.c\n> +++ b/builtin/checkout-index.c\n> @@ -7,6 +7,7 @@\n> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> #include \"builtin.h\"\n> #include \"config.h\"\n> +#include \"dir.h\"\n> #include \"lockfile.h\"\n> #include \"quote.h\"\n> #include \"cache-tree.h\"\n> @@ -116,7 +117,7 @@ static int checkout_file(const char *name, const char *prefix)\n> return -1;\n> }\n>\n> -static int checkout_all(const char *prefix, int prefix_length)\n> +static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n> {\n> int i, errs = 0;\n> struct cache_entry *last_ce = NULL;\n> @@ -125,6 +126,8 @@ static int checkout_all(const char *prefix, int prefix_length)\n> ensure_full_index(&the_index);\n> for (i = 0; i < active_nr ; i++) {\n> struct cache_entry *ce = active_cache[i];\n> + if (!ignore_skip_worktree && ce_skip_worktree(ce))\n> + continue;\n\nSo here I see you let it fall through to the code below that will\nwrite the file to the working tree...but it doesn't clear the\nSKIP_WORKTREE bit in the index when it does so, which I think is a\nbug.\n\n> if (ce_stage(ce) != checkout_stage\n> && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> continue;\n> @@ -176,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> int i;\n> struct lock_file lock_file = LOCK_INIT;\n> int all = 0;\n> + int ignore_skip_worktree = 0;\n> int read_from_stdin = 0;\n> int prefix_length;\n> int force = 0, quiet = 0, not_new = 0;\n> @@ -185,6 +189,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> struct option builtin_checkout_index_options[] = {\n> OPT_BOOL('a', \"all\", &all,\n> N_(\"check out all files in the index\")),\n> + OPT_BOOL(0, \"ignore-skip-worktree-bits\", &ignore_skip_worktree,\n> + N_(\"do not skip files with skip-worktree set\")),\n> OPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n> OPT__QUIET(&quiet,\n> N_(\"no warning for existing files and files not in index\")),\n> @@ -247,6 +253,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n>\n> if (all)\n> die(\"git checkout-index: don't mix '--all' and explicit filenames\");\n> + if (ignore_skip_worktree)\n> + die(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n> if (read_from_stdin)\n> die(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n> p = prefix_path(prefix, prefix_length, arg);\n> @@ -280,7 +288,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> }\n>\n> if (all)\n> - err |= checkout_all(prefix, prefix_length);\n> + err |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n>\n> if (pc_workers > 1)\n> err |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index db7ad41109b..fad61d96107 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -799,14 +799,14 @@ test_expect_success 'checkout-index with folders' '\n> test_all_match test_must_fail git checkout-index -f -- folder1/\n> '\n>\n> -# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n> -# files (even those outside the sparse definition) on disk. However, these files\n> -# don't appear in the percentage of tracked files in git status.\n> -test_expect_failure 'checkout-index --all' '\n> +test_expect_success 'checkout-index --all' '\n> init_repos &&\n>\n> test_all_match git checkout-index --all &&\n> - test_sparse_match test_path_is_missing folder1\n> + test_sparse_match test_path_is_missing folder1 &&\n> +\n> + test_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n> + test_all_match test_path_exists folder1\n\nI added an 'exit 1' here, ran the test and then checked:\n\n$ cd trash\\ directory.t1092-sparse-checkout-compatibility/sparse-checkout/\n$ git ls-files -t | grep folder1/\nS folder1/0/0/0\nS folder1/0/1\nS folder1/a\n\nSo there's some more work to do on this patch.\n"},{"id":"445606","messageId":"CABPp-BFT5Rd9pHGAfK7ymNWXs5AGRu5N+dmdrytSoRdkdKvpRQ@mail.gmail.com","threadId":"57180","inReplyTo":"18c00fc9dd373bd5cfb527cb7d672a5a1b3b0588.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] checkout-index: integrate with sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-06T01:59:45Z","receivedAt":"2022-01-06T02:00:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add repository settings to allow usage of the sparse index.\n>\n> When using the `--all` option, sparse directories are ignored by default due\n> to the `skip-worktree` flag, so there is no need to expand the index. If\n> `--ignore-skip-worktree-bits` is specified, the index is expanded in order\n> to check out all files.\n>\n> When checking out individual files, existing behavior in a full index is to\n> exit with an error if a directory is specified (as the directory name will\n> not match an index entry). However, it is possible in a sparse index to\n> match a directory name to a sparse directory index entry, but checking out\n> that sparse directory still results in an error on checkout. To reduce some\n> potential confusion for users, `checkout_file(...)` explicitly exits with an\n> informative error if provided with a sparse directory name. The test\n> corresponding to this scenario verifies the error message, which now differs\n> between sparse index and non-sparse index checkouts.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/checkout-index.c | 28 ++++++++++++++++++++++--\n> t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++-\n> 2 files changed, 36 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\n> index 2053a80103a..9c5657ccf22 100644\n> --- a/builtin/checkout-index.c\n> +++ b/builtin/checkout-index.c\n> @@ -66,6 +66,7 @@ static int checkout_file(const char *name, const char *prefix)\n> int namelen = strlen(name);\n> int pos = cache_name_pos(name, namelen);\n> int has_same_name = 0;\n> + int is_file = 0;\n> int did_checkout = 0;\n> int errs = 0;\n>\n> @@ -79,6 +80,9 @@ static int checkout_file(const char *name, const char *prefix)\n> break;\n> has_same_name = 1;\n> pos++;\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + break;\n> + is_file = 1;\n> if (ce_stage(ce) != checkout_stage\n> && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> continue;\n> @@ -107,6 +111,8 @@ static int checkout_file(const char *name, const char *prefix)\n> fprintf(stderr, \"git checkout-index: %s \", name);\n> if (!has_same_name)\n> fprintf(stderr, \"is not in the cache\");\n> + else if (!is_file)\n> + fprintf(stderr, \"is a sparse directory\");\n> else if (checkout_stage)\n> fprintf(stderr, \"does not exist at stage %d\",\n> checkout_stage);\n> @@ -122,10 +128,25 @@ static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_w\n> int i, errs = 0;\n> struct cache_entry *last_ce = NULL;\n>\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(&the_index);\n> for (i = 0; i < active_nr ; i++) {\n> struct cache_entry *ce = active_cache[i];\n> +\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + if (!ce_skip_worktree(ce))\n> + BUG(\"sparse directory '%s' does not have skip-worktree set\", ce->name);\n> +\n> + /*\n> + * If the current entry is a sparse directory and skip-worktree\n> + * entries are being checked out, expand the index and continue\n> + * the loop on the current index position (now pointing to the\n> + * first entry inside the expanded sparse directory).\n> + */\n> + if (ignore_skip_worktree) {\n> + ensure_full_index(&the_index);\n> + ce = active_cache[i];\n> + }\n\nSo while iterating through the index, we reach an entry and decide to\nexpand the index. This would be unsafe if our iterator became\ninvalid, but the only way that would happen is if there was a sparse\ndirectory entry earlier in the index, and by construction of this loop\nwe expand upon the first sparse directory entry we see. Since you\nreassign ce = active_cache[i] immediately after expanding the index\nand the sparse directory's sub-entries have to go to that same spot,\nyou're actually at the next valid item to iterate over. Slightly\ntricky, but makes sense.\n\n> + }\n> +\n> if (!ignore_skip_worktree && ce_skip_worktree(ce))\n> continue;\n> if (ce_stage(ce) != checkout_stage\n> @@ -218,6 +239,9 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> git_config(git_default_config, NULL);\n> prefix_length = prefix ? strlen(prefix) : 0;\n>\n> + prepare_repo_settings(the_repository);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> if (read_cache() < 0) {\n> die(\"invalid cache\");\n> }\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index fad61d96107..6ecf1f2bf8e 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -796,7 +796,14 @@ test_expect_success 'checkout-index with folders' '\n> test_all_match test_must_fail git checkout-index -f -- deep/ &&\n>\n> # Outside checkout definition\n> - test_all_match test_must_fail git checkout-index -f -- folder1/\n> + # Note: although all tests fail (as expected), the messaging differs. For\n> + # non-sparse index checkouts, the error is that the \"file\" does not appear\n> + # in the index; for sparse checkouts, the error is explicitly that the\n> + # entry is a sparse directory.\n> + run_on_all test_must_fail git checkout-index -f -- folder1/ &&\n> + test_cmp full-checkout-err sparse-checkout-err &&\n> + ! test_cmp full-checkout-err sparse-index-err &&\n> + grep \"is a sparse directory\" sparse-index-err\n> '\n>\n> test_expect_success 'checkout-index --all' '\n> @@ -965,6 +972,8 @@ test_expect_success 'sparse-index is not expanded' '\n> echo >>sparse-index/untracked.txt &&\n> ensure_not_expanded add . &&\n>\n> + ensure_not_expanded checkout-index -f a &&\n> + ensure_not_expanded checkout-index -f --all &&\n> for ref in update-deep update-folder1 update-folder2 update-deep\n> do\n> echo >>sparse-index/README.md &&\n> --\n> gitgitgadget\n\nPatch looks good to me.\n"},{"id":"445626","messageId":"ff3faad1-f5e9-c4db-712d-11f8110d1945@github.com","threadId":"57180","inReplyTo":"CABPp-BGCCzgNJEaZ9fgHCkW52ws-ef+00GBhy+LZ5Tx+XkhqDw@mail.gmail.com","subject":"Re: [PATCH 5/9] checkout-index: add --ignore-skip-worktree-bits option","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-06T15:07:11Z","receivedAt":"2022-01-06T15:07:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Update `checkout-index --all` to no longer refresh files that have the\n>> `skip-worktree` bit set. The newly-added `--ignore-skip-worktree-bits`\n>> option, when used with `--all`, maintains the old behavior and checks out\n>> all files regardless of `skip-worktree`.\n>>\n>> The ability to toggle whether files should be checked-out based on\n>> `skip-worktree` already exists in `git checkout` and `git restore` (both of\n>> which have an `--ignore-skip-worktree-bits` option).\n> \n> I learned something new.\n> \n> And ick, what a name. Why not --ignore-sparsity or something? Oh well...\n> \n>> Adding the option to\n>> `checkout-index` (and changing the corresponding default behavior to respect\n>> the `skip-worktree` bit) is especially helpful for sparse-checkout: it\n>> prevents inadvertent creation of *all* files outside the sparse definition\n>> on disk and eliminates the need to expand a sparse index by default when\n>> using the `--all` option.\n>>\n>> Internal usage of `checkout-index` in `git stash` and `git filter-branch` do\n>> not make explicit use of files with `skip-worktree` enabled, so\n>> `--ignore-skip-worktree-bits` is not added to them.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> Documentation/git-checkout-index.txt | 11 +++++++++--\n>> builtin/checkout-index.c | 12 ++++++++++--\n>> t/t1092-sparse-checkout-compatibility.sh | 10 +++++-----\n>> 3 files changed, 24 insertions(+), 9 deletions(-)\n>>\n>> diff --git a/Documentation/git-checkout-index.txt b/Documentation/git-checkout-index.txt\n>> index 4d33e7be0f5..2815f3d4b19 100644\n>> --- a/Documentation/git-checkout-index.txt\n>> +++ b/Documentation/git-checkout-index.txt\n>> @@ -12,6 +12,7 @@ SYNOPSIS\n>> 'git checkout-index' [-u] [-q] [-a] [-f] [-n] [--prefix=]\n>> [--stage=|all]\n>> [--temp]\n>> + [--ignore-skip-worktree-bits]\n>> [-z] [--stdin]\n>> [--] [...]\n>>\n>> @@ -37,8 +38,9 @@ OPTIONS\n>>\n>> -a::\n>> --all::\n>> - checks out all files in the index. Cannot be used\n>> - together with explicit filenames.\n>> + checks out all files in the index except for those with the\n>> + skip-worktree bit set (see `--ignore-skip-worktree-bits`).\n>> + Cannot be used together with explicit filenames.\n>>\n>> -n::\n>> --no-create::\n>> @@ -59,6 +61,11 @@ OPTIONS\n>> write the content to temporary files. The temporary name\n>> associations will be written to stdout.\n>>\n>> +--ignore-skip-worktree-bits::\n>> + Check out all files, including those with the skip-worktree bit\n>> + set. Note: may only be used with `--all`; skip-worktree is\n>> + ignored when explicit filenames are specified.\n> \n> Why this restriction? What if the user ran\n> git checkout-index -- '*.c'\n> That's not an explicit filename, but a glob.\n> \n\n`checkout-index` doesn't accept globs/pathspecs, so every provided argument\nmust correspond exactly to an entry in the index. \n\nI originally restricted '--ignore-skip-worktree-bits' to only work with\n'--all' because I wanted changes to the current behavior of `checkout-index`\nin a sparse checkout to be as minimal as possible. However, if this is more\n\"unexpected inconsistency\" than \"I'm glad checkout-index with filenames\nstill works the way it did before\", I'm happy to change it. \n\n>> +\n>> --stdin::\n>> Instead of taking list of paths from the command line,\n>> read list of paths from the standard input. Paths are\n>> diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\n>> index e21620d964e..2053a80103a 100644\n>> --- a/builtin/checkout-index.c\n>> +++ b/builtin/checkout-index.c\n>> @@ -7,6 +7,7 @@\n>> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n>> #include \"builtin.h\"\n>> #include \"config.h\"\n>> +#include \"dir.h\"\n>> #include \"lockfile.h\"\n>> #include \"quote.h\"\n>> #include \"cache-tree.h\"\n>> @@ -116,7 +117,7 @@ static int checkout_file(const char *name, const char *prefix)\n>> return -1;\n>> }\n>>\n>> -static int checkout_all(const char *prefix, int prefix_length)\n>> +static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n>> {\n>> int i, errs = 0;\n>> struct cache_entry *last_ce = NULL;\n>> @@ -125,6 +126,8 @@ static int checkout_all(const char *prefix, int prefix_length)\n>> ensure_full_index(&the_index);\n>> for (i = 0; i < active_nr ; i++) {\n>> struct cache_entry *ce = active_cache[i];\n>> + if (!ignore_skip_worktree && ce_skip_worktree(ce))\n>> + continue;\n> \n> So here I see you let it fall through to the code below that will\n> write the file to the working tree...but it doesn't clear the\n> SKIP_WORKTREE bit in the index when it does so, which I think is a\n> bug.\n> \n\nI disagree, mainly because updating a flag seems inconsistent with how\n`checkout-index` otherwise works. Specifically, `checkout-index` creates or\nreplaces a file on disk (not even necessarily in the git working directory)\nbased on the file's state in the index, but doesn't modify the index in the\nprocess. The only exception is '-u', which is effectively a shortcut for\nrunning `git update-index --refresh` [1]. \n\nIf a user wants to to checkout a file *and* update `skip-worktree`, I think\nexplicitly using `update-index` would be a more appropriate way to do that\n(similar to the example [2] in the `checkout-index` documentation):\n\n $ git checkout-index outside-cone/file\n $ git update-index --no-skip-worktree outside-cone/file\n\n[1] https://lore.kernel.org/git/7vis1kvqac.fsf@assigned-by-dhcp.cox.net/\n[2] https://git-scm.com/docs/git-checkout-index#Documentation/git-checkout-index.txt-Toupdateandrefreshonlythefilesalreadycheckedout\n\n>> if (ce_stage(ce) != checkout_stage\n>> && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n>> continue;\n>> @@ -176,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n>> int i;\n>> struct lock_file lock_file = LOCK_INIT;\n>> int all = 0;\n>> + int ignore_skip_worktree = 0;\n>> int read_from_stdin = 0;\n>> int prefix_length;\n>> int force = 0, quiet = 0, not_new = 0;\n>> @@ -185,6 +189,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n>> struct option builtin_checkout_index_options[] = {\n>> OPT_BOOL('a', \"all\", &all,\n>> N_(\"check out all files in the index\")),\n>> + OPT_BOOL(0, \"ignore-skip-worktree-bits\", &ignore_skip_worktree,\n>> + N_(\"do not skip files with skip-worktree set\")),\n>> OPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n>> OPT__QUIET(&quiet,\n>> N_(\"no warning for existing files and files not in index\")),\n>> @@ -247,6 +253,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n>>\n>> if (all)\n>> die(\"git checkout-index: don't mix '--all' and explicit filenames\");\n>> + if (ignore_skip_worktree)\n>> + die(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n>> if (read_from_stdin)\n>> die(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n>> p = prefix_path(prefix, prefix_length, arg);\n>> @@ -280,7 +288,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n>> }\n>>\n>> if (all)\n>> - err |= checkout_all(prefix, prefix_length);\n>> + err |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n>>\n>> if (pc_workers > 1)\n>> err |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index db7ad41109b..fad61d96107 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -799,14 +799,14 @@ test_expect_success 'checkout-index with folders' '\n>> test_all_match test_must_fail git checkout-index -f -- folder1/\n>> '\n>>\n>> -# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n>> -# files (even those outside the sparse definition) on disk. However, these files\n>> -# don't appear in the percentage of tracked files in git status.\n>> -test_expect_failure 'checkout-index --all' '\n>> +test_expect_success 'checkout-index --all' '\n>> init_repos &&\n>>\n>> test_all_match git checkout-index --all &&\n>> - test_sparse_match test_path_is_missing folder1\n>> + test_sparse_match test_path_is_missing folder1 &&\n>> +\n>> + test_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n>> + test_all_match test_path_exists folder1\n> \n> I added an 'exit 1' here, ran the test and then checked:\n> \n> $ cd trash\\ directory.t1092-sparse-checkout-compatibility/sparse-checkout/\n> $ git ls-files -t | grep folder1/\n> S folder1/0/0/0\n> S folder1/0/1\n> S folder1/a\n> \n> So there's some more work to do on this patch.\n\nUnless I'm misreading your comment, this is exactly the behavior I would\nexpect in this test: all files (even those with `skip-worktree` set, per\n'--ignore-skip-worktree-bits') are created on-disk, with `skip-worktree`\nunmodified.\n"},{"id":"445723","messageId":"CABPp-BFvPDKYhDQGJ8-qKojs+aTXa2ZTKTGJb+Aa4wER8Ug8Ow@mail.gmail.com","threadId":"57180","inReplyTo":"CABPp-BFqTqK3grMLv_odKGwDhUaU8p1epNzv825kTqLAgMrK9g@mail.gmail.com","subject":"Re: [PATCH 4/9] checkout-index: expand sparse checkout compatibility tests","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-07T16:21:13Z","receivedAt":"2022-01-07T16:21:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Jan 5, 2022 at 1:04 PM Elijah Newren wrote:\n>\n> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> wrote:\n> >\n> > From: Victoria Dye \n> >\n...\n> > +'\n> > +\n> > +# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n> > +# files (even those outside the sparse definition) on disk. However, these files\n> > +# don't appear in the percentage of tracked files in git status.\n>\n> Ah, so you _are_ noticing the present-despite-SKIP_WORKTREE files. I\n> think it might be nice to test for these a bit earlier, but it's nice\n> to see some test here for them.\n>\n> I think that present-despite-SKIP_WORKTREE files is an erroneous\n> condition, one we should avoid triggering, and one we should help\n> users clean up from. For checkout-index, the fact that it currently\n> makes more of these files is buggy. It should either (1) clear the\n> SKIP_WORKTREE bit when it writes the files to the working tree, or (2)\n> avoid writing the files to the working tree.\n>\n> And if we choose (1), there's already a --no-create option we could\n> piggy-back on to allow folks to not write the SKIP_WORKTREE files.\n>\n> > +test_expect_failure 'checkout-index --all' '\n> > + init_repos &&\n> > +\n> > + test_all_match git checkout-index --all &&\n> > + test_sparse_match test_path_is_missing folder1\n>\n> Ah, it looks like you're choosing (2). That may be fine, but an\n> interesting anecdote:\n>\n> While attempting to adopt sparse checkouts at $DAYJOB (and\n> particularly using cone mode), we found the code structure just didn't\n> quite work. We needed some directories to be ignored for sparse\n> checkouts to be meaningful at all, but we had some files that were\n> siblings to those directories that were needed for builds to function.\n> We came up with a hack to \"add a few files back\", using \"git\n> checkout-index -- $FILENAME\". We expected that hack to write the\n> listed file(s) to the working tree -- though I think we also had logic\n> to then run the stuff we needed and then delete these temporary files.\n> We used this hack starting in Feb 2020, and eventually restructured\n> the code to not need this hack in Feb 2021. I'll have to mull over\n> whether your choice of option (2) might cause us some problems if\n> someone (a) uses a new git version to (b) access an old version of our\n> code and (c) really wants to work with a sparse checkout (since the\n> \"checkout-index\" stuff was part of the build logic checked into the\n> code). I think your change here is fine (because not using sparse\n> checkouts is an option, we told folks it was experimental, and those\n> are old versions that are only getting security fixes in special\n> circumstances), but let me think about it for a bit...\n\nI brought this up a day or two ago with my co-workers and discussed\nthe combination of conditions needed to trigger any problems. Their\nresponse was \"meh, it's pretty unlikely that anyone ever hits it, the\nfeature was labeled as experimental anyway, and there are multiple\neasy workarounds for the user to choose from\". So, no need to worry\nabout this angle.\n"},{"id":"445725","messageId":"CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com","threadId":"57180","inReplyTo":"ff3faad1-f5e9-c4db-712d-11f8110d1945@github.com","subject":"Re: [PATCH 5/9] checkout-index: add --ignore-skip-worktree-bits option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-07T16:35:19Z","receivedAt":"2022-01-07T16:35:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 6, 2022 at 7:07 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n...\n> >> +--ignore-skip-worktree-bits::\n> >> + Check out all files, including those with the skip-worktree bit\n> >> + set. Note: may only be used with `--all`; skip-worktree is\n> >> + ignored when explicit filenames are specified.\n> >\n> > Why this restriction? What if the user ran\n> > git checkout-index -- '*.c'\n> > That's not an explicit filename, but a glob.\n> >\n>\n> `checkout-index` doesn't accept globs/pathspecs, so every provided argument\n> must correspond exactly to an entry in the index.\n\nAh, my mistake; thanks for pointing that out. I learned something else new.\n\n> I originally restricted '--ignore-skip-worktree-bits' to only work with\n> '--all' because I wanted changes to the current behavior of `checkout-index`\n> in a sparse checkout to be as minimal as possible. However, if this is more\n> \"unexpected inconsistency\" than \"I'm glad checkout-index with filenames\n> still works the way it did before\", I'm happy to change it.\n\nThat's good context. I'm not sure I have a real strong opinion on\nthis side of things, but digging around I did note that this behavior\nis somewhat inconsistent with checkout/restore:\n\n$ git checkout HEAD sparse-dir/filename\nerror: pathspec 'sparse-dir/filename' did not match any file(s) known to git\n\n$ git ls-files -t sparse-dir/filename\nS sparse-dir/filename\n\n(The error message in checkout/restore might be worth fixing up, since\nit's actually a lie as shown by the `ls-files -t` command below it.\nBut that's obviously outside the scope of this series.)\n\nSo we should probably either require the --ignore-skip-worktree-bits\nflag even with individual paths (which might also slightly reduce my\nobjections to the other items below), or explain in the commit message\nwhy checkout and checkout-index treat paths outside the sparsity\nspecification differently.\n\n> >> +\n> >> --stdin::\n> >> Instead of taking list of paths from the command line,\n> >> read list of paths from the standard input. Paths are\n> >> diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\n> >> index e21620d964e..2053a80103a 100644\n> >> --- a/builtin/checkout-index.c\n> >> +++ b/builtin/checkout-index.c\n> >> @@ -7,6 +7,7 @@\n> >> #define USE_THE_INDEX_COMPATIBILITY_MACROS\n> >> #include \"builtin.h\"\n> >> #include \"config.h\"\n> >> +#include \"dir.h\"\n> >> #include \"lockfile.h\"\n> >> #include \"quote.h\"\n> >> #include \"cache-tree.h\"\n> >> @@ -116,7 +117,7 @@ static int checkout_file(const char *name, const char *prefix)\n> >> return -1;\n> >> }\n> >>\n> >> -static int checkout_all(const char *prefix, int prefix_length)\n> >> +static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n> >> {\n> >> int i, errs = 0;\n> >> struct cache_entry *last_ce = NULL;\n> >> @@ -125,6 +126,8 @@ static int checkout_all(const char *prefix, int prefix_length)\n> >> ensure_full_index(&the_index);\n> >> for (i = 0; i < active_nr ; i++) {\n> >> struct cache_entry *ce = active_cache[i];\n> >> + if (!ignore_skip_worktree && ce_skip_worktree(ce))\n> >> + continue;\n> >\n> > So here I see you let it fall through to the code below that will\n> > write the file to the working tree...but it doesn't clear the\n> > SKIP_WORKTREE bit in the index when it does so, which I think is a\n> > bug.\n> >\n>\n> I disagree, mainly because updating a flag seems inconsistent with how\n> `checkout-index` otherwise works. Specifically, `checkout-index` creates or\n> replaces a file on disk (not even necessarily in the git working directory)\n> based on the file's state in the index, but doesn't modify the index in the\n> process. The only exception is '-u', which is effectively a shortcut for\n> running `git update-index --refresh` [1].\n>\n> If a user wants to to checkout a file *and* update `skip-worktree`, I think\n> explicitly using `update-index` would be a more appropriate way to do that\n> (similar to the example [2] in the `checkout-index` documentation):\n>\n> $ git checkout-index outside-cone/file\n> $ git update-index --no-skip-worktree outside-cone/file\n\nI understand the desire to make a minimal change and only do\noperations the command previously did, but I'm also worried about\nintroducing or exacerbating inconsistencies for users. Matheus'\npatches to grep stalled for nearly a year, in part because of\ncomplications of how to handle sparse-checkouts appropriately in all\ncases[1][2] (with trying to sanely figure out how to sanely handle\npresent-despite-SKIP_WORKTREE files being one of the complications).\nHis rm/add follow-ups also took months because of those kinds of\nissues[3]. We've had to add ugly logic to merge-ort to attempt to\nhandle present-despite-SKIP_WORKTREE files[4], and basically just been\nforced to give up in merge-recursive knowing full well that we'll\nsometimes silently discard user modifications. Despite stash\nessentially being a merge, it needed extra code (beyond what was in\nmerge-ort and merge-recursive) to manually tweak SKIP_WORKTREE bits in\norder to avoid a few different bugs that'd result in an early abort\nwith a partial stash application[5]. But beyond implementation\ncomplexity for other commands to deal with these problematic files, it\nwill also cause problems for users:\n\nSince present-despite-SKIP_WORKTREE files are not reported by status,\ndiff, grep, or anything, and `git sparse-checkout reapply` will NOT\nattempt to remove or notify the user about them, and commands like\nswitch/checkout/pull/merge/rebase etc. will not update them (unless\nthere are conflicts in that path), the contents of these files will\nremain stagnant and likely reflect a version of the file from weeks\nago on a different branch altogether. When users then either change\ntheir sparsity patterns or disable sparse-checkouts, now they see\nfiles with \"all kinds of changes\" (because it was the version of a\nfile from a different branch or commit altogether). What are users\nsupposed to do with that? They have to do something, or else\nswitch/checkout/pull will now start erroring out, and the issue that\ncause the problem for the users is totally disconnected from the\ncommand that eventually gives them errors, potentially by days or\nweeks. It's nearly impossible to help users debug why this happened\nwith such a disconnect.\n\nHowever...I've been complaining about these kinds of problems for well\nover a year, because the only solution I knew of was to try to\nminimize each way we could get into this broken state (of having\npresent-despite-SKIP_WORKTREE files) and ask folks to test and modify\ncommands to do something semi-sane when we unfortunately find\nourselves in this broken state anyway. I've gone round and round with\nMatheus trying to figure them out. And back and forth many times with\nStolee. And now...it feels like there'll be no end. So, I decided to\nbite the bullet and attempt to see just how hard it would be to\nactually detect/notify/correct this broken state for users. I think I\nmay have a clever solution. As long as others don't feel it's too\nheavy handed, it could just obviate all these discussions and all the\nspecial case code and simply the tests significantly and drop my\nobjections to your patch. I'll submit the series later today.\n\n[1] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t\nand the dates on the thread; also Matheus and I had several\nconversations off-list trying to resolve the issues over that time\n[2] ...it finally kind of got unstuck after\nhttps://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[3] See e.g. https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t\nand quotes like \"The core functionality of sparse-checkout has always\nbeen only partially implemented\", a statement I still believe is true\ntoday.\n[4] See commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE\nhandling with conflicted entries\", 2021-03-20)\n[5] See commit ba359fd507 (\"stash: fix stash application in\nsparse-checkouts\", 2020-12-01)\n\n> [1] https://lore.kernel.org/git/7vis1kvqac.fsf@assigned-by-dhcp.cox.net/\n> [2] https://git-scm.com/docs/git-checkout-index#Documentation/git-checkout-index.txt-Toupdateandrefreshonlythefilesalreadycheckedout\n>\n> >> if (ce_stage(ce) != checkout_stage\n> >> && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> >> continue;\n> >> @@ -176,6 +179,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> >> int i;\n> >> struct lock_file lock_file = LOCK_INIT;\n> >> int all = 0;\n> >> + int ignore_skip_worktree = 0;\n> >> int read_from_stdin = 0;\n> >> int prefix_length;\n> >> int force = 0, quiet = 0, not_new = 0;\n> >> @@ -185,6 +189,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> >> struct option builtin_checkout_index_options[] = {\n> >> OPT_BOOL('a', \"all\", &all,\n> >> N_(\"check out all files in the index\")),\n> >> + OPT_BOOL(0, \"ignore-skip-worktree-bits\", &ignore_skip_worktree,\n> >> + N_(\"do not skip files with skip-worktree set\")),\n> >> OPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n> >> OPT__QUIET(&quiet,\n> >> N_(\"no warning for existing files and files not in index\")),\n> >> @@ -247,6 +253,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> >>\n> >> if (all)\n> >> die(\"git checkout-index: don't mix '--all' and explicit filenames\");\n> >> + if (ignore_skip_worktree)\n> >> + die(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n> >> if (read_from_stdin)\n> >> die(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n> >> p = prefix_path(prefix, prefix_length, arg);\n> >> @@ -280,7 +288,7 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> >> }\n> >>\n> >> if (all)\n> >> - err |= checkout_all(prefix, prefix_length);\n> >> + err |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n> >>\n> >> if (pc_workers > 1)\n> >> err |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n> >> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> >> index db7ad41109b..fad61d96107 100755\n> >> --- a/t/t1092-sparse-checkout-compatibility.sh\n> >> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> >> @@ -799,14 +799,14 @@ test_expect_success 'checkout-index with folders' '\n> >> test_all_match test_must_fail git checkout-index -f -- folder1/\n> >> '\n> >>\n> >> -# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n> >> -# files (even those outside the sparse definition) on disk. However, these files\n> >> -# don't appear in the percentage of tracked files in git status.\n> >> -test_expect_failure 'checkout-index --all' '\n> >> +test_expect_success 'checkout-index --all' '\n> >> init_repos &&\n> >>\n> >> test_all_match git checkout-index --all &&\n> >> - test_sparse_match test_path_is_missing folder1\n> >> + test_sparse_match test_path_is_missing folder1 &&\n> >> +\n> >> + test_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n> >> + test_all_match test_path_exists folder1\n> >\n> > I added an 'exit 1' here, ran the test and then checked:\n> >\n> > $ cd trash\\ directory.t1092-sparse-checkout-compatibility/sparse-checkout/\n> > $ git ls-files -t | grep folder1/\n> > S folder1/0/0/0\n> > S folder1/0/1\n> > S folder1/a\n> >\n> > So there's some more work to do on this patch.\n>\n> Unless I'm misreading your comment, this is exactly the behavior I would\n> expect in this test: all files (even those with `skip-worktree` set, per\n> '--ignore-skip-worktree-bits') are created on-disk, with `skip-worktree`\n> unmodified.\n\nIf folks agree with the series I'll submit later today to detect/fix\npresent-despite-SKIP_WORKTREE files, then I'll drop my objection to\nthis behavior. But if folks object to that other series, then I am\nconcerned this behavior gives users a ticking time-bomb without\nsufficient warning. So if that other series isn't acceptable to\nfolks, then I may come back here and argue that either you or I need\nto go back in to checkout-index and make it clear the SKIP_WORKTREE\nbit for any files it writes.\n"},{"id":"445796","messageId":"CABPp-BHadoOToWb6Kp7rUj03ZzKhzK_aFJymT_zApEw8st2ttA@mail.gmail.com","threadId":"57180","inReplyTo":"3b734f89c0f2f473d71a5d3c85faec6d48a0e350.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-08T23:57:48Z","receivedAt":"2022-01-08T23:58:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Introduce tests for a variety of `git update-index` use cases, including\n> performance scenarios.\n\nMakes sense.\n\n> Tests for `update-index add/remove` are specifically\n> focused on how `git stash` uses `git update-index` as a subcommand to\n> prepare for sparse index integration with `stash` in a future series.\n\nThis is possibly a tangent, but I'd rather that if we were trying to\nfix `git stash`, that we instead would do so by making it stop forking\nsubprocesses and having it call internal API instead. See for\nexample, a4031f6dc0 (\"Merge branch 'en/stash-apply-sparse-checkout'\ninto maint\", 2021-02-05) which did this. The fact that it forks so\nmany subprocesses is a bug, and comes from the fact that stash is a\npartial conversion from shell to C. I think the subprocess forking is\npart of the problem that causes issues for sparse-checkouts, as I\nspelled out in patches 2 & 3 of the series I mentioned above.\n\nHowever, that doesn't affect this series.\n\n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Victoria Dye \n> ---\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1092-sparse-checkout-compatibility.sh | 125 +++++++++++++++++++++++\n> 2 files changed, 126 insertions(+)\n>\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index 54f8602f3c1..7dbed330160 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n> test_perf_on_all git blame $SPARSE_CONE/a\n> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> test_perf_on_all git checkout-index -f --all\n> +test_perf_on_all git update-index --add --remove\n>\n> test_done\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 6ecf1f2bf8e..6804ab23a27 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -630,6 +630,131 @@ test_expect_success 'reset with wildcard pathspec' '\n> test_all_match git ls-files -s -- folder1\n> '\n>\n> +test_expect_success 'update-index modify outside sparse definition' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> +\n> + # Create & modify folder1/a\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n> + run_on_all ../edit-contents folder1/a &&\n\nAs I've mentioned to Stolee, I'd rather these were explicitly marked\nas intentionally setting up an erroneous condition.\n\nHowever, that might not matter if my other series gets accepted (the\none I promised to send out yesterday, but then spent all day\nresponding to emails instead). Hopefully I'll send it soon.\n\n> +\n> + # If file has skip-worktree enabled, update-index does not modify the\n> + # index entry\n> + test_sparse_match git update-index folder1/a &&\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_must_be_empty sparse-checkout-out &&\n> +\n> + # When skip-worktree is disabled (even on files outside sparse cone), file\n> + # is updated in the index\n> + test_sparse_match git update-index --no-skip-worktree folder1/a &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_all_match git update-index folder1/a &&\n> + test_all_match git status --porcelain=v2\n\nThese make sense.\n\n> +'\n> +\n> +test_expect_success 'update-index --add outside sparse definition' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n> +\n> + # Create folder1, add new file\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_all ../edit-contents folder1/b &&\n> +\n> + # Similar to `git add`, the untracked out-of-cone file is added to the index\n> + # identically across sparse and non-sparse checkouts\n> + test_all_match git update-index --add folder1/b &&\n> + test_all_match git status --porcelain=v2\n\nThe comment is not correct:\n\n$ git add out-of-cone/file\nThe following paths and/or pathspecs matched paths that exist\noutside of your sparse-checkout definition, so will not be\nupdated in the index:\nout-of-cone/file\nhint: If you intend to update such entries, try one of the following:\nhint: * Use the --sparse option.\nhint: * Disable or modify the sparsity rules.\nhint: Disable this message with \"git config advice.updateSparsePath false\"\n\nI might buy that `git update-index` is lower level and should be\nconsidered the same as `git add --sparse`, but the comment should\nmention that and try to sell why update-index should be equivalent to\nthat instead of to `git add`.\n\n> +'\n> +\n> +test_expect_success 'update-index --remove outside sparse definition' '\n> + init_repos &&\n\n> +\n> + # When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n> + # not removed from the index if they do not exist on disk\n> + test_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n> + test_all_match git status --porcelain=v2 &&\n\nThe file is present despite being marked to be missing, we're ignoring\nthe intention of the marking, and we ask for it to be removed, so we\ndon't remove it?\n\nThe levels of negation are _very_ confusing. It took me a while to\nunravel this. I think the logic is something like this\n * folder1/a is marked as SKIP_WORKTREE, meaning it's not supposed to\nbe in the worktree.\n * and it's not.\n * We are stating that we're ignoring SKIP_WORKTREE, though, so this\nlooks like a regular file that has been deleted.\nSo, what would `git update-index --remove $FILE` do for a normal $FILE\ndeleted from the working copy? According to the docs:\n\n --remove\n If a specified file is in the index but is missing then it’s\n removed. Default behavior is to ignore removed file.\n\nSo, the docs say it would remove it. But you don't.\n\n\nAfter digging around and looking at the testcase below, if I had to\nguess what happened, I would say that you figured out what the\nSKIP_WORKTREE behavior was, and assumed that was correct, and added a\nflag that allowed you to request the opposite behavior.\nUnfortunately, I think the pre-existing behavior is buggy.\n\nOf course, there's lots of negation here. Did I get something\nbackwards by chance?\n\n> +\n> + # When the flag is _not_ specified ...\n\nIn my head I'm translating this as:\n\nSKIP_WORKTREE = !worktree\n--ignore-skip-worktree-entries = !!worktree\n\"flag is _not_ specified\" = !!!worktree ?\n\nI'm not sure I would do anything to change it, but just pointing out\nit can be a little hard for others to come up to speed.\n\n> ... , out-of-cone, not-on-disk files are\n> + # removed from the index\n> + rm full-checkout/folder1/a &&\n> + test_all_match git update-index --remove folder1/a &&\n> + test_all_match git status --porcelain=v2 &&\n\nDocumentation/git-update-index.txt defines SKIP_WORKTREE as follows:\n\n\"Skip-worktree bit can be defined in one (long) sentence: When reading\nan entry, if it is marked as skip-worktree, then Git pretends its\nworking directory version is up to date and read the index version\ninstead.\"\n\nIf Git is pretending the file is up-to-date, and `git update-index\n--remove $UP_TO_DATE_FILE` is typically a no-op because the --remove\nflag doesn't do anything when a file is present in the working copy,\nthen why is this the expected behavior?\n\nI know it's the traditional behavior of update-index, but\nSKIP_WORKTREE support in Git has traditionally been filled with holes.\nSo, was this behavior by design (despite contradicting the\ndocumentation), or by accident?\n\n(To be fair, I think the definition given in the manual for\nSKIP_WORKTREE is horrible for other reasons, so I don't like leaning\non it. But I used different logic above in the\n--ignore-skip-worktree-entries case to arrive at the same conclusion\nthat the --remove behavior of update-index seems to be backwards.\nUnless I missed a negation in both cases somewhere? There are so many\nfloating around...)\n\n> + # NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n> + # a skip-worktree file from the index (and disk) when both are specified\n> + test_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n> + test_all_match git status --porcelain=v2\n\nMakes sense.\n\n> +'\n> +\n> +test_expect_success 'update-index with directories' '\n> + init_repos &&\n> +\n> + # update-index will exit silently when provided with a directory name\n> + # containing a trailing slash\n> + test_all_match git update-index deep/ folder1/ &&\n> + grep \"Ignoring path deep/\" sparse-checkout-err &&\n> + grep \"Ignoring path folder1/\" sparse-checkout-err &&\n\nIs this desired behavior or just current behavior?\n\n> +\n> + # When update-index is given a directory name WITHOUT a trailing slash, it will\n> + # behave in different ways depending on the status of the directory on disk:\n> + # * if it exists, the command exits with an error (\"add individual files instead\")\n> + # * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n> + # file and either triggers an error (\"does not exist and --remove not passed\")\n> + # or is ignored completely (when using --remove)\n> + test_all_match test_must_fail git update-index deep &&\n> + run_on_all test_must_fail git update-indexe folder1 &&\n\nThis one will fail for the wrong reason, though -- `update-indexe` is\nnot a valid subcommand. (extra 'e' at the end)\n\n> + test_must_fail git -C full-checkout update-index --remove folder1 &&\n> + test_sparse_match git update-index --remove folder1 &&\n> + test_all_match git status --porcelain=v2\n\nOtherwise these seem reasonable.\n\n> +'\n> +\n> +test_expect_success 'update-index --again file outside sparse definition' '\n> + init_repos &&\n> +\n> + write_script edit-contents <<-\\EOF &&\n> + echo text >>$1\n> + EOF\n\nCopy and paste and forget to remove? edit-contents doesn't seem to be\nused in this test.\n\n> +\n> + test_all_match git checkout -b test-reupdate &&\n> +\n> + # Update HEAD without modifying the index to introduce a difference in\n> + # folder1/a\n> + test_sparse_match git reset --soft update-folder1 &&\n> +\n> + # Because folder1/a differs in the index vs HEAD,\n> + # `git update-index --remove --again` will effectively perform\n> + # `git update-index --remove folder1/a` and remove the folder1/a\n> + test_sparse_match git update-index --remove --again &&\n> + test_sparse_match git status --porcelain=v2\n\nThis might need a --ignore-skip-worktree-entries, as per the\ndiscussion above. Otherwise, this test makes sense.\n\n> +'\n> +\n> +test_expect_success 'update-index --cacheinfo' '\n> + init_repos &&\n> +\n> + deep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n> + folder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n> + folder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n> +\n> + test_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + # Cannot add sparse directory, even in sparse index case\n> + test_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n> +\n> + # Sparse match only - because folder1/a is outside the sparse checkout\n> + # definition (and thus not on-disk), it will appear \"deleted\" in\n> + # unstaged changes.\n> + test_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n> + test_sparse_match git status --porcelain=v2\n\nMakes sense, because the update-index command removes the existing\ncache entry and adds a new one without the SKIP_WORKTREE bit. But it\nmight be worth mentioning that in the commit message. Also, you could\nfollow this up with `git update-index --skip-worktree folder1/a`, and\nthen do a test_all_match git status --porcelain=v2, to show that when\nthe SKIP_WORKTREE bit is restored back to the file, then it again does\nas expected despite not being on-disk.\n\n> +'\n> +\n> test_expect_success 'merge, cherry-pick, and rebase' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n"},{"id":"445798","messageId":"CABPp-BGmaKsvrO1Qxwuj+WgcvuE_NA4agY_HLQ8=csvddRafqw@mail.gmail.com","threadId":"57180","inReplyTo":"c5b98e36516b8f832576816c202be7581f261909.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/9] update-index: integrate with sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T01:49:16Z","receivedAt":"2022-01-09T01:49:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Enable usage of the sparse index with `update-index`. Most variations of\n> `update-index` work without explicitly expanding the index or making any\n> other updates in or outside of `update-index.c`.\n>\n> The one usage requiring additional changes is `--cacheinfo`; if a file\n> inside a sparse directory was specified, the index would not be expanded\n> until after the cache tree is invalidated, leading to a mismatch between the\n> index and cache tree. This scenario is handled by rearranging\n> `add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the\n> index *before* attempting to invalidate the relevant cache tree path,\n> avoiding cache tree/index corruption.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/update-index.c | 3 +++\n> read-cache.c | 10 +++++++---\n> t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n> 3 files changed, 22 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 187203e8bb5..605cc693bbd 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -1077,6 +1077,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>\n> git_config(git_default_config, NULL);\n>\n> + prepare_repo_settings(r);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> /* we will diagnose later if it turns out that we need to update it */\n> newfd = hold_locked_index(&lock_file, 0);\n> if (newfd < 0)\n> diff --git a/read-cache.c b/read-cache.c\n> index cbe73f14e5e..b4600e954b6 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1339,9 +1339,6 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> int skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n> int new_only = option & ADD_CACHE_NEW_ONLY;\n>\n> - if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n> - cache_tree_invalidate_path(istate, ce->name);\n> -\n> /*\n> * If this entry's path sorts after the last entry in the index,\n> * we can avoid searching for it.\n> @@ -1352,6 +1349,13 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> else\n> pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n>\n> + /*\n> + * Cache tree path should be invalidated only after index_name_stage_pos,\n> + * in case it expands a sparse index.\n> + */\n> + if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n> + cache_tree_invalidate_path(istate, ce->name);\n> +\n> /* existing match? Just replace it. */\n> if (pos >= 0) {\n> if (!new_only)\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 6804ab23a27..bc0741c970d 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1216,6 +1216,18 @@ test_expect_success 'sparse index is not expanded: blame' '\n> done\n> '\n>\n> +test_expect_success 'sparse index is not expanded: update-index' '\n> + init_repos &&\n> +\n> + echo \"test\" >sparse-index/README.md &&\n> + echo \"test2\" >sparse-index/a &&\n> + rm -f sparse-index/deep/a &&\n> +\n> + ensure_not_expanded update-index --add README.md &&\n> + ensure_not_expanded update-index a &&\n> + ensure_not_expanded update-index --remove deep/a\n> +'\n\nThe commit message said this change was about --cacheinfo, but this\ntest doesn't use that option. I'm confused; was this a bad patch\nsplitting by chance?\n\n> +\n> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> # in this scenario, but it shouldn't.\n> test_expect_success 'reset mixed and checkout orphan' '\n> --\n> gitgitgadget\n>\n"},{"id":"445800","messageId":"CABPp-BEmT-2+G_PRvhCFkzOc4oP+_5--ESz9=dHAO7gzDsHUOw@mail.gmail.com","threadId":"57180","inReplyTo":"de7fc14356231a60c8cac9aa6f92a7fec1560b6a.1641317820.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 9/9] update-index: reduce scope of index expansion in do_reupdate","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:24:49Z","receivedAt":"2022-01-09T04:25:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Expand the full index (and redo reupdate operation) only if a sparse\n> directory in the index differs from HEAD.\n\nThis was a bit hard to parse and follow for me. Perhaps:\n\nAvoid pre-emptively expanding to a full index in update-index's do_reupdate().\nHowever, if a sparse directory in the index differs from HEAD, we will need to\nthen expand the index and restart the operation.\n\n> Only the index entries that differ\n> between the index and HEAD are updated when performing `git update-index\n> --again`, so unmodified sparse directories are safely skipped. The index\n> does need to be expanded when sparse directories contain changes, though,\n> because `update_one(...)` will not operate on sparse directory index\n> entries.\n>\n> Because the index should only be expanded if a sparse directory is modified,\n> add a test ensuring the index is not expanded when differences only exist\n> within the sparse cone.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/update-index.c | 14 +++++++++++---\n> t/t1092-sparse-checkout-compatibility.sh | 5 ++++-\n> 2 files changed, 15 insertions(+), 4 deletions(-)\n>\n> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> index 605cc693bbd..52ecc714d99 100644\n> --- a/builtin/update-index.c\n> +++ b/builtin/update-index.c\n> @@ -606,7 +606,7 @@ static struct cache_entry *read_one_ent(const char *which,\n> error(\"%s: not in %s branch.\", path, which);\n> return NULL;\n> }\n> - if (mode == S_IFDIR) {\n> + if (!the_index.sparse_index && mode == S_IFDIR) {\n> if (which)\n> error(\"%s: not a blob in %s branch.\", path, which);\n> return NULL;\n> @@ -743,8 +743,6 @@ static int do_reupdate(int ac, const char **av,\n> */\n> has_head = 0;\n> redo:\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(&the_index);\n> for (pos = 0; pos < active_nr; pos++) {\n> const struct cache_entry *ce = active_cache[pos];\n> struct cache_entry *old = NULL;\n> @@ -761,6 +759,16 @@ static int do_reupdate(int ac, const char **av,\n> discard_cache_entry(old);\n> continue; /* unchanged */\n> }\n> +\n> + /* At this point, we know the contents of the sparse directory are\n> + * modified with respect to HEAD, so we expand the index and restart\n> + * to process each path individually\n> + */\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + ensure_full_index(&the_index);\n> + goto redo;\n> + }\n> +\n> /* Be careful. The working tree may not have the\n> * path anymore, in which case, under 'allow_remove',\n> * or worse yet 'allow_replace', active_nr may decrease.\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index bc0741c970d..0863c9747c4 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1225,7 +1225,10 @@ test_expect_success 'sparse index is not expanded: update-index' '\n>\n> ensure_not_expanded update-index --add README.md &&\n> ensure_not_expanded update-index a &&\n> - ensure_not_expanded update-index --remove deep/a\n> + ensure_not_expanded update-index --remove deep/a &&\n> +\n> + ensure_not_expanded reset --soft update-deep &&\n> + ensure_not_expanded update-index --add --remove --again\n> '\n>\n> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> --\n> gitgitgadget\n\nI think the rest makes sense. Thanks for working on these!\n"},{"id":"445801","messageId":"CABPp-BEiUKFW3XCZUgTMG2K65NpCeDruPkvzgsd_=ch+t3t=-g@mail.gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:41:57Z","receivedAt":"2022-01-09T04:42:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> This series continues the work to integrate commands with the sparse index,\n> adding integrations with 'git clean', 'git checkout-index', and 'git\n> update-index'. These three commands, while useful in their own right, are\n> updated mainly because they're used in 'git stash'. A future series will\n> integrate sparse index with 'stash' directly, but its subcommands must be\n> integrated to avoid the performance cost of each one expanding and\n> collapsing the index.\n\nThanks for working on these; it's very nice to see others continuing\nto work on sparse-checkout issues, since it still feels somewhat\nincomplete to me. This work is perhaps less glamorous since you're\nstarting to work on commands less frequently used by users, but I\nstill feel it's very important. Thanks for your diligence.\n\n> The series is broken up into 4 parts:\n>\n> * Patches 1-2 are minor fixups to the 'git reset' sparse index integration\n> in response to discussion [1] that came after the series was ready for\n> merge to 'next'.\n\nThese look good.\n\n> * Patch 3 integrates 'git clean' with the sparse index.\n\nYeah, since clean tends to work with untracked files instead of\ntracked, and SKIP_WORKTREE is all about tracked files, I'd expect the\nconversion would be simple. Nice to see verification that it was.\n\n> * Patches 4-6 integrate 'git checkout-index' with the sparse index and\n> introduce a new '--ignore-skip-worktree-bits' option for use with 'git\n> checkout-index --all'.\n> * This involves changing the behavior of '--all' to respect\n> 'skip-worktree' by default (i.e., it won't check out 'skip-worktree'\n> files). The '--ignore-skip-worktree-bits' option can be specified to\n> force checkout of 'skip-worktree' files, if desired.\n\nI think the basic idea makes sense, and the patches are certainly\nmoving in the right direction.\n\nIf others agree with the series I'm about to submit, then patch 4\nlooks fine as-is and there's just a minor item about\n--ignore-skip-worktree-bits without --all for checkout-index in Patch\n5 (either further behavior changes, or a more detailed commit\nmessage). If others disagree with that other series, then I think\nboth patches 4 & 5 should do more work to avoid problems with\npresent-despite-SKIP_WORKTREE files.\n\nHowever, the issues raised in Patch 7 suggest that Patch 5 might need\nsome reworking (independent of the other series I'm about to submit).\n\nPatch 6 looks good either way.\n\n> * Patches 7-9 integrate 'git update-index' with the sparse index.\n> * Note that, although this integrates the sparse index with\n> '--cacheinfo', sparse directories still cannot be updated using that\n> option (see the prior discussion [2] for more details on why)\n\nPatch 9 might benefit from a small commit message rewording.\n\nI either misunderstood Patch 8, or it has a disconnect of some kind\nbetween the commit message and the testcase.\n\nI've left various comments on patch 7; it'll probably need updates and\nlikely points to further changes needed in patch 5.\n\n\nAnyway, I tend to concentrate on perceived issues when reviewing\nrather than on what looks good, but despite the fact that I think\nthere are some further improvements needed, overall I think this looks\nlike some nice work. Keep it up!\n"},{"id":"445831","messageId":"556f6266-a79f-1d5c-6bd7-79d65223b403@github.com","threadId":"57180","inReplyTo":"CABPp-BGmaKsvrO1Qxwuj+WgcvuE_NA4agY_HLQ8=csvddRafqw@mail.gmail.com","subject":"Re: [PATCH 8/9] update-index: integrate with sparse index","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-10T14:10:19Z","receivedAt":"2022-01-10T14:10:23Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Enable usage of the sparse index with `update-index`. Most variations of\n>> `update-index` work without explicitly expanding the index or making any\n>> other updates in or outside of `update-index.c`.\n>>\n>> The one usage requiring additional changes is `--cacheinfo`; if a file\n>> inside a sparse directory was specified, the index would not be expanded\n>> until after the cache tree is invalidated, leading to a mismatch between the\n>> index and cache tree. This scenario is handled by rearranging\n>> `add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the\n>> index *before* attempting to invalidate the relevant cache tree path,\n>> avoiding cache tree/index corruption.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/update-index.c | 3 +++\n>> read-cache.c | 10 +++++++---\n>> t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n>> 3 files changed, 22 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/builtin/update-index.c b/builtin/update-index.c\n>> index 187203e8bb5..605cc693bbd 100644\n>> --- a/builtin/update-index.c\n>> +++ b/builtin/update-index.c\n>> @@ -1077,6 +1077,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n>>\n>> git_config(git_default_config, NULL);\n>>\n>> + prepare_repo_settings(r);\n>> + the_repository->settings.command_requires_full_index = 0;\n>> +\n>> /* we will diagnose later if it turns out that we need to update it */\n>> newfd = hold_locked_index(&lock_file, 0);\n>> if (newfd < 0)\n>> diff --git a/read-cache.c b/read-cache.c\n>> index cbe73f14e5e..b4600e954b6 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -1339,9 +1339,6 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n>> int skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n>> int new_only = option & ADD_CACHE_NEW_ONLY;\n>>\n>> - if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n>> - cache_tree_invalidate_path(istate, ce->name);\n>> -\n>> /*\n>> * If this entry's path sorts after the last entry in the index,\n>> * we can avoid searching for it.\n>> @@ -1352,6 +1349,13 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n>> else\n>> pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n>>\n>> + /*\n>> + * Cache tree path should be invalidated only after index_name_stage_pos,\n>> + * in case it expands a sparse index.\n>> + */\n>> + if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n>> + cache_tree_invalidate_path(istate, ce->name);\n>> +\n>> /* existing match? Just replace it. */\n>> if (pos >= 0) {\n>> if (!new_only)\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 6804ab23a27..bc0741c970d 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -1216,6 +1216,18 @@ test_expect_success 'sparse index is not expanded: blame' '\n>> done\n>> '\n>>\n>> +test_expect_success 'sparse index is not expanded: update-index' '\n>> + init_repos &&\n>> +\n>> + echo \"test\" >sparse-index/README.md &&\n>> + echo \"test2\" >sparse-index/a &&\n>> + rm -f sparse-index/deep/a &&\n>> +\n>> + ensure_not_expanded update-index --add README.md &&\n>> + ensure_not_expanded update-index a &&\n>> + ensure_not_expanded update-index --remove deep/a\n>> +'\n> \n> The commit message said this change was about --cacheinfo, but this\n> test doesn't use that option. I'm confused; was this a bad patch\n> splitting by chance?\n> \n\nIt was not - the commit message title is \"update-index: integrate with\nsparse index\" and the message starts by saying that this patch enables use\nof the sparse index for *all* of `update-index` (where \"[m]ost variations of\n`update-index` work without...making any other updates in or outside of\n`update-index.c`\").\n\nIt goes on to say that the `--cache-info` option is an exception to the\nabove statement (that most variations work without updates outside\n`update-index.c`) because it requires a slight change to\n`add_index_entry_with_check(...)` to avoid index corruption. That change is\nalso in this patch, but it's not the main \"focus\".\n\nThe test added here is intended to broadly test `update-index` with a sparse\nindex. I'll add a case for `--cacheinfo` in my next re-roll but, for\ncontext, the reason I didn't originally is because I focused on the (as far\nas I could tell) most commonly-used variations of `update-index`.\n\n>> +\n>> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n>> # in this scenario, but it shouldn't.\n>> test_expect_success 'reset mixed and checkout orphan' '\n>> --\n>> gitgitgadget\n>>\n\n"},{"id":"445833","messageId":"159a35ba-7ed0-c601-15bd-54dfda460323@github.com","threadId":"57180","inReplyTo":"CABPp-BHadoOToWb6Kp7rUj03ZzKhzK_aFJymT_zApEw8st2ttA@mail.gmail.com","subject":"Re: [PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-10T15:47:35Z","receivedAt":"2022-01-10T15:47:41Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Introduce tests for a variety of `git update-index` use cases, including\n>> performance scenarios.\n> \n> Makes sense.\n> \n>> Tests for `update-index add/remove` are specifically\n>> focused on how `git stash` uses `git update-index` as a subcommand to\n>> prepare for sparse index integration with `stash` in a future series.\n> \n> This is possibly a tangent, but I'd rather that if we were trying to\n> fix `git stash`, that we instead would do so by making it stop forking\n> subprocesses and having it call internal API instead. See for\n> example, a4031f6dc0 (\"Merge branch 'en/stash-apply-sparse-checkout'\n> into maint\", 2021-02-05) which did this. The fact that it forks so\n> many subprocesses is a bug, and comes from the fact that stash is a\n> partial conversion from shell to C. I think the subprocess forking is\n> part of the problem that causes issues for sparse-checkouts, as I\n> spelled out in patches 2 & 3 of the series I mentioned above.\n> \n> However, that doesn't affect this series.\n> \n>> Co-authored-by: Derrick Stolee \n>> Signed-off-by: Victoria Dye \n>> ---\n>> t/perf/p2000-sparse-operations.sh | 1 +\n>> t/t1092-sparse-checkout-compatibility.sh | 125 +++++++++++++++++++++++\n>> 2 files changed, 126 insertions(+)\n>>\n>> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n>> index 54f8602f3c1..7dbed330160 100755\n>> --- a/t/perf/p2000-sparse-operations.sh\n>> +++ b/t/perf/p2000-sparse-operations.sh\n>> @@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n>> test_perf_on_all git blame $SPARSE_CONE/a\n>> test_perf_on_all git blame $SPARSE_CONE/f3/a\n>> test_perf_on_all git checkout-index -f --all\n>> +test_perf_on_all git update-index --add --remove\n>>\n>> test_done\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 6ecf1f2bf8e..6804ab23a27 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -630,6 +630,131 @@ test_expect_success 'reset with wildcard pathspec' '\n>> test_all_match git ls-files -s -- folder1\n>> '\n>>\n>> +test_expect_success 'update-index modify outside sparse definition' '\n>> + init_repos &&\n>> +\n>> + write_script edit-contents <<-\\EOF &&\n>> + echo text >>$1\n>> + EOF\n>> +\n>> + # Create & modify folder1/a\n>> + run_on_sparse mkdir -p folder1 &&\n>> + run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n>> + run_on_all ../edit-contents folder1/a &&\n> \n> As I've mentioned to Stolee, I'd rather these were explicitly marked\n> as intentionally setting up an erroneous condition.\n> \n> However, that might not matter if my other series gets accepted (the\n> one I promised to send out yesterday, but then spent all day\n> responding to emails instead). Hopefully I'll send it soon.\n> \n>> +\n>> + # If file has skip-worktree enabled, update-index does not modify the\n>> + # index entry\n>> + test_sparse_match git update-index folder1/a &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_must_be_empty sparse-checkout-out &&\n>> +\n>> + # When skip-worktree is disabled (even on files outside sparse cone), file\n>> + # is updated in the index\n>> + test_sparse_match git update-index --no-skip-worktree folder1/a &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_all_match git update-index folder1/a &&\n>> + test_all_match git status --porcelain=v2\n> \n> These make sense.\n> \n>> +'\n>> +\n>> +test_expect_success 'update-index --add outside sparse definition' '\n>> + init_repos &&\n>> +\n>> + write_script edit-contents <<-\\EOF &&\n>> + echo text >>$1\n>> + EOF\n>> +\n>> + # Create folder1, add new file\n>> + run_on_sparse mkdir -p folder1 &&\n>> + run_on_all ../edit-contents folder1/b &&\n>> +\n>> + # Similar to `git add`, the untracked out-of-cone file is added to the index\n>> + # identically across sparse and non-sparse checkouts\n>> + test_all_match git update-index --add folder1/b &&\n>> + test_all_match git status --porcelain=v2\n> \n> The comment is not correct:\n> \n\nIt is correct, but for *untracked* out-of-cone files only. Those files don't\nhave a `skip-worktree` bit because they're not in the index in the first\nplace. The comment is intended to highlight the fact that `update-index`\n(like `git add`, `git status`, etc.) \"decides\" whether to operate on a file\nin a sparse-checkout based on `skip-worktree`, *not* the sparse patterns.\n\nSeeing as the comparison to `git add` makes things more confusing, I'll\nrephrase the test comment.\n\n> $ git add out-of-cone/file\n> The following paths and/or pathspecs matched paths that exist\n> outside of your sparse-checkout definition, so will not be\n> updated in the index:\n> out-of-cone/file\n> hint: If you intend to update such entries, try one of the following:\n> hint: * Use the --sparse option.\n> hint: * Disable or modify the sparsity rules.\n> hint: Disable this message with \"git config advice.updateSparsePath false\"\n> \n> I might buy that `git update-index` is lower level and should be\n> considered the same as `git add --sparse`, but the comment should\n> mention that and try to sell why update-index should be equivalent to\n> that instead of to `git add`.\n> \n\nTracked, out-of-cone files aren't affected by `--add` (the flag allows\n`update-index` to add untracked files), and `update-index out-of-cone/tracked`\nwould ignore the file, so I believe the behavior of `update-index` is\ncurrently more consistent with `git add` than `git add --sparse`.\n\n>> +'\n>> +\n>> +test_expect_success 'update-index --remove outside sparse definition' '\n>> + init_repos &&\n> \n>> +\n>> + # When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n>> + # not removed from the index if they do not exist on disk\n>> + test_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n>> + test_all_match git status --porcelain=v2 &&\n> \n> The file is present despite being marked to be missing, we're ignoring\n> the intention of the marking, and we ask for it to be removed, so we\n> don't remove it?\n> \n> The levels of negation are _very_ confusing. It took me a while to\n> unravel this. I think the logic is something like this\n> * folder1/a is marked as SKIP_WORKTREE, meaning it's not supposed to\n> be in the worktree.\n> * and it's not.\n> * We are stating that we're ignoring SKIP_WORKTREE, though, so this\n> looks like a regular file that has been deleted.\n> So, what would `git update-index --remove $FILE` do for a normal $FILE\n> deleted from the working copy? According to the docs:\n> \n> --remove\n> If a specified file is in the index but is missing then it’s\n> removed. Default behavior is to ignore removed file.\n> \n> So, the docs say it would remove it. But you don't.\n> \n> \n> After digging around and looking at the testcase below, if I had to\n> guess what happened, I would say that you figured out what the\n> SKIP_WORKTREE behavior was, and assumed that was correct, and added a\n> flag that allowed you to request the opposite behavior.\n> Unfortunately, I think the pre-existing behavior is buggy.\n> \n\nI understand why you find it buggy, but I am not making baseless assumptions\nabout the correctness (or lack thereof) of the current behavior. This\nspecific \"gap\" in `update-index --remove` has been discussed in the past [1]\nand was acknowledged as non-ideal and a candidate for change in the future.\nAt the time, the introduction of `--ignore-skip-worktree-entries` [2] was a\n\"safe\" way to ignore `skip-worktree` without changing the default behavior.\n\nPersonally, I think updating the default behavior of `--remove` (and\ncorresponding deprecation of `--[no-]ignore-skip-worktree-entries`) is\nprobably the right way to go. However, I'd like to avoid including it in\nthis series because it deviates pretty substantially from the goal\n\"integrate with sparse index\", and as a result has the potential to\novershadow/derail the rest of the series. If you're alright with (slightly)\ndeferring changes to the behavior of `--remove`, I can submit a separate\nseries for it once this one has stabilized.\n\n[1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n[2] https://lore.kernel.org/git/163b42dfa21c306dc1dc573c5edfc8bda5c99fd0.1572432578.git.gitgitgadget@gmail.com/\n\n> Of course, there's lots of negation here. Did I get something\n> backwards by chance?\n> \n>> +\n>> + # When the flag is _not_ specified ...\n> \n> In my head I'm translating this as:\n> \n> SKIP_WORKTREE = !worktree\n> --ignore-skip-worktree-entries = !!worktree\n> \"flag is _not_ specified\" = !!!worktree ?\n> \n> I'm not sure I would do anything to change it, but just pointing out\n> it can be a little hard for others to come up to speed.\n> \n\nMost of the confusion likely comes from the non-standard behavior of\n`--remove`, but I think I can distill it into (somewhat) straightforward\nstatements about `update-index`:\n\n1. When using the command *without* either `--ignore-skip-worktree-entries`\n OR `--remove`, `skip-worktree` entries provided to the command are\n ignored.\n2. When using the command *with* `--remove` and *without*\n `--ignore-skip-worktree-entries`, `skip-worktree` entries are *not*\n ignored, and are removed from the index.\n3. When both `--remove` and `--ignore-skip-worktree-entries` are specified,\n `skip-worktree` entries are again ignored.\n4. `--ignore-skip-worktree-entries` has no effect without `--remove` also\n specified\n\nThe goal of this test, then, is to exercise conditions 2 & 3 and directly\nshow their effect on `skip-worktree` entries.\n\n>> ... , out-of-cone, not-on-disk files are\n>> + # removed from the index\n>> + rm full-checkout/folder1/a &&\n>> + test_all_match git update-index --remove folder1/a &&\n>> + test_all_match git status --porcelain=v2 &&\n> \n> Documentation/git-update-index.txt defines SKIP_WORKTREE as follows:\n> \n> \"Skip-worktree bit can be defined in one (long) sentence: When reading\n> an entry, if it is marked as skip-worktree, then Git pretends its\n> working directory version is up to date and read the index version\n> instead.\"\n> \n> If Git is pretending the file is up-to-date, and `git update-index\n> --remove $UP_TO_DATE_FILE` is typically a no-op because the --remove\n> flag doesn't do anything when a file is present in the working copy,\n> then why is this the expected behavior?\n> \n> I know it's the traditional behavior of update-index, but\n> SKIP_WORKTREE support in Git has traditionally been filled with holes.\n> So, was this behavior by design (despite contradicting the\n> documentation), or by accident?\n> \n\nAs far as I can tell, it appears to have been intentional in the original\n`skip-worktree` implementation [3], but given Junio & Johannes' discussion\non the `--ignore-skip-worktree-entries` patch [1], the sentiment now would\nprobably lean towards having `--remove` ignore `skip-worktree`.\n\n[1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n (copied from earlier in this message)\n[3] https://lore.kernel.org/git/1250776033-12395-5-git-send-email-pclouds@gmail.com/\n\n\n> (To be fair, I think the definition given in the manual for\n> SKIP_WORKTREE is horrible for other reasons, so I don't like leaning\n> on it. But I used different logic above in the\n> --ignore-skip-worktree-entries case to arrive at the same conclusion\n> that the --remove behavior of update-index seems to be backwards.\n> Unless I missed a negation in both cases somewhere? There are so many\n> floating around...)\n> \n>> + # NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n>> + # a skip-worktree file from the index (and disk) when both are specified\n>> + test_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n>> + test_all_match git status --porcelain=v2\n> \n> Makes sense.\n> \n>> +'\n>> +\n>> +test_expect_success 'update-index with directories' '\n>> + init_repos &&\n>> +\n>> + # update-index will exit silently when provided with a directory name\n>> + # containing a trailing slash\n>> + test_all_match git update-index deep/ folder1/ &&\n>> + grep \"Ignoring path deep/\" sparse-checkout-err &&\n>> + grep \"Ignoring path folder1/\" sparse-checkout-err &&\n> \n> Is this desired behavior or just current behavior?\n> \n>> +\n>> + # When update-index is given a directory name WITHOUT a trailing slash, it will\n>> + # behave in different ways depending on the status of the directory on disk:\n>> + # * if it exists, the command exits with an error (\"add individual files instead\")\n>> + # * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n>> + # file and either triggers an error (\"does not exist and --remove not passed\")\n>> + # or is ignored completely (when using --remove)\n>> + test_all_match test_must_fail git update-index deep &&\n>> + run_on_all test_must_fail git update-indexe folder1 &&\n> \n> This one will fail for the wrong reason, though -- `update-indexe` is\n> not a valid subcommand. (extra 'e' at the end)\n> \n\nThanks for catching that! I'll update in my next re-roll.\n\n>> + test_must_fail git -C full-checkout update-index --remove folder1 &&\n>> + test_sparse_match git update-index --remove folder1 &&\n>> + test_all_match git status --porcelain=v2\n> \n> Otherwise these seem reasonable.\n> \n>> +'\n>> +\n>> +test_expect_success 'update-index --again file outside sparse definition' '\n>> + init_repos &&\n>> +\n>> + write_script edit-contents <<-\\EOF &&\n>> + echo text >>$1\n>> + EOF\n> \n> Copy and paste and forget to remove? edit-contents doesn't seem to be\n> used in this test.\n> \n\nWill remove.\n\n>> +\n>> + test_all_match git checkout -b test-reupdate &&\n>> +\n>> + # Update HEAD without modifying the index to introduce a difference in\n>> + # folder1/a\n>> + test_sparse_match git reset --soft update-folder1 &&\n>> +\n>> + # Because folder1/a differs in the index vs HEAD,\n>> + # `git update-index --remove --again` will effectively perform\n>> + # `git update-index --remove folder1/a` and remove the folder1/a\n>> + test_sparse_match git update-index --remove --again &&\n>> + test_sparse_match git status --porcelain=v2\n> \n> This might need a --ignore-skip-worktree-entries, as per the\n> discussion above. Otherwise, this test makes sense.\n> \n\nThe `--ignore-skip-worktree-entries` option is explicitly omitted because\nthis test needs `update-index` to modify a `skip-worktree` entry. However,\ngiven the debate around what `--remove` should do, I'll update the scenario\nto not use `--remove` or any variation of it.\n\n>> +'\n>> +\n>> +test_expect_success 'update-index --cacheinfo' '\n>> + init_repos &&\n>> +\n>> + deep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n>> + folder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n>> + folder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n>> +\n>> + test_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n>> + test_all_match git status --porcelain=v2 &&\n>> +\n>> + # Cannot add sparse directory, even in sparse index case\n>> + test_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n>> +\n>> + # Sparse match only - because folder1/a is outside the sparse checkout\n>> + # definition (and thus not on-disk), it will appear \"deleted\" in\n>> + # unstaged changes.\n>> + test_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n>> + test_sparse_match git status --porcelain=v2\n> \n> Makes sense, because the update-index command removes the existing\n> cache entry and adds a new one without the SKIP_WORKTREE bit. But it\n> might be worth mentioning that in the commit message. Also, you could\n> follow this up with `git update-index --skip-worktree folder1/a`, and\n> then do a test_all_match git status --porcelain=v2, to show that when\n> the SKIP_WORKTREE bit is restored back to the file, then it again does\n> as expected despite not being on-disk.\n> \n\nI really like this - it helps clarify how `update-index` can be used to\ncorrectly add a sparse-checkout entry to the index with plumbing commands.\nI'll include it in V2.\n\n>> +'\n>> +\n>> test_expect_success 'merge, cherry-pick, and rebase' '\n>> init_repos &&\n>>\n>> --\n>> gitgitgadget\n\n"},{"id":"445834","messageId":"CABPp-BHi0N31TQY8Bt8dq6aWGNdOSFQuQPQ7TtqObZx+iwS_mQ@mail.gmail.com","threadId":"57180","inReplyTo":"556f6266-a79f-1d5c-6bd7-79d65223b403@github.com","subject":"Re: [PATCH 8/9] update-index: integrate with sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-10T15:52:31Z","receivedAt":"2022-01-10T15:52:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jan 10, 2022 at 6:10 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n> >> Enable usage of the sparse index with `update-index`. Most variations of\n> >> `update-index` work without explicitly expanding the index or making any\n> >> other updates in or outside of `update-index.c`.\n> >>\n> >> The one usage requiring additional changes is `--cacheinfo`; if a file\n> >> inside a sparse directory was specified, the index would not be expanded\n> >> until after the cache tree is invalidated, leading to a mismatch between the\n> >> index and cache tree. This scenario is handled by rearranging\n> >> `add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the\n> >> index *before* attempting to invalidate the relevant cache tree path,\n> >> avoiding cache tree/index corruption.\n> >>\n> >> Signed-off-by: Victoria Dye \n> >> ---\n> >> builtin/update-index.c | 3 +++\n> >> read-cache.c | 10 +++++++---\n> >> t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n> >> 3 files changed, 22 insertions(+), 3 deletions(-)\n> >>\n> >> diff --git a/builtin/update-index.c b/builtin/update-index.c\n> >> index 187203e8bb5..605cc693bbd 100644\n> >> --- a/builtin/update-index.c\n> >> +++ b/builtin/update-index.c\n> >> @@ -1077,6 +1077,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n> >>\n> >> git_config(git_default_config, NULL);\n> >>\n> >> + prepare_repo_settings(r);\n> >> + the_repository->settings.command_requires_full_index = 0;\n> >> +\n> >> /* we will diagnose later if it turns out that we need to update it */\n> >> newfd = hold_locked_index(&lock_file, 0);\n> >> if (newfd < 0)\n> >> diff --git a/read-cache.c b/read-cache.c\n> >> index cbe73f14e5e..b4600e954b6 100644\n> >> --- a/read-cache.c\n> >> +++ b/read-cache.c\n> >> @@ -1339,9 +1339,6 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> >> int skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n> >> int new_only = option & ADD_CACHE_NEW_ONLY;\n> >>\n> >> - if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n> >> - cache_tree_invalidate_path(istate, ce->name);\n> >> -\n> >> /*\n> >> * If this entry's path sorts after the last entry in the index,\n> >> * we can avoid searching for it.\n> >> @@ -1352,6 +1349,13 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n> >> else\n> >> pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n> >>\n> >> + /*\n> >> + * Cache tree path should be invalidated only after index_name_stage_pos,\n> >> + * in case it expands a sparse index.\n> >> + */\n> >> + if (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n> >> + cache_tree_invalidate_path(istate, ce->name);\n> >> +\n> >> /* existing match? Just replace it. */\n> >> if (pos >= 0) {\n> >> if (!new_only)\n> >> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> >> index 6804ab23a27..bc0741c970d 100755\n> >> --- a/t/t1092-sparse-checkout-compatibility.sh\n> >> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> >> @@ -1216,6 +1216,18 @@ test_expect_success 'sparse index is not expanded: blame' '\n> >> done\n> >> '\n> >>\n> >> +test_expect_success 'sparse index is not expanded: update-index' '\n> >> + init_repos &&\n> >> +\n> >> + echo \"test\" >sparse-index/README.md &&\n> >> + echo \"test2\" >sparse-index/a &&\n> >> + rm -f sparse-index/deep/a &&\n> >> +\n> >> + ensure_not_expanded update-index --add README.md &&\n> >> + ensure_not_expanded update-index a &&\n> >> + ensure_not_expanded update-index --remove deep/a\n> >> +'\n> >\n> > The commit message said this change was about --cacheinfo, but this\n> > test doesn't use that option. I'm confused; was this a bad patch\n> > splitting by chance?\n> >\n>\n> It was not - the commit message title is \"update-index: integrate with\n> sparse index\" and the message starts by saying that this patch enables use\n> of the sparse index for *all* of `update-index` (where \"[m]ost variations of\n> `update-index` work without...making any other updates in or outside of\n> `update-index.c`\").\n>\n> It goes on to say that the `--cache-info` option is an exception to the\n> above statement (that most variations work without updates outside\n> `update-index.c`) because it requires a slight change to\n> `add_index_entry_with_check(...)` to avoid index corruption. That change is\n> also in this patch, but it's not the main \"focus\".\n>\n> The test added here is intended to broadly test `update-index` with a sparse\n> index. I'll add a case for `--cacheinfo` in my next re-roll but, for\n> context, the reason I didn't originally is because I focused on the (as far\n> as I could tell) most commonly-used variations of `update-index`.\n\nAh, I see. I didn't read closely enough and then saw the majority of\nthe commit message and code changes were about --cacheinfo, but the\ntestcase wasn't. Sorry about that.\n\n> >> +\n> >> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> >> # in this scenario, but it shouldn't.\n> >> test_expect_success 'reset mixed and checkout orphan' '\n> >> --\n> >> gitgitgadget\n> >>\n"},{"id":"445839","messageId":"CABPp-BEXYZxThegY9WvudY0B7KJ1HQm--Kwe5zxQOxvzHMHN1A@mail.gmail.com","threadId":"57180","inReplyTo":"159a35ba-7ed0-c601-15bd-54dfda460323@github.com","subject":"Re: [PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-10T17:11:26Z","receivedAt":"2022-01-10T17:11:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jan 10, 2022 at 7:47 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n> >> Introduce tests for a variety of `git update-index` use cases, including\n> >> performance scenarios.\n> >\n> > Makes sense.\n> >\n> >> Tests for `update-index add/remove` are specifically\n> >> focused on how `git stash` uses `git update-index` as a subcommand to\n> >> prepare for sparse index integration with `stash` in a future series.\n> >\n> > This is possibly a tangent, but I'd rather that if we were trying to\n> > fix `git stash`, that we instead would do so by making it stop forking\n> > subprocesses and having it call internal API instead. See for\n> > example, a4031f6dc0 (\"Merge branch 'en/stash-apply-sparse-checkout'\n> > into maint\", 2021-02-05) which did this. The fact that it forks so\n> > many subprocesses is a bug, and comes from the fact that stash is a\n> > partial conversion from shell to C. I think the subprocess forking is\n> > part of the problem that causes issues for sparse-checkouts, as I\n> > spelled out in patches 2 & 3 of the series I mentioned above.\n> >\n> > However, that doesn't affect this series.\n> >\n> >> Co-authored-by: Derrick Stolee \n> >> Signed-off-by: Victoria Dye \n> >> ---\n> >> t/perf/p2000-sparse-operations.sh | 1 +\n> >> t/t1092-sparse-checkout-compatibility.sh | 125 +++++++++++++++++++++++\n> >> 2 files changed, 126 insertions(+)\n> >>\n> >> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> >> index 54f8602f3c1..7dbed330160 100755\n> >> --- a/t/perf/p2000-sparse-operations.sh\n> >> +++ b/t/perf/p2000-sparse-operations.sh\n> >> @@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n> >> test_perf_on_all git blame $SPARSE_CONE/a\n> >> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> >> test_perf_on_all git checkout-index -f --all\n> >> +test_perf_on_all git update-index --add --remove\n> >>\n> >> test_done\n> >> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> >> index 6ecf1f2bf8e..6804ab23a27 100755\n> >> --- a/t/t1092-sparse-checkout-compatibility.sh\n> >> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> >> @@ -630,6 +630,131 @@ test_expect_success 'reset with wildcard pathspec' '\n> >> test_all_match git ls-files -s -- folder1\n> >> '\n> >>\n> >> +test_expect_success 'update-index modify outside sparse definition' '\n> >> + init_repos &&\n> >> +\n> >> + write_script edit-contents <<-\\EOF &&\n> >> + echo text >>$1\n> >> + EOF\n> >> +\n> >> + # Create & modify folder1/a\n> >> + run_on_sparse mkdir -p folder1 &&\n> >> + run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n> >> + run_on_all ../edit-contents folder1/a &&\n> >\n> > As I've mentioned to Stolee, I'd rather these were explicitly marked\n> > as intentionally setting up an erroneous condition.\n> >\n> > However, that might not matter if my other series gets accepted (the\n> > one I promised to send out yesterday, but then spent all day\n> > responding to emails instead). Hopefully I'll send it soon.\n> >\n> >> +\n> >> + # If file has skip-worktree enabled, update-index does not modify the\n> >> + # index entry\n> >> + test_sparse_match git update-index folder1/a &&\n> >> + test_sparse_match git status --porcelain=v2 &&\n> >> + test_must_be_empty sparse-checkout-out &&\n> >> +\n> >> + # When skip-worktree is disabled (even on files outside sparse cone), file\n> >> + # is updated in the index\n> >> + test_sparse_match git update-index --no-skip-worktree folder1/a &&\n> >> + test_all_match git status --porcelain=v2 &&\n> >> + test_all_match git update-index folder1/a &&\n> >> + test_all_match git status --porcelain=v2\n> >\n> > These make sense.\n> >\n> >> +'\n> >> +\n> >> +test_expect_success 'update-index --add outside sparse definition' '\n> >> + init_repos &&\n> >> +\n> >> + write_script edit-contents <<-\\EOF &&\n> >> + echo text >>$1\n> >> + EOF\n> >> +\n> >> + # Create folder1, add new file\n> >> + run_on_sparse mkdir -p folder1 &&\n> >> + run_on_all ../edit-contents folder1/b &&\n> >> +\n> >> + # Similar to `git add`, the untracked out-of-cone file is added to the index\n> >> + # identically across sparse and non-sparse checkouts\n> >> + test_all_match git update-index --add folder1/b &&\n> >> + test_all_match git status --porcelain=v2\n> >\n> > The comment is not correct:\n> >\n>\n> It is correct, but for *untracked* out-of-cone files only. Those files don't\n> have a `skip-worktree` bit because they're not in the index in the first\n> place.\n\n> The comment is intended to highlight the fact that `update-index`\n> (like `git add`, `git status`, etc.) \"decides\" whether to operate on a file\n> in a sparse-checkout based on `skip-worktree`, *not* the sparse patterns.\n\ngit-update-index may not pay attention to the sparsity patterns for\nuntracked files, but `git add` does. Let me demonstrate. First, I\nset up a simple repo where the following is true:\n\n$ git ls-files -t\nH in-cone/foo.c\nS out-of-cone/tracked\n$ git status --porcelain\n?? out-of-cone/initially-untracked\n\nNow, let's compare `git add` and `git add --sparse`. First, without --sparse:\n\n$ git add out-of-cone/initially-untracked\nThe following paths and/or pathspecs matched paths that exist\noutside of your sparse-checkout definition, so will not be\nupdated in the index:\nout-of-cone/initially-untracked\nhint: If you intend to update such entries, try one of the following:\nhint: * Use the --sparse option.\nhint: * Disable or modify the sparsity rules.\nhint: Disable this message with \"git config advice.updateSparsePath false\"\n$ git ls-files -t\nH in-cone/foo.c\nS out-of-cone/tracked\n\nSo, `git add $UNTRACKED` did not add the file. In contrast:\n\n$ git add --sparse out-of-cone/initially-untracked\n$ git ls-files -t\nH in-cone/foo.c\nH out-of-cone/initially-untracked\nS out-of-cone/tracked\n\nSo, `git add --sparse $UNTRACKED` did add it.\n\n> Seeing as the comparison to `git add` makes things more confusing, I'll\n> rephrase the test comment.\n>\n> > $ git add out-of-cone/file\n> > The following paths and/or pathspecs matched paths that exist\n> > outside of your sparse-checkout definition, so will not be\n> > updated in the index:\n> > out-of-cone/file\n> > hint: If you intend to update such entries, try one of the following:\n> > hint: * Use the --sparse option.\n> > hint: * Disable or modify the sparsity rules.\n> > hint: Disable this message with \"git config advice.updateSparsePath false\"\n> >\n> > I might buy that `git update-index` is lower level and should be\n> > considered the same as `git add --sparse`, but the comment should\n> > mention that and try to sell why update-index should be equivalent to\n> > that instead of to `git add`.\n> >\n>\n> Tracked, out-of-cone files aren't affected by `--add` (the flag allows\n> `update-index` to add untracked files), and `update-index out-of-cone/tracked`\n> would ignore the file.\n\nYes, I believe you're explaining update-index behavior correctly.\n\n> so I believe the behavior of `update-index` is\n> currently more consistent with `git add` than `git add --sparse`.\n\nBut not quite `git add`'s. Just to be clear, let's add update-index\nto the above comparison I did between add and add --sparse. First,\nlet's go back to the initial setup point:\n\n$ git ls-files -t\nH in-cone/foo.c\nS out-of-cone/tracked\n$ git status --porcelain\n?? out-of-cone/initially-untracked\n\nNow, let's try update-index:\n\n$ git update-index --add out-of-cone/initially-untracked\n$ git ls-files -t\nH in-cone/foo.c\nH out-of-cone/initially-untracked\nS out-of-cone/tracked\n\nSo, in other words, `git update-index --add $UNTRACKED` matches the\nbehavior of `git add --sparse $UNTRACKED`, not the behavior of `git\nadd $UNTRACKED`.\n\n> >> +'\n> >> +\n> >> +test_expect_success 'update-index --remove outside sparse definition' '\n> >> + init_repos &&\n> >\n> >> +\n> >> + # When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n> >> + # not removed from the index if they do not exist on disk\n> >> + test_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n> >> + test_all_match git status --porcelain=v2 &&\n> >\n> > The file is present despite being marked to be missing, we're ignoring\n> > the intention of the marking, and we ask for it to be removed, so we\n> > don't remove it?\n> >\n> > The levels of negation are _very_ confusing. It took me a while to\n> > unravel this. I think the logic is something like this\n> > * folder1/a is marked as SKIP_WORKTREE, meaning it's not supposed to\n> > be in the worktree.\n> > * and it's not.\n> > * We are stating that we're ignoring SKIP_WORKTREE, though, so this\n> > looks like a regular file that has been deleted.\n> > So, what would `git update-index --remove $FILE` do for a normal $FILE\n> > deleted from the working copy? According to the docs:\n> >\n> > --remove\n> > If a specified file is in the index but is missing then it’s\n> > removed. Default behavior is to ignore removed file.\n> >\n> > So, the docs say it would remove it. But you don't.\n> >\n> >\n> > After digging around and looking at the testcase below, if I had to\n> > guess what happened, I would say that you figured out what the\n> > SKIP_WORKTREE behavior was, and assumed that was correct, and added a\n> > flag that allowed you to request the opposite behavior.\n> > Unfortunately, I think the pre-existing behavior is buggy.\n> >\n>\n> I understand why you find it buggy, but I am not making baseless assumptions\n> about the correctness (or lack thereof) of the current behavior.\n\nTo be clear, the fact that the behavior was there for a decade would\ntypically be basis enough for an assumption (in my opinion), and I\nwouldn't have faulted folks for making it. I might well have done so\nmyself. My reasoning was just that I was getting confused by the\nnegations and trying to understand the testcase, and when I started to\nunravel it, I found what looked like a possible inconsistency.\n\nAnyway, it's clear here you've actually dug a lot deeper and know the\nhistory here. In contrast, I was making assumptions about the history\n(and ones that weren't correct, though I'd argue my assumptions\nweren't baseless)...\n\n> This\n> specific \"gap\" in `update-index --remove` has been discussed in the past [1]\n> and was acknowledged as non-ideal and a candidate for change in the future.\n> At the time, the introduction of `--ignore-skip-worktree-entries` [2] was a\n> \"safe\" way to ignore `skip-worktree` without changing the default behavior.\n>\n> Personally, I think updating the default behavior of `--remove` (and\n> corresponding deprecation of `--[no-]ignore-skip-worktree-entries`) is\n> probably the right way to go. However, I'd like to avoid including it in\n> this series because it deviates pretty substantially from the goal\n> \"integrate with sparse index\", and as a result has the potential to\n> overshadow/derail the rest of the series. If you're alright with (slightly)\n> deferring changes to the behavior of `--remove`, I can submit a separate\n> series for it once this one has stabilized.\n>\n> [1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n> [2] https://lore.kernel.org/git/163b42dfa21c306dc1dc573c5edfc8bda5c99fd0.1572432578.git.gitgitgadget@gmail.com/\n\nOh, wow.\n\nYeah, fixing it in a later series seems fine. However, could you add\na comment to these testcases that --remove has behavior that violates\nthe definition of `SKIP_WORKTREE`, and that we might want to fix that\nin the future but for now you are just testing the pre-existing\nbehavior for coverage?\n\n(There is a possibility we can't fix it for some reason when we dig\nin. In that case, we should update the documentation for `--remove`\nto call out this special case. But again, that can be for a later\nseries.)\n\n> > Of course, there's lots of negation here. Did I get something\n> > backwards by chance?\n> >\n> >> +\n> >> + # When the flag is _not_ specified ...\n> >\n> > In my head I'm translating this as:\n> >\n> > SKIP_WORKTREE = !worktree\n> > --ignore-skip-worktree-entries = !!worktree\n> > \"flag is _not_ specified\" = !!!worktree ?\n> >\n> > I'm not sure I would do anything to change it, but just pointing out\n> > it can be a little hard for others to come up to speed.\n> >\n>\n> Most of the confusion likely comes from the non-standard behavior of\n> `--remove`, but I think I can distill it into (somewhat) straightforward\n> statements about `update-index`:\n>\n> 1. When using the command *without* either `--ignore-skip-worktree-entries`\n> OR `--remove`, `skip-worktree` entries provided to the command are\n> ignored.\n> 2. When using the command *with* `--remove` and *without*\n> `--ignore-skip-worktree-entries`, `skip-worktree` entries are *not*\n> ignored, and are removed from the index.\n> 3. When both `--remove` and `--ignore-skip-worktree-entries` are specified,\n> `skip-worktree` entries are again ignored.\n> 4. `--ignore-skip-worktree-entries` has no effect without `--remove` also\n> specified\n>\n> The goal of this test, then, is to exercise conditions 2 & 3 and directly\n> show their effect on `skip-worktree` entries.\n\nYeah, it makes sense to have good test coverage. +1.\n\n>\n> >> ... , out-of-cone, not-on-disk files are\n> >> + # removed from the index\n> >> + rm full-checkout/folder1/a &&\n> >> + test_all_match git update-index --remove folder1/a &&\n> >> + test_all_match git status --porcelain=v2 &&\n> >\n> > Documentation/git-update-index.txt defines SKIP_WORKTREE as follows:\n> >\n> > \"Skip-worktree bit can be defined in one (long) sentence: When reading\n> > an entry, if it is marked as skip-worktree, then Git pretends its\n> > working directory version is up to date and read the index version\n> > instead.\"\n> >\n> > If Git is pretending the file is up-to-date, and `git update-index\n> > --remove $UP_TO_DATE_FILE` is typically a no-op because the --remove\n> > flag doesn't do anything when a file is present in the working copy,\n> > then why is this the expected behavior?\n> >\n> > I know it's the traditional behavior of update-index, but\n> > SKIP_WORKTREE support in Git has traditionally been filled with holes.\n> > So, was this behavior by design (despite contradicting the\n> > documentation), or by accident?\n> >\n>\n> As far as I can tell, it appears to have been intentional in the original\n> `skip-worktree` implementation [3], but given Junio & Johannes' discussion\n> on the `--ignore-skip-worktree-entries` patch [1], the sentiment now would\n> probably lean towards having `--remove` ignore `skip-worktree`.\n>\n> [1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n> (copied from earlier in this message)\n> [3] https://lore.kernel.org/git/1250776033-12395-5-git-send-email-pclouds@gmail.com/\n\nAllow me to comment on this history...\n\nI can see sometimes wanting to make a command line option a special\ncase that doesn't follow the general documented rules. Not sure I\nbelieve it in this specific case, but I can see it as being a\npossibility in general. But when you're going to make an option not\nfollow the otherwise documented behavior, then that option's\ndocumentation should have a special callout about how it diverges\nand/or why it's a special case. So, this seems like a double-layered\nproblem to me (not only choosing the wrong behavior, but leaving the\ndocumentation to claim it does something else). It looks like Dscho\npartially tried to fix it, but I would have preferred that Dscho's\ndocumentation comment he added to `--ignore-skip-worktree-entries` be\nadded to `--remove`; it's odd that folks wanting to learn about\n`--remove` behavior need to read the documentation for\n`--ignore-skip-worktree-entries` (even if they aren't using it) in\norder to understand `--remove`'s behavior.\n\n> > (To be fair, I think the definition given in the manual for\n> > SKIP_WORKTREE is horrible for other reasons, so I don't like leaning\n> > on it. But I used different logic above in the\n> > --ignore-skip-worktree-entries case to arrive at the same conclusion\n> > that the --remove behavior of update-index seems to be backwards.\n> > Unless I missed a negation in both cases somewhere? There are so many\n> > floating around...)\n> >\n> >> + # NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n> >> + # a skip-worktree file from the index (and disk) when both are specified\n> >> + test_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n> >> + test_all_match git status --porcelain=v2\n> >\n> > Makes sense.\n> >\n> >> +'\n> >> +\n> >> +test_expect_success 'update-index with directories' '\n> >> + init_repos &&\n> >> +\n> >> + # update-index will exit silently when provided with a directory name\n> >> + # containing a trailing slash\n> >> + test_all_match git update-index deep/ folder1/ &&\n> >> + grep \"Ignoring path deep/\" sparse-checkout-err &&\n> >> + grep \"Ignoring path folder1/\" sparse-checkout-err &&\n> >\n> > Is this desired behavior or just current behavior?\n> >\n> >> +\n> >> + # When update-index is given a directory name WITHOUT a trailing slash, it will\n> >> + # behave in different ways depending on the status of the directory on disk:\n> >> + # * if it exists, the command exits with an error (\"add individual files instead\")\n> >> + # * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n> >> + # file and either triggers an error (\"does not exist and --remove not passed\")\n> >> + # or is ignored completely (when using --remove)\n> >> + test_all_match test_must_fail git update-index deep &&\n> >> + run_on_all test_must_fail git update-indexe folder1 &&\n> >\n> > This one will fail for the wrong reason, though -- `update-indexe` is\n> > not a valid subcommand. (extra 'e' at the end)\n> >\n>\n> Thanks for catching that! I'll update in my next re-roll.\n>\n> >> + test_must_fail git -C full-checkout update-index --remove folder1 &&\n> >> + test_sparse_match git update-index --remove folder1 &&\n> >> + test_all_match git status --porcelain=v2\n> >\n> > Otherwise these seem reasonable.\n> >\n> >> +'\n> >> +\n> >> +test_expect_success 'update-index --again file outside sparse definition' '\n> >> + init_repos &&\n> >> +\n> >> + write_script edit-contents <<-\\EOF &&\n> >> + echo text >>$1\n> >> + EOF\n> >\n> > Copy and paste and forget to remove? edit-contents doesn't seem to be\n> > used in this test.\n> >\n>\n> Will remove.\n>\n> >> +\n> >> + test_all_match git checkout -b test-reupdate &&\n> >> +\n> >> + # Update HEAD without modifying the index to introduce a difference in\n> >> + # folder1/a\n> >> + test_sparse_match git reset --soft update-folder1 &&\n> >> +\n> >> + # Because folder1/a differs in the index vs HEAD,\n> >> + # `git update-index --remove --again` will effectively perform\n> >> + # `git update-index --remove folder1/a` and remove the folder1/a\n> >> + test_sparse_match git update-index --remove --again &&\n> >> + test_sparse_match git status --porcelain=v2\n> >\n> > This might need a --ignore-skip-worktree-entries, as per the\n> > discussion above. Otherwise, this test makes sense.\n> >\n>\n> The `--ignore-skip-worktree-entries` option is explicitly omitted because\n> this test needs `update-index` to modify a `skip-worktree` entry. However,\n> given the debate around what `--remove` should do, I'll update the scenario\n> to not use `--remove` or any variation of it.\n>\n> >> +'\n> >> +\n> >> +test_expect_success 'update-index --cacheinfo' '\n> >> + init_repos &&\n> >> +\n> >> + deep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n> >> + folder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n> >> + folder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n> >> +\n> >> + test_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n> >> + test_all_match git status --porcelain=v2 &&\n> >> +\n> >> + # Cannot add sparse directory, even in sparse index case\n> >> + test_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n> >> +\n> >> + # Sparse match only - because folder1/a is outside the sparse checkout\n> >> + # definition (and thus not on-disk), it will appear \"deleted\" in\n> >> + # unstaged changes.\n> >> + test_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n> >> + test_sparse_match git status --porcelain=v2\n> >\n> > Makes sense, because the update-index command removes the existing\n> > cache entry and adds a new one without the SKIP_WORKTREE bit. But it\n> > might be worth mentioning that in the commit message. Also, you could\n> > follow this up with `git update-index --skip-worktree folder1/a`, and\n> > then do a test_all_match git status --porcelain=v2, to show that when\n> > the SKIP_WORKTREE bit is restored back to the file, then it again does\n> > as expected despite not being on-disk.\n> >\n>\n> I really like this - it helps clarify how `update-index` can be used to\n> correctly add a sparse-checkout entry to the index with plumbing commands.\n> I'll include it in V2.\n>\n> >> +'\n> >> +\n> >> test_expect_success 'merge, cherry-pick, and rebase' '\n> >> init_repos &&\n> >>\n> >> --\n> >> gitgitgadget\n"},{"id":"445850","messageId":"bec659b2-8d69-c69c-73e5-a56ae415874b@github.com","threadId":"57180","inReplyTo":"CABPp-BEXYZxThegY9WvudY0B7KJ1HQm--Kwe5zxQOxvzHMHN1A@mail.gmail.com","subject":"Re: [PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-10T18:01:54Z","receivedAt":"2022-01-10T18:01:58Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Mon, Jan 10, 2022 at 7:47 AM Victoria Dye wrote:\n>>\n>> Elijah Newren wrote:\n>>> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n>>> wrote:\n>>>>\n>>>> From: Victoria Dye \n>>>>\n>>>> Introduce tests for a variety of `git update-index` use cases, including\n>>>> performance scenarios.\n>>>\n>>> Makes sense.\n>>>\n>>>> Tests for `update-index add/remove` are specifically\n>>>> focused on how `git stash` uses `git update-index` as a subcommand to\n>>>> prepare for sparse index integration with `stash` in a future series.\n>>>\n>>> This is possibly a tangent, but I'd rather that if we were trying to\n>>> fix `git stash`, that we instead would do so by making it stop forking\n>>> subprocesses and having it call internal API instead. See for\n>>> example, a4031f6dc0 (\"Merge branch 'en/stash-apply-sparse-checkout'\n>>> into maint\", 2021-02-05) which did this. The fact that it forks so\n>>> many subprocesses is a bug, and comes from the fact that stash is a\n>>> partial conversion from shell to C. I think the subprocess forking is\n>>> part of the problem that causes issues for sparse-checkouts, as I\n>>> spelled out in patches 2 & 3 of the series I mentioned above.\n>>>\n>>> However, that doesn't affect this series.\n>>>\n>>>> Co-authored-by: Derrick Stolee \n>>>> Signed-off-by: Victoria Dye \n>>>> ---\n>>>> t/perf/p2000-sparse-operations.sh | 1 +\n>>>> t/t1092-sparse-checkout-compatibility.sh | 125 +++++++++++++++++++++++\n>>>> 2 files changed, 126 insertions(+)\n>>>>\n>>>> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n>>>> index 54f8602f3c1..7dbed330160 100755\n>>>> --- a/t/perf/p2000-sparse-operations.sh\n>>>> +++ b/t/perf/p2000-sparse-operations.sh\n>>>> @@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n>>>> test_perf_on_all git blame $SPARSE_CONE/a\n>>>> test_perf_on_all git blame $SPARSE_CONE/f3/a\n>>>> test_perf_on_all git checkout-index -f --all\n>>>> +test_perf_on_all git update-index --add --remove\n>>>>\n>>>> test_done\n>>>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>>>> index 6ecf1f2bf8e..6804ab23a27 100755\n>>>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>>>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>>>> @@ -630,6 +630,131 @@ test_expect_success 'reset with wildcard pathspec' '\n>>>> test_all_match git ls-files -s -- folder1\n>>>> '\n>>>>\n>>>> +test_expect_success 'update-index modify outside sparse definition' '\n>>>> + init_repos &&\n>>>> +\n>>>> + write_script edit-contents <<-\\EOF &&\n>>>> + echo text >>$1\n>>>> + EOF\n>>>> +\n>>>> + # Create & modify folder1/a\n>>>> + run_on_sparse mkdir -p folder1 &&\n>>>> + run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n>>>> + run_on_all ../edit-contents folder1/a &&\n>>>\n>>> As I've mentioned to Stolee, I'd rather these were explicitly marked\n>>> as intentionally setting up an erroneous condition.\n>>>\n>>> However, that might not matter if my other series gets accepted (the\n>>> one I promised to send out yesterday, but then spent all day\n>>> responding to emails instead). Hopefully I'll send it soon.\n>>>\n>>>> +\n>>>> + # If file has skip-worktree enabled, update-index does not modify the\n>>>> + # index entry\n>>>> + test_sparse_match git update-index folder1/a &&\n>>>> + test_sparse_match git status --porcelain=v2 &&\n>>>> + test_must_be_empty sparse-checkout-out &&\n>>>> +\n>>>> + # When skip-worktree is disabled (even on files outside sparse cone), file\n>>>> + # is updated in the index\n>>>> + test_sparse_match git update-index --no-skip-worktree folder1/a &&\n>>>> + test_all_match git status --porcelain=v2 &&\n>>>> + test_all_match git update-index folder1/a &&\n>>>> + test_all_match git status --porcelain=v2\n>>>\n>>> These make sense.\n>>>\n>>>> +'\n>>>> +\n>>>> +test_expect_success 'update-index --add outside sparse definition' '\n>>>> + init_repos &&\n>>>> +\n>>>> + write_script edit-contents <<-\\EOF &&\n>>>> + echo text >>$1\n>>>> + EOF\n>>>> +\n>>>> + # Create folder1, add new file\n>>>> + run_on_sparse mkdir -p folder1 &&\n>>>> + run_on_all ../edit-contents folder1/b &&\n>>>> +\n>>>> + # Similar to `git add`, the untracked out-of-cone file is added to the index\n>>>> + # identically across sparse and non-sparse checkouts\n>>>> + test_all_match git update-index --add folder1/b &&\n>>>> + test_all_match git status --porcelain=v2\n>>>\n>>> The comment is not correct:\n>>>\n>>\n>> It is correct, but for *untracked* out-of-cone files only. Those files don't\n>> have a `skip-worktree` bit because they're not in the index in the first\n>> place.\n> \n>> The comment is intended to highlight the fact that `update-index`\n>> (like `git add`, `git status`, etc.) \"decides\" whether to operate on a file\n>> in a sparse-checkout based on `skip-worktree`, *not* the sparse patterns.\n> \n> git-update-index may not pay attention to the sparsity patterns for\n> untracked files, but `git add` does. Let me demonstrate. First, I\n> set up a simple repo where the following is true:\n> \n> $ git ls-files -t\n> H in-cone/foo.c\n> S out-of-cone/tracked\n> $ git status --porcelain\n> ?? out-of-cone/initially-untracked\n> \n> Now, let's compare `git add` and `git add --sparse`. First, without --sparse:\n> \n> $ git add out-of-cone/initially-untracked\n> The following paths and/or pathspecs matched paths that exist\n> outside of your sparse-checkout definition, so will not be\n> updated in the index:\n> out-of-cone/initially-untracked\n> hint: If you intend to update such entries, try one of the following:\n> hint: * Use the --sparse option.\n> hint: * Disable or modify the sparsity rules.\n> hint: Disable this message with \"git config advice.updateSparsePath false\"\n> $ git ls-files -t\n> H in-cone/foo.c\n> S out-of-cone/tracked\n> \n> So, `git add $UNTRACKED` did not add the file. In contrast:\n> \n> $ git add --sparse out-of-cone/initially-untracked\n> $ git ls-files -t\n> H in-cone/foo.c\n> H out-of-cone/initially-untracked\n> S out-of-cone/tracked\n> \n> So, `git add --sparse $UNTRACKED` did add it.\n> \n\nSorry about that - when I wrote the first version of this series in the\n`microsoft/git` fork, `git add` hadn't been updated to reject out-of-cone\nuntracked files as it is in [1]. It's my mistake for not double-checking\nthat it was still the case, apologies for wasting your time on re-explaining\nthis.\n\nIn any case, I'll update the test comment and commit message per your\nsuggestion:\n\n>>> I might buy that `git update-index` is lower level and should be\n>>> considered the same as `git add --sparse`, but the comment should\n>>> mention that and try to sell why update-index should be equivalent to\n>>> that instead of to `git add`.\n\nI'm leaning only slightly towards the current behavior (and will update the\ncomment accordingly), but I'm happy to change it if the reasoning isn't as\nstrong as that of another approach.\n\n[1] 105e8b014b (add: fail when adding an untracked sparse file, 2021-09-24)\n\n>> Seeing as the comparison to `git add` makes things more confusing, I'll\n>> rephrase the test comment.\n>>\n>>> $ git add out-of-cone/file\n>>> The following paths and/or pathspecs matched paths that exist\n>>> outside of your sparse-checkout definition, so will not be\n>>> updated in the index:\n>>> out-of-cone/file\n>>> hint: If you intend to update such entries, try one of the following:\n>>> hint: * Use the --sparse option.\n>>> hint: * Disable or modify the sparsity rules.\n>>> hint: Disable this message with \"git config advice.updateSparsePath false\"\n>>>\n>>> I might buy that `git update-index` is lower level and should be\n>>> considered the same as `git add --sparse`, but the comment should\n>>> mention that and try to sell why update-index should be equivalent to\n>>> that instead of to `git add`.\n>>>\n>>\n>> Tracked, out-of-cone files aren't affected by `--add` (the flag allows\n>> `update-index` to add untracked files), and `update-index out-of-cone/tracked`\n>> would ignore the file.\n> \n> Yes, I believe you're explaining update-index behavior correctly.\n> \n>> so I believe the behavior of `update-index` is\n>> currently more consistent with `git add` than `git add --sparse`.\n> \n> But not quite `git add`'s. Just to be clear, let's add update-index\n> to the above comparison I did between add and add --sparse. First,\n> let's go back to the initial setup point:\n> \n> $ git ls-files -t\n> H in-cone/foo.c\n> S out-of-cone/tracked\n> $ git status --porcelain\n> ?? out-of-cone/initially-untracked\n> \n> Now, let's try update-index:\n> \n> $ git update-index --add out-of-cone/initially-untracked\n> $ git ls-files -t\n> H in-cone/foo.c\n> H out-of-cone/initially-untracked\n> S out-of-cone/tracked\n> \n> So, in other words, `git update-index --add $UNTRACKED` matches the\n> behavior of `git add --sparse $UNTRACKED`, not the behavior of `git\n> add $UNTRACKED`.\n> \n>>>> +'\n>>>> +\n>>>> +test_expect_success 'update-index --remove outside sparse definition' '\n>>>> + init_repos &&\n>>>\n>>>> +\n>>>> + # When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n>>>> + # not removed from the index if they do not exist on disk\n>>>> + test_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n>>>> + test_all_match git status --porcelain=v2 &&\n>>>\n>>> The file is present despite being marked to be missing, we're ignoring\n>>> the intention of the marking, and we ask for it to be removed, so we\n>>> don't remove it?\n>>>\n>>> The levels of negation are _very_ confusing. It took me a while to\n>>> unravel this. I think the logic is something like this\n>>> * folder1/a is marked as SKIP_WORKTREE, meaning it's not supposed to\n>>> be in the worktree.\n>>> * and it's not.\n>>> * We are stating that we're ignoring SKIP_WORKTREE, though, so this\n>>> looks like a regular file that has been deleted.\n>>> So, what would `git update-index --remove $FILE` do for a normal $FILE\n>>> deleted from the working copy? According to the docs:\n>>>\n>>> --remove\n>>> If a specified file is in the index but is missing then it’s\n>>> removed. Default behavior is to ignore removed file.\n>>>\n>>> So, the docs say it would remove it. But you don't.\n>>>\n>>>\n>>> After digging around and looking at the testcase below, if I had to\n>>> guess what happened, I would say that you figured out what the\n>>> SKIP_WORKTREE behavior was, and assumed that was correct, and added a\n>>> flag that allowed you to request the opposite behavior.\n>>> Unfortunately, I think the pre-existing behavior is buggy.\n>>>\n>>\n>> I understand why you find it buggy, but I am not making baseless assumptions\n>> about the correctness (or lack thereof) of the current behavior.\n> \n> To be clear, the fact that the behavior was there for a decade would\n> typically be basis enough for an assumption (in my opinion), and I\n> wouldn't have faulted folks for making it. I might well have done so\n> myself. My reasoning was just that I was getting confused by the\n> negations and trying to understand the testcase, and when I started to\n> unravel it, I found what looked like a possible inconsistency.\n> \n> Anyway, it's clear here you've actually dug a lot deeper and know the\n> history here. In contrast, I was making assumptions about the history\n> (and ones that weren't correct, though I'd argue my assumptions\n> weren't baseless)...\n> \n\nYour assumptions were completely valid, I apologize if my response came off\nas implying otherwise. I'll try to use the comments on the tests to clarify\ntheir behavior as much as possible, hopefully reducing some confusion around\nall the multiple-negative flags & options.\n\n>> This\n>> specific \"gap\" in `update-index --remove` has been discussed in the past [1]\n>> and was acknowledged as non-ideal and a candidate for change in the future.\n>> At the time, the introduction of `--ignore-skip-worktree-entries` [2] was a\n>> \"safe\" way to ignore `skip-worktree` without changing the default behavior.\n>>\n>> Personally, I think updating the default behavior of `--remove` (and\n>> corresponding deprecation of `--[no-]ignore-skip-worktree-entries`) is\n>> probably the right way to go. However, I'd like to avoid including it in\n>> this series because it deviates pretty substantially from the goal\n>> \"integrate with sparse index\", and as a result has the potential to\n>> overshadow/derail the rest of the series. If you're alright with (slightly)\n>> deferring changes to the behavior of `--remove`, I can submit a separate\n>> series for it once this one has stabilized.\n>>\n>> [1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n>> [2] https://lore.kernel.org/git/163b42dfa21c306dc1dc573c5edfc8bda5c99fd0.1572432578.git.gitgitgadget@gmail.com/\n> \n> Oh, wow.\n> \n> Yeah, fixing it in a later series seems fine. However, could you add\n> a comment to these testcases that --remove has behavior that violates\n> the definition of `SKIP_WORKTREE`, and that we might want to fix that\n> in the future but for now you are just testing the pre-existing\n> behavior for coverage?\n> \n\nWill do!\n\n> (There is a possibility we can't fix it for some reason when we dig\n> in. In that case, we should update the documentation for `--remove`\n> to call out this special case. But again, that can be for a later\n> series.)\n> \n>>> Of course, there's lots of negation here. Did I get something\n>>> backwards by chance?\n>>>\n>>>> +\n>>>> + # When the flag is _not_ specified ...\n>>>\n>>> In my head I'm translating this as:\n>>>\n>>> SKIP_WORKTREE = !worktree\n>>> --ignore-skip-worktree-entries = !!worktree\n>>> \"flag is _not_ specified\" = !!!worktree ?\n>>>\n>>> I'm not sure I would do anything to change it, but just pointing out\n>>> it can be a little hard for others to come up to speed.\n>>>\n>>\n>> Most of the confusion likely comes from the non-standard behavior of\n>> `--remove`, but I think I can distill it into (somewhat) straightforward\n>> statements about `update-index`:\n>>\n>> 1. When using the command *without* either `--ignore-skip-worktree-entries`\n>> OR `--remove`, `skip-worktree` entries provided to the command are\n>> ignored.\n>> 2. When using the command *with* `--remove` and *without*\n>> `--ignore-skip-worktree-entries`, `skip-worktree` entries are *not*\n>> ignored, and are removed from the index.\n>> 3. When both `--remove` and `--ignore-skip-worktree-entries` are specified,\n>> `skip-worktree` entries are again ignored.\n>> 4. `--ignore-skip-worktree-entries` has no effect without `--remove` also\n>> specified\n>>\n>> The goal of this test, then, is to exercise conditions 2 & 3 and directly\n>> show their effect on `skip-worktree` entries.\n> \n> Yeah, it makes sense to have good test coverage. +1.\n> \n>>\n>>>> ... , out-of-cone, not-on-disk files are\n>>>> + # removed from the index\n>>>> + rm full-checkout/folder1/a &&\n>>>> + test_all_match git update-index --remove folder1/a &&\n>>>> + test_all_match git status --porcelain=v2 &&\n>>>\n>>> Documentation/git-update-index.txt defines SKIP_WORKTREE as follows:\n>>>\n>>> \"Skip-worktree bit can be defined in one (long) sentence: When reading\n>>> an entry, if it is marked as skip-worktree, then Git pretends its\n>>> working directory version is up to date and read the index version\n>>> instead.\"\n>>>\n>>> If Git is pretending the file is up-to-date, and `git update-index\n>>> --remove $UP_TO_DATE_FILE` is typically a no-op because the --remove\n>>> flag doesn't do anything when a file is present in the working copy,\n>>> then why is this the expected behavior?\n>>>\n>>> I know it's the traditional behavior of update-index, but\n>>> SKIP_WORKTREE support in Git has traditionally been filled with holes.\n>>> So, was this behavior by design (despite contradicting the\n>>> documentation), or by accident?\n>>>\n>>\n>> As far as I can tell, it appears to have been intentional in the original\n>> `skip-worktree` implementation [3], but given Junio & Johannes' discussion\n>> on the `--ignore-skip-worktree-entries` patch [1], the sentiment now would\n>> probably lean towards having `--remove` ignore `skip-worktree`.\n>>\n>> [1] https://lore.kernel.org/git/xmqq36fda3i8.fsf@gitster-ct.c.googlers.com/\n>> (copied from earlier in this message)\n>> [3] https://lore.kernel.org/git/1250776033-12395-5-git-send-email-pclouds@gmail.com/\n> \n> Allow me to comment on this history...\n> \n> I can see sometimes wanting to make a command line option a special\n> case that doesn't follow the general documented rules. Not sure I\n> believe it in this specific case, but I can see it as being a\n> possibility in general. But when you're going to make an option not\n> follow the otherwise documented behavior, then that option's\n> documentation should have a special callout about how it diverges\n> and/or why it's a special case. So, this seems like a double-layered\n> problem to me (not only choosing the wrong behavior, but leaving the\n> documentation to claim it does something else). It looks like Dscho\n> partially tried to fix it, but I would have preferred that Dscho's\n> documentation comment he added to `--ignore-skip-worktree-entries` be\n> added to `--remove`; it's odd that folks wanting to learn about\n> `--remove` behavior need to read the documentation for\n> `--ignore-skip-worktree-entries` (even if they aren't using it) in\n> order to understand `--remove`'s behavior.\n> \n>>> (To be fair, I think the definition given in the manual for\n>>> SKIP_WORKTREE is horrible for other reasons, so I don't like leaning\n>>> on it. But I used different logic above in the\n>>> --ignore-skip-worktree-entries case to arrive at the same conclusion\n>>> that the --remove behavior of update-index seems to be backwards.\n>>> Unless I missed a negation in both cases somewhere? There are so many\n>>> floating around...)\n>>>\n>>>> + # NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n>>>> + # a skip-worktree file from the index (and disk) when both are specified\n>>>> + test_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n>>>> + test_all_match git status --porcelain=v2\n>>>\n>>> Makes sense.\n>>>\n>>>> +'\n>>>> +\n>>>> +test_expect_success 'update-index with directories' '\n>>>> + init_repos &&\n>>>> +\n>>>> + # update-index will exit silently when provided with a directory name\n>>>> + # containing a trailing slash\n>>>> + test_all_match git update-index deep/ folder1/ &&\n>>>> + grep \"Ignoring path deep/\" sparse-checkout-err &&\n>>>> + grep \"Ignoring path folder1/\" sparse-checkout-err &&\n>>>\n>>> Is this desired behavior or just current behavior?\n>>>\n>>>> +\n>>>> + # When update-index is given a directory name WITHOUT a trailing slash, it will\n>>>> + # behave in different ways depending on the status of the directory on disk:\n>>>> + # * if it exists, the command exits with an error (\"add individual files instead\")\n>>>> + # * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n>>>> + # file and either triggers an error (\"does not exist and --remove not passed\")\n>>>> + # or is ignored completely (when using --remove)\n>>>> + test_all_match test_must_fail git update-index deep &&\n>>>> + run_on_all test_must_fail git update-indexe folder1 &&\n>>>\n>>> This one will fail for the wrong reason, though -- `update-indexe` is\n>>> not a valid subcommand. (extra 'e' at the end)\n>>>\n>>\n>> Thanks for catching that! I'll update in my next re-roll.\n>>\n>>>> + test_must_fail git -C full-checkout update-index --remove folder1 &&\n>>>> + test_sparse_match git update-index --remove folder1 &&\n>>>> + test_all_match git status --porcelain=v2\n>>>\n>>> Otherwise these seem reasonable.\n>>>\n>>>> +'\n>>>> +\n>>>> +test_expect_success 'update-index --again file outside sparse definition' '\n>>>> + init_repos &&\n>>>> +\n>>>> + write_script edit-contents <<-\\EOF &&\n>>>> + echo text >>$1\n>>>> + EOF\n>>>\n>>> Copy and paste and forget to remove? edit-contents doesn't seem to be\n>>> used in this test.\n>>>\n>>\n>> Will remove.\n>>\n>>>> +\n>>>> + test_all_match git checkout -b test-reupdate &&\n>>>> +\n>>>> + # Update HEAD without modifying the index to introduce a difference in\n>>>> + # folder1/a\n>>>> + test_sparse_match git reset --soft update-folder1 &&\n>>>> +\n>>>> + # Because folder1/a differs in the index vs HEAD,\n>>>> + # `git update-index --remove --again` will effectively perform\n>>>> + # `git update-index --remove folder1/a` and remove the folder1/a\n>>>> + test_sparse_match git update-index --remove --again &&\n>>>> + test_sparse_match git status --porcelain=v2\n>>>\n>>> This might need a --ignore-skip-worktree-entries, as per the\n>>> discussion above. Otherwise, this test makes sense.\n>>>\n>>\n>> The `--ignore-skip-worktree-entries` option is explicitly omitted because\n>> this test needs `update-index` to modify a `skip-worktree` entry. However,\n>> given the debate around what `--remove` should do, I'll update the scenario\n>> to not use `--remove` or any variation of it.\n>>\n>>>> +'\n>>>> +\n>>>> +test_expect_success 'update-index --cacheinfo' '\n>>>> + init_repos &&\n>>>> +\n>>>> + deep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n>>>> + folder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n>>>> + folder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n>>>> +\n>>>> + test_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n>>>> + test_all_match git status --porcelain=v2 &&\n>>>> +\n>>>> + # Cannot add sparse directory, even in sparse index case\n>>>> + test_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n>>>> +\n>>>> + # Sparse match only - because folder1/a is outside the sparse checkout\n>>>> + # definition (and thus not on-disk), it will appear \"deleted\" in\n>>>> + # unstaged changes.\n>>>> + test_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n>>>> + test_sparse_match git status --porcelain=v2\n>>>\n>>> Makes sense, because the update-index command removes the existing\n>>> cache entry and adds a new one without the SKIP_WORKTREE bit. But it\n>>> might be worth mentioning that in the commit message. Also, you could\n>>> follow this up with `git update-index --skip-worktree folder1/a`, and\n>>> then do a test_all_match git status --porcelain=v2, to show that when\n>>> the SKIP_WORKTREE bit is restored back to the file, then it again does\n>>> as expected despite not being on-disk.\n>>>\n>>\n>> I really like this - it helps clarify how `update-index` can be used to\n>> correctly add a sparse-checkout entry to the index with plumbing commands.\n>> I'll include it in V2.\n>>\n>>>> +'\n>>>> +\n>>>> test_expect_success 'merge, cherry-pick, and rebase' '\n>>>> init_repos &&\n>>>>\n>>>> --\n>>>> gitgitgadget\n\n"},{"id":"445872","messageId":"CABPp-BEgOi-UqH9bP0_OU6G7wCaE7g1GJN-RU5oaWPvwUW+imA@mail.gmail.com","threadId":"57180","inReplyTo":"bec659b2-8d69-c69c-73e5-a56ae415874b@github.com","subject":"Re: [PATCH 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-10T20:03:07Z","receivedAt":"2022-01-10T20:03:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jan 10, 2022 at 10:01 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Mon, Jan 10, 2022 at 7:47 AM Victoria Dye wrote:\n> >>\n> >> Elijah Newren wrote:\n> >>> On Tue, Jan 4, 2022 at 9:37 AM Victoria Dye via GitGitGadget\n> >>> wrote:\n> >>>>\n> >>>> From: Victoria Dye \n> >>>>\n...\n>\n> Sorry about that - when I wrote the first version of this series in the\n> `microsoft/git` fork, `git add` hadn't been updated to reject out-of-cone\n> untracked files as it is in [1]. It's my mistake for not double-checking\n> that it was still the case, apologies for wasting your time on re-explaining\n> this.\n\nNo worries; there's lots of moving parts in the sparse world.\n\n> In any case, I'll update the test comment and commit message per your\n> suggestion:\n>\n> >>> I might buy that `git update-index` is lower level and should be\n> >>> considered the same as `git add --sparse`, but the comment should\n> >>> mention that and try to sell why update-index should be equivalent to\n> >>> that instead of to `git add`.\n>\n> I'm leaning only slightly towards the current behavior (and will update the\n> comment accordingly), but I'm happy to change it if the reasoning isn't as\n> strong as that of another approach.\n>\n> [1] 105e8b014b (add: fail when adding an untracked sparse file, 2021-09-24)\n\nI'm also leaning slightly towards keeping the current behavior and\njust updating the comment.\n\n...\n> >> I understand why you find it buggy, but I am not making baseless assumptions\n> >> about the correctness (or lack thereof) of the current behavior.\n> >\n> > To be clear, the fact that the behavior was there for a decade would\n> > typically be basis enough for an assumption (in my opinion), and I\n> > wouldn't have faulted folks for making it. I might well have done so\n> > myself. My reasoning was just that I was getting confused by the\n> > negations and trying to understand the testcase, and when I started to\n> > unravel it, I found what looked like a possible inconsistency.\n> >\n> > Anyway, it's clear here you've actually dug a lot deeper and know the\n> > history here. In contrast, I was making assumptions about the history\n> > (and ones that weren't correct, though I'd argue my assumptions\n> > weren't baseless)...\n> >\n>\n> Your assumptions were completely valid, I apologize if my response came off\n> as implying otherwise. I'll try to use the comments on the tests to clarify\n> their behavior as much as possible, hopefully reducing some confusion around\n> all the multiple-negative flags & options.\n\nOh, no, not at all. I was just worried that my earlier response might\nhave come across poorly and could be read as criticizing assumptions I\n(incorrectly) _thought_ you were making. I wasn't sure if that was\nactually implied in your wording, but just to be careful I was just\ntrying to specify that I think the level of effort of your submissions\nis totally appropriate -- even if you had been making the assumptions\nI thought you had been. And I was pointing out that I, after all, had\nmade and will continue to make assumptions too. The point of review\nis getting a second pair of eyes, because they can help catch issues;\nand finding those issues often includes double-checking unspoken\nassumptions.\n\nAnyway, I think we're on the same page. Thanks for your hard work\nhere; looking forward to seeing your reroll!\n"},{"id":"445937","messageId":"eefb6ab4c61460d21dca3877b83a0e6c2852c43c.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 1/9] reset: fix validation in sparse index test","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:04:58Z","receivedAt":"2022-01-11T18:06:44Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate t1092 test 'reset with pathspecs outside sparse definition' to verify\nindex contents. The use of `rev-parse` verifies the contents of HEAD, not\nthe index, providing no real validation of the reset results. Conversely,\n`ls-files` reports the contents of the index (OIDs, flags, filenames), which\nare then compared across checkouts to ensure compatible index states.\n\nFixes 741a2c9ffa (reset: expand test coverage for sparse checkouts,\n2021-09-27).\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 6 ++----\n 1 file changed, 2 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 49f70a65692..d5167e7ed69 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -596,13 +596,11 @@ test_expect_success 'reset with pathspecs outside sparse definition' '\n \n \ttest_sparse_match git reset update-folder1 -- folder1 &&\n \tgit -C full-checkout reset update-folder1 -- folder1 &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_all_match git rev-parse HEAD:folder1 &&\n+\ttest_all_match git ls-files -s -- folder1 &&\n \n \ttest_sparse_match git reset update-folder2 -- folder2/a &&\n \tgit -C full-checkout reset update-folder2 -- folder2/a &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_all_match git rev-parse HEAD:folder2/a\n+\ttest_all_match git ls-files -s -- folder2/a\n '\n \n test_expect_success 'reset with wildcard pathspec' '\n-- \ngitgitgadget\n\n"},{"id":"445938","messageId":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.git.1641317820.gitgitgadget@gmail.com","subject":"[PATCH v2 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:04:57Z","receivedAt":"2022-01-11T18:06:44Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"This series continues the work to integrate commands with the sparse index,\nadding integrations with 'git clean', 'git checkout-index', and 'git\nupdate-index'. These three commands, while useful in their own right, are\nupdated mainly because they're used in 'git stash'. A future series will\nintegrate sparse index with 'stash' directly, but its subcommands must be\nintegrated to avoid the performance cost of each one expanding and\ncollapsing the index.\n\nThe series is broken up into 4 parts:\n\n * Patches 1-2 are minor fixups to the 'git reset' sparse index integration\n in response to discussion [1] that came after the series was ready for\n merge to 'next'.\n * Patch 3 integrates 'git clean' with the sparse index.\n * Patches 4-6 integrate 'git checkout-index' with the sparse index and\n introduce a new '--ignore-skip-worktree-bits' option.\n * This involves changing the behavior of 'checkout-index' to respect\n 'skip-worktree' by default (i.e., it won't check out 'skip-worktree'\n files). The '--ignore-skip-worktree-bits' option can be specified to\n force checkout of 'skip-worktree' files, if desired.\n * Patches 7-9 integrate 'git update-index' with the sparse index.\n * Note that, although this integrates the sparse index with\n '--cacheinfo', sparse directories still cannot be updated using that\n option (see the prior discussion [2] for more details on why)\n\n\nChanges since V1\n================\n\n * Changed 'checkout-index' to fail by default when given filenames of files\n with 'skip-worktree' enabled\n * These files can still be forcibly checked-out by using the\n '--ignore-skip-worktree-bits' option\n * Added/updated corresponding t1092 tests\n * Updated t1092 'update-index' tests\n * Mentioned where/why 'skip-worktree' files were manually created on-disk\n for testing purposes\n * Provided explanation as to what '--remove' does, and how it relates to\n '--ignore-skip-worktree-entries'; restructured corresponding test\n * Fixed typo 'update-indexe' -> 'update-index'\n * Removed unused 'edit-contents'\n * Changed '--again' test to not use '--remove' to avoid confusion over\n how/why it updates 'skip-worktree' entries\n * Added \"set skip-worktree\" step to '--cacheinfo' test to illustrate how\n it could be used to add a new outside-of-cone file and remain generally\n compliant with a sparse-checkout definition\n * Added '--cacheinfo' test to \"ensure not expanded\"\n * Moved t1092 test 'sparse index is not expanded: update-index' to avoid\n merge conflict\n * Updated p2000 test for 'update-index': added file argument\n * Without any file arguments, 'update-index' was effectively a no-op\n * Clarified reasoning behind changing/not changing behavior of update-index\n in sparse-checkouts\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/CABPp-BG0iDHf268UAnRyA=0y0T69YTc+bLMdxCmSbrL8s=9ziA@mail.gmail.com/\n\n[2]\nhttps://lore.kernel.org/git/a075091c-d0d4-db5d-fa21-c9d6c90c343e@gmail.com/\n\nVictoria Dye (9):\n reset: fix validation in sparse index test\n reset: reorder wildcard pathspec conditions\n clean: integrate with sparse index\n checkout-index: expand sparse checkout compatibility tests\n checkout-index: add --ignore-skip-worktree-bits option\n checkout-index: integrate with sparse index\n update-index: add tests for sparse-checkout compatibility\n update-index: integrate with sparse index\n update-index: reduce scope of index expansion in do_reupdate\n\n Documentation/git-checkout-index.txt | 10 +-\n builtin/checkout-index.c | 41 +++-\n builtin/clean.c | 3 +\n builtin/reset.c | 12 +-\n builtin/update-index.c | 17 +-\n read-cache.c | 10 +-\n t/perf/p2000-sparse-operations.sh | 2 +\n t/t1092-sparse-checkout-compatibility.sh | 282 ++++++++++++++++++++++-\n 8 files changed, 360 insertions(+), 17 deletions(-)\n\n\nbase-commit: dcc0cd074f0c639a0df20461a301af6d45bd582e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1109%2Fvdye%2Fsparse%2Fupdate-index_checkout-index_clean-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1109/vdye/sparse/update-index_checkout-index_clean-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1109\n\nRange-diff vs v1:\n\n 1: eefb6ab4c61 = 1: eefb6ab4c61 reset: fix validation in sparse index test\n 2: 0194d894c2f = 2: 0194d894c2f reset: reorder wildcard pathspec conditions\n 3: 52aec13d18e = 3: 52aec13d18e clean: integrate with sparse index\n 4: e6a8671f6be ! 4: d964507fdcc checkout-index: expand sparse checkout compatibility tests\n @@ Commit message\n \n Add tests to cover `checkout-index`, with a focus on cases interesting in a\n sparse checkout (e.g., files specified outside sparse checkout definition).\n - New tests are intended to serve as a baseline for expected behavior and\n - performance when integrating `checkout-index` with the sparse index.\n +\n + New tests are intended to serve as a baseline for existing and/or expected\n + behavior and performance when integrating `checkout-index` with the sparse\n + index. Note that the test 'checkout-index --all' is marked as\n + 'test_expect_failure', indicating that `update-index --all` will be modified\n + in a subsequent patch to behave as the test expects.\n \n Signed-off-by: Victoria Dye \n \n 5: ec9a751e8dc ! 5: 601888606d1 checkout-index: add --ignore-skip-worktree-bits option\n @@ Metadata\n ## Commit message ##\n checkout-index: add --ignore-skip-worktree-bits option\n \n - Update `checkout-index --all` to no longer refresh files that have the\n - `skip-worktree` bit set. The newly-added `--ignore-skip-worktree-bits`\n - option, when used with `--all`, maintains the old behavior and checks out\n - all files regardless of `skip-worktree`.\n + Update `checkout-index` to no longer refresh files that have the\n + `skip-worktree` bit set, exiting with an error if `skip-worktree` filenames\n + are directly provided to `checkout-index`. The newly-added\n + `--ignore-skip-worktree-bits` option provides a mechanism to replicate the\n + old behavior, checking out *all* files specified (even those with\n + `skip-worktree` enabled).\n \n The ability to toggle whether files should be checked-out based on\n `skip-worktree` already exists in `git checkout` and `git restore` (both of\n - which have an `--ignore-skip-worktree-bits` option). Adding the option to\n - `checkout-index` (and changing the corresponding default behavior to respect\n - the `skip-worktree` bit) is especially helpful for sparse-checkout: it\n - prevents inadvertent creation of *all* files outside the sparse definition\n - on disk and eliminates the need to expand a sparse index by default when\n - using the `--all` option.\n + which have an `--ignore-skip-worktree-bits` option). The change to, by\n + default, ignore `skip-worktree` files is especially helpful for\n + sparse-checkout; it prevents inadvertent creation of files outside the\n + sparse definition on disk and eliminates the need to expand a sparse index\n + when using the `--all` option.\n \n Internal usage of `checkout-index` in `git stash` and `git filter-branch` do\n not make explicit use of files with `skip-worktree` enabled, so\n `--ignore-skip-worktree-bits` is not added to them.\n \n + Helped-by: Elijah Newren \n Signed-off-by: Victoria Dye \n \n ## Documentation/git-checkout-index.txt ##\n @@ Documentation/git-checkout-index.txt: OPTIONS\n \n +--ignore-skip-worktree-bits::\n +\tCheck out all files, including those with the skip-worktree bit\n -+\tset. Note: may only be used with `--all`; skip-worktree is\n -+\tignored when explicit filenames are specified.\n ++\tset.\n +\n --stdin::\n \tInstead of taking list of paths from the command line,\n @@ builtin/checkout-index.c\n #include \"lockfile.h\"\n #include \"quote.h\"\n #include \"cache-tree.h\"\n +@@\n + #define CHECKOUT_ALL 4\n + static int nul_term_line;\n + static int checkout_stage; /* default to checkout stage0 */\n ++static int ignore_skip_worktree; /* default to 0 */\n + static int to_tempfile;\n + static char topath[4][TEMPORARY_FILENAME_LENGTH + 1];\n + \n @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n - \treturn -1;\n - }\n + \tint namelen = strlen(name);\n + \tint pos = cache_name_pos(name, namelen);\n + \tint has_same_name = 0;\n ++\tint is_skipped = 1;\n + \tint did_checkout = 0;\n + \tint errs = 0;\n \n --static int checkout_all(const char *prefix, int prefix_length)\n -+static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n - {\n - \tint i, errs = 0;\n - \tstruct cache_entry *last_ce = NULL;\n +@@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n + \t\t\tbreak;\n + \t\thas_same_name = 1;\n + \t\tpos++;\n ++\t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n ++\t\t\tbreak;\n ++\t\tis_skipped = 0;\n + \t\tif (ce_stage(ce) != checkout_stage\n + \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n + \t\t\tcontinue;\n +@@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n + \t\tfprintf(stderr, \"git checkout-index: %s \", name);\n + \t\tif (!has_same_name)\n + \t\t\tfprintf(stderr, \"is not in the cache\");\n ++\t\telse if (is_skipped)\n ++\t\t\tfprintf(stderr, \"has skip-worktree enabled; \"\n ++\t\t\t\t\t\"use '--ignore-skip-worktree-bits' to checkout\");\n + \t\telse if (checkout_stage)\n + \t\t\tfprintf(stderr, \"does not exist at stage %d\",\n + \t\t\t\tcheckout_stage);\n @@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n @@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix\n \t\tif (ce_stage(ce) != checkout_stage\n \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n \t\t\tcontinue;\n -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n - \tint i;\n - \tstruct lock_file lock_file = LOCK_INIT;\n - \tint all = 0;\n -+\tint ignore_skip_worktree = 0;\n - \tint read_from_stdin = 0;\n - \tint prefix_length;\n - \tint force = 0, quiet = 0, not_new = 0;\n @@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_checkout_index_options[] = {\n \t\tOPT_BOOL('a', \"all\", &all,\n @@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, co\n \t\tOPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n \t\tOPT__QUIET(&quiet,\n \t\t\tN_(\"no warning for existing files and files not in index\")),\n -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n +\n + ## t/t1092-sparse-checkout-compatibility.sh ##\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index inside sparse definition' '\n + test_expect_success 'checkout-index outside sparse definition' '\n + \tinit_repos &&\n \n - \t\tif (all)\n - \t\t\tdie(\"git checkout-index: don't mix '--all' and explicit filenames\");\n -+\t\tif (ignore_skip_worktree)\n -+\t\t\tdie(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n - \t\tif (read_from_stdin)\n - \t\t\tdie(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n - \t\tp = prefix_path(prefix, prefix_length, arg);\n -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n - \t}\n +-\t# File does not exist on disk yet for sparse checkouts, so checkout-index\n +-\t# succeeds without -f\n +-\ttest_sparse_match git checkout-index -- folder1/a &&\n ++\t# Without --ignore-skip-worktree-bits, outside-of-cone files will trigger\n ++\t# an error\n ++\ttest_sparse_match test_must_fail git checkout-index -- folder1/a &&\n ++\ttest_i18ngrep \"folder1/a has skip-worktree enabled\" sparse-checkout-err &&\n ++\ttest_path_is_missing folder1/a &&\n ++\n ++\t# With --ignore-skip-worktree-bits, outside-of-cone files are checked out\n ++\ttest_sparse_match git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n + \ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n + \ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n \n - \tif (all)\n --\t\terr |= checkout_all(prefix, prefix_length);\n -+\t\terr |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index outside sparse definition' '\n + \trun_on_sparse mkdir -p folder1 &&\n + \trun_on_all cp ../new-a folder1/a &&\n \n - \tif (pc_workers > 1)\n - \t\terr |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n -\n - ## t/t1092-sparse-checkout-compatibility.sh ##\n +-\ttest_all_match test_must_fail git checkout-index -- folder1/a &&\n +-\ttest_all_match git checkout-index -f -- folder1/a &&\n ++\ttest_all_match test_must_fail git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n ++\ttest_all_match git checkout-index -f --ignore-skip-worktree-bits -- folder1/a &&\n + \ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n + \ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n + '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index with folders' '\n \ttest_all_match test_must_fail git checkout-index -f -- folder1/\n '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index wi\n -\ttest_sparse_match test_path_is_missing folder1\n +\ttest_sparse_match test_path_is_missing folder1 &&\n +\n ++\t# --ignore-skip-worktree-bits will cause `skip-worktree` files to be\n ++\t# checked out, causing the outside-of-cone `folder1` to exist on-disk\n +\ttest_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n +\ttest_all_match test_path_exists folder1\n '\n 6: 18c00fc9dd3 ! 6: b4b9086dcdc checkout-index: integrate with sparse index\n @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char\n \tint pos = cache_name_pos(name, namelen);\n \tint has_same_name = 0;\n +\tint is_file = 0;\n + \tint is_skipped = 1;\n \tint did_checkout = 0;\n \tint errs = 0;\n - \n @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n \t\t\tbreak;\n \t\thas_same_name = 1;\n @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char\n +\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n +\t\t\tbreak;\n +\t\tis_file = 1;\n - \t\tif (ce_stage(ce) != checkout_stage\n - \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n - \t\t\tcontinue;\n + \t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n + \t\t\tbreak;\n + \t\tis_skipped = 0;\n @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n \t\tfprintf(stderr, \"git checkout-index: %s \", name);\n \t\tif (!has_same_name)\n \t\t\tfprintf(stderr, \"is not in the cache\");\n +\t\telse if (!is_file)\n +\t\t\tfprintf(stderr, \"is a sparse directory\");\n - \t\telse if (checkout_stage)\n - \t\t\tfprintf(stderr, \"does not exist at stage %d\",\n - \t\t\t\tcheckout_stage);\n -@@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_w\n + \t\telse if (is_skipped)\n + \t\t\tfprintf(stderr, \"has skip-worktree enabled; \"\n + \t\t\t\t\t\"use '--ignore-skip-worktree-bits' to checkout\");\n +@@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length)\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n 7: 3b734f89c0f ! 7: ff32952a21c update-index: add tests for sparse-checkout compatibility\n @@ Commit message\n update-index: add tests for sparse-checkout compatibility\n \n Introduce tests for a variety of `git update-index` use cases, including\n - performance scenarios. Tests for `update-index add/remove` are specifically\n - focused on how `git stash` uses `git update-index` as a subcommand to\n - prepare for sparse index integration with `stash` in a future series.\n + performance scenarios. Tests are intended to exercise `update-index` with\n + options that change the commands interaction with the index (e.g.,\n + `--again`) and with files/directories inside and outside a sparse checkout\n + cone.\n +\n + Of note is that these tests clearly establish the behavior of `git\n + update-index --add` with untracked, outside-of-cone files. Unlike `git add`,\n + which fails with an error when provided with such files, `update-index`\n + succeeds in adding them to the index. Additionally, the `skip-worktree` flag\n + is *not* automatically added to the new entry. Although this is pre-existing\n + behavior, there are a couple of reasons to avoid changing it in favor of\n + consistency with e.g. `git add`:\n +\n + * `update-index` is low-level command for modifying the index; while it can\n + perform operations similar to those of `add`, it traditionally has fewer\n + \"guardrails\" preventing a user from doing something they may not want to\n + do (in this case, adding an outside-of-cone, non-`skip-worktree` file to\n + the index)\n + * `update-index` typically only exits with an error code if it is incapable\n + of performing an operation (e.g., if an internal function call fails);\n + adding a new file outside the sparse checkout definition is still a valid\n + operation, albeit an inadvisable one\n + * `update-index` does not implicitly set flags (e.g., `skip-worktree`) when\n + creating new index entries with `--add`; if flags need to be updated,\n + options like `--[no-]skip-worktree` allow a user to intentionally set them\n +\n + All this to say that, while there are valid reasons to consider changing the\n + treatment of outside-of-cone files in `update-index`, there are also\n + sufficient reasons for leaving it as-is.\n \n Co-authored-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n test_perf_on_all git checkout-index -f --all\n -+test_perf_on_all git update-index --add --remove\n ++test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n \n test_done\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +\tEOF\n +\n +\t# Create & modify folder1/a\n ++\t# Note that this setup is a manual way of reaching the erroneous\n ++\t# condition in which a `skip-worktree` enabled, outside-of-cone file\n ++\t# exists on disk. It is used here to ensure `update-index` is stable\n ++\t# and behaves predictably if such a condition occurs.\n +\trun_on_sparse mkdir -p folder1 &&\n +\trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n +\trun_on_all ../edit-contents folder1/a &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +\trun_on_sparse mkdir -p folder1 &&\n +\trun_on_all ../edit-contents folder1/b &&\n +\n -+\t# Similar to `git add`, the untracked out-of-cone file is added to the index\n -+\t# identically across sparse and non-sparse checkouts\n ++\t# The *untracked* out-of-cone file is added to the index because it does\n ++\t# not have a `skip-worktree` bit to signal that it should be ignored\n ++\t# (unlike in `git add`, which will fail due to the file being outside\n ++\t# the sparse checkout definition).\n +\ttest_all_match git update-index --add folder1/b &&\n +\ttest_all_match git status --porcelain=v2\n +'\n +\n ++# NEEDSWORK: `--remove`, unlike the rest of `update-index`, does not ignore\n ++# `skip-worktree` entries by default and will remove them from the index.\n ++# The `--ignore-skip-worktree-entries` flag must be used in conjunction with\n ++# `--remove` to ignore the `skip-worktree` entries and prevent their removal\n ++# from the index.\n +test_expect_success 'update-index --remove outside sparse definition' '\n +\tinit_repos &&\n +\n -+\t# When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n -+\t# not removed from the index if they do not exist on disk\n ++\t# When --ignore-skip-worktree-entries is _not_ specified:\n ++\t# out-of-cone, not-on-disk files are removed from the index\n ++\ttest_sparse_match git update-index --remove folder1/a &&\n ++\tcat >expect <<-EOF &&\n ++\tD\tfolder1/a\n ++\tEOF\n ++\ttest_sparse_match git diff --cached --name-status &&\n ++\ttest_cmp expect sparse-checkout-out &&\n ++\n ++\t# Reset the state\n ++\ttest_all_match git reset --hard &&\n ++\n ++\t# When --ignore-skip-worktree-entries is specified, out-of-cone\n ++\t# (skip-worktree) files are ignored\n +\ttest_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n -+\ttest_all_match git status --porcelain=v2 &&\n ++\ttest_sparse_match git diff --cached --name-status &&\n ++\ttest_must_be_empty sparse-checkout-out &&\n +\n -+\t# When the flag is _not_ specified, out-of-cone, not-on-disk files are\n -+\t# removed from the index\n -+\trm full-checkout/folder1/a &&\n -+\ttest_all_match git update-index --remove folder1/a &&\n -+\ttest_all_match git status --porcelain=v2 &&\n ++\t# Reset the state\n ++\ttest_all_match git reset --hard &&\n +\n -+\t# NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n ++\t# --force-remove supercedes --ignore-skip-worktree-entries, removing\n +\t# a skip-worktree file from the index (and disk) when both are specified\n -+\ttest_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n -+\ttest_all_match git status --porcelain=v2\n ++\t# with --remove\n ++\ttest_sparse_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n ++\tcat >expect <<-EOF &&\n ++\tD\tfolder1/a\n ++\tEOF\n ++\ttest_sparse_match git diff --cached --name-status &&\n ++\ttest_cmp expect sparse-checkout-out\n +'\n +\n +test_expect_success 'update-index with directories' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +\t# file and either triggers an error (\"does not exist and --remove not passed\")\n +\t# or is ignored completely (when using --remove)\n +\ttest_all_match test_must_fail git update-index deep &&\n -+\trun_on_all test_must_fail git update-indexe folder1 &&\n ++\trun_on_all test_must_fail git update-index folder1 &&\n +\ttest_must_fail git -C full-checkout update-index --remove folder1 &&\n +\ttest_sparse_match git update-index --remove folder1 &&\n +\ttest_all_match git status --porcelain=v2\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +test_expect_success 'update-index --again file outside sparse definition' '\n +\tinit_repos &&\n +\n -+\twrite_script edit-contents <<-\\EOF &&\n -+\techo text >>$1\n -+\tEOF\n -+\n +\ttest_all_match git checkout -b test-reupdate &&\n +\n +\t# Update HEAD without modifying the index to introduce a difference in\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +\ttest_sparse_match git reset --soft update-folder1 &&\n +\n +\t# Because folder1/a differs in the index vs HEAD,\n -+\t# `git update-index --remove --again` will effectively perform\n -+\t# `git update-index --remove folder1/a` and remove the folder1/a\n -+\ttest_sparse_match git update-index --remove --again &&\n -+\ttest_sparse_match git status --porcelain=v2\n ++\t# `git update-index --no-skip-worktree --again` will effectively perform\n ++\t# `git update-index --no-skip-worktree folder1/a` and remove the skip-worktree\n ++\t# flag from folder1/a\n ++\ttest_sparse_match git update-index --no-skip-worktree --again &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\tD\tfolder1/a\n ++\tEOF\n ++\ttest_sparse_match git diff --name-status &&\n ++\ttest_cmp expect sparse-checkout-out\n +'\n +\n +test_expect_success 'update-index --cacheinfo' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n +\t# Cannot add sparse directory, even in sparse index case\n +\ttest_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n +\n -+\t# Sparse match only - because folder1/a is outside the sparse checkout\n -+\t# definition (and thus not on-disk), it will appear \"deleted\" in\n -+\t# unstaged changes.\n -+\ttest_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n -+\ttest_sparse_match git status --porcelain=v2\n ++\t# Sparse match only: the new outside-of-cone entry is added *without* skip-worktree,\n ++\t# so `git status` reports it as \"deleted\" in the worktree\n ++\ttest_sparse_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n ++\ttest_sparse_match git status --porcelain=v2 &&\n ++\tcat >expect <<-EOF &&\n ++\tMD folder1/a\n ++\tEOF\n ++\ttest_sparse_match git status --short -- folder1/a &&\n ++\ttest_cmp expect sparse-checkout-out &&\n ++\n ++\t# To return folder1/a to \"normal\" for a sparse checkout (ignored &\n ++\t# outside-of-cone), add the skip-worktree flag.\n ++\ttest_sparse_match git update-index --skip-worktree folder1/a &&\n ++\tcat >expect <<-EOF &&\n ++\tS folder1/a\n ++\tEOF\n ++\ttest_sparse_match git ls-files -t -- folder1/a &&\n ++\ttest_cmp expect sparse-checkout-out\n +'\n +\n test_expect_success 'merge, cherry-pick, and rebase' '\n 8: c5b98e36516 ! 8: 9ddc51a47d5 update-index: integrate with sparse index\n @@ Metadata\n ## Commit message ##\n update-index: integrate with sparse index\n \n - Enable usage of the sparse index with `update-index`. Most variations of\n + Enable use of the sparse index with `update-index`. Most variations of\n `update-index` work without explicitly expanding the index or making any\n other updates in or outside of `update-index.c`.\n \n @@ read-cache.c: static int add_index_entry_with_check(struct index_state *istate,\n \t\tif (!new_only)\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: blame' '\n - \tdone\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: diff' '\n + \tensure_not_expanded diff --cached\n '\n \n +test_expect_success 'sparse index is not expanded: update-index' '\n +\tinit_repos &&\n +\n ++\tdeep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n ++\tensure_not_expanded update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n ++\n +\techo \"test\" >sparse-index/README.md &&\n +\techo \"test2\" >sparse-index/a &&\n +\trm -f sparse-index/deep/a &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\tensure_not_expanded update-index --remove deep/a\n +'\n +\n - # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n - # in this scenario, but it shouldn't.\n - test_expect_success 'reset mixed and checkout orphan' '\n + test_expect_success 'sparse index is not expanded: blame' '\n + \tinit_repos &&\n + \n 9: de7fc143562 ! 9: 80697e9259e update-index: reduce scope of index expansion in do_reupdate\n @@ Metadata\n ## Commit message ##\n update-index: reduce scope of index expansion in do_reupdate\n \n - Expand the full index (and redo reupdate operation) only if a sparse\n - directory in the index differs from HEAD. Only the index entries that differ\n - between the index and HEAD are updated when performing `git update-index\n - --again`, so unmodified sparse directories are safely skipped. The index\n - does need to be expanded when sparse directories contain changes, though,\n - because `update_one(...)` will not operate on sparse directory index\n - entries.\n + Replace unconditional index expansion in 'do_reupdate()' with one scoped to\n + only where a full index is needed. A full index is only required in\n + 'do_reupdate()' when a sparse directory in the index differs from HEAD; in\n + that case, the index is expanded and the operation restarted.\n \n Because the index should only be expanded if a sparse directory is modified,\n add a test ensuring the index is not expanded when differences only exist\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\tensure_not_expanded update-index --add --remove --again\n '\n \n - # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n + test_expect_success 'sparse index is not expanded: blame' '\n\n-- \ngitgitgadget\n"},{"id":"445939","messageId":"0194d894c2f9c5ae4bbaac4cd6fca89933a04789.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 2/9] reset: reorder wildcard pathspec conditions","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:04:59Z","receivedAt":"2022-01-11T18:06:48Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRearrange conditions in method determining whether index expansion is\nnecessary when a pathspec is specified for `git reset`, placing less\nexpensive condition first. Additionally, add details & examples to related\ncode comments to help with readability.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/reset.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/reset.c b/builtin/reset.c\nindex b1ff699b43a..79b40385b99 100644\n--- a/builtin/reset.c\n+++ b/builtin/reset.c\n@@ -204,10 +204,16 @@ static int pathspec_needs_expanded_index(const struct pathspec *pathspec)\n \t\t\t/*\n \t\t\t * Special case: if the pattern is a path inside the cone\n \t\t\t * followed by only wildcards, the pattern cannot match\n-\t\t\t * partial sparse directories, so we don't expand the index.\n+\t\t\t * partial sparse directories, so we know we don't need to\n+\t\t\t * expand the index.\n+\t\t\t *\n+\t\t\t * Examples:\n+\t\t\t * - in-cone/foo***: doesn't need expanded index\n+\t\t\t * - not-in-cone/bar*: may need expanded index\n+\t\t\t * - **.c: may need expanded index\n \t\t\t */\n-\t\t\tif (path_in_cone_mode_sparse_checkout(item.original, &the_index) &&\n-\t\t\t strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len)\n+\t\t\tif (strspn(item.original + item.nowildcard_len, \"*\") == item.len - item.nowildcard_len &&\n+\t\t\t path_in_cone_mode_sparse_checkout(item.original, &the_index))\n \t\t\t\tcontinue;\n \n \t\t\tfor (pos = 0; pos < active_nr; pos++) {\n-- \ngitgitgadget\n\n"},{"id":"445940","messageId":"d964507fdcc580042c3f7d1e8e3146edd5f6d0ce.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 4/9] checkout-index: expand sparse checkout compatibility tests","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:01Z","receivedAt":"2022-01-11T18:06:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd tests to cover `checkout-index`, with a focus on cases interesting in a\nsparse checkout (e.g., files specified outside sparse checkout definition).\n\nNew tests are intended to serve as a baseline for existing and/or expected\nbehavior and performance when integrating `checkout-index` with the sparse\nindex. Note that the test 'checkout-index --all' is marked as\n'test_expect_failure', indicating that `update-index --all` will be modified\nin a subsequent patch to behave as the test expects.\n\nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 54 ++++++++++++++++++++++++\n 2 files changed, 55 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex cb777c74a24..54f8602f3c1 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -117,5 +117,6 @@ test_perf_on_all git diff\n test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n+test_perf_on_all git checkout-index -f --all\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 05587361452..db7ad41109b 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -755,6 +755,60 @@ test_expect_success 'cherry-pick with conflicts' '\n \ttest_all_match test_must_fail git cherry-pick to-cherry-pick\n '\n \n+test_expect_success 'checkout-index inside sparse definition' '\n+\tinit_repos &&\n+\n+\trun_on_all rm -f deep/a &&\n+\ttest_all_match git checkout-index -- deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\techo test >>new-a &&\n+\trun_on_all cp ../new-a a &&\n+\ttest_all_match test_must_fail git checkout-index -- a &&\n+\ttest_all_match git checkout-index -f -- a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'checkout-index outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# File does not exist on disk yet for sparse checkouts, so checkout-index\n+\t# succeeds without -f\n+\ttest_sparse_match git checkout-index -- folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n+\n+\trun_on_sparse rm -rf folder1 &&\n+\techo test >new-a &&\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_all cp ../new-a folder1/a &&\n+\n+\ttest_all_match test_must_fail git checkout-index -- folder1/a &&\n+\ttest_all_match git checkout-index -f -- folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n+\ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n+'\n+\n+test_expect_success 'checkout-index with folders' '\n+\tinit_repos &&\n+\n+\t# Inside checkout definition\n+\ttest_all_match test_must_fail git checkout-index -f -- deep/ &&\n+\n+\t# Outside checkout definition\n+\ttest_all_match test_must_fail git checkout-index -f -- folder1/\n+'\n+\n+# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n+# files (even those outside the sparse definition) on disk. However, these files\n+# don't appear in the percentage of tracked files in git status.\n+test_expect_failure 'checkout-index --all' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout-index --all &&\n+\ttest_sparse_match test_path_is_missing folder1\n+'\n+\n test_expect_success 'clean' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"445941","messageId":"52aec13d18e339024e350733ee9d0b5c6b9008e6.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 3/9] clean: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:00Z","receivedAt":"2022-01-11T18:06:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRemove full index requirement for `git clean` and test to ensure the index\nis not expanded in `git clean`. Add to existing test for `git clean` to\nverify cleanup of untracked files in sparse directories is consistent\nbetween sparse index and non-sparse index checkouts.\n\nSigned-off-by: Victoria Dye \n---\n builtin/clean.c | 3 +++\n t/t1092-sparse-checkout-compatibility.sh | 21 +++++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex 98a2860409b..5628fc7103e 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -983,6 +983,9 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tdir.flags |= DIR_KEEP_UNTRACKED_CONTENTS;\n \t}\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tif (read_cache() < 0)\n \t\tdie(_(\"index file corrupt\"));\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d5167e7ed69..05587361452 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -764,23 +764,42 @@ test_expect_success 'clean' '\n \ttest_all_match git commit -m \"ignore bogus files\" &&\n \n \trun_on_sparse mkdir folder1 &&\n+\trun_on_all mkdir -p deep/untracked-deep &&\n \trun_on_all touch folder1/bogus &&\n+\trun_on_all touch folder1/untracked &&\n+\trun_on_all touch deep/untracked-deep/bogus &&\n+\trun_on_all touch deep/untracked-deep/untracked &&\n \n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git clean -f &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_exists folder1/bogus &&\n+\trun_on_all test_path_is_missing folder1/untracked &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/untracked &&\n+\n+\ttest_all_match git clean -fd &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_sparse_match ls &&\n+\ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_exists folder1/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n+\trun_on_all test_path_is_missing deep/untracked-deep/untracked &&\n \n \ttest_all_match git clean -xf &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_is_missing folder1/bogus &&\n+\trun_on_all test_path_exists deep/untracked-deep/bogus &&\n \n \ttest_all_match git clean -xdf &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_sparse_match ls &&\n \ttest_sparse_match ls folder1 &&\n+\trun_on_all test_path_is_missing deep/untracked-deep/bogus &&\n \n \ttest_sparse_match test_path_is_dir folder1\n '\n@@ -920,6 +939,8 @@ test_expect_success 'sparse-index is not expanded' '\n \t# Wildcard identifies only full sparse directories, no index expansion\n \tensure_not_expanded reset deepest -- folder\\* &&\n \n+\tensure_not_expanded clean -fd &&\n+\n \tensure_not_expanded checkout -f update-deep &&\n \ttest_config -C sparse-index pull.twohead ort &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"445942","messageId":"b4b9086dcdc62eff27f9520b9495e3b273946e39.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 6/9] checkout-index: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:03Z","receivedAt":"2022-01-11T18:06:52Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd repository settings to allow usage of the sparse index.\n\nWhen using the `--all` option, sparse directories are ignored by default due\nto the `skip-worktree` flag, so there is no need to expand the index. If\n`--ignore-skip-worktree-bits` is specified, the index is expanded in order\nto check out all files.\n\nWhen checking out individual files, existing behavior in a full index is to\nexit with an error if a directory is specified (as the directory name will\nnot match an index entry). However, it is possible in a sparse index to\nmatch a directory name to a sparse directory index entry, but checking out\nthat sparse directory still results in an error on checkout. To reduce some\npotential confusion for users, `checkout_file(...)` explicitly exits with an\ninformative error if provided with a sparse directory name. The test\ncorresponding to this scenario verifies the error message, which now differs\nbetween sparse index and non-sparse index checkouts.\n\nSigned-off-by: Victoria Dye \n---\n builtin/checkout-index.c | 28 ++++++++++++++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++-\n 2 files changed, 36 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 615a118e2f5..97e06e8c52c 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -67,6 +67,7 @@ static int checkout_file(const char *name, const char *prefix)\n \tint namelen = strlen(name);\n \tint pos = cache_name_pos(name, namelen);\n \tint has_same_name = 0;\n+\tint is_file = 0;\n \tint is_skipped = 1;\n \tint did_checkout = 0;\n \tint errs = 0;\n@@ -81,6 +82,9 @@ static int checkout_file(const char *name, const char *prefix)\n \t\t\tbreak;\n \t\thas_same_name = 1;\n \t\tpos++;\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode))\n+\t\t\tbreak;\n+\t\tis_file = 1;\n \t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tbreak;\n \t\tis_skipped = 0;\n@@ -112,6 +116,8 @@ static int checkout_file(const char *name, const char *prefix)\n \t\tfprintf(stderr, \"git checkout-index: %s \", name);\n \t\tif (!has_same_name)\n \t\t\tfprintf(stderr, \"is not in the cache\");\n+\t\telse if (!is_file)\n+\t\t\tfprintf(stderr, \"is a sparse directory\");\n \t\telse if (is_skipped)\n \t\t\tfprintf(stderr, \"has skip-worktree enabled; \"\n \t\t\t\t\t\"use '--ignore-skip-worktree-bits' to checkout\");\n@@ -130,10 +136,25 @@ static int checkout_all(const char *prefix, int prefix_length)\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tif (!ce_skip_worktree(ce))\n+\t\t\t\tBUG(\"sparse directory '%s' does not have skip-worktree set\", ce->name);\n+\n+\t\t\t/*\n+\t\t\t * If the current entry is a sparse directory and skip-worktree\n+\t\t\t * entries are being checked out, expand the index and continue\n+\t\t\t * the loop on the current index position (now pointing to the\n+\t\t\t * first entry inside the expanded sparse directory).\n+\t\t\t */\n+\t\t\tif (ignore_skip_worktree) {\n+\t\t\t\tensure_full_index(&the_index);\n+\t\t\t\tce = active_cache[i];\n+\t\t\t}\n+\t\t}\n+\n \t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (ce_stage(ce) != checkout_stage\n@@ -225,6 +246,9 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tgit_config(git_default_config, NULL);\n \tprefix_length = prefix ? strlen(prefix) : 0;\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tif (read_cache() < 0) {\n \t\tdie(\"invalid cache\");\n \t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 434ef0433c0..0c72c854d84 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -801,7 +801,14 @@ test_expect_success 'checkout-index with folders' '\n \ttest_all_match test_must_fail git checkout-index -f -- deep/ &&\n \n \t# Outside checkout definition\n-\ttest_all_match test_must_fail git checkout-index -f -- folder1/\n+\t# Note: although all tests fail (as expected), the messaging differs. For\n+\t# non-sparse index checkouts, the error is that the \"file\" does not appear\n+\t# in the index; for sparse checkouts, the error is explicitly that the\n+\t# entry is a sparse directory.\n+\trun_on_all test_must_fail git checkout-index -f -- folder1/ &&\n+\ttest_cmp full-checkout-err sparse-checkout-err &&\n+\t! test_cmp full-checkout-err sparse-index-err &&\n+\tgrep \"is a sparse directory\" sparse-index-err\n '\n \n test_expect_success 'checkout-index --all' '\n@@ -972,6 +979,8 @@ test_expect_success 'sparse-index is not expanded' '\n \techo >>sparse-index/untracked.txt &&\n \tensure_not_expanded add . &&\n \n+\tensure_not_expanded checkout-index -f a &&\n+\tensure_not_expanded checkout-index -f --all &&\n \tfor ref in update-deep update-folder1 update-folder2 update-deep\n \tdo\n \t\techo >>sparse-index/README.md &&\n-- \ngitgitgadget\n\n"},{"id":"445943","messageId":"80697e9259e4a2c12fd0546f5f1895fb3068a66c.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 9/9] update-index: reduce scope of index expansion in do_reupdate","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:06Z","receivedAt":"2022-01-11T18:06:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace unconditional index expansion in 'do_reupdate()' with one scoped to\nonly where a full index is needed. A full index is only required in\n'do_reupdate()' when a sparse directory in the index differs from HEAD; in\nthat case, the index is expanded and the operation restarted.\n\nBecause the index should only be expanded if a sparse directory is modified,\nadd a test ensuring the index is not expanded when differences only exist\nwithin the sparse cone.\n\nSigned-off-by: Victoria Dye \n---\n builtin/update-index.c | 14 +++++++++++---\n t/t1092-sparse-checkout-compatibility.sh | 5 ++++-\n 2 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 605cc693bbd..52ecc714d99 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -606,7 +606,7 @@ static struct cache_entry *read_one_ent(const char *which,\n \t\t\terror(\"%s: not in %s branch.\", path, which);\n \t\treturn NULL;\n \t}\n-\tif (mode == S_IFDIR) {\n+\tif (!the_index.sparse_index && mode == S_IFDIR) {\n \t\tif (which)\n \t\t\terror(\"%s: not a blob in %s branch.\", path, which);\n \t\treturn NULL;\n@@ -743,8 +743,6 @@ static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n \t\tstruct cache_entry *old = NULL;\n@@ -761,6 +759,16 @@ static int do_reupdate(int ac, const char **av,\n \t\t\tdiscard_cache_entry(old);\n \t\t\tcontinue; /* unchanged */\n \t\t}\n+\n+\t\t/* At this point, we know the contents of the sparse directory are\n+\t\t * modified with respect to HEAD, so we expand the index and restart\n+\t\t * to process each path individually\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\tensure_full_index(&the_index);\n+\t\t\tgoto redo;\n+\t\t}\n+\n \t\t/* Be careful. The working tree may not have the\n \t\t * path anymore, in which case, under 'allow_remove',\n \t\t * or worse yet 'allow_replace', active_nr may decrease.\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex fceaba7101d..53f84881de7 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1265,7 +1265,10 @@ test_expect_success 'sparse index is not expanded: update-index' '\n \n \tensure_not_expanded update-index --add README.md &&\n \tensure_not_expanded update-index a &&\n-\tensure_not_expanded update-index --remove deep/a\n+\tensure_not_expanded update-index --remove deep/a &&\n+\n+\tensure_not_expanded reset --soft update-deep &&\n+\tensure_not_expanded update-index --add --remove --again\n '\n \n test_expect_success 'sparse index is not expanded: blame' '\n-- \ngitgitgadget\n"},{"id":"445944","messageId":"9ddc51a47d50dcce219b6c88c85797b4818b7456.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 8/9] update-index: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:05Z","receivedAt":"2022-01-11T18:07:02Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of the sparse index with `update-index`. Most variations of\n`update-index` work without explicitly expanding the index or making any\nother updates in or outside of `update-index.c`.\n\nThe one usage requiring additional changes is `--cacheinfo`; if a file\ninside a sparse directory was specified, the index would not be expanded\nuntil after the cache tree is invalidated, leading to a mismatch between the\nindex and cache tree. This scenario is handled by rearranging\n`add_index_entry_with_check`, allowing `index_name_stage_pos` to expand the\nindex *before* attempting to invalidate the relevant cache tree path,\navoiding cache tree/index corruption.\n\nSigned-off-by: Victoria Dye \n---\n builtin/update-index.c | 3 +++\n read-cache.c | 10 +++++++---\n t/t1092-sparse-checkout-compatibility.sh | 15 +++++++++++++++\n 3 files changed, 25 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 187203e8bb5..605cc693bbd 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1077,6 +1077,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \n \tgit_config(git_default_config, NULL);\n \n+\tprepare_repo_settings(r);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \t/* we will diagnose later if it turns out that we need to update it */\n \tnewfd = hold_locked_index(&lock_file, 0);\n \tif (newfd < 0)\ndiff --git a/read-cache.c b/read-cache.c\nindex cbe73f14e5e..b4600e954b6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1339,9 +1339,6 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \tint skip_df_check = option & ADD_CACHE_SKIP_DFCHECK;\n \tint new_only = option & ADD_CACHE_NEW_ONLY;\n \n-\tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n-\t\tcache_tree_invalidate_path(istate, ce->name);\n-\n \t/*\n \t * If this entry's path sorts after the last entry in the index,\n \t * we can avoid searching for it.\n@@ -1352,6 +1349,13 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \telse\n \t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), EXPAND_SPARSE);\n \n+\t/*\n+\t * Cache tree path should be invalidated only after index_name_stage_pos,\n+\t * in case it expands a sparse index.\n+\t */\n+\tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n+\t\tcache_tree_invalidate_path(istate, ce->name);\n+\n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n \t\tif (!new_only)\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 91f849f541e..fceaba7101d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1253,6 +1253,21 @@ test_expect_success 'sparse index is not expanded: diff' '\n \tensure_not_expanded diff --cached\n '\n \n+test_expect_success 'sparse index is not expanded: update-index' '\n+\tinit_repos &&\n+\n+\tdeep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n+\tensure_not_expanded update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n+\n+\techo \"test\" >sparse-index/README.md &&\n+\techo \"test2\" >sparse-index/a &&\n+\trm -f sparse-index/deep/a &&\n+\n+\tensure_not_expanded update-index --add README.md &&\n+\tensure_not_expanded update-index a &&\n+\tensure_not_expanded update-index --remove deep/a\n+'\n+\n test_expect_success 'sparse index is not expanded: blame' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"445945","messageId":"ff32952a21c5005552a2892c4ac1a581f4fd0bb2.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 7/9] update-index: add tests for sparse-checkout compatibility","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:04Z","receivedAt":"2022-01-11T18:07:04Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nIntroduce tests for a variety of `git update-index` use cases, including\nperformance scenarios. Tests are intended to exercise `update-index` with\noptions that change the commands interaction with the index (e.g.,\n`--again`) and with files/directories inside and outside a sparse checkout\ncone.\n\nOf note is that these tests clearly establish the behavior of `git\nupdate-index --add` with untracked, outside-of-cone files. Unlike `git add`,\nwhich fails with an error when provided with such files, `update-index`\nsucceeds in adding them to the index. Additionally, the `skip-worktree` flag\nis *not* automatically added to the new entry. Although this is pre-existing\nbehavior, there are a couple of reasons to avoid changing it in favor of\nconsistency with e.g. `git add`:\n\n* `update-index` is low-level command for modifying the index; while it can\n perform operations similar to those of `add`, it traditionally has fewer\n \"guardrails\" preventing a user from doing something they may not want to\n do (in this case, adding an outside-of-cone, non-`skip-worktree` file to\n the index)\n* `update-index` typically only exits with an error code if it is incapable\n of performing an operation (e.g., if an internal function call fails);\n adding a new file outside the sparse checkout definition is still a valid\n operation, albeit an inadvisable one\n* `update-index` does not implicitly set flags (e.g., `skip-worktree`) when\n creating new index entries with `--add`; if flags need to be updated,\n options like `--[no-]skip-worktree` allow a user to intentionally set them\n\nAll this to say that, while there are valid reasons to consider changing the\ntreatment of outside-of-cone files in `update-index`, there are also\nsufficient reasons for leaving it as-is.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 167 +++++++++++++++++++++++\n 2 files changed, 168 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 54f8602f3c1..2a7106b9495 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -118,5 +118,6 @@ test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n test_perf_on_all git checkout-index -f --all\n+test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n \n test_done\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0c72c854d84..91f849f541e 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -630,6 +630,173 @@ test_expect_success 'reset with wildcard pathspec' '\n \ttest_all_match git ls-files -s -- folder1\n '\n \n+test_expect_success 'update-index modify outside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\t# Create & modify folder1/a\n+\t# Note that this setup is a manual way of reaching the erroneous\n+\t# condition in which a `skip-worktree` enabled, outside-of-cone file\n+\t# exists on disk. It is used here to ensure `update-index` is stable\n+\t# and behaves predictably if such a condition occurs.\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n+\trun_on_all ../edit-contents folder1/a &&\n+\n+\t# If file has skip-worktree enabled, update-index does not modify the\n+\t# index entry\n+\ttest_sparse_match git update-index folder1/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\n+\t# When skip-worktree is disabled (even on files outside sparse cone), file\n+\t# is updated in the index\n+\ttest_sparse_match git update-index --no-skip-worktree folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git update-index folder1/a &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --add outside sparse definition' '\n+\tinit_repos &&\n+\n+\twrite_script edit-contents <<-\\EOF &&\n+\techo text >>$1\n+\tEOF\n+\n+\t# Create folder1, add new file\n+\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_all ../edit-contents folder1/b &&\n+\n+\t# The *untracked* out-of-cone file is added to the index because it does\n+\t# not have a `skip-worktree` bit to signal that it should be ignored\n+\t# (unlike in `git add`, which will fail due to the file being outside\n+\t# the sparse checkout definition).\n+\ttest_all_match git update-index --add folder1/b &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+# NEEDSWORK: `--remove`, unlike the rest of `update-index`, does not ignore\n+# `skip-worktree` entries by default and will remove them from the index.\n+# The `--ignore-skip-worktree-entries` flag must be used in conjunction with\n+# `--remove` to ignore the `skip-worktree` entries and prevent their removal\n+# from the index.\n+test_expect_success 'update-index --remove outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# When --ignore-skip-worktree-entries is _not_ specified:\n+\t# out-of-cone, not-on-disk files are removed from the index\n+\ttest_sparse_match git update-index --remove folder1/a &&\n+\tcat >expect <<-EOF &&\n+\tD\tfolder1/a\n+\tEOF\n+\ttest_sparse_match git diff --cached --name-status &&\n+\ttest_cmp expect sparse-checkout-out &&\n+\n+\t# Reset the state\n+\ttest_all_match git reset --hard &&\n+\n+\t# When --ignore-skip-worktree-entries is specified, out-of-cone\n+\t# (skip-worktree) files are ignored\n+\ttest_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n+\ttest_sparse_match git diff --cached --name-status &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\n+\t# Reset the state\n+\ttest_all_match git reset --hard &&\n+\n+\t# --force-remove supercedes --ignore-skip-worktree-entries, removing\n+\t# a skip-worktree file from the index (and disk) when both are specified\n+\t# with --remove\n+\ttest_sparse_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n+\tcat >expect <<-EOF &&\n+\tD\tfolder1/a\n+\tEOF\n+\ttest_sparse_match git diff --cached --name-status &&\n+\ttest_cmp expect sparse-checkout-out\n+'\n+\n+test_expect_success 'update-index with directories' '\n+\tinit_repos &&\n+\n+\t# update-index will exit silently when provided with a directory name\n+\t# containing a trailing slash\n+\ttest_all_match git update-index deep/ folder1/ &&\n+\tgrep \"Ignoring path deep/\" sparse-checkout-err &&\n+\tgrep \"Ignoring path folder1/\" sparse-checkout-err &&\n+\n+\t# When update-index is given a directory name WITHOUT a trailing slash, it will\n+\t# behave in different ways depending on the status of the directory on disk:\n+\t# * if it exists, the command exits with an error (\"add individual files instead\")\n+\t# * if it does NOT exist (e.g., in a sparse-checkout), it is assumed to be a\n+\t# file and either triggers an error (\"does not exist and --remove not passed\")\n+\t# or is ignored completely (when using --remove)\n+\ttest_all_match test_must_fail git update-index deep &&\n+\trun_on_all test_must_fail git update-index folder1 &&\n+\ttest_must_fail git -C full-checkout update-index --remove folder1 &&\n+\ttest_sparse_match git update-index --remove folder1 &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'update-index --again file outside sparse definition' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-reupdate &&\n+\n+\t# Update HEAD without modifying the index to introduce a difference in\n+\t# folder1/a\n+\ttest_sparse_match git reset --soft update-folder1 &&\n+\n+\t# Because folder1/a differs in the index vs HEAD,\n+\t# `git update-index --no-skip-worktree --again` will effectively perform\n+\t# `git update-index --no-skip-worktree folder1/a` and remove the skip-worktree\n+\t# flag from folder1/a\n+\ttest_sparse_match git update-index --no-skip-worktree --again &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\tcat >expect <<-EOF &&\n+\tD\tfolder1/a\n+\tEOF\n+\ttest_sparse_match git diff --name-status &&\n+\ttest_cmp expect sparse-checkout-out\n+'\n+\n+test_expect_success 'update-index --cacheinfo' '\n+\tinit_repos &&\n+\n+\tdeep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n+\tfolder2_oid=$(git -C full-checkout rev-parse update-folder2:folder2) &&\n+\tfolder1_a_oid=$(git -C full-checkout rev-parse update-folder1:folder1/a) &&\n+\n+\ttest_all_match git update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\t# Cannot add sparse directory, even in sparse index case\n+\ttest_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n+\n+\t# Sparse match only: the new outside-of-cone entry is added *without* skip-worktree,\n+\t# so `git status` reports it as \"deleted\" in the worktree\n+\ttest_sparse_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\tcat >expect <<-EOF &&\n+\tMD folder1/a\n+\tEOF\n+\ttest_sparse_match git status --short -- folder1/a &&\n+\ttest_cmp expect sparse-checkout-out &&\n+\n+\t# To return folder1/a to \"normal\" for a sparse checkout (ignored &\n+\t# outside-of-cone), add the skip-worktree flag.\n+\ttest_sparse_match git update-index --skip-worktree folder1/a &&\n+\tcat >expect <<-EOF &&\n+\tS folder1/a\n+\tEOF\n+\ttest_sparse_match git ls-files -t -- folder1/a &&\n+\ttest_cmp expect sparse-checkout-out\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"445946","messageId":"601888606d1cf7d7752844dbdbc7fac20d4be8c4.1641924306.git.gitgitgadget@gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"[PATCH v2 5/9] checkout-index: add --ignore-skip-worktree-bits option","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-11T18:05:02Z","receivedAt":"2022-01-11T18:07:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nUpdate `checkout-index` to no longer refresh files that have the\n`skip-worktree` bit set, exiting with an error if `skip-worktree` filenames\nare directly provided to `checkout-index`. The newly-added\n`--ignore-skip-worktree-bits` option provides a mechanism to replicate the\nold behavior, checking out *all* files specified (even those with\n`skip-worktree` enabled).\n\nThe ability to toggle whether files should be checked-out based on\n`skip-worktree` already exists in `git checkout` and `git restore` (both of\nwhich have an `--ignore-skip-worktree-bits` option). The change to, by\ndefault, ignore `skip-worktree` files is especially helpful for\nsparse-checkout; it prevents inadvertent creation of files outside the\nsparse definition on disk and eliminates the need to expand a sparse index\nwhen using the `--all` option.\n\nInternal usage of `checkout-index` in `git stash` and `git filter-branch` do\nnot make explicit use of files with `skip-worktree` enabled, so\n`--ignore-skip-worktree-bits` is not added to them.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n Documentation/git-checkout-index.txt | 10 +++++++--\n builtin/checkout-index.c | 13 ++++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 27 +++++++++++++++---------\n 3 files changed, 38 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/git-checkout-index.txt b/Documentation/git-checkout-index.txt\nindex 4d33e7be0f5..01dbd5cbf54 100644\n--- a/Documentation/git-checkout-index.txt\n+++ b/Documentation/git-checkout-index.txt\n@@ -12,6 +12,7 @@ SYNOPSIS\n 'git checkout-index' [-u] [-q] [-a] [-f] [-n] [--prefix=]\n \t\t [--stage=|all]\n \t\t [--temp]\n+\t\t [--ignore-skip-worktree-bits]\n \t\t [-z] [--stdin]\n \t\t [--] [...]\n \n@@ -37,8 +38,9 @@ OPTIONS\n \n -a::\n --all::\n-\tchecks out all files in the index. Cannot be used\n-\ttogether with explicit filenames.\n+\tchecks out all files in the index except for those with the\n+\tskip-worktree bit set (see `--ignore-skip-worktree-bits`).\n+\tCannot be used together with explicit filenames.\n \n -n::\n --no-create::\n@@ -59,6 +61,10 @@ OPTIONS\n \twrite the content to temporary files. The temporary name\n \tassociations will be written to stdout.\n \n+--ignore-skip-worktree-bits::\n+\tCheck out all files, including those with the skip-worktree bit\n+\tset.\n+\n --stdin::\n \tInstead of taking list of paths from the command line,\n \tread list of paths from the standard input. Paths are\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex e21620d964e..615a118e2f5 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -7,6 +7,7 @@\n #define USE_THE_INDEX_COMPATIBILITY_MACROS\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"quote.h\"\n #include \"cache-tree.h\"\n@@ -17,6 +18,7 @@\n #define CHECKOUT_ALL 4\n static int nul_term_line;\n static int checkout_stage; /* default to checkout stage0 */\n+static int ignore_skip_worktree; /* default to 0 */\n static int to_tempfile;\n static char topath[4][TEMPORARY_FILENAME_LENGTH + 1];\n \n@@ -65,6 +67,7 @@ static int checkout_file(const char *name, const char *prefix)\n \tint namelen = strlen(name);\n \tint pos = cache_name_pos(name, namelen);\n \tint has_same_name = 0;\n+\tint is_skipped = 1;\n \tint did_checkout = 0;\n \tint errs = 0;\n \n@@ -78,6 +81,9 @@ static int checkout_file(const char *name, const char *prefix)\n \t\t\tbreak;\n \t\thas_same_name = 1;\n \t\tpos++;\n+\t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n+\t\t\tbreak;\n+\t\tis_skipped = 0;\n \t\tif (ce_stage(ce) != checkout_stage\n \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n \t\t\tcontinue;\n@@ -106,6 +112,9 @@ static int checkout_file(const char *name, const char *prefix)\n \t\tfprintf(stderr, \"git checkout-index: %s \", name);\n \t\tif (!has_same_name)\n \t\t\tfprintf(stderr, \"is not in the cache\");\n+\t\telse if (is_skipped)\n+\t\t\tfprintf(stderr, \"has skip-worktree enabled; \"\n+\t\t\t\t\t\"use '--ignore-skip-worktree-bits' to checkout\");\n \t\telse if (checkout_stage)\n \t\t\tfprintf(stderr, \"does not exist at stage %d\",\n \t\t\t\tcheckout_stage);\n@@ -125,6 +134,8 @@ static int checkout_all(const char *prefix, int prefix_length)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n+\t\tif (!ignore_skip_worktree && ce_skip_worktree(ce))\n+\t\t\tcontinue;\n \t\tif (ce_stage(ce) != checkout_stage\n \t\t && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n \t\t\tcontinue;\n@@ -185,6 +196,8 @@ int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_checkout_index_options[] = {\n \t\tOPT_BOOL('a', \"all\", &all,\n \t\t\tN_(\"check out all files in the index\")),\n+\t\tOPT_BOOL(0, \"ignore-skip-worktree-bits\", &ignore_skip_worktree,\n+\t\t\tN_(\"do not skip files with skip-worktree set\")),\n \t\tOPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n \t\tOPT__QUIET(&quiet,\n \t\t\tN_(\"no warning for existing files and files not in index\")),\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex db7ad41109b..434ef0433c0 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -772,9 +772,14 @@ test_expect_success 'checkout-index inside sparse definition' '\n test_expect_success 'checkout-index outside sparse definition' '\n \tinit_repos &&\n \n-\t# File does not exist on disk yet for sparse checkouts, so checkout-index\n-\t# succeeds without -f\n-\ttest_sparse_match git checkout-index -- folder1/a &&\n+\t# Without --ignore-skip-worktree-bits, outside-of-cone files will trigger\n+\t# an error\n+\ttest_sparse_match test_must_fail git checkout-index -- folder1/a &&\n+\ttest_i18ngrep \"folder1/a has skip-worktree enabled\" sparse-checkout-err &&\n+\ttest_path_is_missing folder1/a &&\n+\n+\t# With --ignore-skip-worktree-bits, outside-of-cone files are checked out\n+\ttest_sparse_match git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n \ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n \ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n \n@@ -783,8 +788,8 @@ test_expect_success 'checkout-index outside sparse definition' '\n \trun_on_sparse mkdir -p folder1 &&\n \trun_on_all cp ../new-a folder1/a &&\n \n-\ttest_all_match test_must_fail git checkout-index -- folder1/a &&\n-\ttest_all_match git checkout-index -f -- folder1/a &&\n+\ttest_all_match test_must_fail git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n+\ttest_all_match git checkout-index -f --ignore-skip-worktree-bits -- folder1/a &&\n \ttest_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n \ttest_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n '\n@@ -799,14 +804,16 @@ test_expect_success 'checkout-index with folders' '\n \ttest_all_match test_must_fail git checkout-index -f -- folder1/\n '\n \n-# NEEDSWORK: even in sparse checkouts, checkout-index --all will create all\n-# files (even those outside the sparse definition) on disk. However, these files\n-# don't appear in the percentage of tracked files in git status.\n-test_expect_failure 'checkout-index --all' '\n+test_expect_success 'checkout-index --all' '\n \tinit_repos &&\n \n \ttest_all_match git checkout-index --all &&\n-\ttest_sparse_match test_path_is_missing folder1\n+\ttest_sparse_match test_path_is_missing folder1 &&\n+\n+\t# --ignore-skip-worktree-bits will cause `skip-worktree` files to be\n+\t# checked out, causing the outside-of-cone `folder1` to exist on-disk\n+\ttest_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n+\ttest_all_match test_path_exists folder1\n '\n \n test_expect_success 'clean' '\n-- \ngitgitgadget\n\n"},{"id":"446083","messageId":"CABPp-BEVM-nhTX+DRf3NN16G-5Wr2KzObCiETqcy+ZYJLR4EWQ@mail.gmail.com","threadId":"57180","inReplyTo":"pull.1109.v2.git.1641924306.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-13T03:02:16Z","receivedAt":"2022-01-13T03:02:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 11, 2022 at 10:05 AM Victoria Dye via GitGitGadget\n wrote:\n>\n> This series continues the work to integrate commands with the sparse index,\n> adding integrations with 'git clean', 'git checkout-index', and 'git\n> update-index'. These three commands, while useful in their own right, are\n> updated mainly because they're used in 'git stash'. A future series will\n> integrate sparse index with 'stash' directly, but its subcommands must be\n> integrated to avoid the performance cost of each one expanding and\n> collapsing the index.\n>\n> The series is broken up into 4 parts:\n>\n> * Patches 1-2 are minor fixups to the 'git reset' sparse index integration\n> in response to discussion [1] that came after the series was ready for\n> merge to 'next'.\n> * Patch 3 integrates 'git clean' with the sparse index.\n> * Patches 4-6 integrate 'git checkout-index' with the sparse index and\n> introduce a new '--ignore-skip-worktree-bits' option.\n> * This involves changing the behavior of 'checkout-index' to respect\n> 'skip-worktree' by default (i.e., it won't check out 'skip-worktree'\n> files). The '--ignore-skip-worktree-bits' option can be specified to\n> force checkout of 'skip-worktree' files, if desired.\n> * Patches 7-9 integrate 'git update-index' with the sparse index.\n> * Note that, although this integrates the sparse index with\n> '--cacheinfo', sparse directories still cannot be updated using that\n> option (see the prior discussion [2] for more details on why)\n>\n>\n> Changes since V1\n> ================\n>\n> * Changed 'checkout-index' to fail by default when given filenames of files\n> with 'skip-worktree' enabled\n> * These files can still be forcibly checked-out by using the\n> '--ignore-skip-worktree-bits' option\n> * Added/updated corresponding t1092 tests\n> * Updated t1092 'update-index' tests\n> * Mentioned where/why 'skip-worktree' files were manually created on-disk\n> for testing purposes\n> * Provided explanation as to what '--remove' does, and how it relates to\n> '--ignore-skip-worktree-entries'; restructured corresponding test\n> * Fixed typo 'update-indexe' -> 'update-index'\n> * Removed unused 'edit-contents'\n> * Changed '--again' test to not use '--remove' to avoid confusion over\n> how/why it updates 'skip-worktree' entries\n> * Added \"set skip-worktree\" step to '--cacheinfo' test to illustrate how\n> it could be used to add a new outside-of-cone file and remain generally\n> compliant with a sparse-checkout definition\n> * Added '--cacheinfo' test to \"ensure not expanded\"\n> * Moved t1092 test 'sparse index is not expanded: update-index' to avoid\n> merge conflict\n> * Updated p2000 test for 'update-index': added file argument\n> * Without any file arguments, 'update-index' was effectively a no-op\n> * Clarified reasoning behind changing/not changing behavior of update-index\n> in sparse-checkouts\n\nNicely done! You've addressed all my (voluminous) feedback from v1;\nthis round looks good to me.\n\nReviewed-by: Elijah Newren \n\n>\n> Thanks!\n>\n> * Victoria\n>\n> [1]\n> https://lore.kernel.org/git/CABPp-BG0iDHf268UAnRyA=0y0T69YTc+bLMdxCmSbrL8s=9ziA@mail.gmail.com/\n>\n> [2]\n> https://lore.kernel.org/git/a075091c-d0d4-db5d-fa21-c9d6c90c343e@gmail.com/\n>\n> Victoria Dye (9):\n> reset: fix validation in sparse index test\n> reset: reorder wildcard pathspec conditions\n> clean: integrate with sparse index\n> checkout-index: expand sparse checkout compatibility tests\n> checkout-index: add --ignore-skip-worktree-bits option\n> checkout-index: integrate with sparse index\n> update-index: add tests for sparse-checkout compatibility\n> update-index: integrate with sparse index\n> update-index: reduce scope of index expansion in do_reupdate\n>\n> Documentation/git-checkout-index.txt | 10 +-\n> builtin/checkout-index.c | 41 +++-\n> builtin/clean.c | 3 +\n> builtin/reset.c | 12 +-\n> builtin/update-index.c | 17 +-\n> read-cache.c | 10 +-\n> t/perf/p2000-sparse-operations.sh | 2 +\n> t/t1092-sparse-checkout-compatibility.sh | 282 ++++++++++++++++++++++-\n> 8 files changed, 360 insertions(+), 17 deletions(-)\n>\n>\n> base-commit: dcc0cd074f0c639a0df20461a301af6d45bd582e\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1109%2Fvdye%2Fsparse%2Fupdate-index_checkout-index_clean-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1109/vdye/sparse/update-index_checkout-index_clean-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1109\n>\n> Range-diff vs v1:\n>\n> 1: eefb6ab4c61 = 1: eefb6ab4c61 reset: fix validation in sparse index test\n> 2: 0194d894c2f = 2: 0194d894c2f reset: reorder wildcard pathspec conditions\n> 3: 52aec13d18e = 3: 52aec13d18e clean: integrate with sparse index\n> 4: e6a8671f6be ! 4: d964507fdcc checkout-index: expand sparse checkout compatibility tests\n> @@ Commit message\n>\n> Add tests to cover `checkout-index`, with a focus on cases interesting in a\n> sparse checkout (e.g., files specified outside sparse checkout definition).\n> - New tests are intended to serve as a baseline for expected behavior and\n> - performance when integrating `checkout-index` with the sparse index.\n> +\n> + New tests are intended to serve as a baseline for existing and/or expected\n> + behavior and performance when integrating `checkout-index` with the sparse\n> + index. Note that the test 'checkout-index --all' is marked as\n> + 'test_expect_failure', indicating that `update-index --all` will be modified\n> + in a subsequent patch to behave as the test expects.\n>\n> Signed-off-by: Victoria Dye \n>\n> 5: ec9a751e8dc ! 5: 601888606d1 checkout-index: add --ignore-skip-worktree-bits option\n> @@ Metadata\n> ## Commit message ##\n> checkout-index: add --ignore-skip-worktree-bits option\n>\n> - Update `checkout-index --all` to no longer refresh files that have the\n> - `skip-worktree` bit set. The newly-added `--ignore-skip-worktree-bits`\n> - option, when used with `--all`, maintains the old behavior and checks out\n> - all files regardless of `skip-worktree`.\n> + Update `checkout-index` to no longer refresh files that have the\n> + `skip-worktree` bit set, exiting with an error if `skip-worktree` filenames\n> + are directly provided to `checkout-index`. The newly-added\n> + `--ignore-skip-worktree-bits` option provides a mechanism to replicate the\n> + old behavior, checking out *all* files specified (even those with\n> + `skip-worktree` enabled).\n>\n> The ability to toggle whether files should be checked-out based on\n> `skip-worktree` already exists in `git checkout` and `git restore` (both of\n> - which have an `--ignore-skip-worktree-bits` option). Adding the option to\n> - `checkout-index` (and changing the corresponding default behavior to respect\n> - the `skip-worktree` bit) is especially helpful for sparse-checkout: it\n> - prevents inadvertent creation of *all* files outside the sparse definition\n> - on disk and eliminates the need to expand a sparse index by default when\n> - using the `--all` option.\n> + which have an `--ignore-skip-worktree-bits` option). The change to, by\n> + default, ignore `skip-worktree` files is especially helpful for\n> + sparse-checkout; it prevents inadvertent creation of files outside the\n> + sparse definition on disk and eliminates the need to expand a sparse index\n> + when using the `--all` option.\n>\n> Internal usage of `checkout-index` in `git stash` and `git filter-branch` do\n> not make explicit use of files with `skip-worktree` enabled, so\n> `--ignore-skip-worktree-bits` is not added to them.\n>\n> + Helped-by: Elijah Newren \n> Signed-off-by: Victoria Dye \n>\n> ## Documentation/git-checkout-index.txt ##\n> @@ Documentation/git-checkout-index.txt: OPTIONS\n>\n> +--ignore-skip-worktree-bits::\n> + Check out all files, including those with the skip-worktree bit\n> -+ set. Note: may only be used with `--all`; skip-worktree is\n> -+ ignored when explicit filenames are specified.\n> ++ set.\n> +\n> --stdin::\n> Instead of taking list of paths from the command line,\n> @@ builtin/checkout-index.c\n> #include \"lockfile.h\"\n> #include \"quote.h\"\n> #include \"cache-tree.h\"\n> +@@\n> + #define CHECKOUT_ALL 4\n> + static int nul_term_line;\n> + static int checkout_stage; /* default to checkout stage0 */\n> ++static int ignore_skip_worktree; /* default to 0 */\n> + static int to_tempfile;\n> + static char topath[4][TEMPORARY_FILENAME_LENGTH + 1];\n> +\n> @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n> - return -1;\n> - }\n> + int namelen = strlen(name);\n> + int pos = cache_name_pos(name, namelen);\n> + int has_same_name = 0;\n> ++ int is_skipped = 1;\n> + int did_checkout = 0;\n> + int errs = 0;\n>\n> --static int checkout_all(const char *prefix, int prefix_length)\n> -+static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_worktree)\n> - {\n> - int i, errs = 0;\n> - struct cache_entry *last_ce = NULL;\n> +@@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n> + break;\n> + has_same_name = 1;\n> + pos++;\n> ++ if (!ignore_skip_worktree && ce_skip_worktree(ce))\n> ++ break;\n> ++ is_skipped = 0;\n> + if (ce_stage(ce) != checkout_stage\n> + && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> + continue;\n> +@@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n> + fprintf(stderr, \"git checkout-index: %s \", name);\n> + if (!has_same_name)\n> + fprintf(stderr, \"is not in the cache\");\n> ++ else if (is_skipped)\n> ++ fprintf(stderr, \"has skip-worktree enabled; \"\n> ++ \"use '--ignore-skip-worktree-bits' to checkout\");\n> + else if (checkout_stage)\n> + fprintf(stderr, \"does not exist at stage %d\",\n> + checkout_stage);\n> @@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length)\n> ensure_full_index(&the_index);\n> for (i = 0; i < active_nr ; i++) {\n> @@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix\n> if (ce_stage(ce) != checkout_stage\n> && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> continue;\n> -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> - int i;\n> - struct lock_file lock_file = LOCK_INIT;\n> - int all = 0;\n> -+ int ignore_skip_worktree = 0;\n> - int read_from_stdin = 0;\n> - int prefix_length;\n> - int force = 0, quiet = 0, not_new = 0;\n> @@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> struct option builtin_checkout_index_options[] = {\n> OPT_BOOL('a', \"all\", &all,\n> @@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, co\n> OPT__FORCE(&force, N_(\"force overwrite of existing files\"), 0),\n> OPT__QUIET(&quiet,\n> N_(\"no warning for existing files and files not in index\")),\n> -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> +\n> + ## t/t1092-sparse-checkout-compatibility.sh ##\n> +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index inside sparse definition' '\n> + test_expect_success 'checkout-index outside sparse definition' '\n> + init_repos &&\n>\n> - if (all)\n> - die(\"git checkout-index: don't mix '--all' and explicit filenames\");\n> -+ if (ignore_skip_worktree)\n> -+ die(\"git checkout-index: don't mix '--ignore-skip-worktree-bits' and explicit filenames\");\n> - if (read_from_stdin)\n> - die(\"git checkout-index: don't mix '--stdin' and explicit filenames\");\n> - p = prefix_path(prefix, prefix_length, arg);\n> -@@ builtin/checkout-index.c: int cmd_checkout_index(int argc, const char **argv, const char *prefix)\n> - }\n> +- # File does not exist on disk yet for sparse checkouts, so checkout-index\n> +- # succeeds without -f\n> +- test_sparse_match git checkout-index -- folder1/a &&\n> ++ # Without --ignore-skip-worktree-bits, outside-of-cone files will trigger\n> ++ # an error\n> ++ test_sparse_match test_must_fail git checkout-index -- folder1/a &&\n> ++ test_i18ngrep \"folder1/a has skip-worktree enabled\" sparse-checkout-err &&\n> ++ test_path_is_missing folder1/a &&\n> ++\n> ++ # With --ignore-skip-worktree-bits, outside-of-cone files are checked out\n> ++ test_sparse_match git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n> + test_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n> + test_cmp sparse-checkout/folder1/a full-checkout/folder1/a &&\n>\n> - if (all)\n> -- err |= checkout_all(prefix, prefix_length);\n> -+ err |= checkout_all(prefix, prefix_length, ignore_skip_worktree);\n> +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index outside sparse definition' '\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_all cp ../new-a folder1/a &&\n>\n> - if (pc_workers > 1)\n> - err |= run_parallel_checkout(&state, pc_workers, pc_threshold,\n> -\n> - ## t/t1092-sparse-checkout-compatibility.sh ##\n> +- test_all_match test_must_fail git checkout-index -- folder1/a &&\n> +- test_all_match git checkout-index -f -- folder1/a &&\n> ++ test_all_match test_must_fail git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n> ++ test_all_match git checkout-index -f --ignore-skip-worktree-bits -- folder1/a &&\n> + test_cmp sparse-checkout/folder1/a sparse-index/folder1/a &&\n> + test_cmp sparse-checkout/folder1/a full-checkout/folder1/a\n> + '\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index with folders' '\n> test_all_match test_must_fail git checkout-index -f -- folder1/\n> '\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'checkout-index wi\n> - test_sparse_match test_path_is_missing folder1\n> + test_sparse_match test_path_is_missing folder1 &&\n> +\n> ++ # --ignore-skip-worktree-bits will cause `skip-worktree` files to be\n> ++ # checked out, causing the outside-of-cone `folder1` to exist on-disk\n> + test_all_match git checkout-index --ignore-skip-worktree-bits --all &&\n> + test_all_match test_path_exists folder1\n> '\n> 6: 18c00fc9dd3 ! 6: b4b9086dcdc checkout-index: integrate with sparse index\n> @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char\n> int pos = cache_name_pos(name, namelen);\n> int has_same_name = 0;\n> + int is_file = 0;\n> + int is_skipped = 1;\n> int did_checkout = 0;\n> int errs = 0;\n> -\n> @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n> break;\n> has_same_name = 1;\n> @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char\n> + if (S_ISSPARSEDIR(ce->ce_mode))\n> + break;\n> + is_file = 1;\n> - if (ce_stage(ce) != checkout_stage\n> - && (CHECKOUT_ALL != checkout_stage || !ce_stage(ce)))\n> - continue;\n> + if (!ignore_skip_worktree && ce_skip_worktree(ce))\n> + break;\n> + is_skipped = 0;\n> @@ builtin/checkout-index.c: static int checkout_file(const char *name, const char *prefix)\n> fprintf(stderr, \"git checkout-index: %s \", name);\n> if (!has_same_name)\n> fprintf(stderr, \"is not in the cache\");\n> + else if (!is_file)\n> + fprintf(stderr, \"is a sparse directory\");\n> - else if (checkout_stage)\n> - fprintf(stderr, \"does not exist at stage %d\",\n> - checkout_stage);\n> -@@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length, int ignore_skip_w\n> + else if (is_skipped)\n> + fprintf(stderr, \"has skip-worktree enabled; \"\n> + \"use '--ignore-skip-worktree-bits' to checkout\");\n> +@@ builtin/checkout-index.c: static int checkout_all(const char *prefix, int prefix_length)\n> int i, errs = 0;\n> struct cache_entry *last_ce = NULL;\n>\n> 7: 3b734f89c0f ! 7: ff32952a21c update-index: add tests for sparse-checkout compatibility\n> @@ Commit message\n> update-index: add tests for sparse-checkout compatibility\n>\n> Introduce tests for a variety of `git update-index` use cases, including\n> - performance scenarios. Tests for `update-index add/remove` are specifically\n> - focused on how `git stash` uses `git update-index` as a subcommand to\n> - prepare for sparse index integration with `stash` in a future series.\n> + performance scenarios. Tests are intended to exercise `update-index` with\n> + options that change the commands interaction with the index (e.g.,\n> + `--again`) and with files/directories inside and outside a sparse checkout\n> + cone.\n> +\n> + Of note is that these tests clearly establish the behavior of `git\n> + update-index --add` with untracked, outside-of-cone files. Unlike `git add`,\n> + which fails with an error when provided with such files, `update-index`\n> + succeeds in adding them to the index. Additionally, the `skip-worktree` flag\n> + is *not* automatically added to the new entry. Although this is pre-existing\n> + behavior, there are a couple of reasons to avoid changing it in favor of\n> + consistency with e.g. `git add`:\n> +\n> + * `update-index` is low-level command for modifying the index; while it can\n> + perform operations similar to those of `add`, it traditionally has fewer\n> + \"guardrails\" preventing a user from doing something they may not want to\n> + do (in this case, adding an outside-of-cone, non-`skip-worktree` file to\n> + the index)\n> + * `update-index` typically only exits with an error code if it is incapable\n> + of performing an operation (e.g., if an internal function call fails);\n> + adding a new file outside the sparse checkout definition is still a valid\n> + operation, albeit an inadvisable one\n> + * `update-index` does not implicitly set flags (e.g., `skip-worktree`) when\n> + creating new index entries with `--add`; if flags need to be updated,\n> + options like `--[no-]skip-worktree` allow a user to intentionally set them\n> +\n> + All this to say that, while there are valid reasons to consider changing the\n> + treatment of outside-of-cone files in `update-index`, there are also\n> + sufficient reasons for leaving it as-is.\n>\n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Victoria Dye \n> @@ t/perf/p2000-sparse-operations.sh: test_perf_on_all git diff --cached\n> test_perf_on_all git blame $SPARSE_CONE/a\n> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> test_perf_on_all git checkout-index -f --all\n> -+test_perf_on_all git update-index --add --remove\n> ++test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n>\n> test_done\n>\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> + EOF\n> +\n> + # Create & modify folder1/a\n> ++ # Note that this setup is a manual way of reaching the erroneous\n> ++ # condition in which a `skip-worktree` enabled, outside-of-cone file\n> ++ # exists on disk. It is used here to ensure `update-index` is stable\n> ++ # and behaves predictably if such a condition occurs.\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n> + run_on_all ../edit-contents folder1/a &&\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> + run_on_sparse mkdir -p folder1 &&\n> + run_on_all ../edit-contents folder1/b &&\n> +\n> -+ # Similar to `git add`, the untracked out-of-cone file is added to the index\n> -+ # identically across sparse and non-sparse checkouts\n> ++ # The *untracked* out-of-cone file is added to the index because it does\n> ++ # not have a `skip-worktree` bit to signal that it should be ignored\n> ++ # (unlike in `git add`, which will fail due to the file being outside\n> ++ # the sparse checkout definition).\n> + test_all_match git update-index --add folder1/b &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> ++# NEEDSWORK: `--remove`, unlike the rest of `update-index`, does not ignore\n> ++# `skip-worktree` entries by default and will remove them from the index.\n> ++# The `--ignore-skip-worktree-entries` flag must be used in conjunction with\n> ++# `--remove` to ignore the `skip-worktree` entries and prevent their removal\n> ++# from the index.\n> +test_expect_success 'update-index --remove outside sparse definition' '\n> + init_repos &&\n> +\n> -+ # When `--ignore-skip-worktree-entries` is specified, out-of-cone files are\n> -+ # not removed from the index if they do not exist on disk\n> ++ # When --ignore-skip-worktree-entries is _not_ specified:\n> ++ # out-of-cone, not-on-disk files are removed from the index\n> ++ test_sparse_match git update-index --remove folder1/a &&\n> ++ cat >expect <<-EOF &&\n> ++ D folder1/a\n> ++ EOF\n> ++ test_sparse_match git diff --cached --name-status &&\n> ++ test_cmp expect sparse-checkout-out &&\n> ++\n> ++ # Reset the state\n> ++ test_all_match git reset --hard &&\n> ++\n> ++ # When --ignore-skip-worktree-entries is specified, out-of-cone\n> ++ # (skip-worktree) files are ignored\n> + test_sparse_match git update-index --remove --ignore-skip-worktree-entries folder1/a &&\n> -+ test_all_match git status --porcelain=v2 &&\n> ++ test_sparse_match git diff --cached --name-status &&\n> ++ test_must_be_empty sparse-checkout-out &&\n> +\n> -+ # When the flag is _not_ specified, out-of-cone, not-on-disk files are\n> -+ # removed from the index\n> -+ rm full-checkout/folder1/a &&\n> -+ test_all_match git update-index --remove folder1/a &&\n> -+ test_all_match git status --porcelain=v2 &&\n> ++ # Reset the state\n> ++ test_all_match git reset --hard &&\n> +\n> -+ # NOTE: --force-remove supercedes --ignore-skip-worktree-entries, removing\n> ++ # --force-remove supercedes --ignore-skip-worktree-entries, removing\n> + # a skip-worktree file from the index (and disk) when both are specified\n> -+ test_all_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n> -+ test_all_match git status --porcelain=v2\n> ++ # with --remove\n> ++ test_sparse_match git update-index --force-remove --ignore-skip-worktree-entries folder1/a &&\n> ++ cat >expect <<-EOF &&\n> ++ D folder1/a\n> ++ EOF\n> ++ test_sparse_match git diff --cached --name-status &&\n> ++ test_cmp expect sparse-checkout-out\n> +'\n> +\n> +test_expect_success 'update-index with directories' '\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> + # file and either triggers an error (\"does not exist and --remove not passed\")\n> + # or is ignored completely (when using --remove)\n> + test_all_match test_must_fail git update-index deep &&\n> -+ run_on_all test_must_fail git update-indexe folder1 &&\n> ++ run_on_all test_must_fail git update-index folder1 &&\n> + test_must_fail git -C full-checkout update-index --remove folder1 &&\n> + test_sparse_match git update-index --remove folder1 &&\n> + test_all_match git status --porcelain=v2\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> +test_expect_success 'update-index --again file outside sparse definition' '\n> + init_repos &&\n> +\n> -+ write_script edit-contents <<-\\EOF &&\n> -+ echo text >>$1\n> -+ EOF\n> -+\n> + test_all_match git checkout -b test-reupdate &&\n> +\n> + # Update HEAD without modifying the index to introduce a difference in\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> + test_sparse_match git reset --soft update-folder1 &&\n> +\n> + # Because folder1/a differs in the index vs HEAD,\n> -+ # `git update-index --remove --again` will effectively perform\n> -+ # `git update-index --remove folder1/a` and remove the folder1/a\n> -+ test_sparse_match git update-index --remove --again &&\n> -+ test_sparse_match git status --porcelain=v2\n> ++ # `git update-index --no-skip-worktree --again` will effectively perform\n> ++ # `git update-index --no-skip-worktree folder1/a` and remove the skip-worktree\n> ++ # flag from folder1/a\n> ++ test_sparse_match git update-index --no-skip-worktree --again &&\n> ++ test_sparse_match git status --porcelain=v2 &&\n> ++\n> ++ cat >expect <<-EOF &&\n> ++ D folder1/a\n> ++ EOF\n> ++ test_sparse_match git diff --name-status &&\n> ++ test_cmp expect sparse-checkout-out\n> +'\n> +\n> +test_expect_success 'update-index --cacheinfo' '\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'reset with wildca\n> + # Cannot add sparse directory, even in sparse index case\n> + test_all_match test_must_fail git update-index --add --cacheinfo 040000 $folder2_oid folder2/ &&\n> +\n> -+ # Sparse match only - because folder1/a is outside the sparse checkout\n> -+ # definition (and thus not on-disk), it will appear \"deleted\" in\n> -+ # unstaged changes.\n> -+ test_all_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n> -+ test_sparse_match git status --porcelain=v2\n> ++ # Sparse match only: the new outside-of-cone entry is added *without* skip-worktree,\n> ++ # so `git status` reports it as \"deleted\" in the worktree\n> ++ test_sparse_match git update-index --add --cacheinfo 100644 $folder1_a_oid folder1/a &&\n> ++ test_sparse_match git status --porcelain=v2 &&\n> ++ cat >expect <<-EOF &&\n> ++ MD folder1/a\n> ++ EOF\n> ++ test_sparse_match git status --short -- folder1/a &&\n> ++ test_cmp expect sparse-checkout-out &&\n> ++\n> ++ # To return folder1/a to \"normal\" for a sparse checkout (ignored &\n> ++ # outside-of-cone), add the skip-worktree flag.\n> ++ test_sparse_match git update-index --skip-worktree folder1/a &&\n> ++ cat >expect <<-EOF &&\n> ++ S folder1/a\n> ++ EOF\n> ++ test_sparse_match git ls-files -t -- folder1/a &&\n> ++ test_cmp expect sparse-checkout-out\n> +'\n> +\n> test_expect_success 'merge, cherry-pick, and rebase' '\n> 8: c5b98e36516 ! 8: 9ddc51a47d5 update-index: integrate with sparse index\n> @@ Metadata\n> ## Commit message ##\n> update-index: integrate with sparse index\n>\n> - Enable usage of the sparse index with `update-index`. Most variations of\n> + Enable use of the sparse index with `update-index`. Most variations of\n> `update-index` work without explicitly expanding the index or making any\n> other updates in or outside of `update-index.c`.\n>\n> @@ read-cache.c: static int add_index_entry_with_check(struct index_state *istate,\n> if (!new_only)\n>\n> ## t/t1092-sparse-checkout-compatibility.sh ##\n> -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: blame' '\n> - done\n> +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: diff' '\n> + ensure_not_expanded diff --cached\n> '\n>\n> +test_expect_success 'sparse index is not expanded: update-index' '\n> + init_repos &&\n> +\n> ++ deep_a_oid=$(git -C full-checkout rev-parse update-deep:deep/a) &&\n> ++ ensure_not_expanded update-index --cacheinfo 100644 $deep_a_oid deep/a &&\n> ++\n> + echo \"test\" >sparse-index/README.md &&\n> + echo \"test2\" >sparse-index/a &&\n> + rm -f sparse-index/deep/a &&\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> + ensure_not_expanded update-index --remove deep/a\n> +'\n> +\n> - # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> - # in this scenario, but it shouldn't.\n> - test_expect_success 'reset mixed and checkout orphan' '\n> + test_expect_success 'sparse index is not expanded: blame' '\n> + init_repos &&\n> +\n> 9: de7fc143562 ! 9: 80697e9259e update-index: reduce scope of index expansion in do_reupdate\n> @@ Metadata\n> ## Commit message ##\n> update-index: reduce scope of index expansion in do_reupdate\n>\n> - Expand the full index (and redo reupdate operation) only if a sparse\n> - directory in the index differs from HEAD. Only the index entries that differ\n> - between the index and HEAD are updated when performing `git update-index\n> - --again`, so unmodified sparse directories are safely skipped. The index\n> - does need to be expanded when sparse directories contain changes, though,\n> - because `update_one(...)` will not operate on sparse directory index\n> - entries.\n> + Replace unconditional index expansion in 'do_reupdate()' with one scoped to\n> + only where a full index is needed. A full index is only required in\n> + 'do_reupdate()' when a sparse directory in the index differs from HEAD; in\n> + that case, the index is expanded and the operation restarted.\n>\n> Because the index should only be expanded if a sparse directory is modified,\n> add a test ensuring the index is not expanded when differences only exist\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> + ensure_not_expanded update-index --add --remove --again\n> '\n>\n> - # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> + test_expect_success 'sparse index is not expanded: blame' '\n>\n> --\n> gitgitgadget\n"},{"id":"447097","messageId":"771fa224-4319-c827-d28d-51e767b0050e@gmail.com","threadId":"57180","inReplyTo":"CABPp-BEVM-nhTX+DRf3NN16G-5Wr2KzObCiETqcy+ZYJLR4EWQ@mail.gmail.com","subject":"Re: [PATCH v2 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2022-01-27T16:36:03Z","receivedAt":"2022-01-27T16:36:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"n 1/12/2022 10:02 PM, Elijah Newren wrote:\n> On Tue, Jan 11, 2022 at 10:05 AM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> This series continues the work to integrate commands with the sparse index,\n>> adding integrations with 'git clean', 'git checkout-index', and 'git\n>> update-index'. These three commands, while useful in their own right, are\n>> updated mainly because they're used in 'git stash'. A future series will\n>> integrate sparse index with 'stash' directly, but its subcommands must be\n>> integrated to avoid the performance cost of each one expanding and\n>> collapsing the index.\n>>\n>> The series is broken up into 4 parts:\n>>\n>> * Patches 1-2 are minor fixups to the 'git reset' sparse index integration\n>> in response to discussion [1] that came after the series was ready for\n>> merge to 'next'.\n>> * Patch 3 integrates 'git clean' with the sparse index.\n>> * Patches 4-6 integrate 'git checkout-index' with the sparse index and\n>> introduce a new '--ignore-skip-worktree-bits' option.\n>> * This involves changing the behavior of 'checkout-index' to respect\n>> 'skip-worktree' by default (i.e., it won't check out 'skip-worktree'\n>> files). The '--ignore-skip-worktree-bits' option can be specified to\n>> force checkout of 'skip-worktree' files, if desired.\n>> * Patches 7-9 integrate 'git update-index' with the sparse index.\n>> * Note that, although this integrates the sparse index with\n>> '--cacheinfo', sparse directories still cannot be updated using that\n>> option (see the prior discussion [2] for more details on why)\n>>\n>>\n>> Changes since V1\n>> ================\n>>\n>> * Changed 'checkout-index' to fail by default when given filenames of files\n>> with 'skip-worktree' enabled\n>> * These files can still be forcibly checked-out by using the\n>> '--ignore-skip-worktree-bits' option\n>> * Added/updated corresponding t1092 tests\n>> * Updated t1092 'update-index' tests\n>> * Mentioned where/why 'skip-worktree' files were manually created on-disk\n>> for testing purposes\n>> * Provided explanation as to what '--remove' does, and how it relates to\n>> '--ignore-skip-worktree-entries'; restructured corresponding test\n>> * Fixed typo 'update-indexe' -> 'update-index'\n>> * Removed unused 'edit-contents'\n>> * Changed '--again' test to not use '--remove' to avoid confusion over\n>> how/why it updates 'skip-worktree' entries\n>> * Added \"set skip-worktree\" step to '--cacheinfo' test to illustrate how\n>> it could be used to add a new outside-of-cone file and remain generally\n>> compliant with a sparse-checkout definition\n>> * Added '--cacheinfo' test to \"ensure not expanded\"\n>> * Moved t1092 test 'sparse index is not expanded: update-index' to avoid\n>> merge conflict\n>> * Updated p2000 test for 'update-index': added file argument\n>> * Without any file arguments, 'update-index' was effectively a no-op\n>> * Clarified reasoning behind changing/not changing behavior of update-index\n>> in sparse-checkouts\n> \n> Nicely done! You've addressed all my (voluminous) feedback from v1;\n> this round looks good to me.\n> \n> Reviewed-by: Elijah Newren \n\nThank you for that review, Elijah. I took this opportunity to\nreread the series as well as the range-diff and I agree that\nthis version is ready to go.\n\nThanks,\n-Stolee\n"},{"id":"447124","messageId":"xmqqilu50wuf.fsf@gitster.g","threadId":"57180","inReplyTo":"771fa224-4319-c827-d28d-51e767b0050e@gmail.com","subject":"Re: [PATCH v2 0/9] Sparse index: integrate with 'clean', 'checkout-index', 'update-index'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-01-27T20:04:08Z","receivedAt":"2022-01-27T20:04:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> this version is ready to go.\n\nThanks, all. I marked the topic as \"Will merge to 'next'\" in the\ndraft of \"What's cooking\" report.\n\n"}]}