{"thread":{"id":"57214","subject":"[RFC PATCH 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs","startedAt":"2022-01-09T04:57:46Z","lastAt":"2022-01-11T23:09:58Z","messageCount":11,"participants":["Elijah Newren","Victoria Dye"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"445802","messageId":"20220109045732.2497526-1-newren@gmail.com","threadId":"57214","inReplyTo":null,"subject":"[RFC PATCH 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:27Z","receivedAt":"2022-01-09T04:57:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"(Maintainer note: This series builds on vd/sparse-clean-etc, because it\ntweaks one of the testcases added there.)\n\nFiles in the present-despite-SKIP_WORKTREE state have caused no ends of\ndiscussions and bugs[1,2,3,4,5,6,...and lots of others]. Trying to\naddress the big issue of discovering & recovering from this state has\nbefuddled me for over a year because I was worried we'd need additional\ncode at every skip_worktree-checking path in the code (and they are all\nover the place), and that we'd make the code significantly slower unless\nwe plumbed a bunch of additional information all over the place to allow\nsome reasonable optimizations.\n\nThis series tries to solve the problem a bit differently by automatic\nearly discovery and recovery; as it result, it greatly simplifies the\nlandscape, reduces our testing matrix burden, and fixes a large swath of\nbugs. It does have a cost, though. See the commit message of patch 3\n(the crux of this series) for the details.\n\nQuick overview:\n\n * Patches 1 & 2 add a test to demonstrate accidental deletion of\n possibly-modified files, and then fix the bug.\n * Patch 3 is the crux of this series; a small amount of code with a\n huge commit message\n * Patch 4 updates the documentation\n * Patch 5 adds some optimizations to reduce the performance impact of\n patch 3\n\n[1] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n[2] https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[3] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[4] commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE handling with conflicted entries\", 2021-03-20)\n[5] commit ba359fd507 (\"stash: fix stash application in sparse-checkouts\", 2020-12-01)\n[6] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n\n[Final note: This series and test results can also be seen at\nhttps://github.com/gitgitgadget/git/pull/1113; gitgitgadget wanted me to\n\"rewrap\" the lines with >80 character URLS in commit messages, and I\nrefuse to remove the URLs or break them by inserting newlines, so I'm\nsending it manually instead.]\n\nElijah Newren (5):\n t1011: add testcase demonstrating accidental loss of user\n modifications\n unpack-trees: fix accidental loss of user changes\n repo_read_index: ensure SKIP_WORKTREE means skip worktree\n Update documentation related to sparsity and the skip-worktree bit\n Accelerate ensure_skip_worktree_means_skip_worktree by caching\n\n Documentation/git-read-tree.txt | 12 ++-\n Documentation/git-sparse-checkout.txt | 76 ++++++++------\n Documentation/git-update-index.txt | 55 +++++++---\n repository.c | 7 ++\n sparse-index.c | 123 +++++++++++++++++++++++\n sparse-index.h | 1 +\n t/t1011-read-tree-sparse-checkout.sh | 23 ++++-\n t/t1092-sparse-checkout-compatibility.sh | 16 +--\n t/t3705-add-sparse-checkout.sh | 2 +\n t/t6428-merge-conflicts-sparse.sh | 23 +----\n t/t7012-skip-worktree-writing.sh | 44 ++------\n t/t7817-grep-sparse-checkout.sh | 11 +-\n unpack-trees.c | 4 +-\n 13 files changed, 284 insertions(+), 113 deletions(-)\n\n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445803","messageId":"20220109045732.2497526-2-newren@gmail.com","threadId":"57214","inReplyTo":"20220109045732.2497526-1-newren@gmail.com","subject":"[RFC PATCH 1/5] t1011: add testcase demonstrating accidental loss of user modifications","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:28Z","receivedAt":"2022-01-09T04:57:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"If a user has a file with local modifications that is not marked as\nSKIP_WORKTREE, but the sparsity patterns are such that it should be\nmarked that way, and the user then invokes a command like\n\n * git checkout -q HEAD^\n\nor\n\n * git read-tree -mu HEAD^\n\nThen the file will be deleted along with all the users' modifications.\nAdd a testcase demonstrating this problem.\n\nNote: This bug only triggers if something other than 'HEAD' is given;\nif the commands above had specified 'HEAD', then the users' file would\nbe left alone.\n\nSigned-off-by: Elijah Newren \n---\n t/t1011-read-tree-sparse-checkout.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 24092c09a9..1b2395b8a8 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -187,6 +187,27 @@ test_expect_success 'read-tree updates worktree, absent case' '\n \ttest ! -f init.t\n '\n \n+test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n+test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\techo sub/added >.git/info/sparse-checkout &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445804","messageId":"20220109045732.2497526-3-newren@gmail.com","threadId":"57214","inReplyTo":"20220109045732.2497526-1-newren@gmail.com","subject":"[RFC PATCH 2/5] unpack-trees: fix accidental loss of user changes","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:29Z","receivedAt":"2022-01-09T04:57:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"For sparse-checkouts, we don't want unpack-trees to error out on files\nthat are missing from the worktree, so there has traditionally been\nlogic to make it skip the verify_uptodate() check for these.\nUnfortunately, it was skipping the verify_uptodate() check for files\nthat were expected to *become* SKIP_WORKTREE. For files that were not\nalready SKIP_WORKTREE, that can cause us to later delete the file in\napply_sparse_checkout(). Only skip the check for files that were\nalready SKIP_WORKTREE as well to avoid lightly discarding important\nchanges users may have made to files.\n\nNote 1: unpack-trees.c is already a bit complex, and the logic around\nCE_SKIP_WORKTREE and CE_NEW_SKIP_WORKTREE in that file are no exception.\nI also tried just replacing CE_NEW_SKIP_WORKTREE with CE_SKIP_WORKTREE\nin the verify_uptodate() check instead of checking for both flags, and\nfound that it also fixed this bug and passed all the tests. I also\nattempted to devise a few testcases that might trip either variant of my\nfix and was unable to find any problems. It may be that just checking\nCE_SKIP_WORKTREE is a better fix, but I'm not sure. I thought it\nwas a bit safer to strictly reduce the number of cases where we skip the\nup-to-date check rather than just toggling which kind of cases skip it,\nand thus went with the current variant of the fix.\n\nNote 2: I also wondered if verify_absent() might have a similar bug, but\ndespite my attempts to try to devise a testcase that would trigger such\na thing, I couldn't find any problematic testcases. Thus, this patch\nmakes no attempt to apply similar changes to verify_absent() and\nverify_absent_if_directory().\n\nSigned-off-by: Elijah Newren \n---\n t/t1011-read-tree-sparse-checkout.sh | 2 +-\n unpack-trees.c | 4 +++-\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 1b2395b8a8..4ed0885bf2 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -197,7 +197,7 @@ test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n \tgrep -q dirty init.t\n '\n \n-test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+test_expect_success 'read-tree will not throw away dirty changes, sparse' '\n \techo \"/*\" >.git/info/sparse-checkout &&\n \tread_tree_u_must_succeed -m -u HEAD &&\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 98e2f2e0e6..6d9d89c662 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2059,7 +2059,9 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\tif (!o->skip_sparse_checkout &&\n+\t (ce->ce_flags & CE_SKIP_WORKTREE) &&\n+\t (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n }\n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445805","messageId":"20220109045732.2497526-4-newren@gmail.com","threadId":"57214","inReplyTo":"20220109045732.2497526-1-newren@gmail.com","subject":"[RFC PATCH 3/5] repo_read_index: ensure SKIP_WORKTREE means skip worktree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:30Z","receivedAt":"2022-01-09T04:57:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"The fix is short (~30 lines), but the description is not. Sorry.\n\nThere is a set of problems caused by files in what I'll refer to as the\n\"present-despite-SKIP_WORKTREE\" state. This commit aims to not just fix\nthese problems, but remove the entire class as a possibility -- for\nthose using sparse checkouts. But first, we need to understand the\nproblems this class presents. A quick outline:\n\n * Problems\n * User facing issues\n * Problem space complexity\n * Maintenance and code correctness challenges\n * SKIP_WORKTREE expectations in Git\n * Suggested solution\n * Pros/Cons of suggested solution\n * Notes on testcase modifications\n\n=== User facing issues ===\n\nThere are various ways for users to get files to be present in the\nworking copy despite having the SKIP_WORKTREE bit set for that file in\nthe index. This may come from:\n * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n * users grabbing files from elsewhere and writing them to the worktree\n (perhaps even cached in their editor)\n * users attempting to \"abort\" a sparse-checkout operation with a\n not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n working tree is not atomic)[3].\n\nOnce users have present-despite-SKIP_WORKTREE files, any modifications\nusers make to these files will be ignored, possibly to users' confusion.\n\nFurther:\n * these files will not be updated by by standard commands\n (switch/checkout/pull/merge/rebase will leave them alone unless\n conflicts happen -- and even then, the conflicted file may be\n written somewhere else to avoid overwriting the SKIP_WORKTREE file\n that is present and in the way)\n * there is nothing in Git that users can use to discover such\n files (status, diff, grep, etc. all ignore it)\n * there is no reasonable mechanism to \"recover\" from such a condition\n (neither `git sparse-checkout reapply` nor `git reset --hard` will\n correct it).\n\nSo, not only are users modifications ignored, but the files get\nprogressively more stale over time. At some point in the future, they\nmay change their sparseness specification or disable sparse-checkouts.\nAt that time, all present-despite-SKIP_WORKTREE files will show up as\nhaving lots of modifications because they represent a version from a\ndifferent branch or commit. These might include user-made local changes\nfrom days before, but the only way to tell is to have users look through\nthem all closely.\n\nIf these users come to others for help, there will be no logs that\nexplain the issue; it's just a mysterious list of changes. Users might\nadamantly claim (correctly, as it turns out) that they didn't modify\nthese files, while others presume they did.\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n[2] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n\n=== Problem space complexity ===\n\nSKIP_WORKTREE has been part of Git for over a decade. Duy did lots of\nwork on it initially, and several others have since come along and put\nlots of work into it. Stolee spent most of 2021 on the sparse-index,\nwith lots of bugfixes along the way including to non-sparse-index cases\nas we are still trying to get sparse checkouts to behave reasonably.\nBasically every codepath throughout the treat needs to be aware of an\nadditional type of file: tracked-but-not-present. The extra type\nresults in lots of extra testcases and lots of extra code everywhere.\n\nBut, the sad thing is that we actually have more than one extra type.\nWe have tracked, tracked-but-not-present (SKIP_WORKTREE), and\ntracked-but-promised-to-not-be-present-but-is-present-anyway\n(present-despite-SKIP_WORKTREE). Two types is a monumental amount of\neffort to support, and adding a third feels a bit like insanity[4].\n\n[4] Some examples of which can be seen at\n https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n\n=== Maintenance and code correctness challenges ===\n\nMatheus' patches to grep stalled for nearly a year, in part because of\ncomplications of how to handle sparse-checkouts appropriately in all\ncases[5][6] (with trying to sanely figure out how to sanely handle\npresent-despite-SKIP_WORKTREE files being one of the complications).\nHis rm/add follow-ups also took months because of those kinds of\nissues[7]. The corner cases with things like submodules and\nSKIP_WORKTREE with the addition of present-despite-SKIP_WORKTREE start\nbecoming really complex[8].\n\nWe've had to add ugly logic to merge-ort to attempt to handle\npresent-despite-SKIP_WORKTREE files[9], and basically just been forced\nto give up in merge-recursive knowing full well that we'll sometimes\nsilently discard user modifications. Despite stash essentially being a\nmerge, it needed extra code (beyond what was in merge-ort and\nmerge-recursive) to manually tweak SKIP_WORKTREE bits in order to avoid\na few different bugs that'd result in an early abort with a partial\nstash application[10].\n\n[5] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t\n and the dates on the thread; also Matheus and I had several\n conversations off-list trying to resolve the issues over that time\n[6] ...it finally kind of got unstuck after\n https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[7] See for example\n https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t\n and quotes like \"The core functionality of sparse-checkout has always\n been only partially implemented\", a statement I still believe is true\n today.\n[8] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[9] See commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE\n handling with conflicted entries\", 2021-03-20)\n[10] See commit ba359fd507 (\"stash: fix stash application in\n sparse-checkouts\", 2020-12-01)\n\n=== SKIP_WORKTREE expectations in Git ===\n\nA couple quotes:\n\nFrom [11] (before the \"sparse-checkout\" command existed):\n If it needs too many special cases, hacks, and conditionals, then it\n is not worth the complexity---if it is easier to write a correct code\n by allowing Git to populate working tree files, it is perfectly fine\n to do so.\n\n In a sense, the sparse checkout \"feature\" itself is a hack by itself,\n and that is why I think this part should be \"best effort\" as well.\n\nFrom the git-sparse-checkout manual (still present today):\n\n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n THE FUTURE.\n\n[11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n\n=== Suggested solution ===\n\nSKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\nthe name of the option implies, to allow the file to NOT be in the\nworktree but consider it to be unchanged rather than deleted.\n\nThe suggests a simple solution: present-despite-SKIP_WORKTREE files\nshould not exist, for those using sparse-checkouts.\n\nEnforce this at index loading time by checking if core.sparseCheckout is\ntrue; if so, check files in the index with the SKIP_WORKTREE bit set to\nverify that they are absent from the working tree. If they are present,\nunset the bit (in memory, though any commands that write to the index\nwill record the update).\n\nUsers can, of course, can get the SKIP_WORKTREE bit back such as by\nrunning `git sparse-checkout reapply` (if they have ensured the file is\nunmodified and doesn't match the specified sparsity patterns).\n\n=== Pros/Cons of suggested solution ===\n\nPros:\n\n * Solves the user visible problems reported above, which I've been\n complaining about for nearly a year but couldn't find a solution to.\n * Much easier behavior in sparse-checkouts for users to reason about\n * Very simple, ~30 lines of code.\n * Significantly simplifies some ugly testcases, and obviates the need\n to test an entire class of potential issues.\n * Reduces code complexity, reasoning, and maintenance. Avoids\n disagreements about weird corner cases[12].\n * It has been reported that some users might be (ab)using\n SKIP_WORKTREE as a let-me-modify-but-keep-the-file-in-the-worktree\n mechanism[13, and a few other similar references]. These users know\n of multiple caveats and shortcomings in doing so; perhaps not\n surprising given the \"SKIP_WORKTREE expecations\" section above.\n However, these users use `git update-index --skip-worktree`, and not\n `git sparse-checkout` or core.sparseCheckout=true. As such, these\n users would be unaffected by this change and can continue abusing\n the system as before.\n\n[12] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[13] https://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree\n\nCons:\n\n * When core.sparseCheckout is enabled, this adds a performance cost to\n reading the index. I'll defer discussion of this cost to a subsequent\n patch, since I have some optimizations to add.\n\n=== Notes on testcase modifications ===\n\nThe good:\n * t1011: Compare to two cases above it ('read-tree will not throw away\n dirty changes, non-sparse'); since the file is present, it should\n match the non-sparse case now\n * t1092: sparse-index & sparse-checkout now match full-worktree\n behavior in more cases! Yaay for consistency!\n * t6428, t7012: look at how much simpler the tests become! Merge and\n stash can just fail early telling the user there's a file in the\n way, instead of not noticing until it's about to write a file and\n then have to implement sudden crash avoidance. Hurray for sanity!\n * t7817: sparse behavior better matches full tree behavior. Hurray\n for sanity!\n\nThe confusing:\n * t3705: These changes were ONLY needed on Windows, but they don't\n hurt other platforms. Let's discuss each individually:\n\n * core.sparseCheckout should be false by default. Nothing in this\n testcase toggles that until many, many tests later. However,\n early tests (#5 in particular) were testing `update-index\n --skip-worktree` behavior in a non-sparse-checkout, but the\n Windows tests in CI were behaving as if core.sparseCheckout=true\n had been specified somewhere. I do not have access to a Windows\n machine. But I just manually did what should have been a no-op\n and turned the config off. And it fixed the test.\n * I have no idea why the leftover .gitattributes file from this\n test was causing failures for test #18 on Windows, but only with\n these changes of mine. Test #18 was checking for empty stderr,\n and specifically wanted to know that some error completely\n unrelated to file endings did not appear. The leftover\n .gitattributes file thus caused some spurious stderr unrelated to\n the thing being checked. Since other tests did not intend to\n test normalization, just proactively remove the .gitattributes\n file. I'm certain this is cleaner and better, I'm just unsure\n why/how this didn't trigger problems before.\n\nSigned-off-by: Elijah Newren \n---\n repository.c | 7 ++++\n sparse-index.c | 22 ++++++++++++\n sparse-index.h | 1 +\n t/t1011-read-tree-sparse-checkout.sh | 2 +-\n t/t1092-sparse-checkout-compatibility.sh | 16 ++++-----\n t/t3705-add-sparse-checkout.sh | 2 ++\n t/t6428-merge-conflicts-sparse.sh | 23 +++----------\n t/t7012-skip-worktree-writing.sh | 44 ++++--------------------\n t/t7817-grep-sparse-checkout.sh | 11 ++++--\n 9 files changed, 62 insertions(+), 66 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex 34610c5a33..dfd1911902 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -301,6 +301,13 @@ int repo_read_index(struct repository *repo)\n \tif (repo->settings.command_requires_full_index)\n \t\tensure_full_index(repo->index);\n \n+\t/*\n+\t * If sparse checkouts are in use, check whether paths with the\n+\t * SKIP_WORKTREE attribute are missing from the worktree; if not,\n+\t * clear that attribute for that path.\n+\t */\n+\tensure_skip_worktree_means_skip_worktree(repo->index);\n+\n \treturn res;\n }\n \ndiff --git a/sparse-index.c b/sparse-index.c\nindex a1d505d50e..79d50e444c 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,6 +341,28 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n+{\n+\tint i;\n+\tif (!core_apply_sparse_checkout)\n+\t\treturn;\n+\n+restart:\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tstruct stat st;\n+\n+\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t\tensure_full_index(istate);\n+\t\t\t\tgoto restart;\n+\t\t\t}\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t}\n+\t}\n+}\n+\n+\n /*\n * This static global helps avoid infinite recursion between\n * expand_to_path() and index_file_exists().\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 656bd835b2..1007859ed4 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -5,6 +5,7 @@ struct index_state;\n #define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n int convert_to_sparse(struct index_state *istate, int flags);\n void ensure_correct_sparsity(struct index_state *istate);\n+void ensure_skip_worktree_means_skip_worktree(struct index_state *istate);\n \n /*\n * Some places in the codebase expect to search for a specific path.\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 4ed0885bf2..dd957be1b7 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -212,7 +212,7 @@ test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n \techo dirty >init.t &&\n-\tread_tree_u_must_succeed -m -u HEAD^ &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n \tgrep -q dirty init.t &&\n \trm init.t\n '\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0863c9747c..6f8538bb4c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -370,7 +370,7 @@ test_expect_success 'status/add: outside sparse cone' '\n \twrite_script edit-contents <<-\\EOF &&\n \techo text >>$1\n \tEOF\n-\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/a &&\n \trun_on_all ../edit-contents folder1/new &&\n \n \ttest_sparse_match git status --porcelain=v2 &&\n@@ -379,8 +379,8 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n-\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_all_match git add --refresh folder1/a &&\n+\ttest_must_be_empty sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n@@ -642,11 +642,11 @@ test_expect_success 'update-index modify outside sparse definition' '\n \trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n \trun_on_all ../edit-contents folder1/a &&\n \n-\t# If file has skip-worktree enabled, update-index does not modify the\n-\t# index entry\n-\ttest_sparse_match git update-index folder1/a &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_must_be_empty sparse-checkout-out &&\n+\t# If file has skip-worktree enabled, but the file is present, it is\n+\t# treated the same as if skip-worktree is disabled\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git update-index folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n \n \t# When skip-worktree is disabled (even on files outside sparse cone), file\n \t# is updated in the index\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex f3143c9290..61506c1d7c 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit config core.sparseCheckout false &&\n \tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n@@ -126,6 +127,7 @@ test_expect_success 'git add --chmod does not update sparse entries' '\n '\n \n test_expect_success 'git add --renormalize does not update sparse entries' '\n+\ttest_when_finished rm .gitattributes &&\n \ttest_config core.autocrlf false &&\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\ndiff --git a/t/t6428-merge-conflicts-sparse.sh b/t/t6428-merge-conflicts-sparse.sh\nindex 7e8bf497f8..142c9aaabc 100755\n--- a/t/t6428-merge-conflicts-sparse.sh\n+++ b/t/t6428-merge-conflicts-sparse.sh\n@@ -112,7 +112,7 @@ test_expect_success 'conflicting entries written to worktree even if sparse' '\n \t)\n '\n \n-test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handled reasonably' '\n+test_expect_success 'present-despite-SKIP_WORKTREE handled reasonably' '\n \ttest_setup_numerals in_the_way &&\n \t(\n \t\tcd numerals_in_the_way &&\n@@ -132,26 +132,13 @@ test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handl\n \n \t\ttest_must_fail git merge -s recursive B^0 &&\n \n-\t\tgit ls-files -t >index_files &&\n-\t\ttest_cmp expected-index index_files &&\n+\t\ttest_path_is_missing .git/MERGE_HEAD &&\n \n-\t\ttest_path_is_file README &&\n \t\ttest_path_is_file numerals &&\n \n-\t\ttest_cmp expected-merge numerals &&\n-\n-\t\t# There should still be a file with \"foobar\" in it\n-\t\tgrep foobar * &&\n-\n-\t\t# 5 other files:\n-\t\t# * expected-merge\n-\t\t# * expected-index\n-\t\t# * index_files\n-\t\t# * others\n-\t\t# * whatever name was given to the numerals file that had\n-\t\t# \"foobar\" in it\n-\t\tgit ls-files -o >others &&\n-\t\ttest_line_count = 5 others\n+\t\t# numerals should still have \"foobar\" in it\n+\t\techo foobar >expect &&\n+\t\ttest_cmp expect numerals\n \t)\n '\n \ndiff --git a/t/t7012-skip-worktree-writing.sh b/t/t7012-skip-worktree-writing.sh\nindex a1080b94e3..cb9f1a6981 100755\n--- a/t/t7012-skip-worktree-writing.sh\n+++ b/t/t7012-skip-worktree-writing.sh\n@@ -171,50 +171,20 @@ test_expect_success 'stash restore in sparse checkout' '\n \n \t\t# Put a file in the working directory in the way\n \t\techo in the way >modified &&\n-\t\tgit stash apply &&\n+\t\ttest_must_fail git stash apply 2>error&&\n \n-\t\t# Ensure stash vivifies modifies paths...\n-\t\tcat >expect <<-EOF &&\n-\t\tH addme\n-\t\tH modified\n-\t\tH removeme\n-\t\tH subdir/A\n-\t\tS untouched\n-\t\tEOF\n-\t\tgit ls-files -t >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\tgrep \"changes.*would be overwritten by merge\" error &&\n \n-\t\t# ...and that the paths show up in status as changed...\n-\t\tcat >expect <<-EOF &&\n-\t\tA addme\n-\t\t M modified\n-\t\t D removeme\n-\t\t M subdir/A\n-\t\t?? actual\n-\t\t?? expect\n-\t\t?? modified.stash.XXXXXX\n-\t\tEOF\n-\t\tgit status --porcelain | \\\n-\t\t\tsed -e s/stash......./stash.XXXXXX/ >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\techo in the way >expect &&\n+\t\ttest_cmp expect modified &&\n+\t\tgit diff --quiet HEAD \":!modified\" &&\n \n \t\t# ...and that working directory reflects the files correctly\n-\t\ttest_path_is_file addme &&\n+\t\ttest_path_is_missing addme &&\n \t\ttest_path_is_file modified &&\n \t\ttest_path_is_missing removeme &&\n \t\ttest_path_is_file subdir/A &&\n-\t\ttest_path_is_missing untouched &&\n-\n-\t\t# ...including that we have the expected \"modified\" file...\n-\t\tcat >expect <<-EOF &&\n-\t\tmodified\n-\t\ttweaked\n-\t\tEOF\n-\t\ttest_cmp expect modified &&\n-\n-\t\t# ...and that the other \"modified\" file is still present...\n-\t\techo in the way >expect &&\n-\t\ttest_cmp expect modified.stash.*\n+\t\ttest_path_is_missing untouched\n \t)\n '\n \ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex 590b99bbb6..eb59564565 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -83,10 +83,13 @@ test_expect_success 'setup' '\n \n # The test below covers a special case: the sparsity patterns exclude '/b' and\n # sparse checkout is enabled, but the path exists in the working tree (e.g.\n-# manually created after `git sparse-checkout init`). git grep should skip it.\n+# manually created after `git sparse-checkout init`). Although b is marked\n+# as SKIP_WORKTREE, git grep should notice it IS present in the worktree and\n+# report it.\n test_expect_success 'working tree grep honors sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n+\tb:new-text\n \tEOF\n \ttest_when_finished \"rm -f b\" &&\n \techo \"new-text\" >b &&\n@@ -126,12 +129,16 @@ test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit'\n '\n \n # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n-# patterns, so grep should not recurse into it.\n+# patterns. We also explicitly mark it as SKIP_WORKTREE in case it got cleared\n+# by previous git commands. Thus sub2 starts as SKIP_WORKTREE but since it is\n+# present in the working tree, grep should recurse into it.\n test_expect_success 'grep --recurse-submodules honors sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tsub/B/b:text\n+\tsub2/a:text\n \tEOF\n+\tgit update-index --skip-worktree sub2 &&\n \tgit grep --recurse-submodules \"text\" >actual &&\n \ttest_cmp expect actual\n '\n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445806","messageId":"20220109045732.2497526-5-newren@gmail.com","threadId":"57214","inReplyTo":"20220109045732.2497526-1-newren@gmail.com","subject":"[RFC PATCH 4/5] Update documentation related to sparsity and the skip-worktree bit","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:31Z","receivedAt":"2022-01-09T04:57:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Make several small updates, to address a few documentation issues\nI spotted:\n * sparse-checkout focused on \"patterns\" even though the inputs (and\n outputs in the case of `list`) are directories in cone-mode\n * The description section of the sparse-checkout documentation\n was a bit sparse (no pun intended), and focused more on internal\n mechanics rather than end user usage. This made sense in the\n early days when the command was even more experimental, but let's\n adjust a bit to try to make it more approachable to end users who\n may want to consider using it. Keep the scary backward\n compatibility warning, though; we're still hard at work trying to\n fix up commands to behave reasonably in sparse checkouts.\n * both read-tree and update-index tried to describe how to use the\n skip-worktree bit, but both predated the sparse-checkout command.\n The sparse-checkout command is a far easier mechanism to use and\n for users trying to reduce the size of their working tree, we\n should recommend users to look at it instead.\n * The update-index documentation pointed out that assume-unchanged\n and skip-worktree sounded similar but had different purposes.\n However, it made no attempt to explain the differences, only to\n point out that they were different. Explain the differences.\n * The update-index documentation focused much more on (internal?)\n implementation details than on end-user usage. Try to explain\n its purpose better for users of update-index, rather than\n fellow developers trying to work with the SKIP_WORKTREE bit.\n * Clarify that when core.sparseCheckout=true, we treat a file's\n presence in the working tree as being an override to the\n SKIP_WORKTREE bit (i.e. in sparse checkouts when the file is\n present we ignore the SKIP_WORKTREE bit).\n\nNote that this commit, like many touching documentation, is best viewed\nwith the `--color-words` option to diff/log.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-read-tree.txt | 12 +++--\n Documentation/git-sparse-checkout.txt | 76 ++++++++++++++++-----------\n Documentation/git-update-index.txt | 55 ++++++++++++++-----\n 3 files changed, 97 insertions(+), 46 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 8c3aceb832..99bb387134 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -375,9 +375,14 @@ have finished your work-in-progress), attempt the merge again.\n SPARSE CHECKOUT\n ---------------\n \n+Note: The `update-index` and `read-tree` primitives for supporting the\n+skip-worktree bit predated the introduction of\n+linkgit:git-sparse-checkout[1]. Users are encouraged to use\n+`sparse-checkout` in preference to these low-level primitives.\n+\n \"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to\n+tell Git whether a file in the working directory is worth looking at.\n \n 'git read-tree' and other merge-based commands ('git merge', 'git\n checkout'...) can help maintaining the skip-worktree bitmap and working\n@@ -385,7 +390,8 @@ directory update. `$GIT_DIR/info/sparse-checkout` is used to\n define the skip-worktree reference bitmap. When 'git read-tree' needs\n to update the working directory, it resets the skip-worktree bit in the index\n based on this file, which uses the same syntax as .gitignore files.\n-If an entry matches a pattern in this file, skip-worktree will not be\n+If an entry matches a pattern in this file, or the entry corresponds to\n+a file present in the working tree, then skip-worktree will not be\n set on that entry. Otherwise, skip-worktree will be set.\n \n Then it compares the new skip-worktree value with the previous one. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b81dbe0654..3da3d5a100 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -3,9 +3,7 @@ git-sparse-checkout(1)\n \n NAME\n ----\n-git-sparse-checkout - Initialize and modify the sparse-checkout\n-configuration, which reduces the checkout to a set of paths\n-given by a list of patterns.\n+git-sparse-checkout - Reduce your working tree to a subset of tracked files\n \n \n SYNOPSIS\n@@ -17,8 +15,20 @@ SYNOPSIS\n DESCRIPTION\n -----------\n \n-Initialize and modify the sparse-checkout configuration, which reduces\n-the checkout to a set of paths given by a list of patterns.\n+This command is used to create sparse checkouts, which means that it\n+changes the working tree from having all tracked files present, to only\n+have a subset of them. It can also switch which subset of files are\n+present, or undo and go back to having all tracked files present in the\n+working copy.\n+\n+The subset of files is chosen by providing a list of directories in\n+cone mode (which is recommended), or by providing a list of patterns\n+in non-cone mode.\n+\n+When in a sparse-checkout, other Git commands behave a bit differently.\n+For example, switching branches will not update paths outside the\n+sparse-checkout directories/patterns, and `git commit -a` will not record\n+paths outside the sparse-checkout directories/patterns as deleted.\n \n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n@@ -28,7 +38,7 @@ THE FUTURE.\n COMMANDS\n --------\n 'list'::\n-\tDescribe the patterns in the sparse-checkout file.\n+\tDescribe the directories or patterns in the sparse-checkout file.\n \n 'set'::\n \tEnable the necessary config settings\n@@ -38,20 +48,26 @@ COMMANDS\n \tlist of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n +\n-When the `--stdin` option is provided, the patterns are read from\n-standard in as a newline-delimited list instead of from the arguments.\n+When the `--stdin` option is provided, the directories or patterns are\n+read from standard in as a newline-delimited list instead of from the\n+arguments.\n +\n When `--cone` is passed or `core.sparseCheckoutCone` is enabled, the\n-input list is considered a list of directories instead of\n-sparse-checkout patterns. This allows for better performance with a\n-limited set of patterns (see 'CONE PATTERN SET' below). Note that the\n-set command will write patterns to the sparse-checkout file to include\n-all files contained in those directories (recursively) as well as\n-files that are siblings of ancestor directories. The input format\n-matches the output of `git ls-tree --name-only`. This includes\n-interpreting pathnames that begin with a double quote (\") as C-style\n-quoted strings. This may become the default in the future; --no-cone\n-can be passed to request non-cone mode.\n+input list is considered a list of directories. This allows for\n+better performance with a limited set of patterns (see 'CONE PATTERN\n+SET' below). The input format matches the output of `git ls-tree\n+--name-only`. This includes interpreting pathnames that begin with a\n+double quote (\") as C-style quoted strings. Note that the set command\n+will write patterns to the sparse-checkout file to include all files\n+contained in those directories (recursively) as well as files that are\n+siblings of ancestor directories. This may become the default in the\n+future; --no-cone can be passed to request non-cone mode.\n++\n+When `--no-cone` is passed or `core.sparseCheckoutCone` is not enabled,\n+the input list is considered a list of patterns. This mode is harder\n+to use and less performant, and is thus not recommended. See the\n+\"Sparse Checkout\" section of linkgit:git-read-tree[1] and the \"Pattern\n+Set\" sections below for more details.\n +\n Use the `--[no-]sparse-index` option to use a sparse index (the\n default is to not use it). A sparse index reduces the size of the\n@@ -69,11 +85,10 @@ understand the sparse directory entries index extension and may fail to\n interact with your repository until it is disabled.\n \n 'add'::\n-\tUpdate the sparse-checkout file to include additional patterns.\n-\tBy default, these patterns are read from the command-line arguments,\n-\tbut they can be read from stdin using the `--stdin` option. When\n-\t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n-\tas directory names as in the 'set' subcommand.\n+\tUpdate the sparse-checkout file to include additional directories\n+\t(in cone mode) or patterns (in non-cone mode). By default, these\n+\tdirectories or patterns are read from the command-line arguments,\n+\tbut they can be read from stdin using the `--stdin` option.\n \n 'reapply'::\n \tReapply the sparsity pattern rules to paths in the working tree.\n@@ -117,13 +132,14 @@ decreased in utility.\n SPARSE CHECKOUT\n ---------------\n \n-\"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at. If\n-the skip-worktree bit is set, then the file is ignored in the working\n-directory. Git will avoid populating the contents of those files, which\n-makes a sparse checkout helpful when working in a repository with many\n-files, but only a few are important to the current user.\n+\"Sparse checkout\" allows populating the working directory sparsely. It\n+uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell Git\n+whether a file in the working directory is worth looking at. If the\n+skip-worktree bit is set, and the file is not present in the working tree,\n+then its absence is ignored. Git will avoid populating the contents of\n+those files, which makes a sparse checkout helpful when working in a\n+repository with many files, but only a few are important to the current\n+user.\n \n The `$GIT_DIR/info/sparse-checkout` file is used to define the\n skip-worktree reference bitmap. When Git updates the working\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d9..568dbfe76b 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -351,6 +351,10 @@ unchanged\". Note that \"assume unchanged\" bit is *not* set if\n the index (use `git update-index --really-refresh` if you want\n to mark them as \"assume unchanged\").\n \n+Sometimes users confuse the assume-unchanged bit with the\n+skip-worktree bit. See the final paragraph in the \"Skip-worktree bit\"\n+section below for an explanation of the differences.\n+\n \n EXAMPLES\n --------\n@@ -392,22 +396,47 @@ M foo.c\n SKIP-WORKTREE BIT\n -----------------\n \n-Skip-worktree bit can be defined in one (long) sentence: When reading\n-an entry, if it is marked as skip-worktree, then Git pretends its\n-working directory version is up to date and read the index version\n-instead.\n+Skip-worktree bit can be defined in one (long) sentence: Tell git to\n+avoid writing the file to the working directory when reasonably\n+possible, and treat the file as unchanged when it is not\n+present in the working directory.\n \n-To elaborate, \"reading\" means checking for file existence, reading\n-file attributes or file content. The working directory version may be\n-present or absent. If present, its content may match against the index\n-version or not. Writing is not affected by this bit, content safety\n-is still first priority. Note that Git _can_ update working directory\n-file, that is marked skip-worktree, if it is safe to do so (i.e.\n-working directory version matches index version)\n+Note that not all git commands will pay attention to this bit, and\n+some only partially support it.\n+\n+The update-index flags and the read-tree capabilities relating to the\n+skip-worktree bit predated the introduction of the\n+linkgit:git-sparse-checkout[1] command, which provides a much easier\n+way to configure and handle the skip-worktree bits. If you want to\n+reduce your working tree to only deal with a subset of the files in\n+the repository, we strongly encourage the use of\n+linkgit:git-sparse-checkout[1] in preference to the low-level\n+update-index and read-tree primitives.\n+\n+The primary purpose of the skip-worktree bit is to enable sparse\n+checkouts, i.e. to have working directories with only a subset of\n+paths present. When the skip-worktree bit is set, Git commands (such\n+as `switch`, `pull`, `merge`) will avoid writing these files.\n+However, these commands will sometimes write these files anyway in\n+important cases such as conflicts during a merge or rebase. Git\n+commands will also avoid treating the lack of such files as an\n+intentional deletion; for example `git add -u` will not not stage a\n+deletion for these files and `git commit -a` will not make a commit\n+deleting them either.\n \n Although this bit looks similar to assume-unchanged bit, its goal is\n-different from assume-unchanged bit's. Skip-worktree also takes\n-precedence over assume-unchanged bit when both are set.\n+different. The assume-unchanged bit is for leaving the file in the\n+working tree but having Git omit checking it for changes and presuming\n+that the file has not been changed (though if it can determine without\n+stat'ing the file that it has changed, it is free to record the\n+changes). skip-worktree tells Git to ignore the absence of the file,\n+avoid updating it when possible with commands that normally update\n+much of the working directory (e.g. `checkout`, `switch`, `pull`,\n+etc.), and not have its absence be recorded in commits. Note that in\n+sparse checkouts (setup by `git sparse-checkout` or by configuring\n+core.sparseCheckout to true), if a file is marked as skip-worktree in\n+the index but is found in the working tree, Git will clear the\n+skip-worktree bit for that file.\n \n SPLIT INDEX\n -----------\n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445807","messageId":"20220109045732.2497526-6-newren@gmail.com","threadId":"57214","inReplyTo":"20220109045732.2497526-1-newren@gmail.com","subject":"[RFC PATCH 5/5] Accelerate ensure_skip_worktree_means_skip_worktree by caching","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-09T04:57:32Z","receivedAt":"2022-01-09T04:57:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Rather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\nfact that entire directories will often be missing, especially for cone\nmode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\nclear tracked sparse dirs\", 2021-09-08). If we have already determined\nthat the parent directory of a file (or any other previous ancestor)\ndoes not exist, then we already know the file cannot exist and do not\nneed to lstat() it separately.\n\nGranted, the cost of ensure_skip_worktree_means_skip_worktree() might\nbe considered a bit high for non-cone mode since it might now lstat()\nevery SKIP_WORKTREE path when the index is loaded (an O(N) cost, with\nN the number of SKIP_WORKTREE paths), but non-cone mode users already\nhave to deal with the O(N*M) cost (with N=the number of tracked files\nand M=the number of sparsity patterns), so this should be reasonable.\n\nSigned-off-by: Elijah Newren \n---\n sparse-index.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 103 insertions(+), 2 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 79d50e444c..608782e255 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,18 +341,117 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+struct path_cache_entry {\n+\tstruct hashmap_entry ent;\n+\tconst char *path;\n+\tint path_length;\n+\tint is_present;\n+};\n+\n+static int path_cache_cmp(const void *unused,\n+\t\t\t const struct hashmap_entry *entry1,\n+\t\t\t const struct hashmap_entry *entry2,\n+\t\t\t const void *also_unused)\n+{\n+\tconst struct path_cache_entry *e1, *e2;\n+\n+\te1 = container_of(entry1, const struct path_cache_entry, ent);\n+\te2 = container_of(entry2, const struct path_cache_entry, ent);\n+\tif (e1->path_length != e2->path_length)\n+\t\treturn e1->path_length - e2->path_length;\n+\treturn memcmp(e1->path, e2->path, e1->path_length);\n+}\n+\n+static struct path_cache_entry *find_path_cache_entry(struct hashmap *map,\n+\t\t\t\t\t\t const char *str,\n+\t\t\t\t\t\t int str_length)\n+{\n+\tstruct path_cache_entry entry;\n+\thashmap_entry_init(&entry.ent, memhash(str, str_length));\n+\tentry.path = str;\n+\tentry.path_length = str_length;\n+\treturn hashmap_get_entry(map, &entry, ent, NULL);\n+}\n+\n+static void record(struct hashmap *path_cache,\n+\t\t struct mem_pool *pool,\n+\t\t const char *path,\n+\t\t int path_length,\n+\t\t int found)\n+{\n+\tstruct path_cache_entry *entry;\n+\n+\tentry = mem_pool_alloc(pool, sizeof(*entry));\n+\thashmap_entry_init(&entry->ent, memhash(path, path_length));\n+\tentry->path = path;\n+\tentry->path_length = path_length;\n+\tentry->is_present = found;\n+\thashmap_add(path_cache, &entry->ent);\n+}\n+\n+static int path_found(struct hashmap *path_cache, struct mem_pool *pool,\n+\t\t const char *path, int path_length)\n+{\n+\tstruct stat st;\n+\tint found;\n+\tconst char *dirsep = path + path_length - 1;\n+\tconst char *tmp;\n+\n+\t/* Find directory separator; memrchr is sadly glibc-specific */\n+\twhile (dirsep > path && *dirsep != '/')\n+\t\tdirsep--;\n+\n+\t/* If parent of path doesn't exist, no point lstat'ing path... */\n+\tif (dirsep > path) {\n+\t\tstruct path_cache_entry *entry;\n+\t\tint new_length, parent_found;\n+\n+\t\t/* First, check if path's parent's existence was cached */\n+\t\tnew_length = dirsep - path;\n+\t\tentry = find_path_cache_entry(path_cache, path, new_length);\n+\t\tif (entry)\n+\t\t\tparent_found = entry->is_present;\n+\t\telse\n+\t\t\tparent_found = path_found(path_cache, pool,\n+\t\t\t\t\t\t path, new_length);\n+\n+\t\tif (!parent_found) {\n+\t\t\t/* path can't exist if parent dir doesn't */\n+\t\t\trecord(path_cache, pool, path, path_length, 0);\n+\t\t\treturn 0;\n+\t\t} /* else parent was found so must check path itself too... */\n+\t}\n+\n+\t/* Okay, parent dir exists, so we have to check original path */\n+\n+\t/* Make sure we have a NUL-terminated string to pass to lstat */\n+\ttmp = path;\n+\tif (path[path_length])\n+\t\ttmp = mem_pool_strndup(pool, path, path_length);\n+\t/* Determine if path exists */\n+\tfound = !lstat(tmp, &st);\n+\n+\trecord(path_cache, pool, path, path_length, found);\n+\treturn found;\n+}\n+\n void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n {\n+\tstruct hashmap path_cache = HASHMAP_INIT(path_cache_cmp, NULL);\n+\tstruct mem_pool pool;\n+\n \tint i;\n+\n \tif (!core_apply_sparse_checkout)\n \t\treturn;\n \n+\tmem_pool_init(&pool, 32*1024);\n restart:\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tstruct stat st;\n \n-\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\tif (ce_skip_worktree(ce) &&\n+\t\t path_found(&path_cache, &pool, ce->name, strlen(ce->name))) {\n \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n \t\t\t\tensure_full_index(istate);\n \t\t\t\tgoto restart;\n@@ -360,6 +459,8 @@ void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n \t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \t\t}\n \t}\n+\thashmap_clear(&path_cache);\n+\tmem_pool_discard(&pool, 0);\n }\n \n \n-- \n2.34.1.442.ge63c19bdd2.dirty\n\n"},{"id":"445874","messageId":"38b69b5e-b461-23cc-155e-26fd58e5f714@github.com","threadId":"57214","inReplyTo":"20220109045732.2497526-4-newren@gmail.com","subject":"Re: [RFC PATCH 3/5] repo_read_index: ensure SKIP_WORKTREE means skip worktree","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-10T20:38:18Z","receivedAt":"2022-01-10T20:38:22Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> The fix is short (~30 lines), but the description is not. Sorry.\n> \n> There is a set of problems caused by files in what I'll refer to as the\n> \"present-despite-SKIP_WORKTREE\" state. This commit aims to not just fix\n> these problems, but remove the entire class as a possibility -- for\n> those using sparse checkouts. But first, we need to understand the\n> problems this class presents. A quick outline:\n> \n> * Problems\n> * User facing issues\n> * Problem space complexity\n> * Maintenance and code correctness challenges\n> * SKIP_WORKTREE expectations in Git\n> * Suggested solution\n> * Pros/Cons of suggested solution\n> * Notes on testcase modifications\n> \n> === User facing issues ===\n> \n> There are various ways for users to get files to be present in the\n> working copy despite having the SKIP_WORKTREE bit set for that file in\n> the index. This may come from:\n> * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n> * users grabbing files from elsewhere and writing them to the worktree\n> (perhaps even cached in their editor)\n> * users attempting to \"abort\" a sparse-checkout operation with a\n> not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n> working tree is not atomic)[3].\n> \n> Once users have present-despite-SKIP_WORKTREE files, any modifications\n> users make to these files will be ignored, possibly to users' confusion.\n> \n> Further:\n> * these files will not be updated by by standard commands\n> (switch/checkout/pull/merge/rebase will leave them alone unless\n> conflicts happen -- and even then, the conflicted file may be\n> written somewhere else to avoid overwriting the SKIP_WORKTREE file\n> that is present and in the way)\n> * there is nothing in Git that users can use to discover such\n> files (status, diff, grep, etc. all ignore it)\n> * there is no reasonable mechanism to \"recover\" from such a condition\n> (neither `git sparse-checkout reapply` nor `git reset --hard` will\n> correct it).\n> \n\nJust to add to this, files like these always force sparse index expansion in\n`git status` (and probably some other commands?), ruining a lot of the\nperformance gains of using sparse index in the first place.\n\n> So, not only are users modifications ignored, but the files get\n> progressively more stale over time. At some point in the future, they\n> may change their sparseness specification or disable sparse-checkouts.\n> At that time, all present-despite-SKIP_WORKTREE files will show up as\n> having lots of modifications because they represent a version from a\n> different branch or commit. These might include user-made local changes\n> from days before, but the only way to tell is to have users look through\n> them all closely.\n> \n> If these users come to others for help, there will be no logs that\n> explain the issue; it's just a mysterious list of changes. Users might\n> adamantly claim (correctly, as it turns out) that they didn't modify\n> these files, while others presume they did.\n> \n> [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n> [2] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n> [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n> \n> === Problem space complexity ===\n> \n> SKIP_WORKTREE has been part of Git for over a decade. Duy did lots of\n> work on it initially, and several others have since come along and put\n> lots of work into it. Stolee spent most of 2021 on the sparse-index,\n> with lots of bugfixes along the way including to non-sparse-index cases\n> as we are still trying to get sparse checkouts to behave reasonably.\n> Basically every codepath throughout the treat needs to be aware of an\n> additional type of file: tracked-but-not-present. The extra type\n> results in lots of extra testcases and lots of extra code everywhere.\n> \n> But, the sad thing is that we actually have more than one extra type.\n> We have tracked, tracked-but-not-present (SKIP_WORKTREE), and\n> tracked-but-promised-to-not-be-present-but-is-present-anyway\n> (present-despite-SKIP_WORKTREE). Two types is a monumental amount of\n> effort to support, and adding a third feels a bit like insanity[4].\n> \n> [4] Some examples of which can be seen at\n> https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n> \n> === Maintenance and code correctness challenges ===\n> \n> Matheus' patches to grep stalled for nearly a year, in part because of\n> complications of how to handle sparse-checkouts appropriately in all\n> cases[5][6] (with trying to sanely figure out how to sanely handle\n> present-despite-SKIP_WORKTREE files being one of the complications).\n> His rm/add follow-ups also took months because of those kinds of\n> issues[7]. The corner cases with things like submodules and\n> SKIP_WORKTREE with the addition of present-despite-SKIP_WORKTREE start\n> becoming really complex[8].\n> \n> We've had to add ugly logic to merge-ort to attempt to handle\n> present-despite-SKIP_WORKTREE files[9], and basically just been forced\n> to give up in merge-recursive knowing full well that we'll sometimes\n> silently discard user modifications. Despite stash essentially being a\n> merge, it needed extra code (beyond what was in merge-ort and\n> merge-recursive) to manually tweak SKIP_WORKTREE bits in order to avoid\n> a few different bugs that'd result in an early abort with a partial\n> stash application[10].\n> \n> [5] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t\n> and the dates on the thread; also Matheus and I had several\n> conversations off-list trying to resolve the issues over that time\n> [6] ...it finally kind of got unstuck after\n> https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n> [7] See for example\n> https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t\n> and quotes like \"The core functionality of sparse-checkout has always\n> been only partially implemented\", a statement I still believe is true\n> today.\n> [8] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n> [9] See commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE\n> handling with conflicted entries\", 2021-03-20)\n> [10] See commit ba359fd507 (\"stash: fix stash application in\n> sparse-checkouts\", 2020-12-01)\n> \n> === SKIP_WORKTREE expectations in Git ===\n> \n> A couple quotes:\n> \n> From [11] (before the \"sparse-checkout\" command existed):\n> If it needs too many special cases, hacks, and conditionals, then it\n> is not worth the complexity---if it is easier to write a correct code\n> by allowing Git to populate working tree files, it is perfectly fine\n> to do so.\n> \n> In a sense, the sparse checkout \"feature\" itself is a hack by itself,\n> and that is why I think this part should be \"best effort\" as well.\n> \n> From the git-sparse-checkout manual (still present today):\n> \n> THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n> COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n> THE FUTURE.\n> \n> [11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n> \n> === Suggested solution ===\n> \n> SKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\n> the name of the option implies, to allow the file to NOT be in the\n> worktree but consider it to be unchanged rather than deleted.\n> \n> The suggests a simple solution: present-despite-SKIP_WORKTREE files\n> should not exist, for those using sparse-checkouts.\n> \n> Enforce this at index loading time by checking if core.sparseCheckout is\n> true; if so, check files in the index with the SKIP_WORKTREE bit set to\n> verify that they are absent from the working tree. If they are present,\n> unset the bit (in memory, though any commands that write to the index\n> will record the update).\n> \n\nSince this solution is specific to a sparse-checkout, should this automatic\nunsetting only be done if the file is outside the sparse checkout\ndefinition? Otherwise, the `sparse-checkout reapply` cleanup suggested below\ndoesn't return the original `skip-worktree` state. \n\nAdmittedly, I imagine it's unlikely that someone is simultaneously using a\nsparse checkout and manually SKIP_WORKTREE-ing files *inside* the sparse\ncheckout definition. But, given that you're not unsetting the flag for\nnon-sparse-checkout SKIP_WORKTREE files, it seems like an additional\nconstraint based on sparse checkout patterns would be consistent with\nother parts of this patch.\n\n> Users can, of course, can get the SKIP_WORKTREE bit back such as by\n> running `git sparse-checkout reapply` (if they have ensured the file is\n> unmodified and doesn't match the specified sparsity patterns).\n> \n\nThere are some performance implications of this solution in a sparse\nindex-enabled checkout. Any time an out-of-cone file is no longer\nSKIP_WORKTREE, its parent directory lineage will be added to the sparse index,\nand performance would progressively (silently) degrade as more out-of-cone\nfiles were added.\n\nThat said, a lot of my concern would be alleviated with some kind of warning\nindicating that a file just had SKIP_WORKTREE removed, including a mention\nof fixing it with `git sparse-checkout reapply`. \n\nIt would be *extra* nice if `git status` could tell a user that they have\nnon-SKIP_WORKTREE files outside the sparse definition, but I think that's\nless critical and probably better suited as a separate series.\n\n> === Pros/Cons of suggested solution ===\n> \n> Pros:\n> \n> * Solves the user visible problems reported above, which I've been\n> complaining about for nearly a year but couldn't find a solution to.\n> * Much easier behavior in sparse-checkouts for users to reason about\n> * Very simple, ~30 lines of code.\n> * Significantly simplifies some ugly testcases, and obviates the need\n> to test an entire class of potential issues.\n> * Reduces code complexity, reasoning, and maintenance. Avoids\n> disagreements about weird corner cases[12].\n> * It has been reported that some users might be (ab)using\n> SKIP_WORKTREE as a let-me-modify-but-keep-the-file-in-the-worktree\n> mechanism[13, and a few other similar references]. These users know\n> of multiple caveats and shortcomings in doing so; perhaps not\n> surprising given the \"SKIP_WORKTREE expecations\" section above.\n> However, these users use `git update-index --skip-worktree`, and not\n> `git sparse-checkout` or core.sparseCheckout=true. As such, these\n> users would be unaffected by this change and can continue abusing\n> the system as before.\n> \n> [12] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n> [13] https://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree\n> \n> Cons:\n> \n> * When core.sparseCheckout is enabled, this adds a performance cost to\n> reading the index. I'll defer discussion of this cost to a subsequent\n> patch, since I have some optimizations to add.\n> \n> === Notes on testcase modifications ===\n> \n> The good:\n> * t1011: Compare to two cases above it ('read-tree will not throw away\n> dirty changes, non-sparse'); since the file is present, it should\n> match the non-sparse case now\n> * t1092: sparse-index & sparse-checkout now match full-worktree\n> behavior in more cases! Yaay for consistency!\n> * t6428, t7012: look at how much simpler the tests become! Merge and\n> stash can just fail early telling the user there's a file in the\n> way, instead of not noticing until it's about to write a file and\n> then have to implement sudden crash avoidance. Hurray for sanity!\n> * t7817: sparse behavior better matches full tree behavior. Hurray\n> for sanity!\n> \n> The confusing:\n> * t3705: These changes were ONLY needed on Windows, but they don't\n> hurt other platforms. Let's discuss each individually:\n> \n> * core.sparseCheckout should be false by default. Nothing in this\n> testcase toggles that until many, many tests later. However,\n> early tests (#5 in particular) were testing `update-index\n> --skip-worktree` behavior in a non-sparse-checkout, but the\n> Windows tests in CI were behaving as if core.sparseCheckout=true\n> had been specified somewhere. I do not have access to a Windows\n> machine. But I just manually did what should have been a no-op\n> and turned the config off. And it fixed the test.\n> * I have no idea why the leftover .gitattributes file from this\n> test was causing failures for test #18 on Windows, but only with\n> these changes of mine. Test #18 was checking for empty stderr,\n> and specifically wanted to know that some error completely\n> unrelated to file endings did not appear. The leftover\n> .gitattributes file thus caused some spurious stderr unrelated to\n> the thing being checked. Since other tests did not intend to\n> test normalization, just proactively remove the .gitattributes\n> file. I'm certain this is cleaner and better, I'm just unsure\n> why/how this didn't trigger problems before.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> repository.c | 7 ++++\n> sparse-index.c | 22 ++++++++++++\n> sparse-index.h | 1 +\n> t/t1011-read-tree-sparse-checkout.sh | 2 +-\n> t/t1092-sparse-checkout-compatibility.sh | 16 ++++-----\n> t/t3705-add-sparse-checkout.sh | 2 ++\n> t/t6428-merge-conflicts-sparse.sh | 23 +++----------\n> t/t7012-skip-worktree-writing.sh | 44 ++++--------------------\n> t/t7817-grep-sparse-checkout.sh | 11 ++++--\n> 9 files changed, 62 insertions(+), 66 deletions(-)\n> \n> diff --git a/repository.c b/repository.c\n> index 34610c5a33..dfd1911902 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -301,6 +301,13 @@ int repo_read_index(struct repository *repo)\n> \tif (repo->settings.command_requires_full_index)\n> \t\tensure_full_index(repo->index);\n> \n> +\t/*\n> +\t * If sparse checkouts are in use, check whether paths with the\n> +\t * SKIP_WORKTREE attribute are missing from the worktree; if not,\n> +\t * clear that attribute for that path.\n> +\t */\n> +\tensure_skip_worktree_means_skip_worktree(repo->index);\n> +\n> \treturn res;\n> }\n> \n> diff --git a/sparse-index.c b/sparse-index.c\n> index a1d505d50e..79d50e444c 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -341,6 +341,28 @@ void ensure_correct_sparsity(struct index_state *istate)\n> \t\tensure_full_index(istate);\n> }\n> \n> +void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n\nI can feel the frustration behind this name. :) \n\nHowever, a more descriptive one would make the code easier to follow, e.g.\n'clear_skip_worktree_from_present_files' (or something else indicating what\nit does to the index).\n\n> +{\n> +\tint i;\n> +\tif (!core_apply_sparse_checkout)\n> +\t\treturn;\n> +\n> +restart:\n> +\tfor (i = 0; i < istate->cache_nr; i++) {\n> +\t\tstruct cache_entry *ce = istate->cache[i];\n> +\t\tstruct stat st;\n> +\n> +\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n> +\t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> +\t\t\t\tensure_full_index(istate);\n> +\t\t\t\tgoto restart;\n> +\t\t\t}\n> +\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n> +\t\t}\n> +\t}\n> +}\n> +\n> +\n> /*\n> * This static global helps avoid infinite recursion between\n> * expand_to_path() and index_file_exists().\n> diff --git a/sparse-index.h b/sparse-index.h\n> index 656bd835b2..1007859ed4 100644\n> --- a/sparse-index.h\n> +++ b/sparse-index.h\n> @@ -5,6 +5,7 @@ struct index_state;\n> #define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n> int convert_to_sparse(struct index_state *istate, int flags);\n> void ensure_correct_sparsity(struct index_state *istate);\n> +void ensure_skip_worktree_means_skip_worktree(struct index_state *istate);\n> \n> /*\n> * Some places in the codebase expect to search for a specific path.\n> diff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\n> index 4ed0885bf2..dd957be1b7 100755\n> --- a/t/t1011-read-tree-sparse-checkout.sh\n> +++ b/t/t1011-read-tree-sparse-checkout.sh\n> @@ -212,7 +212,7 @@ test_expect_success 'read-tree updates worktree, dirty case' '\n> \techo sub/added >.git/info/sparse-checkout &&\n> \tgit checkout -f top &&\n> \techo dirty >init.t &&\n> -\tread_tree_u_must_succeed -m -u HEAD^ &&\n> +\tread_tree_u_must_fail -m -u HEAD^ &&\n> \tgrep -q dirty init.t &&\n> \trm init.t\n> '\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 0863c9747c..6f8538bb4c 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -370,7 +370,7 @@ test_expect_success 'status/add: outside sparse cone' '\n> \twrite_script edit-contents <<-\\EOF &&\n> \techo text >>$1\n> \tEOF\n> -\trun_on_sparse ../edit-contents folder1/a &&\n> +\trun_on_all ../edit-contents folder1/a &&\n> \trun_on_all ../edit-contents folder1/new &&\n> \n> \ttest_sparse_match git status --porcelain=v2 &&\n> @@ -379,8 +379,8 @@ test_expect_success 'status/add: outside sparse cone' '\n> \ttest_sparse_match test_must_fail git add folder1/a &&\n> \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> \ttest_sparse_unstaged folder1/a &&\n> -\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n> -\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> +\ttest_all_match git add --refresh folder1/a &&\n> +\ttest_must_be_empty sparse-checkout-err &&\n> \ttest_sparse_unstaged folder1/a &&\n> \ttest_sparse_match test_must_fail git add folder1/new &&\n> \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> @@ -642,11 +642,11 @@ test_expect_success 'update-index modify outside sparse definition' '\n> \trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n> \trun_on_all ../edit-contents folder1/a &&\n> \n> -\t# If file has skip-worktree enabled, update-index does not modify the\n> -\t# index entry\n> -\ttest_sparse_match git update-index folder1/a &&\n> -\ttest_sparse_match git status --porcelain=v2 &&\n> -\ttest_must_be_empty sparse-checkout-out &&\n> +\t# If file has skip-worktree enabled, but the file is present, it is\n> +\t# treated the same as if skip-worktree is disabled\n> +\ttest_all_match git status --porcelain=v2 &&\n> +\ttest_all_match git update-index folder1/a &&\n> +\ttest_all_match git status --porcelain=v2 &&\n> \n> \t# When skip-worktree is disabled (even on files outside sparse cone), file\n> \t# is updated in the index\n> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n> index f3143c9290..61506c1d7c 100755\n> --- a/t/t3705-add-sparse-checkout.sh\n> +++ b/t/t3705-add-sparse-checkout.sh\n> @@ -19,6 +19,7 @@ setup_sparse_entry () {\n> \tfi &&\n> \tgit add sparse_entry &&\n> \tgit update-index --skip-worktree sparse_entry &&\n> +\tgit config core.sparseCheckout false &&\n> \tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n> \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n> }\n> @@ -126,6 +127,7 @@ test_expect_success 'git add --chmod does not update sparse entries' '\n> '\n> \n> test_expect_success 'git add --renormalize does not update sparse entries' '\n> +\ttest_when_finished rm .gitattributes &&\n> \ttest_config core.autocrlf false &&\n> \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n> \techo \"sparse_entry text=auto\" >.gitattributes &&\n> diff --git a/t/t6428-merge-conflicts-sparse.sh b/t/t6428-merge-conflicts-sparse.sh\n> index 7e8bf497f8..142c9aaabc 100755\n> --- a/t/t6428-merge-conflicts-sparse.sh\n> +++ b/t/t6428-merge-conflicts-sparse.sh\n> @@ -112,7 +112,7 @@ test_expect_success 'conflicting entries written to worktree even if sparse' '\n> \t)\n> '\n> \n> -test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handled reasonably' '\n> +test_expect_success 'present-despite-SKIP_WORKTREE handled reasonably' '\n> \ttest_setup_numerals in_the_way &&\n> \t(\n> \t\tcd numerals_in_the_way &&\n> @@ -132,26 +132,13 @@ test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handl\n> \n> \t\ttest_must_fail git merge -s recursive B^0 &&\n> \n> -\t\tgit ls-files -t >index_files &&\n> -\t\ttest_cmp expected-index index_files &&\n> +\t\ttest_path_is_missing .git/MERGE_HEAD &&\n> \n> -\t\ttest_path_is_file README &&\n> \t\ttest_path_is_file numerals &&\n> \n> -\t\ttest_cmp expected-merge numerals &&\n> -\n> -\t\t# There should still be a file with \"foobar\" in it\n> -\t\tgrep foobar * &&\n> -\n> -\t\t# 5 other files:\n> -\t\t# * expected-merge\n> -\t\t# * expected-index\n> -\t\t# * index_files\n> -\t\t# * others\n> -\t\t# * whatever name was given to the numerals file that had\n> -\t\t# \"foobar\" in it\n> -\t\tgit ls-files -o >others &&\n> -\t\ttest_line_count = 5 others\n> +\t\t# numerals should still have \"foobar\" in it\n> +\t\techo foobar >expect &&\n> +\t\ttest_cmp expect numerals\n> \t)\n> '\n> \n> diff --git a/t/t7012-skip-worktree-writing.sh b/t/t7012-skip-worktree-writing.sh\n> index a1080b94e3..cb9f1a6981 100755\n> --- a/t/t7012-skip-worktree-writing.sh\n> +++ b/t/t7012-skip-worktree-writing.sh\n> @@ -171,50 +171,20 @@ test_expect_success 'stash restore in sparse checkout' '\n> \n> \t\t# Put a file in the working directory in the way\n> \t\techo in the way >modified &&\n> -\t\tgit stash apply &&\n> +\t\ttest_must_fail git stash apply 2>error&&\n> \n> -\t\t# Ensure stash vivifies modifies paths...\n> -\t\tcat >expect <<-EOF &&\n> -\t\tH addme\n> -\t\tH modified\n> -\t\tH removeme\n> -\t\tH subdir/A\n> -\t\tS untouched\n> -\t\tEOF\n> -\t\tgit ls-files -t >actual &&\n> -\t\ttest_cmp expect actual &&\n> +\t\tgrep \"changes.*would be overwritten by merge\" error &&\n> \n> -\t\t# ...and that the paths show up in status as changed...\n> -\t\tcat >expect <<-EOF &&\n> -\t\tA addme\n> -\t\t M modified\n> -\t\t D removeme\n> -\t\t M subdir/A\n> -\t\t?? actual\n> -\t\t?? expect\n> -\t\t?? modified.stash.XXXXXX\n> -\t\tEOF\n> -\t\tgit status --porcelain | \\\n> -\t\t\tsed -e s/stash......./stash.XXXXXX/ >actual &&\n> -\t\ttest_cmp expect actual &&\n> +\t\techo in the way >expect &&\n> +\t\ttest_cmp expect modified &&\n> +\t\tgit diff --quiet HEAD \":!modified\" &&\n> \n> \t\t# ...and that working directory reflects the files correctly\n> -\t\ttest_path_is_file addme &&\n> +\t\ttest_path_is_missing addme &&\n> \t\ttest_path_is_file modified &&\n> \t\ttest_path_is_missing removeme &&\n> \t\ttest_path_is_file subdir/A &&\n> -\t\ttest_path_is_missing untouched &&\n> -\n> -\t\t# ...including that we have the expected \"modified\" file...\n> -\t\tcat >expect <<-EOF &&\n> -\t\tmodified\n> -\t\ttweaked\n> -\t\tEOF\n> -\t\ttest_cmp expect modified &&\n> -\n> -\t\t# ...and that the other \"modified\" file is still present...\n> -\t\techo in the way >expect &&\n> -\t\ttest_cmp expect modified.stash.*\n> +\t\ttest_path_is_missing untouched\n> \t)\n> '\n> \n> diff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\n> index 590b99bbb6..eb59564565 100755\n> --- a/t/t7817-grep-sparse-checkout.sh\n> +++ b/t/t7817-grep-sparse-checkout.sh\n> @@ -83,10 +83,13 @@ test_expect_success 'setup' '\n> \n> # The test below covers a special case: the sparsity patterns exclude '/b' and\n> # sparse checkout is enabled, but the path exists in the working tree (e.g.\n> -# manually created after `git sparse-checkout init`). git grep should skip it.\n> +# manually created after `git sparse-checkout init`). Although b is marked\n> +# as SKIP_WORKTREE, git grep should notice it IS present in the worktree and\n> +# report it.\n> test_expect_success 'working tree grep honors sparse checkout' '\n> \tcat >expect <<-EOF &&\n> \ta:text\n> +\tb:new-text\n> \tEOF\n> \ttest_when_finished \"rm -f b\" &&\n> \techo \"new-text\" >b &&\n> @@ -126,12 +129,16 @@ test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit'\n> '\n> \n> # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n> -# patterns, so grep should not recurse into it.\n> +# patterns. We also explicitly mark it as SKIP_WORKTREE in case it got cleared\n> +# by previous git commands. Thus sub2 starts as SKIP_WORKTREE but since it is\n> +# present in the working tree, grep should recurse into it.\n> test_expect_success 'grep --recurse-submodules honors sparse checkout in submodule' '\n> \tcat >expect <<-EOF &&\n> \ta:text\n> \tsub/B/b:text\n> +\tsub2/a:text\n> \tEOF\n> +\tgit update-index --skip-worktree sub2 &&\n> \tgit grep --recurse-submodules \"text\" >actual &&\n> \ttest_cmp expect actual\n> '\n\n"},{"id":"445948","messageId":"950da537-941f-dfb4-460a-1aa4c68e0750@github.com","threadId":"57214","inReplyTo":"20220109045732.2497526-6-newren@gmail.com","subject":"Re: [RFC PATCH 5/5] Accelerate ensure_skip_worktree_means_skip_worktree by caching","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-11T18:30:13Z","receivedAt":"2022-01-11T18:30:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> Rather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\n> fact that entire directories will often be missing, especially for cone\n> mode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\n> clear tracked sparse dirs\", 2021-09-08). If we have already determined\n> that the parent directory of a file (or any other previous ancestor)\n> does not exist, then we already know the file cannot exist and do not\n> need to lstat() it separately.\n> \n> Granted, the cost of ensure_skip_worktree_means_skip_worktree() might\n> be considered a bit high for non-cone mode since it might now lstat()\n> every SKIP_WORKTREE path when the index is loaded (an O(N) cost, with\n> N the number of SKIP_WORKTREE paths), but non-cone mode users already\n> have to deal with the O(N*M) cost (with N=the number of tracked files\n> and M=the number of sparsity patterns), so this should be reasonable.\n> \n\nDid you write/run any performance tests to see how this optimization changed\nthe execution time? If not, running the `p2000` performance tests against\nthe patch series base, [3/5], and [5/5] would provide some really helpful\ninsight into the cost of `ensure_skip_worktree_means_skip_worktree`, then\nhow much this optimization improves it.\n\n> Signed-off-by: Elijah Newren \n> ---\n> sparse-index.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 103 insertions(+), 2 deletions(-)\n> \n> diff --git a/sparse-index.c b/sparse-index.c\n> index 79d50e444c..608782e255 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -341,18 +341,117 @@ void ensure_correct_sparsity(struct index_state *istate)\n> \t\tensure_full_index(istate);\n> }\n> \n> +struct path_cache_entry {\n> +\tstruct hashmap_entry ent;\n> +\tconst char *path;\n> +\tint path_length;\n> +\tint is_present;\n> +};\n> +\n> +static int path_cache_cmp(const void *unused,\n> +\t\t\t const struct hashmap_entry *entry1,\n> +\t\t\t const struct hashmap_entry *entry2,\n> +\t\t\t const void *also_unused)\n> +{\n> +\tconst struct path_cache_entry *e1, *e2;\n> +\n> +\te1 = container_of(entry1, const struct path_cache_entry, ent);\n> +\te2 = container_of(entry2, const struct path_cache_entry, ent);\n> +\tif (e1->path_length != e2->path_length)\n> +\t\treturn e1->path_length - e2->path_length;\n> +\treturn memcmp(e1->path, e2->path, e1->path_length);\n> +}\n> +\n> +static struct path_cache_entry *find_path_cache_entry(struct hashmap *map,\n> +\t\t\t\t\t\t const char *str,\n> +\t\t\t\t\t\t int str_length)\n> +{\n> +\tstruct path_cache_entry entry;\n> +\thashmap_entry_init(&entry.ent, memhash(str, str_length));\n> +\tentry.path = str;\n> +\tentry.path_length = str_length;\n> +\treturn hashmap_get_entry(map, &entry, ent, NULL);\n> +}\n> +\n> +static void record(struct hashmap *path_cache,\n> +\t\t struct mem_pool *pool,\n> +\t\t const char *path,\n> +\t\t int path_length,\n> +\t\t int found)\n> +{\n> +\tstruct path_cache_entry *entry;\n> +\n> +\tentry = mem_pool_alloc(pool, sizeof(*entry));\n> +\thashmap_entry_init(&entry->ent, memhash(path, path_length));\n> +\tentry->path = path;\n> +\tentry->path_length = path_length;\n> +\tentry->is_present = found;\n> +\thashmap_add(path_cache, &entry->ent);\n> +}\n> +\n> +static int path_found(struct hashmap *path_cache, struct mem_pool *pool,\n> +\t\t const char *path, int path_length)\n> +{\n> +\tstruct stat st;\n> +\tint found;\n> +\tconst char *dirsep = path + path_length - 1;\n> +\tconst char *tmp;\n> +\n> +\t/* Find directory separator; memrchr is sadly glibc-specific */\n> +\twhile (dirsep > path && *dirsep != '/')\n> +\t\tdirsep--;\n> +\n> +\t/* If parent of path doesn't exist, no point lstat'ing path... */\n> +\tif (dirsep > path) {\n> +\t\tstruct path_cache_entry *entry;\n> +\t\tint new_length, parent_found;\n> +\n> +\t\t/* First, check if path's parent's existence was cached */\n> +\t\tnew_length = dirsep - path;\n> +\t\tentry = find_path_cache_entry(path_cache, path, new_length);\n> +\t\tif (entry)\n> +\t\t\tparent_found = entry->is_present;\n> +\t\telse\n> +\t\t\tparent_found = path_found(path_cache, pool,\n> +\t\t\t\t\t\t path, new_length);\n> +\n> +\t\tif (!parent_found) {\n> +\t\t\t/* path can't exist if parent dir doesn't */\n> +\t\t\trecord(path_cache, pool, path, path_length, 0);\n> +\t\t\treturn 0;\n> +\t\t} /* else parent was found so must check path itself too... */\n> +\t}\n> +\n> +\t/* Okay, parent dir exists, so we have to check original path */\n> +\n> +\t/* Make sure we have a NUL-terminated string to pass to lstat */\n> +\ttmp = path;\n> +\tif (path[path_length])\n> +\t\ttmp = mem_pool_strndup(pool, path, path_length);\n> +\t/* Determine if path exists */\n> +\tfound = !lstat(tmp, &st);\n> +\n> +\trecord(path_cache, pool, path, path_length, found);\n> +\treturn found;\n> +}\n> +\n> void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n> {\n> +\tstruct hashmap path_cache = HASHMAP_INIT(path_cache_cmp, NULL);\n> +\tstruct mem_pool pool;\n> +\n> \tint i;\n> +\n> \tif (!core_apply_sparse_checkout)\n> \t\treturn;\n> \n> +\tmem_pool_init(&pool, 32*1024);\n> restart:\n> \tfor (i = 0; i < istate->cache_nr; i++) {\n> \t\tstruct cache_entry *ce = istate->cache[i];\n> -\t\tstruct stat st;\n> \n> -\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n> +\t\tif (ce_skip_worktree(ce) &&\n> +\t\t path_found(&path_cache, &pool, ce->name, strlen(ce->name))) {\n> \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n> \t\t\t\tensure_full_index(istate);\n> \t\t\t\tgoto restart;\n> @@ -360,6 +459,8 @@ void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n> \t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n> \t\t}\n> \t}\n> +\thashmap_clear(&path_cache);\n> +\tmem_pool_discard(&pool, 0);\n> }\n> \n> \n\n"},{"id":"445955","messageId":"CABPp-BGaK1JBUrEnqynoeNAsrVmMWO_-94OswDai4KQdsjGrTw@mail.gmail.com","threadId":"57214","inReplyTo":"38b69b5e-b461-23cc-155e-26fd58e5f714@github.com","subject":"Re: [RFC PATCH 3/5] repo_read_index: ensure SKIP_WORKTREE means skip worktree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-11T19:27:29Z","receivedAt":"2022-01-11T19:27:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jan 10, 2022 at 12:38 PM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n...\n> > === User facing issues ===\n...\n> > Further:\n> > * these files will not be updated by by standard commands\n> > (switch/checkout/pull/merge/rebase will leave them alone unless\n> > conflicts happen -- and even then, the conflicted file may be\n> > written somewhere else to avoid overwriting the SKIP_WORKTREE file\n> > that is present and in the way)\n> > * there is nothing in Git that users can use to discover such\n> > files (status, diff, grep, etc. all ignore it)\n> > * there is no reasonable mechanism to \"recover\" from such a condition\n> > (neither `git sparse-checkout reapply` nor `git reset --hard` will\n> > correct it).\n> >\n>\n> Just to add to this, files like these always force sparse index expansion in\n> `git status` (and probably some other commands?), ruining a lot of the\n> performance gains of using sparse index in the first place.\n\nOh, good point. Another reason this state is just bad.\n\n...\n> > === Suggested solution ===\n> >\n> > SKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\n> > the name of the option implies, to allow the file to NOT be in the\n> > worktree but consider it to be unchanged rather than deleted.\n> >\n> > The suggests a simple solution: present-despite-SKIP_WORKTREE files\n> > should not exist, for those using sparse-checkouts.\n> >\n> > Enforce this at index loading time by checking if core.sparseCheckout is\n> > true; if so, check files in the index with the SKIP_WORKTREE bit set to\n> > verify that they are absent from the working tree. If they are present,\n> > unset the bit (in memory, though any commands that write to the index\n> > will record the update).\n> >\n>\n> Since this solution is specific to a sparse-checkout, should this automatic\n> unsetting only be done if the file is outside the sparse checkout\n> definition? Otherwise, the `sparse-checkout reapply` cleanup suggested below\n> doesn't return the original `skip-worktree` state.\n\nThat's an interesting distinction, but I think there are multiple\nproblems with it:\n 1) I think you might be assuming this state would only be entered by\na deliberate user choice, but that's far from the case\n 2) Allowing present-despite-SKIP_WORKTREE only for in-cone paths\nstill allows the state to exist, which has multiple ramifications:\n 2A) while it solves user facing problems for out-of-cone\npresent-despite-SKIP_WORKTREE files, it leaves user facing problems\nfor in-cone ones\n 2B) it doesn't solve any of the complexity or need for special\ntesting outlined in the commit message since the bad state is still\npossible for some paths\n 2C) Related to (2B), your series would need to do more work to\nmake checkout-index sane\n 3) It's basically impossible to keep this kind of skip-worktree\nstate in a sparse-checkout anyway.\n\nLet's look at (1), (2A), and (3) in more detail, since those may not be obvious:\n\n=== (1) present-despite-SKIP_WORKTREE on in-cone can be triggered\naccidentally ===\n\nUsers can directly edit $GIT_DIR/info/sparse-checkout. We documented\nhow for years. People wrote tools that did so. `git sparse-checkout`\ncame much later. So, they could get into this state by having a\nsparse-checkout already, and then editing\n$GIT_DIR/info/sparse-checkout such that the files which used to be\nout-of-cone are now in-cone even though the working directory doesn't\nmatch. People could also get into this state without knowing about\nthe $GIT_DIR/info/sparse-checkout file, with an in-opportune Ctrl-C\nduring the middle of a `git sparse-checkout ...` command of some sort.\nI actually think it's more likely that people accidentally get into\nthis state than deliberately. However, for sake of argument, let's\npresume people could only get into this state intentionally. I think\nwe'd still want to make my changes for this class of files because of\nthe other reasons, so let's move on to those...\n\n=== (2A) present-despite-SKIP_WORKTREE on in-cone has user-facing\nproblems too ===\n\nSo, we have a user who has some file(s) that are marked SKIP_WORKTREE\ndespite being in-cone. Let's assume that the in-cone file has some\ndirty change(s). First, note that the in-cone file is already\nSKIP_WORKTREE:\n\n $ git ls-files -t\n S in-cone/foo.c\n S out-of-cone/tracked\n\nNow this user tries to switch to another branch which does differ in\nin-cone/foo.c (or try to do a rebase that involves in-cone/foo.c,\nor...):\n\n $ git checkout other-branch-modifying-in-cone\n error: Your local changes to the following files would be\noverwritten by checkout:\n in-cone/foo.c\n Please commit your changes or stash them before you switch branches.\n Aborting\n\nOh, thinks the user, maybe I should check status before switching branches:\n\n $ git status --porcelain\n $\n\nThe user perhaps finds it odd that the file reported as having \"local\nchanges\" doesn't show up in status, but decides to do what the error\nmessage reported anyway:\n\n $ git stash save foobar\n No local changes to save\n\nIf neither status nor stash think there are any changes, why does\ncheckout? Why can't I switch branches?? If they attempt to switch\nbranches again anyway, they'll get the same local-changes error.\nPerhaps at this point they notice there were two possible solutions in\nthe error message, so they decide to try committing the changes:\n\n $ git commit -am 'changes'\n On branch main\n You are in a sparse checkout with 0% of tracked files present.\n\n nothing to commit, working tree clean\n\nThree commands tell me my working tree is clean, but checkout says\notherwise?!?!? Why can't I switch branches??? Stupid Git! (And if\nsomeone points them at `git add $FILENAME`, then they get another\nerror message and are told to use `git add --sparse $FILENAME`...)\n\nIn contrast, with my patch, none of this craziness happens and a\nlocally modified file is detected and reported as locally modified by\nevery command within a sparse checkout. It's so much saner for users.\n\nBut, for sake of argument, let's ignore all of the reasons so far and\ninstead pretend there are users who deliberately create skip-worktree\nfiles in-cone, and thus expect all these weirdnesses. Let's focus on\nthe experience they might expect, and turn to item (3):\n\n=== (3) present-despite-SKIP_WORKTREE on in-cone files will be\naggressively cleared _anyway_, due to the fundamental design of\nsparse-checkout ===\n\nLet's say I have a simple sparse-checkout with just a couple files:\n\n $ git ls-files -t\n H in-cone/foo.c\n S out-of-cone/tracked\n\nNow, I decide to make in-cone/foo.c skip-worktree, but keep it in the\nworking tree and dirty its contents:\n\n $ git update-index --skip-worktree in-cone/foo.c\n $ echo dirty >in-cone/foo.c\n $ git ls-files -t\n S in-cone/foo.c\n S out-of-cone/tracked\n\nNow, status won't report our dirty modifications, commit won't include\nthem, etc. Perhaps that's what the user wanted by marking it as\nskip-worktree. But as soon as they invoke any command that calls\nunpack_trees() in a way that might update the working copy (but which\nwouldn't need to touch the dirty files marked skip-worktree), the\nskip-worktree status is going to be dropped:\n\n $ git checkout -q HEAD^0\n warning: The following paths were already present and thus not\nupdated despite sparse patterns:\n in-cone/foo.c\n\n After fixing the above paths, you may want to run `git\nsparse-checkout reapply`.\n Switched to branch 'other-mod'\n $ git ls-files -t\n H in-cone/foo.c\n S out-of-cone/tracked\n\nThis is not special to detaching HEAD; I could have switched to\nanother branch and seen the same thing. Or rebased or cherry-picked\nor merge commits that didn't modify the skip-worktree file, and they'd\nall unset the skip-worktree bit for this file. This isn't a bug,\neither; it's by design. sparsity patterns have to be reapplied when\nswitching branches in general (or rebasing or...), because that\noperation might bring in new files and so we need to know whether\nthose new files should be SKIP_WORKTREE. You might say we could\nattempt to limit the SKIP_WORKTREE bit flipping to \"just\" new files,\nbut due to the possibility of conflicts in earlier merges/rebases\ncausing files to lose the SKIP_WORKTREE bit, we wanted future\noperations that were busy updating the bits anyway to update those\nfiles and clear the bit (and this pre-dated the work Stolee and I did\non sparse-checkouts, btw).\n\nSo, this means that checkout, merge, rebase, cherry-pick, etc. are\ngoing to be clearing SKIP_WORKTREE bits for in-cone files all the\ntime.\n\nSo, even if users did try to deliberately get into such a state,\nthere's no point attempting to preserve the bit since so many commands\nare going to be aggressively clearing it anyway.\n\n> Admittedly, I imagine it's unlikely that someone is simultaneously using a\n> sparse checkout and manually SKIP_WORKTREE-ing files *inside* the sparse\n> checkout definition. But, given that you're not unsetting the flag for\n> non-sparse-checkout SKIP_WORKTREE files, it seems like an additional\n> constraint based on sparse checkout patterns would be consistent with\n> other parts of this patch.\n>\n> > Users can, of course, can get the SKIP_WORKTREE bit back such as by\n> > running `git sparse-checkout reapply` (if they have ensured the file is\n> > unmodified and doesn't match the specified sparsity patterns).\n> >\n>\n> There are some performance implications of this solution in a sparse\n> index-enabled checkout. Any time an out-of-cone file is no longer\n> SKIP_WORKTREE, its parent directory lineage will be added to the sparse index,\n> and performance would progressively (silently) degrade as more out-of-cone\n> files were added.\n\nCan I restate this a bit?\n\n\"\"\"\nThe point of sparse-index is to allow operations to be faster due to\nneeding to track fewer items in the index. Its performance is nearly\nlinearly correlated with the number of paths it has to track, so fewer\npaths is better. If a user needs to work with more paths than they\npreviously did, then their sparse-index will be correspondingly\nslower.\n\nAnd users who start modifying files, are obviously working with more\npaths than they previously did.\n\"\"\"\n\nBased on that restating, I think the only thing that could be\nconsidered a problem is your implication that the performance\ndegradation would silently persist. If users don't undo or commit\nthose local modifications, then they really are working with more\nfiles and I think it's fine for the sparse-index to remain expanded to\ninclude those files. In contrast, if users undo those local changes\n(or commit them so they are no longer dirty), then future\ncheckout/rebase/merge/etc. operations will automatically (re-)set the\nSKIP_WORKTREE bit on those files, and the sparse-index can shrink\nagain. These dirty files would be very similar to files that have\nconflicts during a merge: both have their SKIP_WORKTREE bit cleared\ndespite being outside the sparsity paths. Continuing the comparison,\nafter a user resolves a conflict and commits the changes, the\npreviously conflicted file will automatically have its SKIP_WORKTREE\nbit set again by a future checkout/rebase/merge/etc. Or users can\nmanually get it to be set earlier with a `git sparse-checkout\nreapply`. So the performance degradation from the additional paths\nthe user is working with is only temporary, and only lasts roughly as\nlong as the user is still working with these additional paths. To me,\nthis feels like expected and wanted behavior.\n\n> That said, a lot of my concern would be alleviated with some kind of warning\n> indicating that a file just had SKIP_WORKTREE removed, including a mention\n> of fixing it with `git sparse-checkout reapply`.\n\nThat might be interesting, but there's two problems here:\n * It'd affect more types of files and situations than intended. In\nparticular, given [11], it could be quite noisy due to triggering on\nfiles other than those manually tweaked by users and even include\nfiles never involved in conflicts either.\n * It'd repeatedly trigger on the exact same files, possibly muddying\nother output unexpectedly. Given that we clear the bit in memory but\nnot necessarily on disk, users would repeatedly see the same warning\nfor the same files during sequences of read-only operations (like `git\ndiff` or `git log`) and they warnings would only go away once the user\nran a command that re-wrote the index (such as `git status`).\n\nI think the point of your suggestion was to help users recover from\npotentially persisting performance degradation; is that fair? If so,\nthen as I highlighted above, I don't think we have any such persisting\nproblem.\n\n[11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n\n> It would be *extra* nice if `git status` could tell a user that they have\n> non-SKIP_WORKTREE files outside the sparse definition, but I think that's\n> less critical and probably better suited as a separate series.\n\nOh, interesting idea. But I agree, it'd be better suited to a separate series.\n\n> > diff --git a/sparse-index.c b/sparse-index.c\n> > index a1d505d50e..79d50e444c 100644\n> > --- a/sparse-index.c\n> > +++ b/sparse-index.c\n> > @@ -341,6 +341,28 @@ void ensure_correct_sparsity(struct index_state *istate)\n> > ensure_full_index(istate);\n> > }\n> >\n> > +void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n>\n> I can feel the frustration behind this name. :)\n\n:-)\n\n> However, a more descriptive one would make the code easier to follow, e.g.\n> 'clear_skip_worktree_from_present_files' (or something else indicating what\n> it does to the index).\n\nYeah, I did spend some time trying to come up with a better name (I\nrealized it wasn't the best), but failed. Your suggestion seems\nobvious in hindsight. I like it; I'll make that change.\n"},{"id":"445974","messageId":"CABPp-BHEuLB9u1ggDUsTZ_85S52XScM4znM7dnrQQ1iKnXxmhw@mail.gmail.com","threadId":"57214","inReplyTo":"950da537-941f-dfb4-460a-1aa4c68e0750@github.com","subject":"Re: [RFC PATCH 5/5] Accelerate ensure_skip_worktree_means_skip_worktree by caching","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-11T22:04:45Z","receivedAt":"2022-01-11T22:05:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jan 11, 2022 at 10:30 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > Rather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\n> > fact that entire directories will often be missing, especially for cone\n> > mode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\n> > clear tracked sparse dirs\", 2021-09-08). If we have already determined\n> > that the parent directory of a file (or any other previous ancestor)\n> > does not exist, then we already know the file cannot exist and do not\n> > need to lstat() it separately.\n> >\n> > Granted, the cost of ensure_skip_worktree_means_skip_worktree() might\n> > be considered a bit high for non-cone mode since it might now lstat()\n> > every SKIP_WORKTREE path when the index is loaded (an O(N) cost, with\n> > N the number of SKIP_WORKTREE paths), but non-cone mode users already\n> > have to deal with the O(N*M) cost (with N=the number of tracked files\n> > and M=the number of sparsity patterns), so this should be reasonable.\n> >\n>\n> Did you write/run any performance tests to see how this optimization changed\n> the execution time? If not, running the `p2000` performance tests against\n> the patch series base, [3/5], and [5/5] would provide some really helpful\n> insight into the cost of `ensure_skip_worktree_means_skip_worktree`, then\n> how much this optimization improves it.\n\nI haven't[1]. You bring up a very good point; I'll add it for the next round.\n\n[1] Long, probably irrelevant story about why: My original patches\nwere actually going to go further and just remove the\npresent-despite-SKIP_WORKTREE files in _all_ cases, sparse-checkout or\nnot. It had not occurred to me while writing the patches to make it\nspecific to sparse-checkouts. Because of that, I figured it was\nbetter to get feedback on if the idea was acceptable and spent a lot\nmore time concentrating on making the case. Then I realized near the\nend that folks who don't use sparse-checkout or SKIP_WORKTREE might be\nannoyed at the overhead also being added for them, for a feature\nthey'll never even use. I decided to back off a bit, and make it\nsparse-checkout specific. Then I realized that backing off might just\nkeep all users happy anyway (the folks who intentionally use\npresent-despite-SKIP_WORKTREE paths, despite their many warts, can\nkeep doings so) and edited a lot of my commit messages, docs, and\ncover letter. And by then it was late Saturday night and I had\npromised to send out my series on Friday. Since I had already marked\nmy cover letter as RFC anyway, I just decided to temporarily punt on\ngetting performance numbers...\n"},{"id":"445979","messageId":"58ce4feb-cd02-d6a4-93cd-df0aff083db2@github.com","threadId":"57214","inReplyTo":"CABPp-BGaK1JBUrEnqynoeNAsrVmMWO_-94OswDai4KQdsjGrTw@mail.gmail.com","subject":"Re: [RFC PATCH 3/5] repo_read_index: ensure SKIP_WORKTREE means skip worktree","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-11T23:09:54Z","receivedAt":"2022-01-11T23:09:58Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Mon, Jan 10, 2022 at 12:38 PM Victoria Dye wrote:\n>>\n>> Elijah Newren wrote:\n> ...\n>>> === User facing issues ===\n> ...\n>>> Further:\n>>> * these files will not be updated by by standard commands\n>>> (switch/checkout/pull/merge/rebase will leave them alone unless\n>>> conflicts happen -- and even then, the conflicted file may be\n>>> written somewhere else to avoid overwriting the SKIP_WORKTREE file\n>>> that is present and in the way)\n>>> * there is nothing in Git that users can use to discover such\n>>> files (status, diff, grep, etc. all ignore it)\n>>> * there is no reasonable mechanism to \"recover\" from such a condition\n>>> (neither `git sparse-checkout reapply` nor `git reset --hard` will\n>>> correct it).\n>>>\n>>\n>> Just to add to this, files like these always force sparse index expansion in\n>> `git status` (and probably some other commands?), ruining a lot of the\n>> performance gains of using sparse index in the first place.\n> \n> Oh, good point. Another reason this state is just bad.\n> \n> ...\n>>> === Suggested solution ===\n>>>\n>>> SKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\n>>> the name of the option implies, to allow the file to NOT be in the\n>>> worktree but consider it to be unchanged rather than deleted.\n>>>\n>>> The suggests a simple solution: present-despite-SKIP_WORKTREE files\n>>> should not exist, for those using sparse-checkouts.\n>>>\n>>> Enforce this at index loading time by checking if core.sparseCheckout is\n>>> true; if so, check files in the index with the SKIP_WORKTREE bit set to\n>>> verify that they are absent from the working tree. If they are present,\n>>> unset the bit (in memory, though any commands that write to the index\n>>> will record the update).\n>>>\n>>\n>> Since this solution is specific to a sparse-checkout, should this automatic\n>> unsetting only be done if the file is outside the sparse checkout\n>> definition? Otherwise, the `sparse-checkout reapply` cleanup suggested below\n>> doesn't return the original `skip-worktree` state.\n> \n> That's an interesting distinction, but I think there are multiple\n> problems with it:\n> 1) I think you might be assuming this state would only be entered by\n> a deliberate user choice, but that's far from the case\n> 2) Allowing present-despite-SKIP_WORKTREE only for in-cone paths\n> still allows the state to exist, which has multiple ramifications:\n> 2A) while it solves user facing problems for out-of-cone\n> present-despite-SKIP_WORKTREE files, it leaves user facing problems\n> for in-cone ones\n> 2B) it doesn't solve any of the complexity or need for special\n> testing outlined in the commit message since the bad state is still\n> possible for some paths\n> 2C) Related to (2B), your series would need to do more work to\n> make checkout-index sane\n> 3) It's basically impossible to keep this kind of skip-worktree\n> state in a sparse-checkout anyway.\n> \n> Let's look at (1), (2A), and (3) in more detail, since those may not be obvious:\n> \n> === (1) present-despite-SKIP_WORKTREE on in-cone can be triggered\n> accidentally ===\n> \n> Users can directly edit $GIT_DIR/info/sparse-checkout. We documented\n> how for years. People wrote tools that did so. `git sparse-checkout`\n> came much later. So, they could get into this state by having a\n> sparse-checkout already, and then editing\n> $GIT_DIR/info/sparse-checkout such that the files which used to be\n> out-of-cone are now in-cone even though the working directory doesn't\n> match. People could also get into this state without knowing about\n> the $GIT_DIR/info/sparse-checkout file, with an in-opportune Ctrl-C\n> during the middle of a `git sparse-checkout ...` command of some sort.\n> I actually think it's more likely that people accidentally get into\n> this state than deliberately. However, for sake of argument, let's\n> presume people could only get into this state intentionally. I think\n> we'd still want to make my changes for this class of files because of\n> the other reasons, so let's move on to those...\n> \n> === (2A) present-despite-SKIP_WORKTREE on in-cone has user-facing\n> problems too ===\n> \n> So, we have a user who has some file(s) that are marked SKIP_WORKTREE\n> despite being in-cone. Let's assume that the in-cone file has some\n> dirty change(s). First, note that the in-cone file is already\n> SKIP_WORKTREE:\n> \n> $ git ls-files -t\n> S in-cone/foo.c\n> S out-of-cone/tracked\n> \n> Now this user tries to switch to another branch which does differ in\n> in-cone/foo.c (or try to do a rebase that involves in-cone/foo.c,\n> or...):\n> \n> $ git checkout other-branch-modifying-in-cone\n> error: Your local changes to the following files would be\n> overwritten by checkout:\n> in-cone/foo.c\n> Please commit your changes or stash them before you switch branches.\n> Aborting\n> \n> Oh, thinks the user, maybe I should check status before switching branches:\n> \n> $ git status --porcelain\n> $\n> \n> The user perhaps finds it odd that the file reported as having \"local\n> changes\" doesn't show up in status, but decides to do what the error\n> message reported anyway:\n> \n> $ git stash save foobar\n> No local changes to save\n> \n> If neither status nor stash think there are any changes, why does\n> checkout? Why can't I switch branches?? If they attempt to switch\n> branches again anyway, they'll get the same local-changes error.\n> Perhaps at this point they notice there were two possible solutions in\n> the error message, so they decide to try committing the changes:\n> \n> $ git commit -am 'changes'\n> On branch main\n> You are in a sparse checkout with 0% of tracked files present.\n> \n> nothing to commit, working tree clean\n> \n> Three commands tell me my working tree is clean, but checkout says\n> otherwise?!?!? Why can't I switch branches??? Stupid Git! (And if\n> someone points them at `git add $FILENAME`, then they get another\n> error message and are told to use `git add --sparse $FILENAME`...)\n> \n> In contrast, with my patch, none of this craziness happens and a\n> locally modified file is detected and reported as locally modified by\n> every command within a sparse checkout. It's so much saner for users.\n> \n\nThis particular issue also happens in non-sparse-checkout usage of\nSKIP_WORKTREE, but point (1) (+ a general desire to make sparse-checkout\nless \"experimental\") is a good reason to ensure the UX in a sparse-checkout\nisn't this frustrating.\n\n> But, for sake of argument, let's ignore all of the reasons so far and\n> instead pretend there are users who deliberately create skip-worktree\n> files in-cone, and thus expect all these weirdnesses. Let's focus on\n> the experience they might expect, and turn to item (3):\n> \n> === (3) present-despite-SKIP_WORKTREE on in-cone files will be\n> aggressively cleared _anyway_, due to the fundamental design of\n> sparse-checkout ===\n> \n> Let's say I have a simple sparse-checkout with just a couple files:\n> \n> $ git ls-files -t\n> H in-cone/foo.c\n> S out-of-cone/tracked\n> \n> Now, I decide to make in-cone/foo.c skip-worktree, but keep it in the\n> working tree and dirty its contents:\n> \n> $ git update-index --skip-worktree in-cone/foo.c\n> $ echo dirty >in-cone/foo.c\n> $ git ls-files -t\n> S in-cone/foo.c\n> S out-of-cone/tracked\n> \n> Now, status won't report our dirty modifications, commit won't include\n> them, etc. Perhaps that's what the user wanted by marking it as\n> skip-worktree. But as soon as they invoke any command that calls\n> unpack_trees() in a way that might update the working copy (but which\n> wouldn't need to touch the dirty files marked skip-worktree), the\n> skip-worktree status is going to be dropped:\n> \n> $ git checkout -q HEAD^0\n> warning: The following paths were already present and thus not\n> updated despite sparse patterns:\n> in-cone/foo.c\n> \n> After fixing the above paths, you may want to run `git\n> sparse-checkout reapply`.\n> Switched to branch 'other-mod'\n> $ git ls-files -t\n> H in-cone/foo.c\n> S out-of-cone/tracked\n> \n> This is not special to detaching HEAD; I could have switched to\n> another branch and seen the same thing. Or rebased or cherry-picked\n> or merge commits that didn't modify the skip-worktree file, and they'd\n> all unset the skip-worktree bit for this file. This isn't a bug,\n> either; it's by design. sparsity patterns have to be reapplied when\n> switching branches in general (or rebasing or...), because that\n> operation might bring in new files and so we need to know whether\n> those new files should be SKIP_WORKTREE. You might say we could\n> attempt to limit the SKIP_WORKTREE bit flipping to \"just\" new files,\n> but due to the possibility of conflicts in earlier merges/rebases\n> causing files to lose the SKIP_WORKTREE bit, we wanted future\n> operations that were busy updating the bits anyway to update those\n> files and clear the bit (and this pre-dated the work Stolee and I did\n> on sparse-checkouts, btw).\n> \n> So, this means that checkout, merge, rebase, cherry-pick, etc. are\n> going to be clearing SKIP_WORKTREE bits for in-cone files all the\n> time.\n> \n> So, even if users did try to deliberately get into such a state,\n> there's no point attempting to preserve the bit since so many commands\n> are going to be aggressively clearing it anyway.\n> \n\nAhh, I should have remembered that - it's also really helpful context for\nunderstanding why you can/should aggressively unset SKIP_WORKTREE (because\ngit will aggressively re-set it if a file is up-to-date).\n\n>> Admittedly, I imagine it's unlikely that someone is simultaneously using a\n>> sparse checkout and manually SKIP_WORKTREE-ing files *inside* the sparse\n>> checkout definition. But, given that you're not unsetting the flag for\n>> non-sparse-checkout SKIP_WORKTREE files, it seems like an additional\n>> constraint based on sparse checkout patterns would be consistent with\n>> other parts of this patch.\n>>\n>>> Users can, of course, can get the SKIP_WORKTREE bit back such as by\n>>> running `git sparse-checkout reapply` (if they have ensured the file is\n>>> unmodified and doesn't match the specified sparsity patterns).\n>>>\n>>\n>> There are some performance implications of this solution in a sparse\n>> index-enabled checkout. Any time an out-of-cone file is no longer\n>> SKIP_WORKTREE, its parent directory lineage will be added to the sparse index,\n>> and performance would progressively (silently) degrade as more out-of-cone\n>> files were added.\n> \n> Can I restate this a bit?\n> \n> \"\"\"\n> The point of sparse-index is to allow operations to be faster due to\n> needing to track fewer items in the index. Its performance is nearly\n> linearly correlated with the number of paths it has to track, so fewer\n> paths is better. If a user needs to work with more paths than they\n> previously did, then their sparse-index will be correspondingly\n> slower.\n> \n> And users who start modifying files, are obviously working with more\n> paths than they previously did.\n> \"\"\"\n> \n> Based on that restating, I think the only thing that could be\n> considered a problem is your implication that the performance\n> degradation would silently persist. If users don't undo or commit\n> those local modifications, then they really are working with more\n> files and I think it's fine for the sparse-index to remain expanded to\n> include those files. In contrast, if users undo those local changes\n> (or commit them so they are no longer dirty), then future\n> checkout/rebase/merge/etc. operations will automatically (re-)set the\n> SKIP_WORKTREE bit on those files, and the sparse-index can shrink\n> again. These dirty files would be very similar to files that have\n> conflicts during a merge: both have their SKIP_WORKTREE bit cleared\n> despite being outside the sparsity paths. Continuing the comparison,\n> after a user resolves a conflict and commits the changes, the\n> previously conflicted file will automatically have its SKIP_WORKTREE\n> bit set again by a future checkout/rebase/merge/etc. Or users can\n> manually get it to be set earlier with a `git sparse-checkout\n> reapply`. So the performance degradation from the additional paths\n> the user is working with is only temporary, and only lasts roughly as\n> long as the user is still working with these additional paths. To me,\n> this feels like expected and wanted behavior.\n> \n>> That said, a lot of my concern would be alleviated with some kind of warning\n>> indicating that a file just had SKIP_WORKTREE removed, including a mention\n>> of fixing it with `git sparse-checkout reapply`.\n> \n> That might be interesting, but there's two problems here:\n> * It'd affect more types of files and situations than intended. In\n> particular, given [11], it could be quite noisy due to triggering on\n> files other than those manually tweaked by users and even include\n> files never involved in conflicts either.\n> * It'd repeatedly trigger on the exact same files, possibly muddying\n> other output unexpectedly. Given that we clear the bit in memory but\n> not necessarily on disk, users would repeatedly see the same warning\n> for the same files during sequences of read-only operations (like `git\n> diff` or `git log`) and they warnings would only go away once the user\n> ran a command that re-wrote the index (such as `git status`).\n> \n> I think the point of your suggestion was to help users recover from\n> potentially persisting performance degradation; is that fair? If so,\n> then as I highlighted above, I don't think we have any such persisting\n> problem.\n> \n\nYes, the performance degradation was my main concern, but I agree that it\nwouldn't be a persistent problem based on your explanation. Thanks for the\nclarification!\n\n> [11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n> \n>> It would be *extra* nice if `git status` could tell a user that they have\n>> non-SKIP_WORKTREE files outside the sparse definition, but I think that's\n>> less critical and probably better suited as a separate series.\n> \n> Oh, interesting idea. But I agree, it'd be better suited to a separate series.\n> \n>>> diff --git a/sparse-index.c b/sparse-index.c\n>>> index a1d505d50e..79d50e444c 100644\n>>> --- a/sparse-index.c\n>>> +++ b/sparse-index.c\n>>> @@ -341,6 +341,28 @@ void ensure_correct_sparsity(struct index_state *istate)\n>>> ensure_full_index(istate);\n>>> }\n>>>\n>>> +void ensure_skip_worktree_means_skip_worktree(struct index_state *istate)\n>>\n>> I can feel the frustration behind this name. :)\n> \n> :-)\n> \n>> However, a more descriptive one would make the code easier to follow, e.g.\n>> 'clear_skip_worktree_from_present_files' (or something else indicating what\n>> it does to the index).\n> \n> Yeah, I did spend some time trying to come up with a better name (I\n> realized it wasn't the best), but failed. Your suggestion seems\n> obvious in hindsight. I like it; I'll make that change.\n\n"}]}