{"thread":{"id":"57236","subject":"[PATCH 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts)","startedAt":"2022-01-13T16:43:54Z","lastAt":"2022-02-22T23:17:07Z","messageCount":30,"participants":["Elijah Newren via GitGitGadget","Elijah Newren","Victoria Dye","Ævar Arnfjörð Bjarmason","Jonathan Nieder","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"446160","messageId":"pull.1114.git.1642092230.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":null,"subject":"[PATCH 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts)","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:45Z","receivedAt":"2022-01-13T16:43:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"(Maintainer note: This series builds on (v2 of) vd/sparse-clean-etc, because\nit tweaks one of the testcases added there.)\n\n(Note 2: There was a previous RFC round of this series at\nhttps://lore.kernel.org/git/20220109045732.2497526-1-newren@gmail.com/.)\n\nFiles in the present-despite-SKIP_WORKTREE state have caused no ends of\ndiscussions and bugs[1,2,3,4,5,6,...and lots of others]. Trying to address\nthe big issue of discovering & recovering from this state has befuddled me\nfor over a year because I was worried we'd need additional code at every\nskip_worktree-checking path in the code (and they are all over the place),\nand that we'd make the code significantly slower unless we plumbed a bunch\nof additional information all over the place to allow some reasonable\noptimizations.\n\nThis series tries to solve the problem a bit differently by automatic early\ndiscovery and recovery; as it result, it greatly simplifies the landscape,\nreduces our testing matrix burden, and fixes a large swath of bugs. And I\nfigured out how to get the perf cost down to essentially negligible.\n\nChanges since RFC version:\n\n * updated the commit messages as per suggestions from Victoria, including\n   adding performance measurements\n * renamed the new function to use a clearer name\n * replaced the final patch with a different optimization, which is both\n   simpler and performs quite a bit better (the cost for my previous patch 5\n   was already decent in many cases, but had a few cases where the cost was\n   significant).\n\nQuick overview:\n\n * Patches 1 & 2 add a test to demonstrate accidental deletion of\n   possibly-modified files, and then fix the bug.\n * Patch 3 is the crux of this series; a small amount of code with a huge\n   commit message\n * Patch 4 updates the documentation\n * Patch 5 adds an optimization to reduce the performance impact of patch 3\n\n[1]\nhttps://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n[2]\nhttps://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[3]\nhttps://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[4] commit 66b209b (\"merge-ort: implement CE_SKIP_WORKTREE handling with\nconflicted entries\", 2021-03-20) [5] commit ba359fd (\"stash: fix stash\napplication in sparse-checkouts\", 2020-12-01) [6]\nhttps://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n\nElijah Newren (5):\n  t1011: add testcase demonstrating accidental loss of user\n    modifications\n  unpack-trees: fix accidental loss of user changes\n  repo_read_index: clear SKIP_WORKTREE bit from files present in\n    worktree\n  Update documentation related to sparsity and the skip-worktree bit\n  Accelerate clear_skip_worktree_from_present_files() by caching\n\n Documentation/git-read-tree.txt          | 12 +++-\n Documentation/git-sparse-checkout.txt    | 76 ++++++++++++++----------\n Documentation/git-update-index.txt       | 57 +++++++++++++-----\n repository.c                             |  7 +++\n sparse-index.c                           | 73 +++++++++++++++++++++++\n sparse-index.h                           |  1 +\n t/t1011-read-tree-sparse-checkout.sh     | 23 ++++++-\n t/t1092-sparse-checkout-compatibility.sh | 16 ++---\n t/t3705-add-sparse-checkout.sh           |  2 +\n t/t6428-merge-conflicts-sparse.sh        | 23 ++-----\n t/t7012-skip-worktree-writing.sh         | 44 +++-----------\n t/t7817-grep-sparse-checkout.sh          | 11 +++-\n unpack-trees.c                           |  4 +-\n 13 files changed, 235 insertions(+), 114 deletions(-)\n\n\nbase-commit: 80697e9259e4a2c12fd0546f5f1895fb3068a66c\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1114%2Fnewren%2Ffix-present-despite-skip-worktree-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1114/newren/fix-present-despite-skip-worktree-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1114\n-- \ngitgitgadget\n"},{"id":"446161","messageId":"c553d558c2f154efbed9c14fd5abce52bf5ab247.1642092230.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH 1/5] t1011: add testcase demonstrating accidental loss of user modifications","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:46Z","receivedAt":"2022-01-13T16:43:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nIf a user has a file with local modifications that is not marked as\nSKIP_WORKTREE, but the sparsity patterns are such that it should be\nmarked that way, and the user then invokes a command like\n\n   * git checkout -q HEAD^\n\nor\n\n   * git read-tree -mu HEAD^\n\nThen the file will be deleted along with all the users' modifications.\nAdd a testcase demonstrating this problem.\n\nNote: This bug only triggers if something other than 'HEAD' is given;\nif the commands above had specified 'HEAD', then the users' file would\nbe left alone.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n t/t1011-read-tree-sparse-checkout.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 24092c09a95..1b2395b8a82 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -187,6 +187,27 @@ test_expect_success 'read-tree updates worktree, absent case' '\n \ttest ! -f init.t\n '\n \n+test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n+test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\techo sub/added >.git/info/sparse-checkout &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n-- \ngitgitgadget\n\n"},{"id":"446162","messageId":"1e3958576e2a132180f8c5d20acc1315d3062119.1642092230.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH 2/5] unpack-trees: fix accidental loss of user changes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:47Z","receivedAt":"2022-01-13T16:43:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nFor sparse-checkouts, we don't want unpack-trees to error out on files\nthat are missing from the worktree, so there has traditionally been\nlogic to make it skip the verify_uptodate() check for these.\nUnfortunately, it was skipping the verify_uptodate() check for files\nthat were expected to *become* SKIP_WORKTREE.  For files that were not\nalready SKIP_WORKTREE, that can cause us to later delete the file in\napply_sparse_checkout().  Only skip the check for files that were\nalready SKIP_WORKTREE as well to avoid lightly discarding important\nchanges users may have made to files.\n\nNote 1: unpack-trees.c is already a bit complex, and the logic around\nCE_SKIP_WORKTREE and CE_NEW_SKIP_WORKTREE in that file are no exception.\nI also tried just replacing CE_NEW_SKIP_WORKTREE with CE_SKIP_WORKTREE\nin the verify_uptodate() check instead of checking for both flags, and\nfound that it also fixed this bug and passed all the tests.  I also\nattempted to devise a few testcases that might trip either variant of my\nfix and was unable to find any problems.  It may be that just checking\nCE_SKIP_WORKTREE is a better fix, but I'm not sure.  I thought it\nwas a bit safer to strictly reduce the number of cases where we skip the\nup-to-date check rather than just toggling which kind of cases skip it,\nand thus went with the current variant of the fix.\n\nNote 2: I also wondered if verify_absent() might have a similar bug, but\ndespite my attempts to try to devise a testcase that would trigger such\na thing, I couldn't find any problematic testcases.  Thus, this patch\nmakes no attempt to apply similar changes to verify_absent() and\nverify_absent_if_directory().\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n t/t1011-read-tree-sparse-checkout.sh | 2 +-\n unpack-trees.c                       | 4 +++-\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 1b2395b8a82..4ed0885bf2f 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -197,7 +197,7 @@ test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n \tgrep -q dirty init.t\n '\n \n-test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+test_expect_success 'read-tree will not throw away dirty changes, sparse' '\n \techo \"/*\" >.git/info/sparse-checkout &&\n \tread_tree_u_must_succeed -m -u HEAD &&\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 98e2f2e0e6f..6d9d89c662a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2059,7 +2059,9 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n int verify_uptodate(const struct cache_entry *ce,\n \t\t    struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\tif (!o->skip_sparse_checkout &&\n+\t    (ce->ce_flags & CE_SKIP_WORKTREE) &&\n+\t    (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n }\n-- \ngitgitgadget\n\n"},{"id":"446163","messageId":"b263cc75b7d4f426fb051d2cae5ae0fabeebb9c9.1642092230.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:48Z","receivedAt":"2022-01-13T16:43:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nThe fix is short (~30 lines), but the description is not.  Sorry.\n\nThere is a set of problems caused by files in what I'll refer to as the\n\"present-despite-SKIP_WORKTREE\" state.  This commit aims to not just fix\nthese problems, but remove the entire class as a possibility -- for\nthose using sparse checkouts.  But first, we need to understand the\nproblems this class presents.  A quick outline:\n\n   * Problems\n     * User facing issues\n     * Problem space complexity\n     * Maintenance and code correctness challenges\n   * SKIP_WORKTREE expectations in Git\n   * Suggested solution\n   * Pros/Cons of suggested solution\n   * Notes on testcase modifications\n\n=== User facing issues ===\n\nThere are various ways for users to get files to be present in the\nworking copy despite having the SKIP_WORKTREE bit set for that file in\nthe index.  This may come from:\n  * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n  * users grabbing files from elsewhere and writing them to the worktree\n    (perhaps even cached in their editor)\n  * users attempting to \"abort\" a sparse-checkout operation with a\n    not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n    working tree is not atomic)[3].\n\nOnce users have present-despite-SKIP_WORKTREE files, any modifications\nusers make to these files will be ignored, possibly to users' confusion.\n\nFurther:\n  * these files will degrade performance for the sparse-index case due\n    to requiring the index to be expanded (see commit 55dfcf9591\n    (\"sparse-checkout: clear tracked sparse dirs\", 2021-09-08) for why\n    we try to delete entire directories outside the sparse cone).\n  * these files will not be updated by by standard commands\n    (switch/checkout/pull/merge/rebase will leave them alone unless\n    conflicts happen -- and even then, the conflicted file may be\n    written somewhere else to avoid overwriting the SKIP_WORKTREE file\n    that is present and in the way)\n  * there is nothing in Git that users can use to discover such\n    files (status, diff, grep, etc. all ignore it)\n  * there is no reasonable mechanism to \"recover\" from such a condition\n    (neither `git sparse-checkout reapply` nor `git reset --hard` will\n    correct it).\n\nSo, not only are users modifications ignored, but the files get\nprogressively more stale over time.  At some point in the future, they\nmay change their sparseness specification or disable sparse-checkouts.\nAt that time, all present-despite-SKIP_WORKTREE files will show up as\nhaving lots of modifications because they represent a version from a\ndifferent branch or commit.  These might include user-made local changes\nfrom days before, but the only way to tell is to have users look through\nthem all closely.\n\nIf these users come to others for help, there will be no logs that\nexplain the issue; it's just a mysterious list of changes.  Users might\nadamantly claim (correctly, as it turns out) that they didn't modify\nthese files, while others presume they did.\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n[2] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n\n=== Problem space complexity ===\n\nSKIP_WORKTREE has been part of Git for over a decade.  Duy did lots of\nwork on it initially, and several others have since come along and put\nlots of work into it.  Stolee spent most of 2021 on the sparse-index,\nwith lots of bugfixes along the way including to non-sparse-index cases\nas we are still trying to get sparse checkouts to behave reasonably.\nBasically every codepath throughout the treat needs to be aware of an\nadditional type of file: tracked-but-not-present.  The extra type\nresults in lots of extra testcases and lots of extra code everywhere.\n\nBut, the sad thing is that we actually have more than one extra type.\nWe have tracked, tracked-but-not-present (SKIP_WORKTREE), and\ntracked-but-promised-to-not-be-present-but-is-present-anyway\n(present-despite-SKIP_WORKTREE).  Two types is a monumental amount of\neffort to support, and adding a third feels a bit like insanity[4].\n\n[4] Some examples of which can be seen at\n    https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n\n=== Maintenance and code correctness challenges ===\n\nMatheus' patches to grep stalled for nearly a year, in part because of\ncomplications of how to handle sparse-checkouts appropriately in all\ncases[5][6] (with trying to sanely figure out how to sanely handle\npresent-despite-SKIP_WORKTREE files being one of the complications).\nHis rm/add follow-ups also took months because of those kinds of\nissues[7].  The corner cases with things like submodules and\nSKIP_WORKTREE with the addition of present-despite-SKIP_WORKTREE start\nbecoming really complex[8].\n\nWe've had to add ugly logic to merge-ort to attempt to handle\npresent-despite-SKIP_WORKTREE files[9], and basically just been forced\nto give up in merge-recursive knowing full well that we'll sometimes\nsilently discard user modifications.  Despite stash essentially being a\nmerge, it needed extra code (beyond what was in merge-ort and\nmerge-recursive) to manually tweak SKIP_WORKTREE bits in order to avoid\na few different bugs that'd result in an early abort with a partial\nstash application[10].\n\n[5] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t\n    and the dates on the thread; also Matheus and I had several\n    conversations off-list trying to resolve the issues over that time\n[6] ...it finally kind of got unstuck after\n    https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[7] See for example\n    https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t\n    and quotes like \"The core functionality of sparse-checkout has always\n    been only partially implemented\", a statement I still believe is true\n    today.\n[8] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[9] See commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE\n    handling with conflicted entries\", 2021-03-20)\n[10] See commit ba359fd507 (\"stash: fix stash application in\n     sparse-checkouts\", 2020-12-01)\n\n=== SKIP_WORKTREE expectations in Git ===\n\nA couple quotes:\n\nFrom [11] (before the \"sparse-checkout\" command existed):\n   If it needs too many special cases, hacks, and conditionals, then it\n   is not worth the complexity---if it is easier to write a correct code\n   by allowing Git to populate working tree files, it is perfectly fine\n   to do so.\n\n   In a sense, the sparse checkout \"feature\" itself is a hack by itself,\n   and that is why I think this part should be \"best effort\" as well.\n\nFrom the git-sparse-checkout manual (still present today):\n\n   THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n   COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n   THE FUTURE.\n\n[11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n\n=== Suggested solution ===\n\nSKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\nthe name of the option implies, to allow the file to NOT be in the\nworktree but consider it to be unchanged rather than deleted.\n\nThe suggests a simple solution: present-despite-SKIP_WORKTREE files\nshould not exist, for those using sparse-checkouts.\n\nEnforce this at index loading time by checking if core.sparseCheckout is\ntrue; if so, check files in the index with the SKIP_WORKTREE bit set to\nverify that they are absent from the working tree.  If they are present,\nunset the bit (in memory, though any commands that write to the index\nwill record the update).\n\nUsers can, of course, can get the SKIP_WORKTREE bit back such as by\nrunning `git sparse-checkout reapply` (if they have ensured the file is\nunmodified and doesn't match the specified sparsity patterns).\n\n=== Pros/Cons of suggested solution ===\n\nPros:\n\n  * Solves the user visible problems reported above, which I've been\n    complaining about for nearly a year but couldn't find a solution to.\n  * Helps prevent slow performance degradation with a sparse-index.\n  * Much easier behavior in sparse-checkouts for users to reason about\n  * Very simple, ~30 lines of code.\n  * Significantly simplifies some ugly testcases, and obviates the need\n    to test an entire class of potential issues.\n  * Reduces code complexity, reasoning, and maintenance.  Avoids\n    disagreements about weird corner cases[12].\n  * It has been reported that some users might be (ab)using\n    SKIP_WORKTREE as a let-me-modify-but-keep-the-file-in-the-worktree\n    mechanism[13, and a few other similar references].  These users know\n    of multiple caveats and shortcomings in doing so; perhaps not\n    surprising given the \"SKIP_WORKTREE expecations\" section above.\n    However, these users use `git update-index --skip-worktree`, and not\n    `git sparse-checkout` or core.sparseCheckout=true.  As such, these\n    users would be unaffected by this change and can continue abusing\n    the system as before.\n\n[12] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[13] https://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree\n\nCons:\n\n  * When core.sparseCheckout is enabled, this adds a performance cost to\n    reading the index.  I'll defer discussion of this cost to a subsequent\n    patch, since I have some optimizations to add.\n\n=== Notes on testcase modifications ===\n\nThe good:\n  * t1011: Compare to two cases above it ('read-tree will not throw away\n    dirty changes, non-sparse'); since the file is present, it should\n    match the non-sparse case now\n  * t1092: sparse-index & sparse-checkout now match full-worktree\n    behavior in more cases!  Yaay for consistency!\n  * t6428, t7012: look at how much simpler the tests become!  Merge and\n    stash can just fail early telling the user there's a file in the\n    way, instead of not noticing until it's about to write a file and\n    then have to implement sudden crash avoidance.  Hurray for sanity!\n  * t7817: sparse behavior better matches full tree behavior.  Hurray\n    for sanity!\n\nThe confusing:\n  * t3705: These changes were ONLY needed on Windows, but they don't\n    hurt other platforms.  Let's discuss each individually:\n\n    * core.sparseCheckout should be false by default.  Nothing in this\n      testcase toggles that until many, many tests later.  However,\n      early tests (#5 in particular) were testing `update-index\n      --skip-worktree` behavior in a non-sparse-checkout, but the\n      Windows tests in CI were behaving as if core.sparseCheckout=true\n      had been specified somewhere.  I do not have access to a Windows\n      machine.  But I just manually did what should have been a no-op\n      and turned the config off.  And it fixed the test.\n    * I have no idea why the leftover .gitattributes file from this\n      test was causing failures for test #18 on Windows, but only with\n      these changes of mine.  Test #18 was checking for empty stderr,\n      and specifically wanted to know that some error completely\n      unrelated to file endings did not appear.  The leftover\n      .gitattributes file thus caused some spurious stderr unrelated to\n      the thing being checked.  Since other tests did not intend to\n      test normalization, just proactively remove the .gitattributes\n      file.  I'm certain this is cleaner and better, I'm just unsure\n      why/how this didn't trigger problems before.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n repository.c                             |  7 ++++\n sparse-index.c                           | 21 +++++++++++\n sparse-index.h                           |  1 +\n t/t1011-read-tree-sparse-checkout.sh     |  2 +-\n t/t1092-sparse-checkout-compatibility.sh | 16 ++++-----\n t/t3705-add-sparse-checkout.sh           |  2 ++\n t/t6428-merge-conflicts-sparse.sh        | 23 +++----------\n t/t7012-skip-worktree-writing.sh         | 44 ++++--------------------\n t/t7817-grep-sparse-checkout.sh          | 11 ++++--\n 9 files changed, 61 insertions(+), 66 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex 34610c5a33e..dddee32258f 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -301,6 +301,13 @@ int repo_read_index(struct repository *repo)\n \tif (repo->settings.command_requires_full_index)\n \t\tensure_full_index(repo->index);\n \n+\t/*\n+\t * If sparse checkouts are in use, check whether paths with the\n+\t * SKIP_WORKTREE attribute are missing from the worktree; if not,\n+\t * clear that attribute for that path.\n+\t */\n+\tclear_skip_worktree_from_present_files(repo->index);\n+\n \treturn res;\n }\n \ndiff --git a/sparse-index.c b/sparse-index.c\nindex a1d505d50e9..b82648b10ee 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,6 +341,27 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+void clear_skip_worktree_from_present_files(struct index_state *istate)\n+{\n+\tint i;\n+\tif (!core_apply_sparse_checkout)\n+\t\treturn;\n+\n+restart:\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tstruct stat st;\n+\n+\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t\tensure_full_index(istate);\n+\t\t\t\tgoto restart;\n+\t\t\t}\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t}\n+\t}\n+}\n+\n /*\n  * This static global helps avoid infinite recursion between\n  * expand_to_path() and index_file_exists().\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 656bd835b25..633d4fb7e31 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -5,6 +5,7 @@ struct index_state;\n #define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n int convert_to_sparse(struct index_state *istate, int flags);\n void ensure_correct_sparsity(struct index_state *istate);\n+void clear_skip_worktree_from_present_files(struct index_state *istate);\n \n /*\n  * Some places in the codebase expect to search for a specific path.\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 4ed0885bf2f..dd957be1b78 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -212,7 +212,7 @@ test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n \techo dirty >init.t &&\n-\tread_tree_u_must_succeed -m -u HEAD^ &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n \tgrep -q dirty init.t &&\n \trm init.t\n '\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 53f84881de7..df2839e8e2c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -370,7 +370,7 @@ test_expect_success 'status/add: outside sparse cone' '\n \twrite_script edit-contents <<-\\EOF &&\n \techo text >>$1\n \tEOF\n-\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/a &&\n \trun_on_all ../edit-contents folder1/new &&\n \n \ttest_sparse_match git status --porcelain=v2 &&\n@@ -379,8 +379,8 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n-\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_all_match git add --refresh folder1/a &&\n+\ttest_must_be_empty sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n@@ -646,11 +646,11 @@ test_expect_success 'update-index modify outside sparse definition' '\n \trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n \trun_on_all ../edit-contents folder1/a &&\n \n-\t# If file has skip-worktree enabled, update-index does not modify the\n-\t# index entry\n-\ttest_sparse_match git update-index folder1/a &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_must_be_empty sparse-checkout-out &&\n+\t# If file has skip-worktree enabled, but the file is present, it is\n+\t# treated the same as if skip-worktree is disabled\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git update-index folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n \n \t# When skip-worktree is disabled (even on files outside sparse cone), file\n \t# is updated in the index\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex f3143c92908..61506c1d7ce 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit config core.sparseCheckout false &&\n \tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n@@ -126,6 +127,7 @@ test_expect_success 'git add --chmod does not update sparse entries' '\n '\n \n test_expect_success 'git add --renormalize does not update sparse entries' '\n+\ttest_when_finished rm .gitattributes &&\n \ttest_config core.autocrlf false &&\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\ndiff --git a/t/t6428-merge-conflicts-sparse.sh b/t/t6428-merge-conflicts-sparse.sh\nindex 7e8bf497f82..142c9aaabc5 100755\n--- a/t/t6428-merge-conflicts-sparse.sh\n+++ b/t/t6428-merge-conflicts-sparse.sh\n@@ -112,7 +112,7 @@ test_expect_success 'conflicting entries written to worktree even if sparse' '\n \t)\n '\n \n-test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handled reasonably' '\n+test_expect_success 'present-despite-SKIP_WORKTREE handled reasonably' '\n \ttest_setup_numerals in_the_way &&\n \t(\n \t\tcd numerals_in_the_way &&\n@@ -132,26 +132,13 @@ test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handl\n \n \t\ttest_must_fail git merge -s recursive B^0 &&\n \n-\t\tgit ls-files -t >index_files &&\n-\t\ttest_cmp expected-index index_files &&\n+\t\ttest_path_is_missing .git/MERGE_HEAD &&\n \n-\t\ttest_path_is_file README &&\n \t\ttest_path_is_file numerals &&\n \n-\t\ttest_cmp expected-merge numerals &&\n-\n-\t\t# There should still be a file with \"foobar\" in it\n-\t\tgrep foobar * &&\n-\n-\t\t# 5 other files:\n-\t\t#   * expected-merge\n-\t\t#   * expected-index\n-\t\t#   * index_files\n-\t\t#   * others\n-\t\t#   * whatever name was given to the numerals file that had\n-\t\t#     \"foobar\" in it\n-\t\tgit ls-files -o >others &&\n-\t\ttest_line_count = 5 others\n+\t\t# numerals should still have \"foobar\" in it\n+\t\techo foobar >expect &&\n+\t\ttest_cmp expect numerals\n \t)\n '\n \ndiff --git a/t/t7012-skip-worktree-writing.sh b/t/t7012-skip-worktree-writing.sh\nindex a1080b94e38..cb9f1a6981e 100755\n--- a/t/t7012-skip-worktree-writing.sh\n+++ b/t/t7012-skip-worktree-writing.sh\n@@ -171,50 +171,20 @@ test_expect_success 'stash restore in sparse checkout' '\n \n \t\t# Put a file in the working directory in the way\n \t\techo in the way >modified &&\n-\t\tgit stash apply &&\n+\t\ttest_must_fail git stash apply 2>error&&\n \n-\t\t# Ensure stash vivifies modifies paths...\n-\t\tcat >expect <<-EOF &&\n-\t\tH addme\n-\t\tH modified\n-\t\tH removeme\n-\t\tH subdir/A\n-\t\tS untouched\n-\t\tEOF\n-\t\tgit ls-files -t >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\tgrep \"changes.*would be overwritten by merge\" error &&\n \n-\t\t# ...and that the paths show up in status as changed...\n-\t\tcat >expect <<-EOF &&\n-\t\tA  addme\n-\t\t M modified\n-\t\t D removeme\n-\t\t M subdir/A\n-\t\t?? actual\n-\t\t?? expect\n-\t\t?? modified.stash.XXXXXX\n-\t\tEOF\n-\t\tgit status --porcelain | \\\n-\t\t\tsed -e s/stash......./stash.XXXXXX/ >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\techo in the way >expect &&\n+\t\ttest_cmp expect modified &&\n+\t\tgit diff --quiet HEAD \":!modified\" &&\n \n \t\t# ...and that working directory reflects the files correctly\n-\t\ttest_path_is_file    addme &&\n+\t\ttest_path_is_missing addme &&\n \t\ttest_path_is_file    modified &&\n \t\ttest_path_is_missing removeme &&\n \t\ttest_path_is_file    subdir/A &&\n-\t\ttest_path_is_missing untouched &&\n-\n-\t\t# ...including that we have the expected \"modified\" file...\n-\t\tcat >expect <<-EOF &&\n-\t\tmodified\n-\t\ttweaked\n-\t\tEOF\n-\t\ttest_cmp expect modified &&\n-\n-\t\t# ...and that the other \"modified\" file is still present...\n-\t\techo in the way >expect &&\n-\t\ttest_cmp expect modified.stash.*\n+\t\ttest_path_is_missing untouched\n \t)\n '\n \ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex 590b99bbb6f..eb595645657 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -83,10 +83,13 @@ test_expect_success 'setup' '\n \n # The test below covers a special case: the sparsity patterns exclude '/b' and\n # sparse checkout is enabled, but the path exists in the working tree (e.g.\n-# manually created after `git sparse-checkout init`). git grep should skip it.\n+# manually created after `git sparse-checkout init`).  Although b is marked\n+# as SKIP_WORKTREE, git grep should notice it IS present in the worktree and\n+# report it.\n test_expect_success 'working tree grep honors sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n+\tb:new-text\n \tEOF\n \ttest_when_finished \"rm -f b\" &&\n \techo \"new-text\" >b &&\n@@ -126,12 +129,16 @@ test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit'\n '\n \n # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n-# patterns, so grep should not recurse into it.\n+# patterns.  We also explicitly mark it as SKIP_WORKTREE in case it got cleared\n+# by previous git commands.  Thus sub2 starts as SKIP_WORKTREE but since it is\n+# present in the working tree, grep should recurse into it.\n test_expect_success 'grep --recurse-submodules honors sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tsub/B/b:text\n+\tsub2/a:text\n \tEOF\n+\tgit update-index --skip-worktree sub2 &&\n \tgit grep --recurse-submodules \"text\" >actual &&\n \ttest_cmp expect actual\n '\n-- \ngitgitgadget\n\n"},{"id":"446164","messageId":"c74ad19616ed1e0971d5de2a3da795dc264af074.1642092230.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH 4/5] Update documentation related to sparsity and the skip-worktree bit","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:49Z","receivedAt":"2022-01-13T16:43:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nMake several small updates, to address a few documentation issues\nI spotted:\n  * sparse-checkout focused on \"patterns\" even though the inputs (and\n    outputs in the case of `list`) are directories in cone-mode\n  * The description section of the sparse-checkout documentation\n    was a bit sparse (no pun intended), and focused more on internal\n    mechanics rather than end user usage.  This made sense in the\n    early days when the command was even more experimental, but let's\n    adjust a bit to try to make it more approachable to end users who\n    may want to consider using it.  Keep the scary backward\n    compatibility warning, though; we're still hard at work trying to\n    fix up commands to behave reasonably in sparse checkouts.\n  * both read-tree and update-index tried to describe how to use the\n    skip-worktree bit, but both predated the sparse-checkout command.\n    The sparse-checkout command is a far easier mechanism to use and\n    for users trying to reduce the size of their working tree, we\n    should recommend users to look at it instead.\n  * The update-index documentation pointed out that assume-unchanged\n    and skip-worktree sounded similar but had different purposes.\n    However, it made no attempt to explain the differences, only to\n    point out that they were different.  Explain the differences.\n  * The update-index documentation focused much more on (internal?)\n    implementation details than on end-user usage.  Try to explain\n    its purpose better for users of update-index, rather than\n    fellow developers trying to work with the SKIP_WORKTREE bit.\n  * Clarify that when core.sparseCheckout=true, we treat a file's\n    presence in the working tree as being an override to the\n    SKIP_WORKTREE bit (i.e. in sparse checkouts when the file is\n    present we ignore the SKIP_WORKTREE bit).\n\nNote that this commit, like many touching documentation, is best viewed\nwith the `--color-words` option to diff/log.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n Documentation/git-read-tree.txt       | 12 +++--\n Documentation/git-sparse-checkout.txt | 76 ++++++++++++++++-----------\n Documentation/git-update-index.txt    | 57 +++++++++++++++-----\n 3 files changed, 98 insertions(+), 47 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 8c3aceb8324..99bb387134d 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -375,9 +375,14 @@ have finished your work-in-progress), attempt the merge again.\n SPARSE CHECKOUT\n ---------------\n \n+Note: The `update-index` and `read-tree` primitives for supporting the\n+skip-worktree bit predated the introduction of\n+linkgit:git-sparse-checkout[1].  Users are encouraged to use\n+`sparse-checkout` in preference to these low-level primitives.\n+\n \"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to\n+tell Git whether a file in the working directory is worth looking at.\n \n 'git read-tree' and other merge-based commands ('git merge', 'git\n checkout'...) can help maintaining the skip-worktree bitmap and working\n@@ -385,7 +390,8 @@ directory update. `$GIT_DIR/info/sparse-checkout` is used to\n define the skip-worktree reference bitmap. When 'git read-tree' needs\n to update the working directory, it resets the skip-worktree bit in the index\n based on this file, which uses the same syntax as .gitignore files.\n-If an entry matches a pattern in this file, skip-worktree will not be\n+If an entry matches a pattern in this file, or the entry corresponds to\n+a file present in the working tree, then skip-worktree will not be\n set on that entry. Otherwise, skip-worktree will be set.\n \n Then it compares the new skip-worktree value with the previous one. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b81dbe06543..3da3d5a1007 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -3,9 +3,7 @@ git-sparse-checkout(1)\n \n NAME\n ----\n-git-sparse-checkout - Initialize and modify the sparse-checkout\n-configuration, which reduces the checkout to a set of paths\n-given by a list of patterns.\n+git-sparse-checkout - Reduce your working tree to a subset of tracked files\n \n \n SYNOPSIS\n@@ -17,8 +15,20 @@ SYNOPSIS\n DESCRIPTION\n -----------\n \n-Initialize and modify the sparse-checkout configuration, which reduces\n-the checkout to a set of paths given by a list of patterns.\n+This command is used to create sparse checkouts, which means that it\n+changes the working tree from having all tracked files present, to only\n+have a subset of them.  It can also switch which subset of files are\n+present, or undo and go back to having all tracked files present in the\n+working copy.\n+\n+The subset of files is chosen by providing a list of directories in\n+cone mode (which is recommended), or by providing a list of patterns\n+in non-cone mode.\n+\n+When in a sparse-checkout, other Git commands behave a bit differently.\n+For example, switching branches will not update paths outside the\n+sparse-checkout directories/patterns, and `git commit -a` will not record\n+paths outside the sparse-checkout directories/patterns as deleted.\n \n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n@@ -28,7 +38,7 @@ THE FUTURE.\n COMMANDS\n --------\n 'list'::\n-\tDescribe the patterns in the sparse-checkout file.\n+\tDescribe the directories or patterns in the sparse-checkout file.\n \n 'set'::\n \tEnable the necessary config settings\n@@ -38,20 +48,26 @@ COMMANDS\n \tlist of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n +\n-When the `--stdin` option is provided, the patterns are read from\n-standard in as a newline-delimited list instead of from the arguments.\n+When the `--stdin` option is provided, the directories or patterns are\n+read from standard in as a newline-delimited list instead of from the\n+arguments.\n +\n When `--cone` is passed or `core.sparseCheckoutCone` is enabled, the\n-input list is considered a list of directories instead of\n-sparse-checkout patterns.  This allows for better performance with a\n-limited set of patterns (see 'CONE PATTERN SET' below).  Note that the\n-set command will write patterns to the sparse-checkout file to include\n-all files contained in those directories (recursively) as well as\n-files that are siblings of ancestor directories. The input format\n-matches the output of `git ls-tree --name-only`.  This includes\n-interpreting pathnames that begin with a double quote (\") as C-style\n-quoted strings.  This may become the default in the future; --no-cone\n-can be passed to request non-cone mode.\n+input list is considered a list of directories.  This allows for\n+better performance with a limited set of patterns (see 'CONE PATTERN\n+SET' below).  The input format matches the output of `git ls-tree\n+--name-only`.  This includes interpreting pathnames that begin with a\n+double quote (\") as C-style quoted strings.  Note that the set command\n+will write patterns to the sparse-checkout file to include all files\n+contained in those directories (recursively) as well as files that are\n+siblings of ancestor directories. This may become the default in the\n+future; --no-cone can be passed to request non-cone mode.\n++\n+When `--no-cone` is passed or `core.sparseCheckoutCone` is not enabled,\n+the input list is considered a list of patterns.  This mode is harder\n+to use and less performant, and is thus not recommended.  See the\n+\"Sparse Checkout\" section of linkgit:git-read-tree[1] and the \"Pattern\n+Set\" sections below for more details.\n +\n Use the `--[no-]sparse-index` option to use a sparse index (the\n default is to not use it).  A sparse index reduces the size of the\n@@ -69,11 +85,10 @@ understand the sparse directory entries index extension and may fail to\n interact with your repository until it is disabled.\n \n 'add'::\n-\tUpdate the sparse-checkout file to include additional patterns.\n-\tBy default, these patterns are read from the command-line arguments,\n-\tbut they can be read from stdin using the `--stdin` option. When\n-\t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n-\tas directory names as in the 'set' subcommand.\n+\tUpdate the sparse-checkout file to include additional directories\n+\t(in cone mode) or patterns (in non-cone mode).  By default, these\n+\tdirectories or patterns are read from the command-line arguments,\n+\tbut they can be read from stdin using the `--stdin` option.\n \n 'reapply'::\n \tReapply the sparsity pattern rules to paths in the working tree.\n@@ -117,13 +132,14 @@ decreased in utility.\n SPARSE CHECKOUT\n ---------------\n \n-\"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at. If\n-the skip-worktree bit is set, then the file is ignored in the working\n-directory. Git will avoid populating the contents of those files, which\n-makes a sparse checkout helpful when working in a repository with many\n-files, but only a few are important to the current user.\n+\"Sparse checkout\" allows populating the working directory sparsely.  It\n+uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell Git\n+whether a file in the working directory is worth looking at. If the\n+skip-worktree bit is set, and the file is not present in the working tree,\n+then its absence is ignored. Git will avoid populating the contents of\n+those files, which makes a sparse checkout helpful when working in a\n+repository with many files, but only a few are important to the current\n+user.\n \n The `$GIT_DIR/info/sparse-checkout` file is used to define the\n skip-worktree reference bitmap. When Git updates the working\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..568dbfe76b8 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -351,6 +351,10 @@ unchanged\".  Note that \"assume unchanged\" bit is *not* set if\n the index (use `git update-index --really-refresh` if you want\n to mark them as \"assume unchanged\").\n \n+Sometimes users confuse the assume-unchanged bit with the\n+skip-worktree bit.  See the final paragraph in the \"Skip-worktree bit\"\n+section below for an explanation of the differences.\n+\n \n EXAMPLES\n --------\n@@ -392,22 +396,47 @@ M foo.c\n SKIP-WORKTREE BIT\n -----------------\n \n-Skip-worktree bit can be defined in one (long) sentence: When reading\n-an entry, if it is marked as skip-worktree, then Git pretends its\n-working directory version is up to date and read the index version\n-instead.\n-\n-To elaborate, \"reading\" means checking for file existence, reading\n-file attributes or file content. The working directory version may be\n-present or absent. If present, its content may match against the index\n-version or not. Writing is not affected by this bit, content safety\n-is still first priority. Note that Git _can_ update working directory\n-file, that is marked skip-worktree, if it is safe to do so (i.e.\n-working directory version matches index version)\n+Skip-worktree bit can be defined in one (long) sentence: Tell git to\n+avoid writing the file to the working directory when reasonably\n+possible, and treat the file as unchanged when it is not\n+present in the working directory.\n+\n+Note that not all git commands will pay attention to this bit, and\n+some only partially support it.\n+\n+The update-index flags and the read-tree capabilities relating to the\n+skip-worktree bit predated the introduction of the\n+linkgit:git-sparse-checkout[1] command, which provides a much easier\n+way to configure and handle the skip-worktree bits.  If you want to\n+reduce your working tree to only deal with a subset of the files in\n+the repository, we strongly encourage the use of\n+linkgit:git-sparse-checkout[1] in preference to the low-level\n+update-index and read-tree primitives.\n+\n+The primary purpose of the skip-worktree bit is to enable sparse\n+checkouts, i.e. to have working directories with only a subset of\n+paths present.  When the skip-worktree bit is set, Git commands (such\n+as `switch`, `pull`, `merge`) will avoid writing these files.\n+However, these commands will sometimes write these files anyway in\n+important cases such as conflicts during a merge or rebase.  Git\n+commands will also avoid treating the lack of such files as an\n+intentional deletion; for example `git add -u` will not not stage a\n+deletion for these files and `git commit -a` will not make a commit\n+deleting them either.\n \n Although this bit looks similar to assume-unchanged bit, its goal is\n-different from assume-unchanged bit's. Skip-worktree also takes\n-precedence over assume-unchanged bit when both are set.\n+different.  The assume-unchanged bit is for leaving the file in the\n+working tree but having Git omit checking it for changes and presuming\n+that the file has not been changed (though if it can determine without\n+stat'ing the file that it has changed, it is free to record the\n+changes).  skip-worktree tells Git to ignore the absence of the file,\n+avoid updating it when possible with commands that normally update\n+much of the working directory (e.g. `checkout`, `switch`, `pull`,\n+etc.), and not have its absence be recorded in commits.  Note that in\n+sparse checkouts (setup by `git sparse-checkout` or by configuring\n+core.sparseCheckout to true), if a file is marked as skip-worktree in\n+the index but is found in the working tree, Git will clear the\n+skip-worktree bit for that file.\n \n SPLIT INDEX\n -----------\n-- \ngitgitgadget\n\n"},{"id":"446165","messageId":"e68028ebe0afc1bb9e623efbdd30de5a8f0740bf.1642092230.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-13T16:43:50Z","receivedAt":"2022-01-13T16:44:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nTrying to clear the skip-worktree bit from files that are present does\npresent some computational overhead, for sparse-checkouts.  (We do not\ndo the bit clearing in non-sparse-checkouts.)  Optimize it as follows:\n\nRather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\nfact that entire directories will often be missing, especially for cone\nmode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\nclear tracked sparse dirs\", 2021-09-08).  If we have already determined\nthat the parent directory of a file (or other previous ancestor) does\nnot exist, then the file cannot exist either so we do not need to\nlstat() it separately.\n\nTimings for p2000 included below, reformatted to fit in normal commit\nmessage line lengths, which compare three things:\n  * Timings before this series\n  * Timings of the unoptimized version of\n    clear_skip_worktree_from_present_files() from a few commits ago\n  * Timings after the optimization in this commit\n\n(NOTE: t/perf/ appears to have timing resolution only down to 0.01 s,\nwhich presents significant measurement error when timings only differ by\n0.01s.  I don't trust any such timings below, and yet all the optimized\nresults differ by at most 0.01s.)\n\nTest        Before Series    Unoptimized              Optimized\n-----------------------------------------------------------------------------\n*git status*\nfull-v3     0.15(0.10+0.06)  0.32(0.16+0.17) +113.3%  0.16(0.10+0.07) +6.7%\nfull-v4     0.15(0.11+0.05)  0.32(0.17+0.16) +113.3%  0.16(0.11+0.05) +6.7%\nsparse-v3   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.02+0.05) +0.0%\nsparse-v4   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.03+0.05) +0.0%\n\n*git add -A*\nfull-v3     0.40(0.30+0.07)  0.56(0.36+0.17) +40.0%   0.39(0.30+0.07) -2.5%\nfull-v4     0.37(0.28+0.07)  0.54(0.37+0.16) +45.9%   0.38(0.29+0.07) +2.7%\nsparse-v3   0.06(0.04+0.05)  0.08(0.05+0.05) +33.3%   0.06(0.05+0.04) +0.0%\nsparse-v4   0.05(0.03+0.05)  0.05(0.04+0.04) +0.0%    0.06(0.04+0.05) +20.0%\n\n*git add .*\nfull-v3     0.40(0.31+0.07)  0.57(0.37+0.17) +42.5%   0.41(0.30+0.08) +2.5%\nfull-v4     0.38(0.30+0.06)  0.55(0.37+0.16) +44.7%   0.38(0.30+0.06) +0.0%\nsparse-v3   0.06(0.04+0.05)  0.06(0.05+0.04) +0.0%    0.06(0.03+0.05) +0.0%\nsparse-v4   0.06(0.05+0.05)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.06) +0.0%\n\n*git commit -a -m A*\nfull-v3     0.41(0.32+0.06)  0.58(0.39+0.17) +41.5%   0.42(0.32+0.07) +2.4%\nfull-v4     0.39(0.30+0.07)  0.56(0.38+0.17) +43.6%   0.40(0.31+0.07) +2.6%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.04(0.03+0.05)  0.04(0.03+0.05) +0.0%    0.04(0.03+0.04) +0.0%\n\n*git checkout -f -*\nfull-v3     0.56(0.46+0.07)  0.73(0.55+0.16) +30.4%   0.57(0.47+0.08) +1.8%\nfull-v4     0.54(0.45+0.07)  0.71(0.53+0.17) +31.5%   0.55(0.45+0.07) +1.9%\nsparse-v3   0.06(0.04+0.04)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.05) +0.0%\nsparse-v4   0.05(0.05+0.04)  0.05(0.04+0.05) +0.0%    0.06(0.04+0.05) +20.0%\n\n*git reset*\nfull-v3     0.34(0.26+0.05)  0.51(0.34+0.15) +50.0%   0.34(0.26+0.06) +0.0%\nfull-v4     0.32(0.24+0.06)  0.49(0.32+0.15) +53.1%   0.33(0.25+0.06) +3.1%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.03(0.03+0.04)  0.03(0.02+0.04) +0.0%    0.03(0.03+0.04) +0.0%\n\n*git reset --hard*\nfull-v3     0.57(0.46+0.07)  0.90(0.61+0.25) +57.9%   0.57(0.45+0.08) +0.0%\nfull-v4     0.54(0.46+0.05)  0.88(0.59+0.26) +63.0%   0.55(0.45+0.07) +1.9%\nsparse-v3   0.07(0.03+0.03)  0.07(0.04+0.03) +0.0%    0.07(0.03+0.03) +0.0%\nsparse-v4   0.06(0.03+0.03)  0.06(0.04+0.02) +0.0%    0.06(0.03+0.03) +0.0%\n\n*git reset -- does-not-exist*\nfull-v3     0.35(0.27+0.06)  0.52(0.32+0.17) +48.6%   0.35(0.27+0.06) +0.0%\nfull-v4     0.33(0.26+0.05)  0.50(0.33+0.15) +51.5%   0.33(0.26+0.06) +0.0%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.04(0.02+0.04)  0.03(0.02+0.04) -25.0%   0.03(0.02+0.04) -25.0%\n\n*git diff*\nfull-v3     0.07(0.04+0.04)  0.24(0.11+0.14) +242.9%  0.07(0.04+0.04) +0.0%\nfull-v4     0.07(0.03+0.05)  0.24(0.13+0.12) +242.9%  0.08(0.04+0.05) +14.3%\nsparse-v3   0.02(0.01+0.04)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.05) +0.0%\nsparse-v4   0.02(0.02+0.03)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.04) +0.0%\n\n*git diff --cached*\nfull-v3     0.05(0.03+0.02)  0.22(0.12+0.09) +340.0%  0.05(0.03+0.01) +0.0%\nfull-v4     0.05(0.03+0.01)  0.23(0.12+0.11) +360.0%  0.05(0.03+0.02) +0.0%\nsparse-v3   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\nsparse-v4   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\n\n*git blame f2/f4/a*\nfull-v3     0.18(0.13+0.05)  0.52(0.29+0.23) +188.9%  0.19(0.15+0.04) +5.6%\nfull-v4     0.19(0.15+0.04)  0.52(0.28+0.23) +173.7%  0.19(0.14+0.04) +0.0%\nsparse-v3   0.10(0.08+0.02)  0.10(0.09+0.01) +0.0%    0.10(0.09+0.01) +0.0%\nsparse-v4   0.10(0.08+0.02)  0.10(0.08+0.02) +0.0%    0.10(0.08+0.02) +0.0%\n\n*git blame f2/f4/f3/a*\nfull-v3     0.45(0.36+0.08)  0.78(0.51+0.27) +73.3%   0.45(0.37+0.08) +0.0%\nfull-v4     0.45(0.37+0.08)  0.78(0.51+0.26) +73.3%   0.45(0.37+0.08) +0.0%\nsparse-v3   0.36(0.32+0.04)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\nsparse-v4   0.36(0.31+0.05)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\n\n*git checkout-index -f --all*\nfull-v3     0.07(0.02+0.05)  0.24(0.12+0.12) +242.9%  0.08(0.04+0.04) +14.3%\nfull-v4     0.07(0.03+0.04)  0.24(0.11+0.13) +242.9%  0.08(0.03+0.04) +14.3%\nsparse-v3   0.04(0.01+0.03)  0.04(0.00+0.03) +0.0%    0.04(0.01+0.03) +0.0%\nsparse-v4   0.04(0.01+0.02)  0.04(0.01+0.03) +0.0%    0.04(0.01+0.02) +0.0%\n\n*git update-index --add --remove f2/f4/a*\nfull-v3     0.29(0.23+0.02)  0.46(0.30+0.12) +58.6%   0.30(0.24+0.02) +3.4%\nfull-v4     0.27(0.22+0.02)  0.45(0.29+0.12) +66.7%   0.28(0.22+0.03) +3.7%\nsparse-v3   0.02(0.02+0.00)  0.02(0.01+0.00) +0.0%    0.02(0.01+0.00) +0.0%\nsparse-v4   0.02(0.02+0.00)  0.02(0.02+0.00) +0.0%    0.02(0.02+0.00) +0.0%\n\nSo, with the optimization, the extra work appears to be essentially 0\nfor sparse-checkouts that are also using sparse-indexes (even before my\noptimization), and the extra work appears to be just marginally more\nthan 0 for sparse-checkouts that are using full indexes.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n sparse-index.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 54 insertions(+), 2 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b82648b10ee..eed170cd8f7 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,18 +341,70 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+static int path_found(const char *path, const char **dirname, size_t *dir_len,\n+\t\t      int *dir_found)\n+{\n+\tstruct stat st;\n+\tchar *newdir;\n+\tchar *tmp;\n+\n+\t/*\n+\t * If dirname corresponds to a directory that doesn't exist, and this\n+\t * path starts with dirname, then path can't exist.\n+\t */\n+\tif (!*dir_found && !memcmp(path, *dirname, *dir_len))\n+\t\treturn 0;\n+\n+\t/*\n+\t * If path itself exists, return 1.\n+\t */\n+\tif (!lstat(path, &st))\n+\t\treturn 1;\n+\n+\t/*\n+\t * Otherwise, path does not exist so we'll return 0...but we'll first\n+\t * determine some info about its parent directory so we can avoid\n+\t * lstat calls for future cache entries.\n+\t */\n+\tnewdir = strrchr(path, '/');\n+\tif (!newdir)\n+\t\treturn 0; /* Didn't find a parent dir; just return 0 now. */\n+\n+\t/*\n+\t * If path starts with directory (which we already lstat'ed and found),\n+\t * then no need to lstat parent directory again.\n+\t */\n+\tif (*dir_found && *dirname && memcmp(path, *dirname, *dir_len))\n+\t\treturn 0;\n+\n+\t/* Free previous dirname, and cache path's dirname */\n+\t*dirname = path;\n+\t*dir_len = newdir - path + 1;\n+\n+\ttmp = xstrndup(path, *dir_len);\n+\t*dir_found = !lstat(tmp, &st);\n+\tfree(tmp);\n+\n+\treturn 0;\n+}\n+\n void clear_skip_worktree_from_present_files(struct index_state *istate)\n {\n+\tconst char *last_dirname = NULL;\n+\tsize_t dir_len = 0;\n+\tint dir_found = 1;\n+\n \tint i;\n+\n \tif (!core_apply_sparse_checkout)\n \t\treturn;\n \n restart:\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tstruct stat st;\n \n-\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\tif (ce_skip_worktree(ce) &&\n+\t\t    path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n \t\t\t\tensure_full_index(istate);\n \t\t\t\tgoto restart;\n-- \ngitgitgadget\n"},{"id":"446190","messageId":"CABPp-BGHWJ9-E-OBu_be4dSovZKzS5PYEsc4DsA4VOjOZkcGMA@mail.gmail.com","threadId":"57236","inReplyTo":"e68028ebe0afc1bb9e623efbdd30de5a8f0740bf.1642092230.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-01-13T23:35:03Z","receivedAt":"2022-01-13T23:35:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 13, 2022 at 8:43 AM Elijah Newren via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> From: Elijah Newren <newren@gmail.com>\n>\n> Trying to clear the skip-worktree bit from files that are present does\n> present some computational overhead, for sparse-checkouts.  (We do not\n> do the bit clearing in non-sparse-checkouts.)  Optimize it as follows:\n>\n...\n> (NOTE: t/perf/ appears to have timing resolution only down to 0.01 s,\n> which presents significant measurement error when timings only differ by\n> 0.01s.  I don't trust any such timings below, and yet all the optimized\n> results differ by at most 0.01s.)\n\nTo follow up on this using using a tool with higher precision for a\nfew selected cases:\n\n> Test        Before Series    Unoptimized              Optimized\n> -----------------------------------------------------------------------------\n...\n> *git add -A*\n> full-v3     0.40(0.30+0.07)  0.56(0.36+0.17) +40.0%   0.39(0.30+0.07) -2.5%\n\nIn hyperfine, the results were:\n\nbefore-series,full-v3$ hyperfine '~/floss/git/git add -A'\nBenchmark #1: ~/floss/git/git add -A\n  Time (mean ± σ):     129.4 ms ±   3.0 ms    [User: 84.0 ms, System: 59.9 ms]\n  Range (min … max):   122.5 ms … 138.3 ms    24 runs\n\npost-optim,full-v3$ hyperfine '~/floss/git/git add -A'\nBenchmark #1: ~/floss/git/git add -A\n  Time (mean ± σ):     133.4 ms ±   3.0 ms    [User: 85.6 ms, System: 61.4 ms]\n  Range (min … max):   129.7 ms … 142.7 ms    22 runs\n\nSince 133.4/129.4 ~ 1.0309, we see that we have +3.1% on the execution\ntiming due to changes in this series.  Notably, the number is actually\npositive as we'd expect since we are clearly doing more work, whereas\nthe t/perf system reports -2.5% due to measurement inaccuracy.\n\n> full-v4     0.37(0.28+0.07)  0.54(0.37+0.16) +45.9%   0.38(0.29+0.07) +2.7%\n> sparse-v3   0.06(0.04+0.05)  0.08(0.05+0.05) +33.3%   0.06(0.05+0.04) +0.0%\n> sparse-v4   0.05(0.03+0.05)  0.05(0.04+0.04) +0.0%    0.06(0.04+0.05) +20.0%\n\nbefore-series,sparse-v4$ hyperfine '~/floss/git/git add -A'\nBenchmark #1: ~/floss/git/git add -A\n  Time (mean ± σ):      37.0 ms ±   1.1 ms    [User: 23.1 ms, System: 47.8 ms]\n  Range (min … max):    35.6 ms …  42.1 ms    72 runs\n\npost-optim,sparse-v4$ hyperfine '~/floss/git/git add -A'\nBenchmark #1: ~/floss/git/git add -A\n  Time (mean ± σ):      37.3 ms ±   0.9 ms    [User: 22.8 ms, System: 48.7 ms]\n  Range (min … max):    36.2 ms …  40.7 ms    72 runs\n\n37.3/37.0 ~ 1.008, so a +0.8% on execution timing.  Hard to measure,\nand nice that hyperfine automatically chooses to run this 72 times\njust to ensure the data is more reliable.  (Also, the +0.8% is much\nmore believable than the +20.0% from lack of measurement accuracy,\nespecially combined with data below)\n\n> *git diff*\n> full-v3     0.07(0.04+0.04)  0.24(0.11+0.14) +242.9%  0.07(0.04+0.04) +0.0%\n> full-v4     0.07(0.03+0.05)  0.24(0.13+0.12) +242.9%  0.08(0.04+0.05) +14.3%\n\nbefore-series,full-v4$ hyperfine '~/floss/git/git diff'\nBenchmark #1: ~/floss/git/git diff\n  Time (mean ± σ):      69.9 ms ±   1.9 ms    [User: 37.5 ms, System: 47.6 ms]\n  Range (min … max):    66.5 ms …  74.0 ms    44 runs\n\npost-optim,full-v4$ hyperfine '~/floss/git/git diff'\nBenchmark #1: ~/floss/git/git diff\n  Time (mean ± σ):      73.0 ms ±   1.7 ms    [User: 40.7 ms, System: 47.1 ms]\n  Range (min … max):    70.3 ms …  76.2 ms    40 runs\n\nSo 73.0/69.9 ~ 1.044, so a +4.4% to the execution timing -- much less\nthan the +14.3% reported due to lack of measurement accuracy.\n\n(If I had used ~/floss/git/bin-wrappers/git instead of\n~/floss/git/git, the timings would have been going from 72.3ms to\n75.1ms, representing a +3.9% increase instead.  Probably doesn't\nmatter, but just in case folks were curious about what if I used the\nbin-wrappers.)\n\n> sparse-v3   0.02(0.01+0.04)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.05) +0.0%\n> sparse-v4   0.02(0.02+0.03)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.04) +0.0%\n\nHere, the timings went from 17.5ms to 17.6ms, or +0.6%.  It's really\nhard to measure the overhead of the changes of my patch series when\nsparse-index is in use, which makes sense since the extra loop only\nhas to check the toplevel directory, not any of the entries under it.\nBut it does show just a minor cost in runtime due to the new added\ncheck.\n\n\n\nI think the original commit message makes it clear that with the (new)\noptimization, the timings are quite reasonable, but maybe this little\nextra flavor helps for anyone who was just looking at the percentage\nchanges.  The timings and timing differences really need to be bigger\nthan the precision in order to trust the percentage differences, which\njust wasn't the case for t/perf here.  But t/perf was good enough to\nshow that the timings after my series are roughly the same as before\nthe series (never more than 0.01s slower).\n"},{"id":"446222","messageId":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.git.1642092230.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts)","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:38Z","receivedAt":"2022-01-14T15:59:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"(Maintainer note: This series builds on (v2 of) vd/sparse-clean-etc, because\nit tweaks one of the testcases added there.)\n\n(Note 2: There was a previous RFC round of this series at\nhttps://lore.kernel.org/git/20220109045732.2497526-1-newren@gmail.com/.)\n\nFiles in the present-despite-SKIP_WORKTREE state have caused no ends of\ndiscussions and bugs[1,2,3,4,5,6,...and lots of others]. Trying to address\nthe big issue of discovering & recovering from this state has befuddled me\nfor over a year because I was worried we'd need additional code at every\nskip_worktree-checking path in the code (and they are all over the place),\nand that we'd make the code significantly slower unless we plumbed a bunch\nof additional information all over the place to allow some reasonable\noptimizations.\n\nThis series tries to solve the problem a bit differently by automatic early\ndiscovery and recovery; as a result, it greatly simplifies the landscape,\nreduces our testing matrix burden, and fixes a large swath of bugs. And I\nfigured out how to get the perf cost down to essentially negligible.\n\nChanges since v1 (or v2 if you count RFC as v1):\n\n * now includes some fixes for testcases from\n   ds/fetch-pull-with-sparse-index (which topic marked its own new tests as\n   potentially suboptimal; with my series, the sparse behavior now matches\n   the full tree behavior on that test. Wahoo!). Note that Junio's version\n   of vd/sparse-clean-etc already includes ds/fetch-pull-with-sparse-index,\n   so no need to merge in anything extra.\n\nChanges since RFC version:\n\n * updated the commit messages as per suggestions from Victoria, including\n   adding performance measurements\n * renamed the new function to use a clearer name\n * replaced the final patch with a different optimization, which is both\n   simpler and performs quite a bit better (the cost for my previous patch 5\n   was already decent in many cases, but had a few cases where the cost was\n   significant).\n\nQuick overview:\n\n * Patches 1 & 2 add a test to demonstrate accidental deletion of\n   possibly-modified files, and then fix the bug.\n * Patch 3 is the crux of this series; a small amount of code with a huge\n   commit message\n * Patch 4 updates the documentation\n * Patch 5 adds an optimization to reduce the performance impact of patch 3\n\n[1]\nhttps://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n[2]\nhttps://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[3]\nhttps://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[4] commit 66b209b (\"merge-ort: implement CE_SKIP_WORKTREE handling with\nconflicted entries\", 2021-03-20) [5] commit ba359fd (\"stash: fix stash\napplication in sparse-checkouts\", 2020-12-01) [6]\nhttps://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n\nElijah Newren (5):\n  t1011: add testcase demonstrating accidental loss of user\n    modifications\n  unpack-trees: fix accidental loss of user changes\n  repo_read_index: clear SKIP_WORKTREE bit from files present in\n    worktree\n  Update documentation related to sparsity and the skip-worktree bit\n  Accelerate clear_skip_worktree_from_present_files() by caching\n\n Documentation/git-read-tree.txt          | 12 +++-\n Documentation/git-sparse-checkout.txt    | 76 ++++++++++++++----------\n Documentation/git-update-index.txt       | 57 +++++++++++++-----\n repository.c                             |  7 +++\n sparse-index.c                           | 73 +++++++++++++++++++++++\n sparse-index.h                           |  1 +\n t/t1011-read-tree-sparse-checkout.sh     | 23 ++++++-\n t/t1092-sparse-checkout-compatibility.sh | 41 ++++++-------\n t/t3705-add-sparse-checkout.sh           |  2 +\n t/t6428-merge-conflicts-sparse.sh        | 23 ++-----\n t/t7012-skip-worktree-writing.sh         | 44 +++-----------\n t/t7817-grep-sparse-checkout.sh          | 11 +++-\n unpack-trees.c                           |  4 +-\n 13 files changed, 246 insertions(+), 128 deletions(-)\n\n\nbase-commit: 48609de3bf32befb69c40c1a2595a98dac0448b4\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1114%2Fnewren%2Ffix-present-despite-skip-worktree-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1114/newren/fix-present-despite-skip-worktree-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1114\n\nRange-diff vs v1:\n\n 1:  c553d558c2f = 1:  d50d804af4e t1011: add testcase demonstrating accidental loss of user modifications\n 2:  1e3958576e2 = 2:  206c638fa90 unpack-trees: fix accidental loss of user changes\n 3:  b263cc75b7d ! 3:  11d46a399d2 repo_read_index: clear SKIP_WORKTREE bit from files present in worktree\n     @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index modi\n       \n       \t# When skip-worktree is disabled (even on files outside sparse cone), file\n       \t# is updated in the index\n     +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'ls-files' '\n     + \ttest_cmp dense sparse &&\n     + \n     + \t# Set up a strange condition of having a file edit\n     +-\t# outside of the sparse-checkout cone. This is just\n     +-\t# to verify that sparse-checkout and sparse-index\n     +-\t# behave the same in this case.\n     ++\t# outside of the sparse-checkout cone. We want to verify\n     ++\t# that all modes handle this the same, and detect the\n     ++\t# modification.\n     + \twrite_script edit-content <<-\\EOF &&\n     +-\tmkdir folder1 &&\n     ++\tmkdir -p folder1 &&\n     + \techo content >>folder1/a\n     + \tEOF\n     +-\trun_on_sparse ../edit-content &&\n     ++\trun_on_all ../edit-content &&\n     + \n     +-\t# ls-files does not currently notice modified files whose\n     +-\t# cache entries are marked SKIP_WORKTREE. This may change\n     +-\t# in the future, but here we test that sparse index does\n     +-\t# not accidentally create a change of behavior.\n     +-\ttest_sparse_match git ls-files --modified &&\n     +-\ttest_must_be_empty sparse-checkout-out &&\n     +-\ttest_must_be_empty sparse-index-out &&\n     ++\ttest_all_match git ls-files --modified &&\n     + \n     + \tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n     +-\ttest_must_be_empty sparse-index-out &&\n     ++\tcat >expect <<-\\EOF &&\n     ++\tfolder1/a\n     ++\tEOF\n     ++\ttest_cmp expect sparse-index-out &&\n     + \n     + \t# Add folder1 to the sparse-checkout cone and\n     + \t# check that ls-files shows the expanded files.\n     + \ttest_sparse_match git sparse-checkout add folder1 &&\n     +-\ttest_sparse_match git ls-files --modified &&\n     ++\ttest_all_match git ls-files --modified &&\n     + \n     + \ttest_all_match git ls-files &&\n     + \tgit -C sparse-index ls-files --sparse >actual &&\n      \n       ## t/t3705-add-sparse-checkout.sh ##\n      @@ t/t3705-add-sparse-checkout.sh: setup_sparse_entry () {\n 4:  c74ad19616e = 4:  0af00779128 Update documentation related to sparsity and the skip-worktree bit\n 5:  e68028ebe0a = 5:  05ac964e630 Accelerate clear_skip_worktree_from_present_files() by caching\n\n-- \ngitgitgadget\n"},{"id":"446223","messageId":"d50d804af4e17ff1613ae40e3b5cf1b1bd0986a0.1642175983.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] t1011: add testcase demonstrating accidental loss of user modifications","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:39Z","receivedAt":"2022-01-14T15:59:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nIf a user has a file with local modifications that is not marked as\nSKIP_WORKTREE, but the sparsity patterns are such that it should be\nmarked that way, and the user then invokes a command like\n\n   * git checkout -q HEAD^\n\nor\n\n   * git read-tree -mu HEAD^\n\nThen the file will be deleted along with all the users' modifications.\nAdd a testcase demonstrating this problem.\n\nNote: This bug only triggers if something other than 'HEAD' is given;\nif the commands above had specified 'HEAD', then the users' file would\nbe left alone.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n t/t1011-read-tree-sparse-checkout.sh | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 24092c09a95..1b2395b8a82 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -187,6 +187,27 @@ test_expect_success 'read-tree updates worktree, absent case' '\n \ttest ! -f init.t\n '\n \n+test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n+test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+\techo \"/*\" >.git/info/sparse-checkout &&\n+\tread_tree_u_must_succeed -m -u HEAD &&\n+\n+\techo dirty >init.t &&\n+\techo sub/added >.git/info/sparse-checkout &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n+\ttest_path_is_file init.t &&\n+\tgrep -q dirty init.t\n+'\n+\n test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n-- \ngitgitgadget\n\n"},{"id":"446224","messageId":"206c638fa904842def0ebe325747464ad58da7f1.1642175983.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] unpack-trees: fix accidental loss of user changes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:40Z","receivedAt":"2022-01-14T15:59:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nFor sparse-checkouts, we don't want unpack-trees to error out on files\nthat are missing from the worktree, so there has traditionally been\nlogic to make it skip the verify_uptodate() check for these.\nUnfortunately, it was skipping the verify_uptodate() check for files\nthat were expected to *become* SKIP_WORKTREE.  For files that were not\nalready SKIP_WORKTREE, that can cause us to later delete the file in\napply_sparse_checkout().  Only skip the check for files that were\nalready SKIP_WORKTREE as well to avoid lightly discarding important\nchanges users may have made to files.\n\nNote 1: unpack-trees.c is already a bit complex, and the logic around\nCE_SKIP_WORKTREE and CE_NEW_SKIP_WORKTREE in that file are no exception.\nI also tried just replacing CE_NEW_SKIP_WORKTREE with CE_SKIP_WORKTREE\nin the verify_uptodate() check instead of checking for both flags, and\nfound that it also fixed this bug and passed all the tests.  I also\nattempted to devise a few testcases that might trip either variant of my\nfix and was unable to find any problems.  It may be that just checking\nCE_SKIP_WORKTREE is a better fix, but I'm not sure.  I thought it\nwas a bit safer to strictly reduce the number of cases where we skip the\nup-to-date check rather than just toggling which kind of cases skip it,\nand thus went with the current variant of the fix.\n\nNote 2: I also wondered if verify_absent() might have a similar bug, but\ndespite my attempts to try to devise a testcase that would trigger such\na thing, I couldn't find any problematic testcases.  Thus, this patch\nmakes no attempt to apply similar changes to verify_absent() and\nverify_absent_if_directory().\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n t/t1011-read-tree-sparse-checkout.sh | 2 +-\n unpack-trees.c                       | 4 +++-\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 1b2395b8a82..4ed0885bf2f 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -197,7 +197,7 @@ test_expect_success 'read-tree will not throw away dirty changes, non-sparse' '\n \tgrep -q dirty init.t\n '\n \n-test_expect_failure 'read-tree will not throw away dirty changes, sparse' '\n+test_expect_success 'read-tree will not throw away dirty changes, sparse' '\n \techo \"/*\" >.git/info/sparse-checkout &&\n \tread_tree_u_must_succeed -m -u HEAD &&\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..96525d2ec26 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2065,7 +2065,9 @@ static int verify_uptodate_1(const struct cache_entry *ce,\n int verify_uptodate(const struct cache_entry *ce,\n \t\t    struct unpack_trees_options *o)\n {\n-\tif (!o->skip_sparse_checkout && (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n+\tif (!o->skip_sparse_checkout &&\n+\t    (ce->ce_flags & CE_SKIP_WORKTREE) &&\n+\t    (ce->ce_flags & CE_NEW_SKIP_WORKTREE))\n \t\treturn 0;\n \treturn verify_uptodate_1(ce, o, ERROR_NOT_UPTODATE_FILE);\n }\n-- \ngitgitgadget\n\n"},{"id":"446225","messageId":"11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:41Z","receivedAt":"2022-01-14T15:59:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nThe fix is short (~30 lines), but the description is not.  Sorry.\n\nThere is a set of problems caused by files in what I'll refer to as the\n\"present-despite-SKIP_WORKTREE\" state.  This commit aims to not just fix\nthese problems, but remove the entire class as a possibility -- for\nthose using sparse checkouts.  But first, we need to understand the\nproblems this class presents.  A quick outline:\n\n   * Problems\n     * User facing issues\n     * Problem space complexity\n     * Maintenance and code correctness challenges\n   * SKIP_WORKTREE expectations in Git\n   * Suggested solution\n   * Pros/Cons of suggested solution\n   * Notes on testcase modifications\n\n=== User facing issues ===\n\nThere are various ways for users to get files to be present in the\nworking copy despite having the SKIP_WORKTREE bit set for that file in\nthe index.  This may come from:\n  * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n  * users grabbing files from elsewhere and writing them to the worktree\n    (perhaps even cached in their editor)\n  * users attempting to \"abort\" a sparse-checkout operation with a\n    not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n    working tree is not atomic)[3].\n\nOnce users have present-despite-SKIP_WORKTREE files, any modifications\nusers make to these files will be ignored, possibly to users' confusion.\n\nFurther:\n  * these files will degrade performance for the sparse-index case due\n    to requiring the index to be expanded (see commit 55dfcf9591\n    (\"sparse-checkout: clear tracked sparse dirs\", 2021-09-08) for why\n    we try to delete entire directories outside the sparse cone).\n  * these files will not be updated by by standard commands\n    (switch/checkout/pull/merge/rebase will leave them alone unless\n    conflicts happen -- and even then, the conflicted file may be\n    written somewhere else to avoid overwriting the SKIP_WORKTREE file\n    that is present and in the way)\n  * there is nothing in Git that users can use to discover such\n    files (status, diff, grep, etc. all ignore it)\n  * there is no reasonable mechanism to \"recover\" from such a condition\n    (neither `git sparse-checkout reapply` nor `git reset --hard` will\n    correct it).\n\nSo, not only are users modifications ignored, but the files get\nprogressively more stale over time.  At some point in the future, they\nmay change their sparseness specification or disable sparse-checkouts.\nAt that time, all present-despite-SKIP_WORKTREE files will show up as\nhaving lots of modifications because they represent a version from a\ndifferent branch or commit.  These might include user-made local changes\nfrom days before, but the only way to tell is to have users look through\nthem all closely.\n\nIf these users come to others for help, there will be no logs that\nexplain the issue; it's just a mysterious list of changes.  Users might\nadamantly claim (correctly, as it turns out) that they didn't modify\nthese files, while others presume they did.\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n[2] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n\n=== Problem space complexity ===\n\nSKIP_WORKTREE has been part of Git for over a decade.  Duy did lots of\nwork on it initially, and several others have since come along and put\nlots of work into it.  Stolee spent most of 2021 on the sparse-index,\nwith lots of bugfixes along the way including to non-sparse-index cases\nas we are still trying to get sparse checkouts to behave reasonably.\nBasically every codepath throughout the treat needs to be aware of an\nadditional type of file: tracked-but-not-present.  The extra type\nresults in lots of extra testcases and lots of extra code everywhere.\n\nBut, the sad thing is that we actually have more than one extra type.\nWe have tracked, tracked-but-not-present (SKIP_WORKTREE), and\ntracked-but-promised-to-not-be-present-but-is-present-anyway\n(present-despite-SKIP_WORKTREE).  Two types is a monumental amount of\neffort to support, and adding a third feels a bit like insanity[4].\n\n[4] Some examples of which can be seen at\n    https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n\n=== Maintenance and code correctness challenges ===\n\nMatheus' patches to grep stalled for nearly a year, in part because of\ncomplications of how to handle sparse-checkouts appropriately in all\ncases[5][6] (with trying to sanely figure out how to sanely handle\npresent-despite-SKIP_WORKTREE files being one of the complications).\nHis rm/add follow-ups also took months because of those kinds of\nissues[7].  The corner cases with things like submodules and\nSKIP_WORKTREE with the addition of present-despite-SKIP_WORKTREE start\nbecoming really complex[8].\n\nWe've had to add ugly logic to merge-ort to attempt to handle\npresent-despite-SKIP_WORKTREE files[9], and basically just been forced\nto give up in merge-recursive knowing full well that we'll sometimes\nsilently discard user modifications.  Despite stash essentially being a\nmerge, it needed extra code (beyond what was in merge-ort and\nmerge-recursive) to manually tweak SKIP_WORKTREE bits in order to avoid\na few different bugs that'd result in an early abort with a partial\nstash application[10].\n\n[5] See https://lore.kernel.org/git/5f3f7ac77039d41d1692ceae4b0c5df3bb45b74a.1612901326.git.matheus.bernardino@usp.br/#t\n    and the dates on the thread; also Matheus and I had several\n    conversations off-list trying to resolve the issues over that time\n[6] ...it finally kind of got unstuck after\n    https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n[7] See for example\n    https://lore.kernel.org/git/CABPp-BHwNoVnooqDFPAsZxBT9aR5Dwk5D9sDRCvYSb8akxAJgA@mail.gmail.com/#t\n    and quotes like \"The core functionality of sparse-checkout has always\n    been only partially implemented\", a statement I still believe is true\n    today.\n[8] https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n[9] See commit 66b209b86a (\"merge-ort: implement CE_SKIP_WORKTREE\n    handling with conflicted entries\", 2021-03-20)\n[10] See commit ba359fd507 (\"stash: fix stash application in\n     sparse-checkouts\", 2020-12-01)\n\n=== SKIP_WORKTREE expectations in Git ===\n\nA couple quotes:\n\nFrom [11] (before the \"sparse-checkout\" command existed):\n   If it needs too many special cases, hacks, and conditionals, then it\n   is not worth the complexity---if it is easier to write a correct code\n   by allowing Git to populate working tree files, it is perfectly fine\n   to do so.\n\n   In a sense, the sparse checkout \"feature\" itself is a hack by itself,\n   and that is why I think this part should be \"best effort\" as well.\n\nFrom the git-sparse-checkout manual (still present today):\n\n   THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n   COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n   THE FUTURE.\n\n[11] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n\n=== Suggested solution ===\n\nSKIP_WORKTREE was written to allow sparse-checkouts, in particular, as\nthe name of the option implies, to allow the file to NOT be in the\nworktree but consider it to be unchanged rather than deleted.\n\nThe suggests a simple solution: present-despite-SKIP_WORKTREE files\nshould not exist, for those using sparse-checkouts.\n\nEnforce this at index loading time by checking if core.sparseCheckout is\ntrue; if so, check files in the index with the SKIP_WORKTREE bit set to\nverify that they are absent from the working tree.  If they are present,\nunset the bit (in memory, though any commands that write to the index\nwill record the update).\n\nUsers can, of course, can get the SKIP_WORKTREE bit back such as by\nrunning `git sparse-checkout reapply` (if they have ensured the file is\nunmodified and doesn't match the specified sparsity patterns).\n\n=== Pros/Cons of suggested solution ===\n\nPros:\n\n  * Solves the user visible problems reported above, which I've been\n    complaining about for nearly a year but couldn't find a solution to.\n  * Helps prevent slow performance degradation with a sparse-index.\n  * Much easier behavior in sparse-checkouts for users to reason about\n  * Very simple, ~30 lines of code.\n  * Significantly simplifies some ugly testcases, and obviates the need\n    to test an entire class of potential issues.\n  * Reduces code complexity, reasoning, and maintenance.  Avoids\n    disagreements about weird corner cases[12].\n  * It has been reported that some users might be (ab)using\n    SKIP_WORKTREE as a let-me-modify-but-keep-the-file-in-the-worktree\n    mechanism[13, and a few other similar references].  These users know\n    of multiple caveats and shortcomings in doing so; perhaps not\n    surprising given the \"SKIP_WORKTREE expecations\" section above.\n    However, these users use `git update-index --skip-worktree`, and not\n    `git sparse-checkout` or core.sparseCheckout=true.  As such, these\n    users would be unaffected by this change and can continue abusing\n    the system as before.\n\n[12] https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[13] https://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree\n\nCons:\n\n  * When core.sparseCheckout is enabled, this adds a performance cost to\n    reading the index.  I'll defer discussion of this cost to a subsequent\n    patch, since I have some optimizations to add.\n\n=== Notes on testcase modifications ===\n\nThe good:\n  * t1011: Compare to two cases above it ('read-tree will not throw away\n    dirty changes, non-sparse'); since the file is present, it should\n    match the non-sparse case now\n  * t1092: sparse-index & sparse-checkout now match full-worktree\n    behavior in more cases!  Yaay for consistency!\n  * t6428, t7012: look at how much simpler the tests become!  Merge and\n    stash can just fail early telling the user there's a file in the\n    way, instead of not noticing until it's about to write a file and\n    then have to implement sudden crash avoidance.  Hurray for sanity!\n  * t7817: sparse behavior better matches full tree behavior.  Hurray\n    for sanity!\n\nThe confusing:\n  * t3705: These changes were ONLY needed on Windows, but they don't\n    hurt other platforms.  Let's discuss each individually:\n\n    * core.sparseCheckout should be false by default.  Nothing in this\n      testcase toggles that until many, many tests later.  However,\n      early tests (#5 in particular) were testing `update-index\n      --skip-worktree` behavior in a non-sparse-checkout, but the\n      Windows tests in CI were behaving as if core.sparseCheckout=true\n      had been specified somewhere.  I do not have access to a Windows\n      machine.  But I just manually did what should have been a no-op\n      and turned the config off.  And it fixed the test.\n    * I have no idea why the leftover .gitattributes file from this\n      test was causing failures for test #18 on Windows, but only with\n      these changes of mine.  Test #18 was checking for empty stderr,\n      and specifically wanted to know that some error completely\n      unrelated to file endings did not appear.  The leftover\n      .gitattributes file thus caused some spurious stderr unrelated to\n      the thing being checked.  Since other tests did not intend to\n      test normalization, just proactively remove the .gitattributes\n      file.  I'm certain this is cleaner and better, I'm just unsure\n      why/how this didn't trigger problems before.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n repository.c                             |  7 ++++\n sparse-index.c                           | 21 +++++++++++\n sparse-index.h                           |  1 +\n t/t1011-read-tree-sparse-checkout.sh     |  2 +-\n t/t1092-sparse-checkout-compatibility.sh | 41 ++++++++++------------\n t/t3705-add-sparse-checkout.sh           |  2 ++\n t/t6428-merge-conflicts-sparse.sh        | 23 +++----------\n t/t7012-skip-worktree-writing.sh         | 44 ++++--------------------\n t/t7817-grep-sparse-checkout.sh          | 11 ++++--\n 9 files changed, 72 insertions(+), 80 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex 34610c5a33e..dddee32258f 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -301,6 +301,13 @@ int repo_read_index(struct repository *repo)\n \tif (repo->settings.command_requires_full_index)\n \t\tensure_full_index(repo->index);\n \n+\t/*\n+\t * If sparse checkouts are in use, check whether paths with the\n+\t * SKIP_WORKTREE attribute are missing from the worktree; if not,\n+\t * clear that attribute for that path.\n+\t */\n+\tclear_skip_worktree_from_present_files(repo->index);\n+\n \treturn res;\n }\n \ndiff --git a/sparse-index.c b/sparse-index.c\nindex a1d505d50e9..b82648b10ee 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,6 +341,27 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+void clear_skip_worktree_from_present_files(struct index_state *istate)\n+{\n+\tint i;\n+\tif (!core_apply_sparse_checkout)\n+\t\treturn;\n+\n+restart:\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tstruct stat st;\n+\n+\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\t\t\tensure_full_index(istate);\n+\t\t\t\tgoto restart;\n+\t\t\t}\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t}\n+\t}\n+}\n+\n /*\n  * This static global helps avoid infinite recursion between\n  * expand_to_path() and index_file_exists().\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 656bd835b25..633d4fb7e31 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -5,6 +5,7 @@ struct index_state;\n #define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n int convert_to_sparse(struct index_state *istate, int flags);\n void ensure_correct_sparsity(struct index_state *istate);\n+void clear_skip_worktree_from_present_files(struct index_state *istate);\n \n /*\n  * Some places in the codebase expect to search for a specific path.\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex 4ed0885bf2f..dd957be1b78 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -212,7 +212,7 @@ test_expect_success 'read-tree updates worktree, dirty case' '\n \techo sub/added >.git/info/sparse-checkout &&\n \tgit checkout -f top &&\n \techo dirty >init.t &&\n-\tread_tree_u_must_succeed -m -u HEAD^ &&\n+\tread_tree_u_must_fail -m -u HEAD^ &&\n \tgrep -q dirty init.t &&\n \trm init.t\n '\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f3a059e5af5..2a04b532f91 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -367,7 +367,7 @@ test_expect_success 'status/add: outside sparse cone' '\n \twrite_script edit-contents <<-\\EOF &&\n \techo text >>$1\n \tEOF\n-\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_all ../edit-contents folder1/a &&\n \trun_on_all ../edit-contents folder1/new &&\n \n \ttest_sparse_match git status --porcelain=v2 &&\n@@ -376,8 +376,8 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n-\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_all_match git add --refresh folder1/a &&\n+\ttest_must_be_empty sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n@@ -643,11 +643,11 @@ test_expect_success 'update-index modify outside sparse definition' '\n \trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n \trun_on_all ../edit-contents folder1/a &&\n \n-\t# If file has skip-worktree enabled, update-index does not modify the\n-\t# index entry\n-\ttest_sparse_match git update-index folder1/a &&\n-\ttest_sparse_match git status --porcelain=v2 &&\n-\ttest_must_be_empty sparse-checkout-out &&\n+\t# If file has skip-worktree enabled, but the file is present, it is\n+\t# treated the same as if skip-worktree is disabled\n+\ttest_all_match git status --porcelain=v2 &&\n+\ttest_all_match git update-index folder1/a &&\n+\ttest_all_match git status --porcelain=v2 &&\n \n \t# When skip-worktree is disabled (even on files outside sparse cone), file\n \t# is updated in the index\n@@ -1331,30 +1331,27 @@ test_expect_success 'ls-files' '\n \ttest_cmp dense sparse &&\n \n \t# Set up a strange condition of having a file edit\n-\t# outside of the sparse-checkout cone. This is just\n-\t# to verify that sparse-checkout and sparse-index\n-\t# behave the same in this case.\n+\t# outside of the sparse-checkout cone. We want to verify\n+\t# that all modes handle this the same, and detect the\n+\t# modification.\n \twrite_script edit-content <<-\\EOF &&\n-\tmkdir folder1 &&\n+\tmkdir -p folder1 &&\n \techo content >>folder1/a\n \tEOF\n-\trun_on_sparse ../edit-content &&\n+\trun_on_all ../edit-content &&\n \n-\t# ls-files does not currently notice modified files whose\n-\t# cache entries are marked SKIP_WORKTREE. This may change\n-\t# in the future, but here we test that sparse index does\n-\t# not accidentally create a change of behavior.\n-\ttest_sparse_match git ls-files --modified &&\n-\ttest_must_be_empty sparse-checkout-out &&\n-\ttest_must_be_empty sparse-index-out &&\n+\ttest_all_match git ls-files --modified &&\n \n \tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n-\ttest_must_be_empty sparse-index-out &&\n+\tcat >expect <<-\\EOF &&\n+\tfolder1/a\n+\tEOF\n+\ttest_cmp expect sparse-index-out &&\n \n \t# Add folder1 to the sparse-checkout cone and\n \t# check that ls-files shows the expanded files.\n \ttest_sparse_match git sparse-checkout add folder1 &&\n-\ttest_sparse_match git ls-files --modified &&\n+\ttest_all_match git ls-files --modified &&\n \n \ttest_all_match git ls-files &&\n \tgit -C sparse-index ls-files --sparse >actual &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 81f3384eeed..95609046c61 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit config core.sparseCheckout false &&\n \tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n@@ -126,6 +127,7 @@ test_expect_success 'git add --chmod does not update sparse entries' '\n '\n \n test_expect_success 'git add --renormalize does not update sparse entries' '\n+\ttest_when_finished rm .gitattributes &&\n \ttest_config core.autocrlf false &&\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\ndiff --git a/t/t6428-merge-conflicts-sparse.sh b/t/t6428-merge-conflicts-sparse.sh\nindex 7e8bf497f82..142c9aaabc5 100755\n--- a/t/t6428-merge-conflicts-sparse.sh\n+++ b/t/t6428-merge-conflicts-sparse.sh\n@@ -112,7 +112,7 @@ test_expect_success 'conflicting entries written to worktree even if sparse' '\n \t)\n '\n \n-test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handled reasonably' '\n+test_expect_success 'present-despite-SKIP_WORKTREE handled reasonably' '\n \ttest_setup_numerals in_the_way &&\n \t(\n \t\tcd numerals_in_the_way &&\n@@ -132,26 +132,13 @@ test_expect_merge_algorithm failure success 'present-despite-SKIP_WORKTREE handl\n \n \t\ttest_must_fail git merge -s recursive B^0 &&\n \n-\t\tgit ls-files -t >index_files &&\n-\t\ttest_cmp expected-index index_files &&\n+\t\ttest_path_is_missing .git/MERGE_HEAD &&\n \n-\t\ttest_path_is_file README &&\n \t\ttest_path_is_file numerals &&\n \n-\t\ttest_cmp expected-merge numerals &&\n-\n-\t\t# There should still be a file with \"foobar\" in it\n-\t\tgrep foobar * &&\n-\n-\t\t# 5 other files:\n-\t\t#   * expected-merge\n-\t\t#   * expected-index\n-\t\t#   * index_files\n-\t\t#   * others\n-\t\t#   * whatever name was given to the numerals file that had\n-\t\t#     \"foobar\" in it\n-\t\tgit ls-files -o >others &&\n-\t\ttest_line_count = 5 others\n+\t\t# numerals should still have \"foobar\" in it\n+\t\techo foobar >expect &&\n+\t\ttest_cmp expect numerals\n \t)\n '\n \ndiff --git a/t/t7012-skip-worktree-writing.sh b/t/t7012-skip-worktree-writing.sh\nindex a1080b94e38..cb9f1a6981e 100755\n--- a/t/t7012-skip-worktree-writing.sh\n+++ b/t/t7012-skip-worktree-writing.sh\n@@ -171,50 +171,20 @@ test_expect_success 'stash restore in sparse checkout' '\n \n \t\t# Put a file in the working directory in the way\n \t\techo in the way >modified &&\n-\t\tgit stash apply &&\n+\t\ttest_must_fail git stash apply 2>error&&\n \n-\t\t# Ensure stash vivifies modifies paths...\n-\t\tcat >expect <<-EOF &&\n-\t\tH addme\n-\t\tH modified\n-\t\tH removeme\n-\t\tH subdir/A\n-\t\tS untouched\n-\t\tEOF\n-\t\tgit ls-files -t >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\tgrep \"changes.*would be overwritten by merge\" error &&\n \n-\t\t# ...and that the paths show up in status as changed...\n-\t\tcat >expect <<-EOF &&\n-\t\tA  addme\n-\t\t M modified\n-\t\t D removeme\n-\t\t M subdir/A\n-\t\t?? actual\n-\t\t?? expect\n-\t\t?? modified.stash.XXXXXX\n-\t\tEOF\n-\t\tgit status --porcelain | \\\n-\t\t\tsed -e s/stash......./stash.XXXXXX/ >actual &&\n-\t\ttest_cmp expect actual &&\n+\t\techo in the way >expect &&\n+\t\ttest_cmp expect modified &&\n+\t\tgit diff --quiet HEAD \":!modified\" &&\n \n \t\t# ...and that working directory reflects the files correctly\n-\t\ttest_path_is_file    addme &&\n+\t\ttest_path_is_missing addme &&\n \t\ttest_path_is_file    modified &&\n \t\ttest_path_is_missing removeme &&\n \t\ttest_path_is_file    subdir/A &&\n-\t\ttest_path_is_missing untouched &&\n-\n-\t\t# ...including that we have the expected \"modified\" file...\n-\t\tcat >expect <<-EOF &&\n-\t\tmodified\n-\t\ttweaked\n-\t\tEOF\n-\t\ttest_cmp expect modified &&\n-\n-\t\t# ...and that the other \"modified\" file is still present...\n-\t\techo in the way >expect &&\n-\t\ttest_cmp expect modified.stash.*\n+\t\ttest_path_is_missing untouched\n \t)\n '\n \ndiff --git a/t/t7817-grep-sparse-checkout.sh b/t/t7817-grep-sparse-checkout.sh\nindex 590b99bbb6f..eb595645657 100755\n--- a/t/t7817-grep-sparse-checkout.sh\n+++ b/t/t7817-grep-sparse-checkout.sh\n@@ -83,10 +83,13 @@ test_expect_success 'setup' '\n \n # The test below covers a special case: the sparsity patterns exclude '/b' and\n # sparse checkout is enabled, but the path exists in the working tree (e.g.\n-# manually created after `git sparse-checkout init`). git grep should skip it.\n+# manually created after `git sparse-checkout init`).  Although b is marked\n+# as SKIP_WORKTREE, git grep should notice it IS present in the worktree and\n+# report it.\n test_expect_success 'working tree grep honors sparse checkout' '\n \tcat >expect <<-EOF &&\n \ta:text\n+\tb:new-text\n \tEOF\n \ttest_when_finished \"rm -f b\" &&\n \techo \"new-text\" >b &&\n@@ -126,12 +129,16 @@ test_expect_success 'grep --cached searches entries with the SKIP_WORKTREE bit'\n '\n \n # Note that sub2/ is present in the worktree but it is excluded by the sparsity\n-# patterns, so grep should not recurse into it.\n+# patterns.  We also explicitly mark it as SKIP_WORKTREE in case it got cleared\n+# by previous git commands.  Thus sub2 starts as SKIP_WORKTREE but since it is\n+# present in the working tree, grep should recurse into it.\n test_expect_success 'grep --recurse-submodules honors sparse checkout in submodule' '\n \tcat >expect <<-EOF &&\n \ta:text\n \tsub/B/b:text\n+\tsub2/a:text\n \tEOF\n+\tgit update-index --skip-worktree sub2 &&\n \tgit grep --recurse-submodules \"text\" >actual &&\n \ttest_cmp expect actual\n '\n-- \ngitgitgadget\n\n"},{"id":"446226","messageId":"0af00779128e594aff0ee4ec5378addeac8e88a2.1642175983.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] Update documentation related to sparsity and the skip-worktree bit","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:42Z","receivedAt":"2022-01-14T15:59:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nMake several small updates, to address a few documentation issues\nI spotted:\n  * sparse-checkout focused on \"patterns\" even though the inputs (and\n    outputs in the case of `list`) are directories in cone-mode\n  * The description section of the sparse-checkout documentation\n    was a bit sparse (no pun intended), and focused more on internal\n    mechanics rather than end user usage.  This made sense in the\n    early days when the command was even more experimental, but let's\n    adjust a bit to try to make it more approachable to end users who\n    may want to consider using it.  Keep the scary backward\n    compatibility warning, though; we're still hard at work trying to\n    fix up commands to behave reasonably in sparse checkouts.\n  * both read-tree and update-index tried to describe how to use the\n    skip-worktree bit, but both predated the sparse-checkout command.\n    The sparse-checkout command is a far easier mechanism to use and\n    for users trying to reduce the size of their working tree, we\n    should recommend users to look at it instead.\n  * The update-index documentation pointed out that assume-unchanged\n    and skip-worktree sounded similar but had different purposes.\n    However, it made no attempt to explain the differences, only to\n    point out that they were different.  Explain the differences.\n  * The update-index documentation focused much more on (internal?)\n    implementation details than on end-user usage.  Try to explain\n    its purpose better for users of update-index, rather than\n    fellow developers trying to work with the SKIP_WORKTREE bit.\n  * Clarify that when core.sparseCheckout=true, we treat a file's\n    presence in the working tree as being an override to the\n    SKIP_WORKTREE bit (i.e. in sparse checkouts when the file is\n    present we ignore the SKIP_WORKTREE bit).\n\nNote that this commit, like many touching documentation, is best viewed\nwith the `--color-words` option to diff/log.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n Documentation/git-read-tree.txt       | 12 +++--\n Documentation/git-sparse-checkout.txt | 76 ++++++++++++++++-----------\n Documentation/git-update-index.txt    | 57 +++++++++++++++-----\n 3 files changed, 98 insertions(+), 47 deletions(-)\n\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex 8c3aceb8324..99bb387134d 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -375,9 +375,14 @@ have finished your work-in-progress), attempt the merge again.\n SPARSE CHECKOUT\n ---------------\n \n+Note: The `update-index` and `read-tree` primitives for supporting the\n+skip-worktree bit predated the introduction of\n+linkgit:git-sparse-checkout[1].  Users are encouraged to use\n+`sparse-checkout` in preference to these low-level primitives.\n+\n \"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to\n+tell Git whether a file in the working directory is worth looking at.\n \n 'git read-tree' and other merge-based commands ('git merge', 'git\n checkout'...) can help maintaining the skip-worktree bitmap and working\n@@ -385,7 +390,8 @@ directory update. `$GIT_DIR/info/sparse-checkout` is used to\n define the skip-worktree reference bitmap. When 'git read-tree' needs\n to update the working directory, it resets the skip-worktree bit in the index\n based on this file, which uses the same syntax as .gitignore files.\n-If an entry matches a pattern in this file, skip-worktree will not be\n+If an entry matches a pattern in this file, or the entry corresponds to\n+a file present in the working tree, then skip-worktree will not be\n set on that entry. Otherwise, skip-worktree will be set.\n \n Then it compares the new skip-worktree value with the previous one. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b81dbe06543..3da3d5a1007 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -3,9 +3,7 @@ git-sparse-checkout(1)\n \n NAME\n ----\n-git-sparse-checkout - Initialize and modify the sparse-checkout\n-configuration, which reduces the checkout to a set of paths\n-given by a list of patterns.\n+git-sparse-checkout - Reduce your working tree to a subset of tracked files\n \n \n SYNOPSIS\n@@ -17,8 +15,20 @@ SYNOPSIS\n DESCRIPTION\n -----------\n \n-Initialize and modify the sparse-checkout configuration, which reduces\n-the checkout to a set of paths given by a list of patterns.\n+This command is used to create sparse checkouts, which means that it\n+changes the working tree from having all tracked files present, to only\n+have a subset of them.  It can also switch which subset of files are\n+present, or undo and go back to having all tracked files present in the\n+working copy.\n+\n+The subset of files is chosen by providing a list of directories in\n+cone mode (which is recommended), or by providing a list of patterns\n+in non-cone mode.\n+\n+When in a sparse-checkout, other Git commands behave a bit differently.\n+For example, switching branches will not update paths outside the\n+sparse-checkout directories/patterns, and `git commit -a` will not record\n+paths outside the sparse-checkout directories/patterns as deleted.\n \n THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n@@ -28,7 +38,7 @@ THE FUTURE.\n COMMANDS\n --------\n 'list'::\n-\tDescribe the patterns in the sparse-checkout file.\n+\tDescribe the directories or patterns in the sparse-checkout file.\n \n 'set'::\n \tEnable the necessary config settings\n@@ -38,20 +48,26 @@ COMMANDS\n \tlist of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n +\n-When the `--stdin` option is provided, the patterns are read from\n-standard in as a newline-delimited list instead of from the arguments.\n+When the `--stdin` option is provided, the directories or patterns are\n+read from standard in as a newline-delimited list instead of from the\n+arguments.\n +\n When `--cone` is passed or `core.sparseCheckoutCone` is enabled, the\n-input list is considered a list of directories instead of\n-sparse-checkout patterns.  This allows for better performance with a\n-limited set of patterns (see 'CONE PATTERN SET' below).  Note that the\n-set command will write patterns to the sparse-checkout file to include\n-all files contained in those directories (recursively) as well as\n-files that are siblings of ancestor directories. The input format\n-matches the output of `git ls-tree --name-only`.  This includes\n-interpreting pathnames that begin with a double quote (\") as C-style\n-quoted strings.  This may become the default in the future; --no-cone\n-can be passed to request non-cone mode.\n+input list is considered a list of directories.  This allows for\n+better performance with a limited set of patterns (see 'CONE PATTERN\n+SET' below).  The input format matches the output of `git ls-tree\n+--name-only`.  This includes interpreting pathnames that begin with a\n+double quote (\") as C-style quoted strings.  Note that the set command\n+will write patterns to the sparse-checkout file to include all files\n+contained in those directories (recursively) as well as files that are\n+siblings of ancestor directories. This may become the default in the\n+future; --no-cone can be passed to request non-cone mode.\n++\n+When `--no-cone` is passed or `core.sparseCheckoutCone` is not enabled,\n+the input list is considered a list of patterns.  This mode is harder\n+to use and less performant, and is thus not recommended.  See the\n+\"Sparse Checkout\" section of linkgit:git-read-tree[1] and the \"Pattern\n+Set\" sections below for more details.\n +\n Use the `--[no-]sparse-index` option to use a sparse index (the\n default is to not use it).  A sparse index reduces the size of the\n@@ -69,11 +85,10 @@ understand the sparse directory entries index extension and may fail to\n interact with your repository until it is disabled.\n \n 'add'::\n-\tUpdate the sparse-checkout file to include additional patterns.\n-\tBy default, these patterns are read from the command-line arguments,\n-\tbut they can be read from stdin using the `--stdin` option. When\n-\t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n-\tas directory names as in the 'set' subcommand.\n+\tUpdate the sparse-checkout file to include additional directories\n+\t(in cone mode) or patterns (in non-cone mode).  By default, these\n+\tdirectories or patterns are read from the command-line arguments,\n+\tbut they can be read from stdin using the `--stdin` option.\n \n 'reapply'::\n \tReapply the sparsity pattern rules to paths in the working tree.\n@@ -117,13 +132,14 @@ decreased in utility.\n SPARSE CHECKOUT\n ---------------\n \n-\"Sparse checkout\" allows populating the working directory sparsely.\n-It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n-Git whether a file in the working directory is worth looking at. If\n-the skip-worktree bit is set, then the file is ignored in the working\n-directory. Git will avoid populating the contents of those files, which\n-makes a sparse checkout helpful when working in a repository with many\n-files, but only a few are important to the current user.\n+\"Sparse checkout\" allows populating the working directory sparsely.  It\n+uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell Git\n+whether a file in the working directory is worth looking at. If the\n+skip-worktree bit is set, and the file is not present in the working tree,\n+then its absence is ignored. Git will avoid populating the contents of\n+those files, which makes a sparse checkout helpful when working in a\n+repository with many files, but only a few are important to the current\n+user.\n \n The `$GIT_DIR/info/sparse-checkout` file is used to define the\n skip-worktree reference bitmap. When Git updates the working\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 2853f168d97..568dbfe76b8 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -351,6 +351,10 @@ unchanged\".  Note that \"assume unchanged\" bit is *not* set if\n the index (use `git update-index --really-refresh` if you want\n to mark them as \"assume unchanged\").\n \n+Sometimes users confuse the assume-unchanged bit with the\n+skip-worktree bit.  See the final paragraph in the \"Skip-worktree bit\"\n+section below for an explanation of the differences.\n+\n \n EXAMPLES\n --------\n@@ -392,22 +396,47 @@ M foo.c\n SKIP-WORKTREE BIT\n -----------------\n \n-Skip-worktree bit can be defined in one (long) sentence: When reading\n-an entry, if it is marked as skip-worktree, then Git pretends its\n-working directory version is up to date and read the index version\n-instead.\n-\n-To elaborate, \"reading\" means checking for file existence, reading\n-file attributes or file content. The working directory version may be\n-present or absent. If present, its content may match against the index\n-version or not. Writing is not affected by this bit, content safety\n-is still first priority. Note that Git _can_ update working directory\n-file, that is marked skip-worktree, if it is safe to do so (i.e.\n-working directory version matches index version)\n+Skip-worktree bit can be defined in one (long) sentence: Tell git to\n+avoid writing the file to the working directory when reasonably\n+possible, and treat the file as unchanged when it is not\n+present in the working directory.\n+\n+Note that not all git commands will pay attention to this bit, and\n+some only partially support it.\n+\n+The update-index flags and the read-tree capabilities relating to the\n+skip-worktree bit predated the introduction of the\n+linkgit:git-sparse-checkout[1] command, which provides a much easier\n+way to configure and handle the skip-worktree bits.  If you want to\n+reduce your working tree to only deal with a subset of the files in\n+the repository, we strongly encourage the use of\n+linkgit:git-sparse-checkout[1] in preference to the low-level\n+update-index and read-tree primitives.\n+\n+The primary purpose of the skip-worktree bit is to enable sparse\n+checkouts, i.e. to have working directories with only a subset of\n+paths present.  When the skip-worktree bit is set, Git commands (such\n+as `switch`, `pull`, `merge`) will avoid writing these files.\n+However, these commands will sometimes write these files anyway in\n+important cases such as conflicts during a merge or rebase.  Git\n+commands will also avoid treating the lack of such files as an\n+intentional deletion; for example `git add -u` will not not stage a\n+deletion for these files and `git commit -a` will not make a commit\n+deleting them either.\n \n Although this bit looks similar to assume-unchanged bit, its goal is\n-different from assume-unchanged bit's. Skip-worktree also takes\n-precedence over assume-unchanged bit when both are set.\n+different.  The assume-unchanged bit is for leaving the file in the\n+working tree but having Git omit checking it for changes and presuming\n+that the file has not been changed (though if it can determine without\n+stat'ing the file that it has changed, it is free to record the\n+changes).  skip-worktree tells Git to ignore the absence of the file,\n+avoid updating it when possible with commands that normally update\n+much of the working directory (e.g. `checkout`, `switch`, `pull`,\n+etc.), and not have its absence be recorded in commits.  Note that in\n+sparse checkouts (setup by `git sparse-checkout` or by configuring\n+core.sparseCheckout to true), if a file is marked as skip-worktree in\n+the index but is found in the working tree, Git will clear the\n+skip-worktree bit for that file.\n \n SPLIT INDEX\n -----------\n-- \ngitgitgadget\n\n"},{"id":"446227","messageId":"05ac964e630a2e72eebaa1818a8807cd7a7d4f7e.1642175983.git.gitgitgadget@gmail.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-01-14T15:59:43Z","receivedAt":"2022-01-14T16:00:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren <newren@gmail.com>\n\nTrying to clear the skip-worktree bit from files that are present does\npresent some computational overhead, for sparse-checkouts.  (We do not\ndo the bit clearing in non-sparse-checkouts.)  Optimize it as follows:\n\nRather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\nfact that entire directories will often be missing, especially for cone\nmode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\nclear tracked sparse dirs\", 2021-09-08).  If we have already determined\nthat the parent directory of a file (or other previous ancestor) does\nnot exist, then the file cannot exist either so we do not need to\nlstat() it separately.\n\nTimings for p2000 included below, reformatted to fit in normal commit\nmessage line lengths, which compare three things:\n  * Timings before this series\n  * Timings of the unoptimized version of\n    clear_skip_worktree_from_present_files() from a few commits ago\n  * Timings after the optimization in this commit\n\n(NOTE: t/perf/ appears to have timing resolution only down to 0.01 s,\nwhich presents significant measurement error when timings only differ by\n0.01s.  I don't trust any such timings below, and yet all the optimized\nresults differ by at most 0.01s.)\n\nTest        Before Series    Unoptimized              Optimized\n-----------------------------------------------------------------------------\n*git status*\nfull-v3     0.15(0.10+0.06)  0.32(0.16+0.17) +113.3%  0.16(0.10+0.07) +6.7%\nfull-v4     0.15(0.11+0.05)  0.32(0.17+0.16) +113.3%  0.16(0.11+0.05) +6.7%\nsparse-v3   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.02+0.05) +0.0%\nsparse-v4   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.03+0.05) +0.0%\n\n*git add -A*\nfull-v3     0.40(0.30+0.07)  0.56(0.36+0.17) +40.0%   0.39(0.30+0.07) -2.5%\nfull-v4     0.37(0.28+0.07)  0.54(0.37+0.16) +45.9%   0.38(0.29+0.07) +2.7%\nsparse-v3   0.06(0.04+0.05)  0.08(0.05+0.05) +33.3%   0.06(0.05+0.04) +0.0%\nsparse-v4   0.05(0.03+0.05)  0.05(0.04+0.04) +0.0%    0.06(0.04+0.05) +20.0%\n\n*git add .*\nfull-v3     0.40(0.31+0.07)  0.57(0.37+0.17) +42.5%   0.41(0.30+0.08) +2.5%\nfull-v4     0.38(0.30+0.06)  0.55(0.37+0.16) +44.7%   0.38(0.30+0.06) +0.0%\nsparse-v3   0.06(0.04+0.05)  0.06(0.05+0.04) +0.0%    0.06(0.03+0.05) +0.0%\nsparse-v4   0.06(0.05+0.05)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.06) +0.0%\n\n*git commit -a -m A*\nfull-v3     0.41(0.32+0.06)  0.58(0.39+0.17) +41.5%   0.42(0.32+0.07) +2.4%\nfull-v4     0.39(0.30+0.07)  0.56(0.38+0.17) +43.6%   0.40(0.31+0.07) +2.6%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.04(0.03+0.05)  0.04(0.03+0.05) +0.0%    0.04(0.03+0.04) +0.0%\n\n*git checkout -f -*\nfull-v3     0.56(0.46+0.07)  0.73(0.55+0.16) +30.4%   0.57(0.47+0.08) +1.8%\nfull-v4     0.54(0.45+0.07)  0.71(0.53+0.17) +31.5%   0.55(0.45+0.07) +1.9%\nsparse-v3   0.06(0.04+0.04)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.05) +0.0%\nsparse-v4   0.05(0.05+0.04)  0.05(0.04+0.05) +0.0%    0.06(0.04+0.05) +20.0%\n\n*git reset*\nfull-v3     0.34(0.26+0.05)  0.51(0.34+0.15) +50.0%   0.34(0.26+0.06) +0.0%\nfull-v4     0.32(0.24+0.06)  0.49(0.32+0.15) +53.1%   0.33(0.25+0.06) +3.1%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.03(0.03+0.04)  0.03(0.02+0.04) +0.0%    0.03(0.03+0.04) +0.0%\n\n*git reset --hard*\nfull-v3     0.57(0.46+0.07)  0.90(0.61+0.25) +57.9%   0.57(0.45+0.08) +0.0%\nfull-v4     0.54(0.46+0.05)  0.88(0.59+0.26) +63.0%   0.55(0.45+0.07) +1.9%\nsparse-v3   0.07(0.03+0.03)  0.07(0.04+0.03) +0.0%    0.07(0.03+0.03) +0.0%\nsparse-v4   0.06(0.03+0.03)  0.06(0.04+0.02) +0.0%    0.06(0.03+0.03) +0.0%\n\n*git reset -- does-not-exist*\nfull-v3     0.35(0.27+0.06)  0.52(0.32+0.17) +48.6%   0.35(0.27+0.06) +0.0%\nfull-v4     0.33(0.26+0.05)  0.50(0.33+0.15) +51.5%   0.33(0.26+0.06) +0.0%\nsparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\nsparse-v4   0.04(0.02+0.04)  0.03(0.02+0.04) -25.0%   0.03(0.02+0.04) -25.0%\n\n*git diff*\nfull-v3     0.07(0.04+0.04)  0.24(0.11+0.14) +242.9%  0.07(0.04+0.04) +0.0%\nfull-v4     0.07(0.03+0.05)  0.24(0.13+0.12) +242.9%  0.08(0.04+0.05) +14.3%\nsparse-v3   0.02(0.01+0.04)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.05) +0.0%\nsparse-v4   0.02(0.02+0.03)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.04) +0.0%\n\n*git diff --cached*\nfull-v3     0.05(0.03+0.02)  0.22(0.12+0.09) +340.0%  0.05(0.03+0.01) +0.0%\nfull-v4     0.05(0.03+0.01)  0.23(0.12+0.11) +360.0%  0.05(0.03+0.02) +0.0%\nsparse-v3   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\nsparse-v4   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\n\n*git blame f2/f4/a*\nfull-v3     0.18(0.13+0.05)  0.52(0.29+0.23) +188.9%  0.19(0.15+0.04) +5.6%\nfull-v4     0.19(0.15+0.04)  0.52(0.28+0.23) +173.7%  0.19(0.14+0.04) +0.0%\nsparse-v3   0.10(0.08+0.02)  0.10(0.09+0.01) +0.0%    0.10(0.09+0.01) +0.0%\nsparse-v4   0.10(0.08+0.02)  0.10(0.08+0.02) +0.0%    0.10(0.08+0.02) +0.0%\n\n*git blame f2/f4/f3/a*\nfull-v3     0.45(0.36+0.08)  0.78(0.51+0.27) +73.3%   0.45(0.37+0.08) +0.0%\nfull-v4     0.45(0.37+0.08)  0.78(0.51+0.26) +73.3%   0.45(0.37+0.08) +0.0%\nsparse-v3   0.36(0.32+0.04)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\nsparse-v4   0.36(0.31+0.05)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\n\n*git checkout-index -f --all*\nfull-v3     0.07(0.02+0.05)  0.24(0.12+0.12) +242.9%  0.08(0.04+0.04) +14.3%\nfull-v4     0.07(0.03+0.04)  0.24(0.11+0.13) +242.9%  0.08(0.03+0.04) +14.3%\nsparse-v3   0.04(0.01+0.03)  0.04(0.00+0.03) +0.0%    0.04(0.01+0.03) +0.0%\nsparse-v4   0.04(0.01+0.02)  0.04(0.01+0.03) +0.0%    0.04(0.01+0.02) +0.0%\n\n*git update-index --add --remove f2/f4/a*\nfull-v3     0.29(0.23+0.02)  0.46(0.30+0.12) +58.6%   0.30(0.24+0.02) +3.4%\nfull-v4     0.27(0.22+0.02)  0.45(0.29+0.12) +66.7%   0.28(0.22+0.03) +3.7%\nsparse-v3   0.02(0.02+0.00)  0.02(0.01+0.00) +0.0%    0.02(0.01+0.00) +0.0%\nsparse-v4   0.02(0.02+0.00)  0.02(0.02+0.00) +0.0%    0.02(0.02+0.00) +0.0%\n\nSo, with the optimization, the extra work appears to be essentially 0\nfor sparse-checkouts that are also using sparse-indexes (even before my\noptimization), and the extra work appears to be just marginally more\nthan 0 for sparse-checkouts that are using full indexes.\n\nSigned-off-by: Elijah Newren <newren@gmail.com>\n---\n sparse-index.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 54 insertions(+), 2 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b82648b10ee..eed170cd8f7 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -341,18 +341,70 @@ void ensure_correct_sparsity(struct index_state *istate)\n \t\tensure_full_index(istate);\n }\n \n+static int path_found(const char *path, const char **dirname, size_t *dir_len,\n+\t\t      int *dir_found)\n+{\n+\tstruct stat st;\n+\tchar *newdir;\n+\tchar *tmp;\n+\n+\t/*\n+\t * If dirname corresponds to a directory that doesn't exist, and this\n+\t * path starts with dirname, then path can't exist.\n+\t */\n+\tif (!*dir_found && !memcmp(path, *dirname, *dir_len))\n+\t\treturn 0;\n+\n+\t/*\n+\t * If path itself exists, return 1.\n+\t */\n+\tif (!lstat(path, &st))\n+\t\treturn 1;\n+\n+\t/*\n+\t * Otherwise, path does not exist so we'll return 0...but we'll first\n+\t * determine some info about its parent directory so we can avoid\n+\t * lstat calls for future cache entries.\n+\t */\n+\tnewdir = strrchr(path, '/');\n+\tif (!newdir)\n+\t\treturn 0; /* Didn't find a parent dir; just return 0 now. */\n+\n+\t/*\n+\t * If path starts with directory (which we already lstat'ed and found),\n+\t * then no need to lstat parent directory again.\n+\t */\n+\tif (*dir_found && *dirname && memcmp(path, *dirname, *dir_len))\n+\t\treturn 0;\n+\n+\t/* Free previous dirname, and cache path's dirname */\n+\t*dirname = path;\n+\t*dir_len = newdir - path + 1;\n+\n+\ttmp = xstrndup(path, *dir_len);\n+\t*dir_found = !lstat(tmp, &st);\n+\tfree(tmp);\n+\n+\treturn 0;\n+}\n+\n void clear_skip_worktree_from_present_files(struct index_state *istate)\n {\n+\tconst char *last_dirname = NULL;\n+\tsize_t dir_len = 0;\n+\tint dir_found = 1;\n+\n \tint i;\n+\n \tif (!core_apply_sparse_checkout)\n \t\treturn;\n \n restart:\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tstruct stat st;\n \n-\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n+\t\tif (ce_skip_worktree(ce) &&\n+\t\t    path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n \t\t\t\tensure_full_index(istate);\n \t\t\t\tgoto restart;\n-- \ngitgitgadget\n"},{"id":"446282","messageId":"e4f45158-8fed-0d57-767d-500d4de37e9a@github.com","threadId":"57236","inReplyTo":"05ac964e630a2e72eebaa1818a8807cd7a7d4f7e.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-15T01:39:48Z","receivedAt":"2022-01-15T01:39:52Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren <newren@gmail.com>\n> \n> Trying to clear the skip-worktree bit from files that are present does\n> present some computational overhead, for sparse-checkouts.  (We do not\n> do the bit clearing in non-sparse-checkouts.)  Optimize it as follows:\n> \n> Rather than lstat()'ing every SKIP_WORKTREE path, take advantage of the\n> fact that entire directories will often be missing, especially for cone\n> mode and even more so ever since commit 55dfcf9591 (\"sparse-checkout:\n> clear tracked sparse dirs\", 2021-09-08).  If we have already determined\n> that the parent directory of a file (or other previous ancestor) does\n> not exist, then the file cannot exist either so we do not need to\n> lstat() it separately.\n> \n> Timings for p2000 included below, reformatted to fit in normal commit\n> message line lengths, which compare three things:\n>   * Timings before this series\n>   * Timings of the unoptimized version of\n>     clear_skip_worktree_from_present_files() from a few commits ago\n>   * Timings after the optimization in this commit\n> \n> (NOTE: t/perf/ appears to have timing resolution only down to 0.01 s,\n> which presents significant measurement error when timings only differ by\n> 0.01s.  I don't trust any such timings below, and yet all the optimized\n> results differ by at most 0.01s.)\n> \n> Test        Before Series    Unoptimized              Optimized\n> -----------------------------------------------------------------------------\n> *git status*\n> full-v3     0.15(0.10+0.06)  0.32(0.16+0.17) +113.3%  0.16(0.10+0.07) +6.7%\n> full-v4     0.15(0.11+0.05)  0.32(0.17+0.16) +113.3%  0.16(0.11+0.05) +6.7%\n> sparse-v3   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.02+0.05) +0.0%\n> sparse-v4   0.04(0.03+0.04)  0.04(0.02+0.05) +0.0%    0.04(0.03+0.05) +0.0%\n> \n> *git add -A*\n> full-v3     0.40(0.30+0.07)  0.56(0.36+0.17) +40.0%   0.39(0.30+0.07) -2.5%\n> full-v4     0.37(0.28+0.07)  0.54(0.37+0.16) +45.9%   0.38(0.29+0.07) +2.7%\n> sparse-v3   0.06(0.04+0.05)  0.08(0.05+0.05) +33.3%   0.06(0.05+0.04) +0.0%\n> sparse-v4   0.05(0.03+0.05)  0.05(0.04+0.04) +0.0%    0.06(0.04+0.05) +20.0%\n> \n> *git add .*\n> full-v3     0.40(0.31+0.07)  0.57(0.37+0.17) +42.5%   0.41(0.30+0.08) +2.5%\n> full-v4     0.38(0.30+0.06)  0.55(0.37+0.16) +44.7%   0.38(0.30+0.06) +0.0%\n> sparse-v3   0.06(0.04+0.05)  0.06(0.05+0.04) +0.0%    0.06(0.03+0.05) +0.0%\n> sparse-v4   0.06(0.05+0.05)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.06) +0.0%\n> \n> *git commit -a -m A*\n> full-v3     0.41(0.32+0.06)  0.58(0.39+0.17) +41.5%   0.42(0.32+0.07) +2.4%\n> full-v4     0.39(0.30+0.07)  0.56(0.38+0.17) +43.6%   0.40(0.31+0.07) +2.6%\n> sparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\n> sparse-v4   0.04(0.03+0.05)  0.04(0.03+0.05) +0.0%    0.04(0.03+0.04) +0.0%\n> \n> *git checkout -f -*\n> full-v3     0.56(0.46+0.07)  0.73(0.55+0.16) +30.4%   0.57(0.47+0.08) +1.8%\n> full-v4     0.54(0.45+0.07)  0.71(0.53+0.17) +31.5%   0.55(0.45+0.07) +1.9%\n> sparse-v3   0.06(0.04+0.04)  0.06(0.04+0.05) +0.0%    0.06(0.04+0.05) +0.0%\n> sparse-v4   0.05(0.05+0.04)  0.05(0.04+0.05) +0.0%    0.06(0.04+0.05) +20.0%\n> \n> *git reset*\n> full-v3     0.34(0.26+0.05)  0.51(0.34+0.15) +50.0%   0.34(0.26+0.06) +0.0%\n> full-v4     0.32(0.24+0.06)  0.49(0.32+0.15) +53.1%   0.33(0.25+0.06) +3.1%\n> sparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\n> sparse-v4   0.03(0.03+0.04)  0.03(0.02+0.04) +0.0%    0.03(0.03+0.04) +0.0%\n> \n> *git reset --hard*\n> full-v3     0.57(0.46+0.07)  0.90(0.61+0.25) +57.9%   0.57(0.45+0.08) +0.0%\n> full-v4     0.54(0.46+0.05)  0.88(0.59+0.26) +63.0%   0.55(0.45+0.07) +1.9%\n> sparse-v3   0.07(0.03+0.03)  0.07(0.04+0.03) +0.0%    0.07(0.03+0.03) +0.0%\n> sparse-v4   0.06(0.03+0.03)  0.06(0.04+0.02) +0.0%    0.06(0.03+0.03) +0.0%\n> \n> *git reset -- does-not-exist*\n> full-v3     0.35(0.27+0.06)  0.52(0.32+0.17) +48.6%   0.35(0.27+0.06) +0.0%\n> full-v4     0.33(0.26+0.05)  0.50(0.33+0.15) +51.5%   0.33(0.26+0.06) +0.0%\n> sparse-v3   0.04(0.03+0.04)  0.04(0.03+0.04) +0.0%    0.04(0.03+0.04) +0.0%\n> sparse-v4   0.04(0.02+0.04)  0.03(0.02+0.04) -25.0%   0.03(0.02+0.04) -25.0%\n> \n> *git diff*\n> full-v3     0.07(0.04+0.04)  0.24(0.11+0.14) +242.9%  0.07(0.04+0.04) +0.0%\n> full-v4     0.07(0.03+0.05)  0.24(0.13+0.12) +242.9%  0.08(0.04+0.05) +14.3%\n> sparse-v3   0.02(0.01+0.04)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.05) +0.0%\n> sparse-v4   0.02(0.02+0.03)  0.02(0.01+0.04) +0.0%    0.02(0.01+0.04) +0.0%\n> \n> *git diff --cached*\n> full-v3     0.05(0.03+0.02)  0.22(0.12+0.09) +340.0%  0.05(0.03+0.01) +0.0%\n> full-v4     0.05(0.03+0.01)  0.23(0.12+0.11) +360.0%  0.05(0.03+0.02) +0.0%\n> sparse-v3   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\n> sparse-v4   0.01(0.00+0.00)  0.01(0.00+0.00) +0.0%    0.01(0.00+0.00) +0.0%\n> \n> *git blame f2/f4/a*\n> full-v3     0.18(0.13+0.05)  0.52(0.29+0.23) +188.9%  0.19(0.15+0.04) +5.6%\n> full-v4     0.19(0.15+0.04)  0.52(0.28+0.23) +173.7%  0.19(0.14+0.04) +0.0%\n> sparse-v3   0.10(0.08+0.02)  0.10(0.09+0.01) +0.0%    0.10(0.09+0.01) +0.0%\n> sparse-v4   0.10(0.08+0.02)  0.10(0.08+0.02) +0.0%    0.10(0.08+0.02) +0.0%\n> \n> *git blame f2/f4/f3/a*\n> full-v3     0.45(0.36+0.08)  0.78(0.51+0.27) +73.3%   0.45(0.37+0.08) +0.0%\n> full-v4     0.45(0.37+0.08)  0.78(0.51+0.26) +73.3%   0.45(0.37+0.08) +0.0%\n> sparse-v3   0.36(0.32+0.04)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\n> sparse-v4   0.36(0.31+0.05)  0.36(0.31+0.05) +0.0%    0.36(0.31+0.04) +0.0%\n> \n> *git checkout-index -f --all*\n> full-v3     0.07(0.02+0.05)  0.24(0.12+0.12) +242.9%  0.08(0.04+0.04) +14.3%\n> full-v4     0.07(0.03+0.04)  0.24(0.11+0.13) +242.9%  0.08(0.03+0.04) +14.3%\n> sparse-v3   0.04(0.01+0.03)  0.04(0.00+0.03) +0.0%    0.04(0.01+0.03) +0.0%\n> sparse-v4   0.04(0.01+0.02)  0.04(0.01+0.03) +0.0%    0.04(0.01+0.02) +0.0%\n> \n> *git update-index --add --remove f2/f4/a*\n> full-v3     0.29(0.23+0.02)  0.46(0.30+0.12) +58.6%   0.30(0.24+0.02) +3.4%\n> full-v4     0.27(0.22+0.02)  0.45(0.29+0.12) +66.7%   0.28(0.22+0.03) +3.7%\n> sparse-v3   0.02(0.02+0.00)  0.02(0.01+0.00) +0.0%    0.02(0.01+0.00) +0.0%\n> sparse-v4   0.02(0.02+0.00)  0.02(0.02+0.00) +0.0%    0.02(0.02+0.00) +0.0%\n> \n> So, with the optimization, the extra work appears to be essentially 0\n> for sparse-checkouts that are also using sparse-indexes (even before my\n> optimization), and the extra work appears to be just marginally more\n> than 0 for sparse-checkouts that are using full indexes.\n> \n\nThank you for including these results! In the cone-mode case, I'm happy to\nsee that the optimization effectively brings the cost of correcting\nSKIP_WORKTREE down to 0. As for the unoptimized version (which, if I'm\nreading this correctly, still approximates some worst-case non-cone cases),\nI think correcting SKIP_WORKTREE is probably still worth the performance\ncost - in the relatively massive test repo used in p2000, it's still only\n~0.2-0.3s across the board.\n\n> Signed-off-by: Elijah Newren <newren@gmail.com>\n> ---\n>  sparse-index.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++--\n>  1 file changed, 54 insertions(+), 2 deletions(-)\n> \n> diff --git a/sparse-index.c b/sparse-index.c\n> index b82648b10ee..eed170cd8f7 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -341,18 +341,70 @@ void ensure_correct_sparsity(struct index_state *istate)\n>  \t\tensure_full_index(istate);\n>  }\n>  \n> +static int path_found(const char *path, const char **dirname, size_t *dir_len,\n> +\t\t      int *dir_found)\n> +{\n> +\tstruct stat st;\n> +\tchar *newdir;\n> +\tchar *tmp;\n> +\n> +\t/*\n> +\t * If dirname corresponds to a directory that doesn't exist, and this\n> +\t * path starts with dirname, then path can't exist.\n> +\t */\n> +\tif (!*dir_found && !memcmp(path, *dirname, *dir_len))\n> +\t\treturn 0;\n> +\n\nI really like how this version of the optimization works, and ultimately\nfind it simpler than the hashmap approach from the RFC. Looks good!\n\n> +\t/*\n> +\t * If path itself exists, return 1.\n> +\t */\n> +\tif (!lstat(path, &st))\n> +\t\treturn 1;\n> +\n> +\t/*\n> +\t * Otherwise, path does not exist so we'll return 0...but we'll first\n> +\t * determine some info about its parent directory so we can avoid\n> +\t * lstat calls for future cache entries.\n> +\t */\n> +\tnewdir = strrchr(path, '/');\n> +\tif (!newdir)\n> +\t\treturn 0; /* Didn't find a parent dir; just return 0 now. */\n> +\n> +\t/*\n> +\t * If path starts with directory (which we already lstat'ed and found),\n> +\t * then no need to lstat parent directory again.\n> +\t */\n> +\tif (*dir_found && *dirname && memcmp(path, *dirname, *dir_len))\n> +\t\treturn 0;\n> +\n> +\t/* Free previous dirname, and cache path's dirname */\n> +\t*dirname = path;\n> +\t*dir_len = newdir - path + 1;\n> +\n> +\ttmp = xstrndup(path, *dir_len);\n> +\t*dir_found = !lstat(tmp, &st);\n> +\tfree(tmp);\n> +\n> +\treturn 0;\n> +}\n> +\n>  void clear_skip_worktree_from_present_files(struct index_state *istate)\n>  {\n> +\tconst char *last_dirname = NULL;\n> +\tsize_t dir_len = 0;\n> +\tint dir_found = 1;\n> +\n>  \tint i;\n> +\n>  \tif (!core_apply_sparse_checkout)\n>  \t\treturn;\n>  \n>  restart:\n>  \tfor (i = 0; i < istate->cache_nr; i++) {\n>  \t\tstruct cache_entry *ce = istate->cache[i];\n> -\t\tstruct stat st;\n>  \n> -\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n> +\t\tif (ce_skip_worktree(ce) &&\n> +\t\t    path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n>  \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n>  \t\t\t\tensure_full_index(istate);\n>  \t\t\t\tgoto restart;\n\n"},{"id":"446283","messageId":"14b6dbeb-5a12-3ac0-9a0e-9cf2c3a381fe@github.com","threadId":"57236","inReplyTo":"pull.1114.v2.git.1642175983.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/5] Remove the present-despite-SKIP_WORKTREE class of bugs (for sparse-checkouts)","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-01-15T01:51:03Z","receivedAt":"2022-01-15T01:51:07Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren via GitGitGadget wrote:\n> (Maintainer note: This series builds on (v2 of) vd/sparse-clean-etc, because\n> it tweaks one of the testcases added there.)\n> \n> (Note 2: There was a previous RFC round of this series at\n> https://lore.kernel.org/git/20220109045732.2497526-1-newren@gmail.com/.)\n> \n> Files in the present-despite-SKIP_WORKTREE state have caused no ends of\n> discussions and bugs[1,2,3,4,5,6,...and lots of others]. Trying to address\n> the big issue of discovering & recovering from this state has befuddled me\n> for over a year because I was worried we'd need additional code at every\n> skip_worktree-checking path in the code (and they are all over the place),\n> and that we'd make the code significantly slower unless we plumbed a bunch\n> of additional information all over the place to allow some reasonable\n> optimizations.\n> \n> This series tries to solve the problem a bit differently by automatic early\n> discovery and recovery; as a result, it greatly simplifies the landscape,\n> reduces our testing matrix burden, and fixes a large swath of bugs. And I\n> figured out how to get the perf cost down to essentially negligible.\n> \n> Changes since v1 (or v2 if you count RFC as v1):\n> \n>  * now includes some fixes for testcases from\n>    ds/fetch-pull-with-sparse-index (which topic marked its own new tests as\n>    potentially suboptimal; with my series, the sparse behavior now matches\n>    the full tree behavior on that test. Wahoo!). Note that Junio's version\n>    of vd/sparse-clean-etc already includes ds/fetch-pull-with-sparse-index,\n>    so no need to merge in anything extra.\n> \n> Changes since RFC version:\n> \n>  * updated the commit messages as per suggestions from Victoria, including\n>    adding performance measurements\n>  * renamed the new function to use a clearer name\n>  * replaced the final patch with a different optimization, which is both\n>    simpler and performs quite a bit better (the cost for my previous patch 5\n>    was already decent in many cases, but had a few cases where the cost was\n>    significant).\n> \n> Quick overview:\n> \n>  * Patches 1 & 2 add a test to demonstrate accidental deletion of\n>    possibly-modified files, and then fix the bug.\n>  * Patch 3 is the crux of this series; a small amount of code with a huge\n>    commit message\n>  * Patch 4 updates the documentation\n>  * Patch 5 adds an optimization to reduce the performance impact of patch 3\n> \nBetwee the RFC and this version, you've made the changes I was looking for\nand answered any remaining questions I had. This series (including patches\n1, 2, and 4 - I didn't have anything substantial to say/ask/add to those)\nlooks good to me.\n\nThank you for working on this!\n\n> [1]\n> https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n> [2]\n> https://lore.kernel.org/git/CABPp-BGJ_Nvi5TmgriD9Bh6eNXE2EDq2f8e8QKXAeYG3BxZafA@mail.gmail.com/\n> [3]\n> https://lore.kernel.org/git/pull.809.git.git.1592356884310.gitgitgadget@gmail.com/\n> [4] commit 66b209b (\"merge-ort: implement CE_SKIP_WORKTREE handling with\n> conflicted entries\", 2021-03-20) [5] commit ba359fd (\"stash: fix stash\n> application in sparse-checkouts\", 2020-12-01) [6]\n> https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n> \n> Elijah Newren (5):\n>   t1011: add testcase demonstrating accidental loss of user\n>     modifications\n>   unpack-trees: fix accidental loss of user changes\n>   repo_read_index: clear SKIP_WORKTREE bit from files present in\n>     worktree\n>   Update documentation related to sparsity and the skip-worktree bit\n>   Accelerate clear_skip_worktree_from_present_files() by caching\n> \n>  Documentation/git-read-tree.txt          | 12 +++-\n>  Documentation/git-sparse-checkout.txt    | 76 ++++++++++++++----------\n>  Documentation/git-update-index.txt       | 57 +++++++++++++-----\n>  repository.c                             |  7 +++\n>  sparse-index.c                           | 73 +++++++++++++++++++++++\n>  sparse-index.h                           |  1 +\n>  t/t1011-read-tree-sparse-checkout.sh     | 23 ++++++-\n>  t/t1092-sparse-checkout-compatibility.sh | 41 ++++++-------\n>  t/t3705-add-sparse-checkout.sh           |  2 +\n>  t/t6428-merge-conflicts-sparse.sh        | 23 ++-----\n>  t/t7012-skip-worktree-writing.sh         | 44 +++-----------\n>  t/t7817-grep-sparse-checkout.sh          | 11 +++-\n>  unpack-trees.c                           |  4 +-\n>  13 files changed, 246 insertions(+), 128 deletions(-)\n> \n> \n> base-commit: 48609de3bf32befb69c40c1a2595a98dac0448b4\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1114%2Fnewren%2Ffix-present-despite-skip-worktree-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1114/newren/fix-present-despite-skip-worktree-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1114\n> \n> Range-diff vs v1:\n> \n>  1:  c553d558c2f = 1:  d50d804af4e t1011: add testcase demonstrating accidental loss of user modifications\n>  2:  1e3958576e2 = 2:  206c638fa90 unpack-trees: fix accidental loss of user changes\n>  3:  b263cc75b7d ! 3:  11d46a399d2 repo_read_index: clear SKIP_WORKTREE bit from files present in worktree\n>      @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index modi\n>        \n>        \t# When skip-worktree is disabled (even on files outside sparse cone), file\n>        \t# is updated in the index\n>      +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'ls-files' '\n>      + \ttest_cmp dense sparse &&\n>      + \n>      + \t# Set up a strange condition of having a file edit\n>      +-\t# outside of the sparse-checkout cone. This is just\n>      +-\t# to verify that sparse-checkout and sparse-index\n>      +-\t# behave the same in this case.\n>      ++\t# outside of the sparse-checkout cone. We want to verify\n>      ++\t# that all modes handle this the same, and detect the\n>      ++\t# modification.\n>      + \twrite_script edit-content <<-\\EOF &&\n>      +-\tmkdir folder1 &&\n>      ++\tmkdir -p folder1 &&\n>      + \techo content >>folder1/a\n>      + \tEOF\n>      +-\trun_on_sparse ../edit-content &&\n>      ++\trun_on_all ../edit-content &&\n>      + \n>      +-\t# ls-files does not currently notice modified files whose\n>      +-\t# cache entries are marked SKIP_WORKTREE. This may change\n>      +-\t# in the future, but here we test that sparse index does\n>      +-\t# not accidentally create a change of behavior.\n>      +-\ttest_sparse_match git ls-files --modified &&\n>      +-\ttest_must_be_empty sparse-checkout-out &&\n>      +-\ttest_must_be_empty sparse-index-out &&\n>      ++\ttest_all_match git ls-files --modified &&\n>      + \n>      + \tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n>      +-\ttest_must_be_empty sparse-index-out &&\n>      ++\tcat >expect <<-\\EOF &&\n>      ++\tfolder1/a\n>      ++\tEOF\n>      ++\ttest_cmp expect sparse-index-out &&\n>      + \n>      + \t# Add folder1 to the sparse-checkout cone and\n>      + \t# check that ls-files shows the expanded files.\n>      + \ttest_sparse_match git sparse-checkout add folder1 &&\n>      +-\ttest_sparse_match git ls-files --modified &&\n>      ++\ttest_all_match git ls-files --modified &&\n>      + \n>      + \ttest_all_match git ls-files &&\n>      + \tgit -C sparse-index ls-files --sparse >actual &&\n>       \n>        ## t/t3705-add-sparse-checkout.sh ##\n>       @@ t/t3705-add-sparse-checkout.sh: setup_sparse_entry () {\n>  4:  c74ad19616e = 4:  0af00779128 Update documentation related to sparsity and the skip-worktree bit\n>  5:  e68028ebe0a = 5:  05ac964e630 Accelerate clear_skip_worktree_from_present_files() by caching\n> \n\n"},{"id":"448562","messageId":"220216.86sfsjur8m.gmgdl@evledraar.gmail.com","threadId":"57236","inReplyTo":"d50d804af4e17ff1613ae40e3b5cf1b1bd0986a0.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/5] t1011: add testcase demonstrating accidental loss of user modifications","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:51:05Z","receivedAt":"2022-02-16T08:53:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren <newren@gmail.com>\n> [...]\n> +\ttest_path_is_file init.t &&\n> +\tgrep -q dirty init.t\n> [...]\n> +\ttest_path_is_file init.t &&\n> +\tgrep -q dirty init.t\n\nMaybe I'm missing something, but can these two just be:\n\n    grep dirty init.t\n\nI.e. won't grep report errors appropriately here, e.g.:\n    \n    $ grep foo t\n    grep: t: Is a directory\n    $ grep foo x\n    grep: x: No such file or directory\n\nThe only prior art I could find was the same pattern in your c449947a79d\n(directory rename detection: files/directories in the way of some\nrenames, 2018-04-19).\n\nIt's probably good to lose the \"-q\" too, unless this output is way too\nverbose without it. In any case the errors wouldn't be affected.\n"},{"id":"448564","messageId":"220216.86o837uqak.gmgdl@evledraar.gmail.com","threadId":"57236","inReplyTo":"11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T08:57:54Z","receivedAt":"2022-02-16T09:14:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren <newren@gmail.com>\n> [...]\n>  \t# Set up a strange condition of having a file edit\n> -\t# outside of the sparse-checkout cone. This is just\n> -\t# to verify that sparse-checkout and sparse-index\n> -\t# behave the same in this case.\n> +\t# outside of the sparse-checkout cone. We want to verify\n> +\t# that all modes handle this the same, and detect the\n> +\t# modification.\n>  \twrite_script edit-content <<-\\EOF &&\n> -\tmkdir folder1 &&\n> +\tmkdir -p folder1 &&\n>  \techo content >>folder1/a\n>  \tEOF\n> -\trun_on_sparse ../edit-content &&\n> +\trun_on_all ../edit-content &&\n\nThe end-state of this series will pass its tests with this on top, only\nthe last \"mkdir -p\" you added for the ls-files test seems to do\nanything:\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 2a04b532f91..160c119e17d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -639,7 +639,7 @@ test_expect_success 'update-index modify outside sparse definition' '\n \t# condition in which a `skip-worktree` enabled, outside-of-cone file\n \t# exists on disk. It is used here to ensure `update-index` is stable\n \t# and behaves predictably if such a condition occurs.\n-\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_sparse mkdir folder1 &&\n \trun_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n \trun_on_all ../edit-contents folder1/a &&\n \n@@ -665,7 +665,7 @@ test_expect_success 'update-index --add outside sparse definition' '\n \tEOF\n \n \t# Create folder1, add new file\n-\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_sparse mkdir folder1 &&\n \trun_on_all ../edit-contents folder1/b &&\n \n \t# The *untracked* out-of-cone file is added to the index because it does\n@@ -949,7 +949,7 @@ test_expect_success 'checkout-index outside sparse definition' '\n \n \trun_on_sparse rm -rf folder1 &&\n \techo test >new-a &&\n-\trun_on_sparse mkdir -p folder1 &&\n+\trun_on_sparse mkdir folder1 &&\n \trun_on_all cp ../new-a folder1/a &&\n \n \ttest_all_match test_must_fail git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n@@ -996,7 +996,7 @@ test_expect_success 'clean' '\n \ttest_all_match git commit -m \"ignore bogus files\" &&\n \n \trun_on_sparse mkdir folder1 &&\n-\trun_on_all mkdir -p deep/untracked-deep &&\n+\trun_on_all mkdir deep/untracked-deep &&\n \trun_on_all touch folder1/bogus &&\n \trun_on_all touch folder1/untracked &&\n \trun_on_all touch deep/untracked-deep/bogus &&\n\nA bit nit-y I guess, but I do think tests are much easier to follow when\nit's clear when we're doing initial setup v.s. using already set-up\ndata. In this case \n\nMore importantnly I think between this and 19a0acc83e4 (t1092: test\ninteresting sparse-checkout scenarios, 2021-01-23) that introduced this\npattern there's a large foot-gun being left in place here by using these\n\"run_on_all\" and \"run_on_sparse\" helpers to run POSIX tooling, as\nopposed to git itself.\n\nUtilities like \"mv\", \"rm\", \"mkdir\" etc. are differently chatty between\nplatform, and this helper captures their stdout/stderr for a later\ntest_cmp.\n\nNow, I think actually there isn't a bug *now* because we clobber the\noutput, and seem to only call test_all_match() and other test_cmp\nhelpers right after we've run \"git\", not these POSIX utilities.\n\nBut since all we want in those cases is just a \"run these commands in\nthese N dirs\" it would be good to split up the helper.\n\n\n"},{"id":"448565","messageId":"220216.86k0dvupv7.gmgdl@evledraar.gmail.com","threadId":"57236","inReplyTo":"0af00779128e594aff0ee4ec5378addeac8e88a2.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/5] Update documentation related to sparsity and the skip-worktree bit","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T09:15:25Z","receivedAt":"2022-02-16T09:24:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren <newren@gmail.com>\n>\n> Make several small updates, to address a few documentation issues\n> I spotted:\n>   * sparse-checkout focused on \"patterns\" even though the inputs (and\n>     outputs in the case of `list`) are directories in cone-mode\n>   * The description section of the sparse-checkout documentation\n>     was a bit sparse (no pun intended), and focused more on internal\n>     mechanics rather than end user usage.  This made sense in the\n>     early days when the command was even more experimental, but let's\n>     adjust a bit to try to make it more approachable to end users who\n>     may want to consider using it.  Keep the scary backward\n>     compatibility warning, though; we're still hard at work trying to\n>     fix up commands to behave reasonably in sparse checkouts.\n>   * both read-tree and update-index tried to describe how to use the\n>     skip-worktree bit, but both predated the sparse-checkout command.\n>     The sparse-checkout command is a far easier mechanism to use and\n>     for users trying to reduce the size of their working tree, we\n>     should recommend users to look at it instead.\n>   * The update-index documentation pointed out that assume-unchanged\n>     and skip-worktree sounded similar but had different purposes.\n>     However, it made no attempt to explain the differences, only to\n>     point out that they were different.  Explain the differences.\n>   * The update-index documentation focused much more on (internal?)\n>     implementation details than on end-user usage.  Try to explain\n>     its purpose better for users of update-index, rather than\n>     fellow developers trying to work with the SKIP_WORKTREE bit.\n>   * Clarify that when core.sparseCheckout=true, we treat a file's\n>     presence in the working tree as being an override to the\n>     SKIP_WORKTREE bit (i.e. in sparse checkouts when the file is\n>     present we ignore the SKIP_WORKTREE bit).\n>\n> Note that this commit, like many touching documentation, is best viewed\n> with the `--color-words` option to diff/log.\n>\n> Signed-off-by: Elijah Newren <newren@gmail.com>\n> ---\n>  Documentation/git-read-tree.txt       | 12 +++--\n>  Documentation/git-sparse-checkout.txt | 76 ++++++++++++++++-----------\n>  Documentation/git-update-index.txt    | 57 +++++++++++++++-----\n>  3 files changed, 98 insertions(+), 47 deletions(-)\n>\n> diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\n> index 8c3aceb8324..99bb387134d 100644\n> --- a/Documentation/git-read-tree.txt\n> +++ b/Documentation/git-read-tree.txt\n> @@ -375,9 +375,14 @@ have finished your work-in-progress), attempt the merge again.\n>  SPARSE CHECKOUT\n>  ---------------\n>  \n> +Note: The `update-index` and `read-tree` primitives for supporting the\n> +skip-worktree bit predated the introduction of\n> +linkgit:git-sparse-checkout[1].  Users are encouraged to use\n> +`sparse-checkout` in preference to these low-level primitives.\n\nI was honestly a bit confused about whether we were really referring to\nthe git-update-index and git-read-tree commands here, or some\nsparse-checkout (re-)usage of the same as \"primitives\", but reading it\nagain & the commit message we're just talking about the commands here.\n\nSo this really just wants to assure readers that they're advised to use\nthe shiny porcelain command instead of the plumbing.\n\nI think we should refer to these as e.g. \"linkgit:git-update-index[1]\"\nnot \"`update-index`\" here, and call them e.g. \"plumbing commands\"\ninstead of \"primitives\" here, which will address that (the reader\nwonders \"what's a primitive?\").\n\n> -Initialize and modify the sparse-checkout configuration, which reduces\n> -the checkout to a set of paths given by a list of patterns.\n> +This command is used to create sparse checkouts, which means that it\n> +changes the working tree from having all tracked files present, to only\n> +have a subset of them.  It can also switch which subset of files are\n> +present, or undo and go back to having all tracked files present in the\n> +working copy.\n\nIn terms of prose I think it's preferred to keep matter-of-fact \"Slices\nand dices apples, making them easier to eat\" instead of \"This command\nslices and dices apples, which means that it's easier to eat them\".\n\nI've forgotten what the linguisting term for that is, but it's more\nconsistent with our docs, and makes for easier reading.\n\n> +The subset of files is chosen by providing a list of directories in\n> +cone mode (which is recommended), or by providing a list of patterns\n> +in non-cone mode.\n\nAs someone with light familiarity with sparse-checkout:\n\nI'm still not sure if this is telling me that it's preferred to list\ndirectories v.s. patterns, or if it's telling me it's better to operate\nin \"cone mode\" v.s. \"non-cone mode\", or some combination of the two.\n\nIOW I think peeling out that \"(which is recommended)\" and making it\nclearly refer to which (or both?) of the two we're talking about would\nbe much better.\n\n> +When in a sparse-checkout, other Git commands behave a bit differently.\n> +For example, switching branches will not update paths outside the\n> +sparse-checkout directories/patterns, and `git commit -a` will not record\n> +paths outside the sparse-checkout directories/patterns as deleted.\n\n(I didn't read through the rest in any detail)\n"},{"id":"448566","messageId":"220216.86fsojup82.gmgdl@evledraar.gmail.com","threadId":"57236","inReplyTo":"05ac964e630a2e72eebaa1818a8807cd7a7d4f7e.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-16T09:32:12Z","receivedAt":"2022-02-16T09:37:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n\n> From: Elijah Newren <newren@gmail.com>\n> [...]\n> +static int path_found(const char *path, const char **dirname, size_t *dir_len,\n> +\t\t      int *dir_found)\n> +{\n> +\tstruct stat st;\n> +\tchar *newdir;\n> +\tchar *tmp;\n> +\n> +\t/*\n> +\t * If dirname corresponds to a directory that doesn't exist, and this\n> +\t * path starts with dirname, then path can't exist.\n> +\t */\n> +\tif (!*dir_found && !memcmp(path, *dirname, *dir_len))\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * If path itself exists, return 1.\n> +\t */\n> +\tif (!lstat(path, &st))\n> +\t\treturn 1;\n> +\n> +\t/*\n> +\t * Otherwise, path does not exist so we'll return 0...but we'll first\n> +\t * determine some info about its parent directory so we can avoid\n> +\t * lstat calls for future cache entries.\n> +\t */\n> +\tnewdir = strrchr(path, '/');\n> +\tif (!newdir)\n> +\t\treturn 0; /* Didn't find a parent dir; just return 0 now. */\n> +\n> +\t/*\n> +\t * If path starts with directory (which we already lstat'ed and found),\n> +\t * then no need to lstat parent directory again.\n> +\t */\n> +\tif (*dir_found && *dirname && memcmp(path, *dirname, *dir_len))\n> +\t\treturn 0;\n\nI really don't care/just asking, but there was a discussion on another\ntopic about guarding calls to the mem*() family when n=0:\nhttps://lore.kernel.org/git/xmqq1r24gsph.fsf@gitster.g/\n\nIs this the same sort of redundancy where we could lose the \"&&\n*dirname\" part, or is it still important because a \"\\0\" dirname would\nhave corresponding non-0 *dir_len?\n\nMore generally ... (see below)...\n\n> +\n> +\t/* Free previous dirname, and cache path's dirname */\n> +\t*dirname = path;\n> +\t*dir_len = newdir - path + 1;\n> +\n> +\ttmp = xstrndup(path, *dir_len);\n> +\t*dir_found = !lstat(tmp, &st);\n\nIn most other places we're a bit more careful about lstat() error handling, e.g.:\n    \n    builtin/init-db.c:              if (lstat(path->buf, &st_git)) {\n    builtin/init-db.c-                      if (errno != ENOENT)\n    builtin/init-db.c-                              die_errno(_(\"cannot stat '%s'\"), path->buf);\n    builtin/init-db.c-              }\n    \nShouldn't we do the same here and at least error() on return values of\n-1 with an accompanying errno that isn't ENOENT?\n\n\n> +\tfree(tmp);\n> +\n> +\treturn 0;\n> +}\n> +\n>  void clear_skip_worktree_from_present_files(struct index_state *istate)\n>  {\n> +\tconst char *last_dirname = NULL;\n> +\tsize_t dir_len = 0;\n> +\tint dir_found = 1;\n> +\n>  \tint i;\n> +\n>  \tif (!core_apply_sparse_checkout)\n>  \t\treturn;\n>  \n>  restart:\n>  \tfor (i = 0; i < istate->cache_nr; i++) {\n>  \t\tstruct cache_entry *ce = istate->cache[i];\n> -\t\tstruct stat st;\n>  \n> -\t\tif (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n> +\t\tif (ce_skip_worktree(ce) &&\n> +\t\t    path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n\n...(continued from above) is the \"path is zero\" part of this even\nreachable? I tried with this on top and ran your tests (and the rest of\nt*sparse*.sh) successfully:\n\t\n\tdiff --git a/sparse-index.c b/sparse-index.c\n\tindex eed170cd8f7..f89c944d8cd 100644\n\t--- a/sparse-index.c\n\t+++ b/sparse-index.c\n\t@@ -403,6 +403,7 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n\t \tfor (i = 0; i < istate->cache_nr; i++) {\n\t \t\tstruct cache_entry *ce = istate->cache[i];\n\t \n\t+\t\tassert(*ce->name);\n\t \t\tif (ce_skip_worktree(ce) &&\n\t \t\t    path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n\t \t\t\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n\nI.e. isn't this undue paranoia about the cache API giving us zero-length\npaths?\n"},{"id":"448606","messageId":"CABPp-BGD4m-QzQ+qqLc4QzP6-HcJDi2RAmzf+=dhxpvPjrRVEQ@mail.gmail.com","threadId":"57236","inReplyTo":"220216.86sfsjur8m.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 1/5] t1011: add testcase demonstrating accidental loss of user modifications","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-16T16:02:38Z","receivedAt":"2022-02-16T16:02:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 16, 2022 at 12:53 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren <newren@gmail.com>\n> > [...]\n> > +     test_path_is_file init.t &&\n> > +     grep -q dirty init.t\n> > [...]\n> > +     test_path_is_file init.t &&\n> > +     grep -q dirty init.t\n>\n> Maybe I'm missing something, but can these two just be:\n>\n>     grep dirty init.t\n>\n> I.e. won't grep report errors appropriately here, e.g.:\n>\n>     $ grep foo t\n>     grep: t: Is a directory\n>     $ grep foo x\n>     grep: x: No such file or directory\n>\n> The only prior art I could find was the same pattern in your c449947a79d\n> (directory rename detection: files/directories in the way of some\n> renames, 2018-04-19).\n>\n> It's probably good to lose the \"-q\" too, unless this output is way too\n> verbose without it. In any case the errors wouldn't be affected.\n\nFair point.  I believe my original test during development was just\ntest_path_is_file, then I came back later and decided I should check\nthe contents weren't lost.  A test_cmp might be a better choice to\nreplace both of these, because we don't really want to check that it\njust has \"dirty\" somewhere in the file, but that it's been left alone\nand only has those contents.\n"},{"id":"448607","messageId":"CABPp-BHsZ=TPfJ-wohpmYVMAD_3umAffNqV+EXn0xr0cDBRzpw@mail.gmail.com","threadId":"57236","inReplyTo":"220216.86o837uqak.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-16T16:08:49Z","receivedAt":"2022-02-16T16:09:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 16, 2022 at 1:14 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren <newren@gmail.com>\n> > [...]\n> >       # Set up a strange condition of having a file edit\n> > -     # outside of the sparse-checkout cone. This is just\n> > -     # to verify that sparse-checkout and sparse-index\n> > -     # behave the same in this case.\n> > +     # outside of the sparse-checkout cone. We want to verify\n> > +     # that all modes handle this the same, and detect the\n> > +     # modification.\n> >       write_script edit-content <<-\\EOF &&\n> > -     mkdir folder1 &&\n> > +     mkdir -p folder1 &&\n> >       echo content >>folder1/a\n> >       EOF\n> > -     run_on_sparse ../edit-content &&\n> > +     run_on_all ../edit-content &&\n>\n> The end-state of this series will pass its tests with this on top, only\n> the last \"mkdir -p\" you added for the ls-files test seems to do\n> anything:\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 2a04b532f91..160c119e17d 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -639,7 +639,7 @@ test_expect_success 'update-index modify outside sparse definition' '\n>         # condition in which a `skip-worktree` enabled, outside-of-cone file\n>         # exists on disk. It is used here to ensure `update-index` is stable\n>         # and behaves predictably if such a condition occurs.\n> -       run_on_sparse mkdir -p folder1 &&\n> +       run_on_sparse mkdir folder1 &&\n>         run_on_sparse cp ../initial-repo/folder1/a folder1/a &&\n>         run_on_all ../edit-contents folder1/a &&\n>\n> @@ -665,7 +665,7 @@ test_expect_success 'update-index --add outside sparse definition' '\n>         EOF\n>\n>         # Create folder1, add new file\n> -       run_on_sparse mkdir -p folder1 &&\n> +       run_on_sparse mkdir folder1 &&\n>         run_on_all ../edit-contents folder1/b &&\n>\n>         # The *untracked* out-of-cone file is added to the index because it does\n> @@ -949,7 +949,7 @@ test_expect_success 'checkout-index outside sparse definition' '\n>\n>         run_on_sparse rm -rf folder1 &&\n>         echo test >new-a &&\n> -       run_on_sparse mkdir -p folder1 &&\n> +       run_on_sparse mkdir folder1 &&\n>         run_on_all cp ../new-a folder1/a &&\n>\n>         test_all_match test_must_fail git checkout-index --ignore-skip-worktree-bits -- folder1/a &&\n> @@ -996,7 +996,7 @@ test_expect_success 'clean' '\n>         test_all_match git commit -m \"ignore bogus files\" &&\n>\n>         run_on_sparse mkdir folder1 &&\n> -       run_on_all mkdir -p deep/untracked-deep &&\n> +       run_on_all mkdir deep/untracked-deep &&\n>         run_on_all touch folder1/bogus &&\n>         run_on_all touch folder1/untracked &&\n>         run_on_all touch deep/untracked-deep/bogus &&\n>\n> A bit nit-y I guess, but I do think tests are much easier to follow when\n> it's clear when we're doing initial setup v.s. using already set-up\n> data. In this case\n>\n> More importantnly I think between this and 19a0acc83e4 (t1092: test\n> interesting sparse-checkout scenarios, 2021-01-23) that introduced this\n> pattern there's a large foot-gun being left in place here by using these\n> \"run_on_all\" and \"run_on_sparse\" helpers to run POSIX tooling, as\n> opposed to git itself.\n>\n> Utilities like \"mv\", \"rm\", \"mkdir\" etc. are differently chatty between\n> platform, and this helper captures their stdout/stderr for a later\n> test_cmp.\n>\n> Now, I think actually there isn't a bug *now* because we clobber the\n> output, and seem to only call test_all_match() and other test_cmp\n> helpers right after we've run \"git\", not these POSIX utilities.\n>\n> But since all we want in those cases is just a \"run these commands in\n> these N dirs\" it would be good to split up the helper.\n\nMaybe, but that sounds like a comment on the series from 13 months ago\nrather than this series.  It's somewhat orthogonal to what I'm doing\nhere...  ;-)\n"},{"id":"448608","messageId":"CABPp-BF2d81f0va7x7fpEAFqJtjBgEr-UBY4QCAxrnjoFmdZ3g@mail.gmail.com","threadId":"57236","inReplyTo":"220216.86k0dvupv7.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 4/5] Update documentation related to sparsity and the skip-worktree bit","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-16T16:21:31Z","receivedAt":"2022-02-16T16:21:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 16, 2022 at 1:23 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren <newren@gmail.com>\n> >\n> > Make several small updates, to address a few documentation issues\n> > I spotted:\n> >   * sparse-checkout focused on \"patterns\" even though the inputs (and\n> >     outputs in the case of `list`) are directories in cone-mode\n> >   * The description section of the sparse-checkout documentation\n> >     was a bit sparse (no pun intended), and focused more on internal\n> >     mechanics rather than end user usage.  This made sense in the\n> >     early days when the command was even more experimental, but let's\n> >     adjust a bit to try to make it more approachable to end users who\n> >     may want to consider using it.  Keep the scary backward\n> >     compatibility warning, though; we're still hard at work trying to\n> >     fix up commands to behave reasonably in sparse checkouts.\n> >   * both read-tree and update-index tried to describe how to use the\n> >     skip-worktree bit, but both predated the sparse-checkout command.\n> >     The sparse-checkout command is a far easier mechanism to use and\n> >     for users trying to reduce the size of their working tree, we\n> >     should recommend users to look at it instead.\n> >   * The update-index documentation pointed out that assume-unchanged\n> >     and skip-worktree sounded similar but had different purposes.\n> >     However, it made no attempt to explain the differences, only to\n> >     point out that they were different.  Explain the differences.\n> >   * The update-index documentation focused much more on (internal?)\n> >     implementation details than on end-user usage.  Try to explain\n> >     its purpose better for users of update-index, rather than\n> >     fellow developers trying to work with the SKIP_WORKTREE bit.\n> >   * Clarify that when core.sparseCheckout=true, we treat a file's\n> >     presence in the working tree as being an override to the\n> >     SKIP_WORKTREE bit (i.e. in sparse checkouts when the file is\n> >     present we ignore the SKIP_WORKTREE bit).\n> >\n> > Note that this commit, like many touching documentation, is best viewed\n> > with the `--color-words` option to diff/log.\n> >\n> > Signed-off-by: Elijah Newren <newren@gmail.com>\n> > ---\n> >  Documentation/git-read-tree.txt       | 12 +++--\n> >  Documentation/git-sparse-checkout.txt | 76 ++++++++++++++++-----------\n> >  Documentation/git-update-index.txt    | 57 +++++++++++++++-----\n> >  3 files changed, 98 insertions(+), 47 deletions(-)\n> >\n> > diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\n> > index 8c3aceb8324..99bb387134d 100644\n> > --- a/Documentation/git-read-tree.txt\n> > +++ b/Documentation/git-read-tree.txt\n> > @@ -375,9 +375,14 @@ have finished your work-in-progress), attempt the merge again.\n> >  SPARSE CHECKOUT\n> >  ---------------\n> >\n> > +Note: The `update-index` and `read-tree` primitives for supporting the\n> > +skip-worktree bit predated the introduction of\n> > +linkgit:git-sparse-checkout[1].  Users are encouraged to use\n> > +`sparse-checkout` in preference to these low-level primitives.\n>\n> I was honestly a bit confused about whether we were really referring to\n> the git-update-index and git-read-tree commands here, or some\n> sparse-checkout (re-)usage of the same as \"primitives\"\n\nNeither, really.\n\n>, but reading it\n> again & the commit message we're just talking about the commands here.\n\nI was trying to talk about just the small building blocks found in\ngit-update-index and git-read-tree commands that relate to sparse\ncheckouts (i.e. the --[no-]skip-worktree and\n--[no-]ignore-skip-worktree-entries options to git-update-index, and\nthe ability to update the working tree according to\n$GITDIR/info/sprse-checkout buried in `git read-tree -mu HEAD`).\n\n> So this really just wants to assure readers that they're advised to use\n> the shiny porcelain command instead of the plumbing.\n>\n> I think we should refer to these as e.g. \"linkgit:git-update-index[1]\"\n> not \"`update-index`\" here, and call them e.g. \"plumbing commands\"\n> instead of \"primitives\" here, which will address that (the reader\n> wonders \"what's a primitive?\").\n\nPerhaps:\n\nNote: The skip-worktree capabilities in linkgit:git-update-index[1]\nand `read-tree` predated the introduction of\nlinkgit:git-sparse-checkout[1].  Users are encouraged to use the\n`sparse-checkout` command in preference to these plumbing commands for\nsparse-checkout/skip-worktree related needs.\n\n?\n\n>\n> > -Initialize and modify the sparse-checkout configuration, which reduces\n> > -the checkout to a set of paths given by a list of patterns.\n> > +This command is used to create sparse checkouts, which means that it\n> > +changes the working tree from having all tracked files present, to only\n> > +have a subset of them.  It can also switch which subset of files are\n> > +present, or undo and go back to having all tracked files present in the\n> > +working copy.\n>\n> In terms of prose I think it's preferred to keep matter-of-fact \"Slices\n> and dices apples, making them easier to eat\" instead of \"This command\n> slices and dices apples, which means that it's easier to eat them\".\n>\n> I've forgotten what the linguisting term for that is, but it's more\n> consistent with our docs, and makes for easier reading.\n\nSo, s/means that it changes/change/ ?\n\n> > +The subset of files is chosen by providing a list of directories in\n> > +cone mode (which is recommended), or by providing a list of patterns\n> > +in non-cone mode.\n>\n> As someone with light familiarity with sparse-checkout:\n>\n> I'm still not sure if this is telling me that it's preferred to list\n> directories v.s. patterns, or if it's telling me it's better to operate\n> in \"cone mode\" v.s. \"non-cone mode\", or some combination of the two.\n>\n> IOW I think peeling out that \"(which is recommended)\" and making it\n> clearly refer to which (or both?) of the two we're talking about would\n> be much better.\n\nMuch easier to fix when I resubmit the patch to change the default for\nsparse-checkout to cone mode (i.e. when I resubmit\nhttps://lore.kernel.org/git/e30119b96dfaf9fdb82039cab84f8b81caacc1de.1644712798.git.gitgitgadget@gmail.com/).\nI can include the above changes too.\n\n> > +When in a sparse-checkout, other Git commands behave a bit differently.\n> > +For example, switching branches will not update paths outside the\n> > +sparse-checkout directories/patterns, and `git commit -a` will not record\n> > +paths outside the sparse-checkout directories/patterns as deleted.\n>\n> (I didn't read through the rest in any detail)\n"},{"id":"448609","messageId":"CABPp-BEog_CBEjx3FBGdUAhjwrPPDuP54HWQssAWnGeUnr0cBg@mail.gmail.com","threadId":"57236","inReplyTo":"220216.86fsojup82.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-16T16:30:42Z","receivedAt":"2022-02-16T16:31:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 16, 2022 at 1:37 AM Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n>\n> > From: Elijah Newren <newren@gmail.com>\n> > [...]\n> > +static int path_found(const char *path, const char **dirname, size_t *dir_len,\n> > +                   int *dir_found)\n> > +{\n> > +     struct stat st;\n> > +     char *newdir;\n> > +     char *tmp;\n> > +\n> > +     /*\n> > +      * If dirname corresponds to a directory that doesn't exist, and this\n> > +      * path starts with dirname, then path can't exist.\n> > +      */\n> > +     if (!*dir_found && !memcmp(path, *dirname, *dir_len))\n> > +             return 0;\n> > +\n> > +     /*\n> > +      * If path itself exists, return 1.\n> > +      */\n> > +     if (!lstat(path, &st))\n> > +             return 1;\n> > +\n> > +     /*\n> > +      * Otherwise, path does not exist so we'll return 0...but we'll first\n> > +      * determine some info about its parent directory so we can avoid\n> > +      * lstat calls for future cache entries.\n> > +      */\n> > +     newdir = strrchr(path, '/');\n> > +     if (!newdir)\n> > +             return 0; /* Didn't find a parent dir; just return 0 now. */\n> > +\n> > +     /*\n> > +      * If path starts with directory (which we already lstat'ed and found),\n> > +      * then no need to lstat parent directory again.\n> > +      */\n> > +     if (*dir_found && *dirname && memcmp(path, *dirname, *dir_len))\n> > +             return 0;\n>\n> I really don't care/just asking, but there was a discussion on another\n> topic about guarding calls to the mem*() family when n=0:\n> https://lore.kernel.org/git/xmqq1r24gsph.fsf@gitster.g/\n>\n> Is this the same sort of redundancy where we could lose the \"&&\n> *dirname\" part, or is it still important because a \"\\0\" dirname would\n> have corresponding non-0 *dir_len?\n\nNo, dirname is a char**, not a char*.  I need to make sure *dirname is\nnon-NULL before passing to memcmp or we get segfaults (and *dirname\nwill be NULL the first time it gets to this line, so the check is\ncritical).\n\n> More generally ... (see below)...\n>\n> > +\n> > +     /* Free previous dirname, and cache path's dirname */\n> > +     *dirname = path;\n> > +     *dir_len = newdir - path + 1;\n> > +\n> > +     tmp = xstrndup(path, *dir_len);\n> > +     *dir_found = !lstat(tmp, &st);\n>\n> In most other places we're a bit more careful about lstat() error handling, e.g.:\n>\n>     builtin/init-db.c:              if (lstat(path->buf, &st_git)) {\n>     builtin/init-db.c-                      if (errno != ENOENT)\n>     builtin/init-db.c-                              die_errno(_(\"cannot stat '%s'\"), path->buf);\n>     builtin/init-db.c-              }\n>\n> Shouldn't we do the same here and at least error() on return values of\n> -1 with an accompanying errno that isn't ENOENT?\n\nIf we should do that everywhere, should we have an xlstat in wrapper.[ch]?\n\n> > +     free(tmp);\n> > +\n> > +     return 0;\n> > +}\n> > +\n> >  void clear_skip_worktree_from_present_files(struct index_state *istate)\n> >  {\n> > +     const char *last_dirname = NULL;\n> > +     size_t dir_len = 0;\n> > +     int dir_found = 1;\n> > +\n> >       int i;\n> > +\n> >       if (!core_apply_sparse_checkout)\n> >               return;\n> >\n> >  restart:\n> >       for (i = 0; i < istate->cache_nr; i++) {\n> >               struct cache_entry *ce = istate->cache[i];\n> > -             struct stat st;\n> >\n> > -             if (ce_skip_worktree(ce) && !lstat(ce->name, &st)) {\n> > +             if (ce_skip_worktree(ce) &&\n> > +                 path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n>\n> ...(continued from above) is the \"path is zero\" part of this even\n> reachable? I tried with this on top and ran your tests (and the rest of\n> t*sparse*.sh) successfully:\n>\n>         diff --git a/sparse-index.c b/sparse-index.c\n>         index eed170cd8f7..f89c944d8cd 100644\n>         --- a/sparse-index.c\n>         +++ b/sparse-index.c\n>         @@ -403,6 +403,7 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n>                 for (i = 0; i < istate->cache_nr; i++) {\n>                         struct cache_entry *ce = istate->cache[i];\n>\n>         +               assert(*ce->name);\n>                         if (ce_skip_worktree(ce) &&\n>                             path_found(ce->name, &last_dirname, &dir_len, &dir_found)) {\n>                                 if (S_ISSPARSEDIR(ce->ce_mode)) {\n>\n> I.e. isn't this undue paranoia about the cache API giving us zero-length\n> paths?\n\nNope, not related at all, for two reasons: the code above was checking\nfor NULL pointers rather than NUL characters, and the argument I was\nchecking was last_dirname, not ce->name.\n"},{"id":"448658","messageId":"CABPp-BHbpO7CUXApVcEuv-Q-+_+oCTh_gmRO9UOwQPYPsbAH+A@mail.gmail.com","threadId":"57236","inReplyTo":"CABPp-BEog_CBEjx3FBGdUAhjwrPPDuP54HWQssAWnGeUnr0cBg@mail.gmail.com","subject":"Re: [PATCH v2 5/5] Accelerate clear_skip_worktree_from_present_files() by caching","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-17T04:40:48Z","receivedAt":"2022-02-17T04:41:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 16, 2022 at 8:30 AM Elijah Newren <newren@gmail.com> wrote:\n>\n> On Wed, Feb 16, 2022 at 1:37 AM Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n> >\n> > On Fri, Jan 14 2022, Elijah Newren via GitGitGadget wrote:\n> >\n...\n> > > +\n> > > +     /* Free previous dirname, and cache path's dirname */\n> > > +     *dirname = path;\n> > > +     *dir_len = newdir - path + 1;\n> > > +\n> > > +     tmp = xstrndup(path, *dir_len);\n> > > +     *dir_found = !lstat(tmp, &st);\n> >\n> > In most other places we're a bit more careful about lstat() error handling, e.g.:\n> >\n> >     builtin/init-db.c:              if (lstat(path->buf, &st_git)) {\n> >     builtin/init-db.c-                      if (errno != ENOENT)\n> >     builtin/init-db.c-                              die_errno(_(\"cannot stat '%s'\"), path->buf);\n> >     builtin/init-db.c-              }\n> >\n> > Shouldn't we do the same here and at least error() on return values of\n> > -1 with an accompanying errno that isn't ENOENT?\n>\n> If we should do that everywhere, should we have an xlstat in wrapper.[ch]?\n\nTurns out we already DO have a file_exists() wrapper function, but\nit's found in dir.c rather than wrapper.[ch].  However, it doesn't\nbother to check errno either...\n"},{"id":"448821","messageId":"YhBCsg2DCEd9FXjE@google.com","threadId":"57236","inReplyTo":"11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-19T01:06:58Z","receivedAt":"2022-02-19T01:07:05Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(cc-ing Jonathan Tan, Jose Lopes, and Jeff Hostetler, vfs experts)\nHi Elijah,\n\nElijah Newren wrote[1]:\n\n> The fix is short (~30 lines), but the description is not.  Sorry.\n>\n> There is a set of problems caused by files in what I'll refer to as the\n> \"present-despite-SKIP_WORKTREE\" state.  This commit aims to not just fix\n> these problems, but remove the entire class as a possibility -- for\n> those using sparse checkouts.  But first, we need to understand the\n> problems this class presents.  A quick outline:\n>\n>    * Problems\n>      * User facing issues\n>      * Problem space complexity\n>      * Maintenance and code correctness challenges\n>    * SKIP_WORKTREE expectations in Git\n>    * Suggested solution\n>    * Pros/Cons of suggested solution\n>    * Notes on testcase modifications\n\nThanks for a clear explanation!  This is very helpful.\n\n> === User facing issues ===\n>\n> There are various ways for users to get files to be present in the\n> working copy despite having the SKIP_WORKTREE bit set for that file in\n> the index.  This may come from:\n>   * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n>   * users grabbing files from elsewhere and writing them to the worktree\n>     (perhaps even cached in their editor)\n>   * users attempting to \"abort\" a sparse-checkout operation with a\n>     not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n>     working tree is not atomic)[3].\n>\n> Once users have present-despite-SKIP_WORKTREE files, any modifications\n> users make to these files will be ignored, possibly to users' confusion.\n[...]\n> The suggests a simple solution: present-despite-SKIP_WORKTREE files\n> should not exist, for those using sparse-checkouts.\n\nThis patch just reached \"next\", so at $DAYJOB a test for our vfsd[2]\nnoticed this change.  The trick behind a Git-based virtual filesystem\nis typically:\n\n- since we control the filesystem, we can pretend all files are\n  already present.  On access, we obtain the file content from the git\n  object store.  On write, we update the sparse-checkout pattern so\n  that Git knows to start tracking the file.\n\n- by keeping the sparse-checkout pattern narrow, we minimize the time\n  commands like \"git status\" need to spend looking for changes in\n  unmodified files.  Controlling the filesystem means we don't need to\n  worry about changes to files that don't match that pattern (since\n  any modification would also trigger a sparse-checkout pattern\n  update).\n\nIf I understand the intent behind this change correctly, it's\nincompatible with that trick.  How would you recommend handling that?\nIn the not too far away future, I'd expect something like the \"VFS\nprojection hook\" to handle this use case, but in the meantime, I would\nexpect this change to break VFS for Git performance.  A few options:\n\n a. We could guard it with a config option.  It would still be a\n    regression for VFS for Git users, but they'd be able to use the\n    config option to restore the expected behavior.  (Or\n    alternatively, such a config option could be disabled by default,\n    but I suspect that would defeat the purpose described for the\n    patch.)\n\n b. We could distinguish between the vfsd and the \"interrupted and\n    forgot to update SKIP_WORKTREE bits in the index\" cases somehow.\n    This sounds complex.\n\n c. Something else?\n\n(b) and (c) aren't sounding obviously good, so (a) seems tempting.\nWhat do you think?\n\nThanks,\nJonathan\n\n[1] https://lore.kernel.org/git/11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com/\n[2] see\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\nfor what I mean by \"vfsd\"\n"},{"id":"448857","messageId":"CABPp-BHU4VYXF8kNvZEwBLu2BYP2Q1c9dYMW_8QfNmvGjB1ZOA@mail.gmail.com","threadId":"57236","inReplyTo":"YhBCsg2DCEd9FXjE@google.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-19T16:42:56Z","receivedAt":"2022-02-19T16:43:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 18, 2022 at 5:07 PM Jonathan Nieder <jrnieder@gmail.com> wrote:\n>\n> (cc-ing Jonathan Tan, Jose Lopes, and Jeff Hostetler, vfs experts)\n> Hi Elijah,\n>\n> Elijah Newren wrote[1]:\n>\n> > The fix is short (~30 lines), but the description is not.  Sorry.\n> >\n> > There is a set of problems caused by files in what I'll refer to as the\n> > \"present-despite-SKIP_WORKTREE\" state.  This commit aims to not just fix\n> > these problems, but remove the entire class as a possibility -- for\n> > those using sparse checkouts.  But first, we need to understand the\n> > problems this class presents.  A quick outline:\n> >\n> >    * Problems\n> >      * User facing issues\n> >      * Problem space complexity\n> >      * Maintenance and code correctness challenges\n> >    * SKIP_WORKTREE expectations in Git\n> >    * Suggested solution\n> >    * Pros/Cons of suggested solution\n> >    * Notes on testcase modifications\n>\n> Thanks for a clear explanation!  This is very helpful.\n>\n> > === User facing issues ===\n> >\n> > There are various ways for users to get files to be present in the\n> > working copy despite having the SKIP_WORKTREE bit set for that file in\n> > the index.  This may come from:\n> >   * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n> >   * users grabbing files from elsewhere and writing them to the worktree\n> >     (perhaps even cached in their editor)\n> >   * users attempting to \"abort\" a sparse-checkout operation with a\n> >     not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n> >     working tree is not atomic)[3].\n> >\n> > Once users have present-despite-SKIP_WORKTREE files, any modifications\n> > users make to these files will be ignored, possibly to users' confusion.\n> [...]\n> > The suggests a simple solution: present-despite-SKIP_WORKTREE files\n> > should not exist, for those using sparse-checkouts.\n>\n> This patch just reached \"next\", so at $DAYJOB a test for our vfsd[2]\n> noticed this change.  The trick behind a Git-based virtual filesystem\n> is typically:\n>\n> - since we control the filesystem, we can pretend all files are\n>   already present.  On access, we obtain the file content from the git\n>   object store.  On write, we update the sparse-checkout pattern so\n>   that Git knows to start tracking the file.\n>\n> - by keeping the sparse-checkout pattern narrow, we minimize the time\n>   commands like \"git status\" need to spend looking for changes in\n>   unmodified files.  Controlling the filesystem means we don't need to\n>   worry about changes to files that don't match that pattern (since\n>   any modification would also trigger a sparse-checkout pattern\n>   update).\n\nSorry for the headache.\n\nLet me try to restate the problem I'm solving, then attempt to put\nyour above situation into my own words, to verify I'm understanding...\n\nSo the primary challenge I was trying to address with this patch\nseries, was that keeping the filesystem and the \"sparsity\" state\nin-sync was just *difficult*.  Things like checkout-index or even\ndiff+apply will write to the working tree without even thinking about\nupdating the sparsity state in the index to match.  And that's only\nlooking at Git-related commands.  People can write new copies of files\nmanually in a myriad of ways.  Detecting and handling that\ninconsistent state between the working tree and index wasn't as easy\nas it looked.\n\nOne way to solve this problem would be to have a vfs, where any time\nsomeone writes to a file, you update the index to clear the\nSKIP_WORKTREE bit for the written file.  Sounds like you have such a\nthing. (...or close to it, since you'd update the sparsity patterns,\nand the tooling to check mismatches of sparsity patterns and the index\nstate are good and those get updated nicely.  We just struggle to have\nsomething that correctly updates between index and working tree\nmismatches of sparsity state).  So, that basically means you wouldn't\nbenefit from this change.\n\nBut for those of us without some kind of vfs that detects writes and\nauto-updates either the sparsity patterns or the SKIP_WORKTREE bit in\nthe index, we want something that will manually check the working tree\nto see if files have been written and update the index accordingly.\n\nAnd, of course, you're trying to do more than just detect\ninconsistencies -- you want the vfs to fully control the sparsity\npatterns and expand them based on dynamic file accesses by the user.\nThat dynamic bit doesn't play well with the non-vfs workaround.\n\n\nDoes that sound right?\n\n> If I understand the intent behind this change correctly, it's\n> incompatible with that trick.  How would you recommend handling that?\n> In the not too far away future, I'd expect something like the \"VFS\n> projection hook\" to handle this use case, but in the meantime, I would\n> expect this change to break VFS for Git performance.  A few options:\n\nSide note: I thought Microsoft's vfs was first named GVFS and then\nbased on naming collisions renamed to VFS for Git.  Sounds like you\nhave something that is probably a bit different, but which you are\nalso calling VFS for Git?  Is there some potential confusion here, or\nare you banking on Microsoft eventually dropping their project?  Or\nthat you'll both keep your projects internal and not share them so the\nnaming collision doesn't matter?  Just curious...\n\n>  a. We could guard it with a config option.  It would still be a\n>     regression for VFS for Git users, but they'd be able to use the\n>     config option to restore the expected behavior.  (Or\n>     alternatively, such a config option could be disabled by default,\n>     but I suspect that would defeat the purpose described for the\n>     patch.)\n>\n>  b. We could distinguish between the vfsd and the \"interrupted and\n>     forgot to update SKIP_WORKTREE bits in the index\" cases somehow.\n>     This sounds complex.\n>\n>  c. Something else?\n>\n> (b) and (c) aren't sounding obviously good, so (a) seems tempting.\n> What do you think?\n\nYeah, I'm having a hard time coming up with a way that either the VFS\ncould recognize these special Git present-despite-skip-worktree checks\nand treat them differently, or having Git recognize that it is running\nunder a special VFS that likes to aggressively and automagically\nexpand the sparsity patterns.  So (a) seems tempting to me too.\n\nGot any name suggestions?  core.avoidPresentDespiteSkipWorktreeCheck\n(defaulting to false)?  core.sparsityConsistencyCheck (defaulting to\ntrue)?\n\nDid your team want to implement that on top of\nen/present-despite-skipped since you can verify if it works for you,\nor did you want me to take a stab at it?  Should be a pretty simple\nchange.\n\n> Thanks,\n> Jonathan\n>\n> [1] https://lore.kernel.org/git/11d46a399d26c913787b704d2b7169cafc28d639.1642175983.git.gitgitgadget@gmail.com/\n> [2] see\n> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n> for what I mean by \"vfsd\"\n"},{"id":"448868","messageId":"YhEzmdhxHC3W8ijE@google.com","threadId":"57236","inReplyTo":"CABPp-BHU4VYXF8kNvZEwBLu2BYP2Q1c9dYMW_8QfNmvGjB1ZOA@mail.gmail.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-19T18:14:49Z","receivedAt":"2022-02-19T18:14:56Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nElijah Newren wrote:\n\n> And, of course, you're trying to do more than just detect\n> inconsistencies -- you want the vfs to fully control the sparsity\n> patterns and expand them based on dynamic file accesses by the user.\n> That dynamic bit doesn't play well with the non-vfs workaround.\n>\n>\n> Does that sound right?\n\nYou captured some of it.  There's a bit more: typically when using\nsparse checkout the traditional way, you will not have files in your\nworking directory that do not match the sparse checkout pattern.  That\nway, the disk usage in the working copy is nice and small.  But with a\nvfsd like described in [2], having other files in the working\ndirectory does not cost disk usage because the corresponding data even\nin compressed (git object) form does not have to be present locally\nuntil the files are accessed.  So a vfsd gives an end user the\nillusion that all files are present, whereas git only operates on a\nsmall subset (the working set).\n\nWith this change, git would start operating on all the files.\n\n[...]\n> Side note: I thought Microsoft's vfs was first named GVFS and then\n> based on naming collisions renamed to VFS for Git.  Sounds like you\n> have something that is probably a bit different, but which you are\n> also calling VFS for Git?\n\nNo, sorry for the lack of clarity.  When I say \"VFS for Git\", I\ngenuinely mean https://vfsforgit.org/, which was authored by Microsoft\nand to my understanding is still used by Microsoft's Windows team and\nis available for anyone to use.  (That page currently returns a cert\nerror because their SSL cert expired 10 days ago.  But hopefully it\nconveys the idea, and the content is still there if you go through the\ninterstitial.)\n\nI agree that it can be kind of confusing to talk about that alongside\nVFSes in general, but I didn't choose the name. :)\n\n[...]\n> On Fri, Feb 18, 2022 at 5:07 PM Jonathan Nieder <jrnieder@gmail.com> wrote:\n\n>>  a. We could guard it with a config option.  It would still be a\n>>     regression for VFS for Git users, but they'd be able to use the\n>>     config option to restore the expected behavior.\n[...]\n>>  b. We could distinguish between the vfsd and the \"interrupted and\n>>     forgot to update SKIP_WORKTREE bits in the index\" cases somehow.\n>>     This sounds complex.\n>>\n>>  c. Something else?\n>>\n>> (b) and (c) aren't sounding obviously good, so (a) seems tempting.\n>> What do you think?\n>\n> Yeah, I'm having a hard time coming up with a way that either the VFS\n> could recognize these special Git present-despite-skip-worktree checks\n> and treat them differently, or having Git recognize that it is running\n> under a special VFS that likes to aggressively and automagically\n> expand the sparsity patterns.  So (a) seems tempting to me too.\n\nThanks.  In a way it feels like giving up (isn't it better when things\nautomagically Just Work?), but I think it's the right thing to do.\n\n> Got any name suggestions?  core.avoidPresentDespiteSkipWorktreeCheck\n> (defaulting to false)?  core.sparsityConsistencyCheck (defaulting to\n> true)?\n>\n> Did your team want to implement that on top of\n> en/present-despite-skipped since you can verify if it works for you,\n> or did you want me to take a stab at it?  Should be a pretty simple\n> change.\n\nMonday is a holiday here; it shouldn't be hard to get a patch out\nlater this week.  Happy to write one but I also won't be at all offended\nif someone else writes it first.\n\nIdeally the config name should describe the intent from the user's\nperspective instead of the implementation.  So something like\n\"sparsecheckout.expectFilesOutsideSparsePattern\".\n\nThanks,\nJonathan\n\n>> [2] see\n>> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n>> for what I mean by \"vfsd\"\n"},{"id":"448879","messageId":"CABPp-BFEt1zxR6xoZmwmY2dtXkK+ti+g-1DHZCgeP2VpzWqqyA@mail.gmail.com","threadId":"57236","inReplyTo":"YhEzmdhxHC3W8ijE@google.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-20T05:28:59Z","receivedAt":"2022-02-20T05:29:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Feb 19, 2022 at 10:14 AM Jonathan Nieder <jrnieder@gmail.com> wrote:\n>\n> Hi,\n>\n> Elijah Newren wrote:\n>\n> > And, of course, you're trying to do more than just detect\n> > inconsistencies -- you want the vfs to fully control the sparsity\n> > patterns and expand them based on dynamic file accesses by the user.\n> > That dynamic bit doesn't play well with the non-vfs workaround.\n> >\n> >\n> > Does that sound right?\n>\n> You captured some of it.  There's a bit more: typically when using\n> sparse checkout the traditional way, you will not have files in your\n> working directory that do not match the sparse checkout pattern.  That\n> way, the disk usage in the working copy is nice and small.  But with a\n> vfsd like described in [2], having other files in the working\n> directory does not cost disk usage because the corresponding data even\n> in compressed (git object) form does not have to be present locally\n> until the files are accessed.  So a vfsd gives an end user the\n> illusion that all files are present, whereas git only operates on a\n> small subset (the working set).\n>\n> With this change, git would start operating on all the files.\n>\n> [...]\n> > Side note: I thought Microsoft's vfs was first named GVFS and then\n> > based on naming collisions renamed to VFS for Git.  Sounds like you\n> > have something that is probably a bit different, but which you are\n> > also calling VFS for Git?\n>\n> No, sorry for the lack of clarity.  When I say \"VFS for Git\", I\n> genuinely mean https://vfsforgit.org/, which was authored by Microsoft\n> and to my understanding is still used by Microsoft's Windows team and\n> is available for anyone to use.  (That page currently returns a cert\n> error because their SSL cert expired 10 days ago.  But hopefully it\n> conveys the idea, and the content is still there if you go through the\n> interstitial.)\n>\n> I agree that it can be kind of confusing to talk about that alongside\n> VFSes in general, but I didn't choose the name. :)\n>\n> [...]\n> > On Fri, Feb 18, 2022 at 5:07 PM Jonathan Nieder <jrnieder@gmail.com> wrote:\n>\n> >>  a. We could guard it with a config option.  It would still be a\n> >>     regression for VFS for Git users, but they'd be able to use the\n> >>     config option to restore the expected behavior.\n> [...]\n> >>  b. We could distinguish between the vfsd and the \"interrupted and\n> >>     forgot to update SKIP_WORKTREE bits in the index\" cases somehow.\n> >>     This sounds complex.\n> >>\n> >>  c. Something else?\n> >>\n> >> (b) and (c) aren't sounding obviously good, so (a) seems tempting.\n> >> What do you think?\n> >\n> > Yeah, I'm having a hard time coming up with a way that either the VFS\n> > could recognize these special Git present-despite-skip-worktree checks\n> > and treat them differently, or having Git recognize that it is running\n> > under a special VFS that likes to aggressively and automagically\n> > expand the sparsity patterns.  So (a) seems tempting to me too.\n>\n> Thanks.  In a way it feels like giving up (isn't it better when things\n> automagically Just Work?), but I think it's the right thing to do.\n>\n> > Got any name suggestions?  core.avoidPresentDespiteSkipWorktreeCheck\n> > (defaulting to false)?  core.sparsityConsistencyCheck (defaulting to\n> > true)?\n> >\n> > Did your team want to implement that on top of\n> > en/present-despite-skipped since you can verify if it works for you,\n> > or did you want me to take a stab at it?  Should be a pretty simple\n> > change.\n>\n> Monday is a holiday here; it shouldn't be hard to get a patch out\n> later this week.  Happy to write one but I also won't be at all offended\n> if someone else writes it first.\n>\n> Ideally the config name should describe the intent from the user's\n> perspective instead of the implementation.  So something like\n> \"sparsecheckout.expectFilesOutsideSparsePattern\".\n>\n> Thanks,\n> Jonathan\n\nI took a stab at it over here:\nhttps://lore.kernel.org/git/pull.1153.git.1645333542011.gitgitgadget@gmail.com/\n\nLet me know if that works for you.\n"},{"id":"448901","messageId":"0979ce9b-d7be-9f84-0942-201626b488a4@github.com","threadId":"57236","inReplyTo":"YhBCsg2DCEd9FXjE@google.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-20T16:56:15Z","receivedAt":"2022-02-20T16:56:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/18/2022 8:06 PM, Jonathan Nieder wrote:\n> (cc-ing Jonathan Tan, Jose Lopes, and Jeff Hostetler, vfs experts)\n> Hi Elijah,\n> \n> Elijah Newren wrote[1]:\n> \n>> The fix is short (~30 lines), but the description is not.  Sorry.\n>>\n>> There is a set of problems caused by files in what I'll refer to as the\n>> \"present-despite-SKIP_WORKTREE\" state.  This commit aims to not just fix\n>> these problems, but remove the entire class as a possibility -- for\n>> those using sparse checkouts.  But first, we need to understand the\n>> problems this class presents.  A quick outline:\n>>\n>>    * Problems\n>>      * User facing issues\n>>      * Problem space complexity\n>>      * Maintenance and code correctness challenges\n>>    * SKIP_WORKTREE expectations in Git\n>>    * Suggested solution\n>>    * Pros/Cons of suggested solution\n>>    * Notes on testcase modifications\n> \n> Thanks for a clear explanation!  This is very helpful.\n> \n>> === User facing issues ===\n>>\n>> There are various ways for users to get files to be present in the\n>> working copy despite having the SKIP_WORKTREE bit set for that file in\n>> the index.  This may come from:\n>>   * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n>>   * users grabbing files from elsewhere and writing them to the worktree\n>>     (perhaps even cached in their editor)\n>>   * users attempting to \"abort\" a sparse-checkout operation with a\n>>     not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n>>     working tree is not atomic)[3].\n>>\n>> Once users have present-despite-SKIP_WORKTREE files, any modifications\n>> users make to these files will be ignored, possibly to users' confusion.\n> [...]\n>> The suggests a simple solution: present-despite-SKIP_WORKTREE files\n>> should not exist, for those using sparse-checkouts.\n> \n> This patch just reached \"next\", so at $DAYJOB a test for our vfsd[2]\n> noticed this change.  The trick behind a Git-based virtual filesystem\n> is typically:\n> \n> - since we control the filesystem, we can pretend all files are\n>   already present.  On access, we obtain the file content from the git\n>   object store.  On write, we update the sparse-checkout pattern so\n>   that Git knows to start tracking the file.\n> \n> - by keeping the sparse-checkout pattern narrow, we minimize the time\n>   commands like \"git status\" need to spend looking for changes in\n>   unmodified files.  Controlling the filesystem means we don't need to\n>   worry about changes to files that don't match that pattern (since\n>   any modification would also trigger a sparse-checkout pattern\n>   update).\n> \n> If I understand the intent behind this change correctly, it's\n> incompatible with that trick.  How would you recommend handling that?\n> In the not too far away future, I'd expect something like the \"VFS\n> projection hook\" to handle this use case, but in the meantime, I would\n> expect this change to break VFS for Git performance.  A few options:\n> \n>  a. We could guard it with a config option.  It would still be a\n>     regression for VFS for Git users, but they'd be able to use the\n>     config option to restore the expected behavior.  (Or\n>     alternatively, such a config option could be disabled by default,\n>     but I suspect that would defeat the purpose described for the\n>     patch.)\n> \n>  b. We could distinguish between the vfsd and the \"interrupted and\n>     forgot to update SKIP_WORKTREE bits in the index\" cases somehow.\n>     This sounds complex.\n> \n>  c. Something else?\n> \n> (b) and (c) aren't sounding obviously good, so (a) seems tempting.\n> What do you think?\n\nJust chiming in here to say that we've dealt with these issues in\nmicrosoft/git by special-casing them behind our core_virtualfilesystem\nglobal. A recent example is the changes to 'git add' to prevent\nadding a file that is marked as sparse (unless --sparse is specified).\nWe always allow this when in the virtualized scenario [1].\n\n[1] https://github.com/microsoft/git/blob/2f6531aced2e77a6c1000a923967ae0105383930/builtin/add.c#L50-L54\n\nThis seems like something that should be on vfsd to handle, and should\nnot prevent upstream Git from making changes that benefit its users.\n\nThanks,\n-Stolee\n"},{"id":"449201","messageId":"YhVu7GH/tcVmao66@google.com","threadId":"57236","inReplyTo":"0979ce9b-d7be-9f84-0942-201626b488a4@github.com","subject":"Re: [PATCH v2 3/5] repo_read_index: clear SKIP_WORKTREE bit from files present in worktree","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-22T23:17:00Z","receivedAt":"2022-02-22T23:17:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDerrick Stolee wrote:\n\n> Just chiming in here to say that we've dealt with these issues in\n> microsoft/git by special-casing them behind our core_virtualfilesystem\n> global. A recent example is the changes to 'git add' to prevent\n> adding a file that is marked as sparse (unless --sparse is specified).\n> We always allow this when in the virtualized scenario [1].\n>\n> [1] https://github.com/microsoft/git/blob/2f6531aced2e77a6c1000a923967ae0105383930/builtin/add.c#L50-L54\n\nOh!  Thanks for that pointer --- it's very helpful.  I hadn't realized\nthat VFS for Git still requires a patched Git.\n\n> This seems like something that should be on vfsd to handle, and should\n> not prevent upstream Git from making changes that benefit its users.\n\nSeparate from the \"should\" questions (in general I'm in favor of\nunpatched Git being able to serve as a solid platform for both Scalar\nand this kind of VFS use case), this context tells me that this is not\nin fact a regression for VFS for Git users, so it's much appreciated.\n\nThanks,\nJonathan\n"}]}