{"thread":{"id":"56179","subject":"[PATCH 0/2] Sparse index: delete ignored files outside sparse cone","startedAt":"2021-07-29T17:27:25Z","lastAt":"2021-09-08T16:12:01Z","messageCount":92,"participants":["Derrick Stolee via GitGitGadget","Elijah Newren","Derrick Stolee","Johannes Schindelin","Eric Sunshine","René Scharfe","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"431499","messageId":"pull.1009.git.1627579637.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":null,"subject":"[PATCH 0/2] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T17:27:14Z","receivedAt":"2021-07-29T17:27:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"My commits are technically based on ds/add-with-sparse-index, but could\napply earlier, I believe.\n\nWe launched an experimental release [1] of the sparse-index feature to our\ninternal users. We immediately discovered a problem due to the isolated way\nin which we tested the sparse index: we were never building the project and\nchanging our sparse-checkout definition.\n\n[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n\nUsers who ran a build in one project and then moved to another still had\nbuild artifacts in their worktree that lived inside the old directories.\nSince the files are marked by the .gitignore patterns, these files were not\nremoved by the 'git sparse-checkout set' command. However, they make the\nsparse-index unusable because every 'git status' command needs to expand the\nsparse-directory entries in order to see if the files are tracked or not.\nThis made the first experimental release actually slower for all users\nbecause of this cost.\n\nThe solution we shipped to these customers was to change the way our fork\nhandles these ignored files. Specifically, instead of Git completely\nignoring the files, we changed Git to understand that with cone-mode\nsparse-checkout patterns, the users is asking for entire directories to be\nremoved from the worktree. The link [1] included earlier has this change.\n\nI believe that this is a reasonable expectation, though I recognize that it\nmight look like breaking the expectations of how .gitignore files work.\n\nTo limit the scope of this change, the focus is limited to behave this way\nonly when the sparse-index is enabled, because otherwise the sparse index\ndoes not behave as intended. With that restriction in mind, the\nimplementation can use the sparse directory entries to assist.\n\nI imagine that when the sparse index is mature enough, we will enable it by\ndefault when cone mode sparse-checkout is enabled, in which case this\nbehavior would expand to all users with cone mode sparse-checkout.\n\nI'm interested in the community's thoughts about this change, as it seems\nlike one that we should make carefully and intentionally.\n\nWhile the rewrite of the t7519 test seems unrelated, it is required to avoid\na test failure with this change that deletes files outside of the cone. By\nmoving the test into repositories not at $TRASH_DIRECTORY, we gain more\ncontrol over the repository structure.\n\nThanks, -Stolee\n\nDerrick Stolee (2):\n t7519: rewrite sparse index test\n sparse-checkout: clear tracked sparse dirs\n\n Documentation/git-sparse-checkout.txt | 6 +++\n builtin/sparse-checkout.c | 73 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 42 +++++++++++++++\n t/t7519-status-fsmonitor.sh | 38 +++++++-------\n 4 files changed, 142 insertions(+), 17 deletions(-)\n\n\nbase-commit: adf5b15ac3d44d92e0438451ef36631ed3ee2a63\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1009\n-- \ngitgitgadget\n"},{"id":"431500","messageId":"29f1633f6d5689f70e689ac5210e550be6b7ee7a.1627579637.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.git.1627579637.gitgitgadget@gmail.com","subject":"[PATCH 1/2] t7519: rewrite sparse index test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T17:27:15Z","receivedAt":"2021-07-29T17:27:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse index is tested with the FS Monitor hook and extension since\nf8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\nwas very fragile because it shared an index across sparse and non-sparse\nbehavior. Since that expansion and contraction could cause the index to\nlose its FS Monitor bitmap and token, behavior is fragile to changes in\n'git sparse-checkout set'.\n\nRewrite the test to use two clones of the original repo: full and\nsparse. This allows us to also keep the test files (actual, expect,\ntrace2.txt) out of the repos we are testing with 'git status'.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex deea88d4431..6f2cf306f66 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n check_sparse_index_behavior () {\n-\tgit status --porcelain=v2 >expect &&\n-\tgit sparse-checkout init --cone --sparse-index &&\n-\tgit sparse-checkout set dir1 dir2 &&\n+\tgit -C full status --porcelain=v2 >expect &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit status --porcelain=v2 >actual &&\n+\t\tgit -C sparse status --porcelain=v2 >actual &&\n \ttest_region $1 index ensure_full_index trace2.txt &&\n \ttest_region fsm_hook query trace2.txt &&\n \ttest_cmp expect actual &&\n-\trm trace2.txt &&\n-\tgit sparse-checkout disable\n+\trm trace2.txt\n }\n \n test_expect_success 'status succeeds with sparse index' '\n-\tgit reset --hard &&\n+\tgit clone . full &&\n+\tgit clone . sparse &&\n+\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n-\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n-\tcheck_sparse_index_behavior ! &&\n-\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \tEOF\n-\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n+\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n \tcheck_sparse_index_behavior ! &&\n \n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1/modified\\0\"\n \tEOF\n \tcheck_sparse_index_behavior ! &&\n \n-\tcp -r dir1 dir1a &&\n-\tgit add dir1a &&\n-\tgit commit -m \"add dir1a\" &&\n+\tgit -C sparse sparse-checkout add dir1a &&\n+\n+\tfor repo in full sparse\n+\tdo\n+\t\tcp -r $repo/dir1 $repo/dir1a &&\n+\t\tgit -C $repo add dir1a &&\n+\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n+\tdone &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n \t# This one modifies outside the sparse-checkout definition\n \t# and hence we expect to expand the sparse-index.\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1a/modified\\0\"\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"431501","messageId":"9212bbf4e3cab20fe49ab8e6dd4ac0277c4f2805.1627579637.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.git.1627579637.gitgitgadget@gmail.com","subject":"[PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-07-29T17:27:16Z","receivedAt":"2021-07-29T17:27:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen changing the scope of a sparse-checkout using cone mode, we might\nhave some tracked directories go out of scope. The current logic removes\nthe tracked files from within those directories, but leaves the ignored\nfiles within those directories. This is a bit unexpected to users who\nhave given input to Git saying they don't need those directories\nanymore.\n\nThis is something that is new to the cone mode pattern type: the user\nhas explicitly said \"I want these directories and _not_ those\ndirectories.\" The typical sparse-checkout patterns more generally apply\nto \"I want files with with these patterns\" so it is natural to leave\nignored files as they are. This focus on directories in cone mode\nprovides us an opportunity to change the behavior.\n\nLeaving these ignored files in the sparse directories makes it\nimpossible to gain performance benefits in the sparse index. When we\ntrack into these directories, we need to know if the files are ignored\nor not, which might depend on the _tracked_ .gitignore file(s) within\nthe sparse directory. This depends on the indexed version of the file,\nso the sparse directory must be expanded.\n\nBy deleting the sparse directories when changing scope (or running 'git\nsparse-checkout reapply') we regain these performance benefits as if the\nrepository was in a clean state.\n\nSince these ignored files are frequently build output or helper files\nfrom IDEs, the users should not need the files now that the tracked\nfiles are removed. If the tracked files reappear, then they will have\nnewer timestamps than the build artifacts, so the artifacts will need to\nbe regenerated anyway.\n\nIf users depend on ignored files within the sparse directories, then\nthey have created a bad shape in their repository. Regardless, such\nshapes would create risk that changing the behavior for all cone mode\nusers might be too risky to take on at the moment. Since this data shape\nmakes it impossible to get performance benefits using the sparse index,\nwe limit the change to only be enabled when the sparse index is enabled.\nUsers can opt out of this behavior by disabline the sparse index.\n\nDepending on user feedback or real-world use, we might want to consider\nexpanding the behavior change to all of cone mode. Since we are\ncurrently restricting to the sparse index case, we can use the existence\nof sparse directory entries in the index as indicators of which\ndirectories should be removed.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 6 +++\n builtin/sparse-checkout.c | 73 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 42 +++++++++++++++\n 3 files changed, 121 insertions(+)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex fdcf43f87cb..c443189f418 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -210,6 +210,12 @@ case-insensitive check. This corrects for case mismatched filenames in the\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n+When the sparse index is enabled through the `index.sparse` config option,\n+the cone mode sparse-checkout patterns will also remove ignored files that\n+are not within the sparse-checkout definition. This is important behavior\n+to preserve the performance of the sparse index, but also matches that\n+cone mode patterns care about directories, not files.\n+\n \n SUBMODULES\n ----------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex a4bdd7c4940..5c03636b6ec 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -15,6 +15,7 @@\n #include \"wt-status.h\"\n #include \"quote.h\"\n #include \"sparse-index.h\"\n+#include \"run-command.h\"\n \n static const char *empty_base = \"\";\n \n@@ -100,6 +101,71 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void clean_tracked_sparse_directories(struct repository *r)\n+{\n+\tint i;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t pathlen;\n+\n+\t/*\n+\t * If we are not using cone mode patterns, then we cannot\n+\t * delete directories outside of the sparse cone.\n+\t */\n+\tif (!r || !r->index || !r->index->sparse_checkout_patterns ||\n+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn;\n+\t/*\n+\t * NEEDSWORK: For now, only use this behavior when index.sparse\n+\t * is enabled. We may want this behavior enabled whenever using\n+\t * cone mode patterns.\n+\t */\n+\tprepare_repo_settings(r);\n+\tif (!r->worktree || !r->settings.sparse_index)\n+\t\treturn;\n+\n+\t/*\n+\t * Since we now depend on the sparse index to enable this\n+\t * behavior, use it to our advantage. This process is more\n+\t * complicated without it.\n+\t */\n+\tconvert_to_sparse(r->index);\n+\n+\tstrbuf_addstr(&path, r->worktree);\n+\tstrbuf_complete(&path, '/');\n+\tpathlen = path.len;\n+\n+\tfor (i = 0; i < r->index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = r->index->cache[i];\n+\n+\t\t/*\n+\t\t * Is this a sparse directory? If so, then definitely\n+\t\t * include it. All contained content is outside of the\n+\t\t * patterns.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t repo_file_exists(r, ce->name)) {\n+\t\t\tstrbuf_setlen(&path, pathlen);\n+\t\t\tstrbuf_addstr(&path, ce->name);\n+\n+\t\t\t/*\n+\t\t\t * Removal is \"best effort\". If something blocks\n+\t\t\t * the deletion, then continue with a warning.\n+\t\t\t */\n+\t\t\tif (remove_dir_recursively(&path, 0))\n+\t\t\t\twarning(_(\"failed to remove directory '%s'\"), path.buf);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&path);\n+\n+\t/*\n+\t * This is temporary: the sparse-checkout builtin is not\n+\t * integrated with the sparse-index yet, so we need to keep\n+\t * it full during the process.\n+\t */\n+\tensure_full_index(r->index);\n+}\n+\n static int update_working_directory(struct pattern_list *pl)\n {\n \tenum update_sparsity_result result;\n@@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)\n \telse\n \t\trollback_lock_file(&lock_file);\n \n+\tclean_tracked_sparse_directories(r);\n+\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\n@@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n {\n \tint result;\n \tint changed_config = 0;\n+\tstruct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));\n \tstruct pattern_list *pl = xcalloc(1, sizeof(*pl));\n \n+\tget_sparse_checkout_patterns(old_pl);\n+\n \tswitch (m) {\n \tcase ADD:\n \t\tif (core_sparse_checkout_cone)\n@@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n \t\tset_config(MODE_NO_PATTERNS);\n \n \tclear_pattern_list(pl);\n+\tclear_pattern_list(old_pl);\n \tfree(pl);\n+\tfree(old_pl);\n \treturn result;\n }\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 38fc8340f5c..43eb314c94a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -642,4 +642,46 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n \tcheck_files repo/deep a deeper1\n '\n \n+test_expect_success 'cone mode clears ignored subdirectories' '\n+\trm repo/.git/info/sparse-checkout &&\n+\n+\t# NEEDSWORK: --sparse-index is required for now\n+\tgit -C repo sparse-checkout init --cone --sparse-index &&\n+\tgit -C repo sparse-checkout set deep/deeper1 &&\n+\n+\tcat >repo/.gitignore <<-\\EOF &&\n+\tobj/\n+\t*.o\n+\tEOF\n+\n+\tgit -C repo add .gitignore &&\n+\tgit -C repo commit -m \".gitignore\" &&\n+\n+\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n+\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n+\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n+\tdo\n+\t\techo ignored >repo/$file || return 1\n+\tdone &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout reapply &&\n+\ttest_path_is_missing repo/folder1 &&\n+\ttest_path_is_missing repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper2 &&\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"431564","messageId":"CABPp-BH1jw6MakBeFADybeEV6LTDRS1OMvqGwMYiLoTBdG_L1w@mail.gmail.com","threadId":"56179","inReplyTo":"pull.1009.git.1627579637.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-30T13:11:43Z","receivedAt":"2021-07-30T13:11:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> My commits are technically based on ds/add-with-sparse-index, but could\n> apply earlier, I believe.\n>\n> We launched an experimental release [1] of the sparse-index feature to our\n> internal users. We immediately discovered a problem due to the isolated way\n> in which we tested the sparse index: we were never building the project and\n> changing our sparse-checkout definition.\n>\n> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n>\n> Users who ran a build in one project and then moved to another still had\n> build artifacts in their worktree that lived inside the old directories.\n> Since the files are marked by the .gitignore patterns, these files were not\n> removed by the 'git sparse-checkout set' command. However, they make the\n> sparse-index unusable because every 'git status' command needs to expand the\n> sparse-directory entries in order to see if the files are tracked or not.\n> This made the first experimental release actually slower for all users\n> because of this cost.\n\nAs highlighted at\nhttps://lore.kernel.org/git/CABPp-BHwRHhYE3BPxCG+fRFWf2dCZ60rnO=ykztquKoOp_x_QQ@mail.gmail.com/\n?\n\n\"\"\"So, although 'git status'\nshouldn't have to check the tracked files anymore, it is still going\nto have to look at each of the *untracked* files and compare to\n.gitignore files in order to correctly classify each file as ignored\nor just plain untracked.\"\"\"\n\nAs I noted there,\n\n\"\"\"about a month ago my\ncolleagues made the tool just auto-invoke [--remove-old-ignores] from other\nsparsify invocations. To my knowledge, there have been no complaints\nabout that being automatically turned on; but there were\ncomplaints/confusion before about the directories being left around.\n(Of course, non-ignored files are still left around by that option.)\"\"\"\n\nI think your changes here shouldn't be limited to a sparse-index but\nshould apply to cone-mode sparse checkouts generally.\n\n> The solution we shipped to these customers was to change the way our fork\n> handles these ignored files. Specifically, instead of Git completely\n> ignoring the files, we changed Git to understand that with cone-mode\n> sparse-checkout patterns, the users is asking for entire directories to be\n> removed from the worktree. The link [1] included earlier has this change.\n\nCould you clarify what you are removing?\n 1) Tracked but now SKIP_WOKRTREE files\n 2) .gitignore'd files\n 3) other untracked files\n\n(1) has always been removed by sparse-checkout. I think you're adding\n(2) but it's not clear if you are also removing (3).\n\n> I believe that this is a reasonable expectation, though I recognize that it\n> might look like breaking the expectations of how .gitignore files work.\n>\n> To limit the scope of this change, the focus is limited to behave this way\n> only when the sparse-index is enabled, because otherwise the sparse index\n> does not behave as intended. With that restriction in mind, the\n> implementation can use the sparse directory entries to assist.\n>\n> I imagine that when the sparse index is mature enough, we will enable it by\n> default when cone mode sparse-checkout is enabled, in which case this\n> behavior would expand to all users with cone mode sparse-checkout.\n\nI'm all in favor of just doing it for all cone mode users, at least\nfor .gitignore'd files.\n\n> I'm interested in the community's thoughts about this change, as it seems\n> like one that we should make carefully and intentionally.\n\nYeah, I approached it pretty cautiously, just adding a new option in\nour tool (`sparsify --remove-old-ignores`) because I wasn't sure if it\nshould be done by default but wanted to provide an easy way for users\nto access it. We got numerous complaints and after some months, added\na config option to run it automatically. We still got complaints and\neventually my colleagues just made it automatically run and provided\nno way to turn it off. No user complaints since.\n"},{"id":"431568","messageId":"CABPp-BGbRbyCYYS+NcYrC-T4hJf7BCoLE2HsXFM4K51A0wSgcg@mail.gmail.com","threadId":"56179","inReplyTo":"9212bbf4e3cab20fe49ab8e6dd4ac0277c4f2805.1627579637.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-07-30T13:52:19Z","receivedAt":"2021-07-30T13:52:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When changing the scope of a sparse-checkout using cone mode, we might\n> have some tracked directories go out of scope. The current logic removes\n> the tracked files from within those directories, but leaves the ignored\n> files within those directories. This is a bit unexpected to users who\n> have given input to Git saying they don't need those directories\n> anymore.\n>\n> This is something that is new to the cone mode pattern type: the user\n> has explicitly said \"I want these directories and _not_ those\n> directories.\" The typical sparse-checkout patterns more generally apply\n> to \"I want files with with these patterns\" so it is natural to leave\n> ignored files as they are. This focus on directories in cone mode\n> provides us an opportunity to change the behavior.\n>\n> Leaving these ignored files in the sparse directories makes it\n> impossible to gain performance benefits in the sparse index. When we\n> track into these directories, we need to know if the files are ignored\n> or not, which might depend on the _tracked_ .gitignore file(s) within\n> the sparse directory. This depends on the indexed version of the file,\n> so the sparse directory must be expanded.\n\nIs this the issue I highlighted at\nhttps://lore.kernel.org/git/CABPp-BHsiLTtv6AuycRrQ5K6q0=ZTJe0kd7uTUL+UT=nxj66zA@mail.gmail.com/,\nthe paragraph \"...I thought the point of add_patterns()...\" or is\nthere more or other things involved here?\n\n> By deleting the sparse directories when changing scope (or running 'git\n> sparse-checkout reapply') we regain these performance benefits as if the\n> repository was in a clean state.\n\n:-)\n\nHowever, note that our repository is probably ~25x smaller than yours\nin terms of number of files, and 'git status' performance really isn't\nthat big of an issue for us. Users complained to us about these\ndirectories sticking around simply because they were confused by the\ndirectories remaining. (Did sparsify not work? What's all that\nleftover stuff? Why do they have to deal with it? etc.)\n\n> Since these ignored files are frequently build output or helper files\n> from IDEs, the users should not need the files now that the tracked\n> files are removed. If the tracked files reappear, then they will have\n> newer timestamps than the build artifacts, so the artifacts will need to\n> be regenerated anyway.\n\nThis is such a good explanation of why automatically removing these\ndirectories was perfectly fine for us to do.\n\n> If users depend on ignored files within the sparse directories, then\n> they have created a bad shape in their repository. Regardless, such\n> shapes would create risk that changing the behavior for all cone mode\n> users might be too risky to take on at the moment. Since this data shape\n> makes it impossible to get performance benefits using the sparse index,\n> we limit the change to only be enabled when the sparse index is enabled.\n> Users can opt out of this behavior by disabline the sparse index.\n\ns/disabline/disabling/, otherwise, I fully agree with this paragraph.\n\n> Depending on user feedback or real-world use, we might want to consider\n> expanding the behavior change to all of cone mode. Since we are\n> currently restricting to the sparse index case, we can use the existence\n> of sparse directory entries in the index as indicators of which\n> directories should be removed.\n\nLet's just expand it to all of cone mode.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 6 +++\n> builtin/sparse-checkout.c | 73 +++++++++++++++++++++++++++\n> t/t1091-sparse-checkout-builtin.sh | 42 +++++++++++++++\n> 3 files changed, 121 insertions(+)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index fdcf43f87cb..c443189f418 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -210,6 +210,12 @@ case-insensitive check. This corrects for case mismatched filenames in the\n> 'git sparse-checkout set' command to reflect the expected cone in the working\n> directory.\n>\n> +When the sparse index is enabled through the `index.sparse` config option,\n> +the cone mode sparse-checkout patterns will also remove ignored files that\n> +are not within the sparse-checkout definition. This is important behavior\n> +to preserve the performance of the sparse index, but also matches that\n> +cone mode patterns care about directories, not files.\n> +\n>\n> SUBMODULES\n> ----------\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index a4bdd7c4940..5c03636b6ec 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -15,6 +15,7 @@\n> #include \"wt-status.h\"\n> #include \"quote.h\"\n> #include \"sparse-index.h\"\n> +#include \"run-command.h\"\n>\n> static const char *empty_base = \"\";\n>\n> @@ -100,6 +101,71 @@ static int sparse_checkout_list(int argc, const char **argv)\n> return 0;\n> }\n>\n> +static void clean_tracked_sparse_directories(struct repository *r)\n> +{\n> + int i;\n> + struct strbuf path = STRBUF_INIT;\n> + size_t pathlen;\n> +\n> + /*\n> + * If we are not using cone mode patterns, then we cannot\n> + * delete directories outside of the sparse cone.\n> + */\n> + if (!r || !r->index || !r->index->sparse_checkout_patterns ||\n> + !r->index->sparse_checkout_patterns->use_cone_patterns)\n> + return;\n> + /*\n> + * NEEDSWORK: For now, only use this behavior when index.sparse\n> + * is enabled. We may want this behavior enabled whenever using\n> + * cone mode patterns.\n> + */\n> + prepare_repo_settings(r);\n> + if (!r->worktree || !r->settings.sparse_index)\n> + return;\n\ns/may/do/ :-)\n\n> +\n> + /*\n> + * Since we now depend on the sparse index to enable this\n> + * behavior, use it to our advantage. This process is more\n> + * complicated without it.\n> + */\n\nAh, the real reason why you limited this change to sparse-index comes out. :-)\n\n> + convert_to_sparse(r->index);\n> +\n> + strbuf_addstr(&path, r->worktree);\n> + strbuf_complete(&path, '/');\n> + pathlen = path.len;\n> +\n> + for (i = 0; i < r->index->cache_nr; i++) {\n> + struct cache_entry *ce = r->index->cache[i];\n> +\n> + /*\n> + * Is this a sparse directory? If so, then definitely\n> + * include it. All contained content is outside of the\n> + * patterns.\n\n\"include\"? I would have used the word \"remove\", but it gets confusing\nbecause the question is if we want to include it in our list of things\nto remove or remove it from the working directory.\n\n> + */\n> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n> + repo_file_exists(r, ce->name)) {\n> + strbuf_setlen(&path, pathlen);\n> + strbuf_addstr(&path, ce->name);\n> +\n> + /*\n> + * Removal is \"best effort\". If something blocks\n> + * the deletion, then continue with a warning.\n> + */\n> + if (remove_dir_recursively(&path, 0))\n> + warning(_(\"failed to remove directory '%s'\"), path.buf);\n\nUm, doesn't this delete untracked files that are not ignored as well\nas the ignored files? If so, was that intentional? I'm fully on\nboard with removing the gitignore'd files, but I'm worried removing\nother untracked files is dangerous.\n\nMy implementation of this concept (in an external tool) was more along\nthe lines of\n\n * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`\noutput and finding common fully-sparse directories\n * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES\n\n\n> + }\n> + }\n> +\n> + strbuf_release(&path);\n> +\n> + /*\n> + * This is temporary: the sparse-checkout builtin is not\n> + * integrated with the sparse-index yet, so we need to keep\n> + * it full during the process.\n> + */\n> + ensure_full_index(r->index);\n> +}\n> +\n> static int update_working_directory(struct pattern_list *pl)\n> {\n> enum update_sparsity_result result;\n> @@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)\n> else\n> rollback_lock_file(&lock_file);\n>\n> + clean_tracked_sparse_directories(r);\n> +\n> r->index->sparse_checkout_patterns = NULL;\n> return result;\n> }\n\n(Adding a comment here just to separate the two blocks below.)\n\n> @@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n> {\n> int result;\n> int changed_config = 0;\n> + struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));\n> struct pattern_list *pl = xcalloc(1, sizeof(*pl));\n>\n> + get_sparse_checkout_patterns(old_pl);\n> +\n> switch (m) {\n> case ADD:\n> if (core_sparse_checkout_cone)\n> @@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n> set_config(MODE_NO_PATTERNS);\n>\n> clear_pattern_list(pl);\n> + clear_pattern_list(old_pl);\n> free(pl);\n> + free(old_pl);\n> return result;\n> }\n\nYou create an old_pl, populate it with get_sparse_checkout_patterns(),\ndo nothing with it, then clear and free it? I'm confused by the above\ntwo blocks.\n\n>\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 38fc8340f5c..43eb314c94a 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -642,4 +642,46 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n> check_files repo/deep a deeper1\n> '\n>\n> +test_expect_success 'cone mode clears ignored subdirectories' '\n> + rm repo/.git/info/sparse-checkout &&\n> +\n> + # NEEDSWORK: --sparse-index is required for now\n> + git -C repo sparse-checkout init --cone --sparse-index &&\n> + git -C repo sparse-checkout set deep/deeper1 &&\n> +\n> + cat >repo/.gitignore <<-\\EOF &&\n> + obj/\n> + *.o\n> + EOF\n> +\n> + git -C repo add .gitignore &&\n> + git -C repo commit -m \".gitignore\" &&\n> +\n> + mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n> + for file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n> + deep/deeper2/obj/a deep/deeper2/file.o file.o\n> + do\n> + echo ignored >repo/$file || return 1\n> + done &&\n> +\n> + git -C repo status --porcelain=v2 >out &&\n> + test_must_be_empty out &&\n> +\n> + git -C repo sparse-checkout reapply &&\n> + test_path_is_missing repo/folder1 &&\n> + test_path_is_missing repo/deep/deeper2 &&\n> + test_path_is_dir repo/obj &&\n> + test_path_is_file repo/file.o &&\n> +\n> + git -C repo status --porcelain=v2 >out &&\n> + test_must_be_empty out &&\n> +\n> + git -C repo sparse-checkout set deep/deeper2 &&\n> + test_path_is_missing repo/deep/deeper1 &&\n> + test_path_is_dir repo/deep/deeper2 &&\n\nWhat's the expectation for repo/obj?\n\n> +\n> + git -C repo status --porcelain=v2 >out &&\n> + test_must_be_empty out\n> +'\n> +\n> test_done\n> --\n"},{"id":"431694","messageId":"76639e16-204d-7812-d4c5-56c70e280bed@gmail.com","threadId":"56179","inReplyTo":"CABPp-BGbRbyCYYS+NcYrC-T4hJf7BCoLE2HsXFM4K51A0wSgcg@mail.gmail.com","subject":"Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-02T14:34:28Z","receivedAt":"2021-08-02T14:34:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2021 9:52 AM, Elijah Newren wrote:\n> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n> wrote:\n...\n>> Leaving these ignored files in the sparse directories makes it\n>> impossible to gain performance benefits in the sparse index. When we\n>> track into these directories, we need to know if the files are ignored\n>> or not, which might depend on the _tracked_ .gitignore file(s) within\n>> the sparse directory. This depends on the indexed version of the file,\n>> so the sparse directory must be expanded.\n> \n> Is this the issue I highlighted at\n> https://lore.kernel.org/git/CABPp-BHsiLTtv6AuycRrQ5K6q0=ZTJe0kd7uTUL+UT=nxj66zA@mail.gmail.com/,\n> the paragraph \"...I thought the point of add_patterns()...\" or is\n> there more or other things involved here?\n\nThis is exactly that point. I'm not sure why I didn't pick up on your\nearlier concerns as something to do _immediately_, but for some reason\nI thought I could delay it until later.\n\n>> If users depend on ignored files within the sparse directories, then\n>> they have created a bad shape in their repository. Regardless, such\n>> shapes would create risk that changing the behavior for all cone mode\n>> users might be too risky to take on at the moment. Since this data shape\n>> makes it impossible to get performance benefits using the sparse index,\n>> we limit the change to only be enabled when the sparse index is enabled.\n>> Users can opt out of this behavior by disabline the sparse index.\n> \n> s/disabline/disabling/, otherwise, I fully agree with this paragraph.\n\nWill fix. Thanks.\n \n>> Depending on user feedback or real-world use, we might want to consider\n>> expanding the behavior change to all of cone mode. Since we are\n>> currently restricting to the sparse index case, we can use the existence\n>> of sparse directory entries in the index as indicators of which\n>> directories should be removed.\n> \n> Let's just expand it to all of cone mode.\n\nI'm open to that. I'll leave this version open for feedback a bit longer\nbefore doing so.\n\n>> + /*\n>> + * NEEDSWORK: For now, only use this behavior when index.sparse\n>> + * is enabled. We may want this behavior enabled whenever using\n>> + * cone mode patterns.\n>> + */\n>> + prepare_repo_settings(r);\n>> + if (!r->worktree || !r->settings.sparse_index)\n>> + return;\n> \n> s/may/do/ :-)\n\nMaybe!\n \n>> +\n>> + /*\n>> + * Since we now depend on the sparse index to enable this\n>> + * behavior, use it to our advantage. This process is more\n>> + * complicated without it.\n>> + */\n> \n> Ah, the real reason why you limited this change to sparse-index comes out. :-)\n> \n>> + convert_to_sparse(r->index);\n>> +\n>> + strbuf_addstr(&path, r->worktree);\n>> + strbuf_complete(&path, '/');\n>> + pathlen = path.len;\n>> +\n>> + for (i = 0; i < r->index->cache_nr; i++) {\n>> + struct cache_entry *ce = r->index->cache[i];\n>> +\n>> + /*\n>> + * Is this a sparse directory? If so, then definitely\n>> + * include it. All contained content is outside of the\n>> + * patterns.\n> \n> \"include\"? I would have used the word \"remove\", but it gets confusing\n> because the question is if we want to include it in our list of things\n> to remove or remove it from the working directory.\n\nThis comment is a bit stale, sorry. Earlier I was populating a list of\nthe directories, but in this version I'm just deleting them immediately.\n\n>> + */\n>> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n>> + repo_file_exists(r, ce->name)) {\n>> + strbuf_setlen(&path, pathlen);\n>> + strbuf_addstr(&path, ce->name);\n>> +\n>> + /*\n>> + * Removal is \"best effort\". If something blocks\n>> + * the deletion, then continue with a warning.\n>> + */\n>> + if (remove_dir_recursively(&path, 0))\n>> + warning(_(\"failed to remove directory '%s'\"), path.buf);\n> \n> Um, doesn't this delete untracked files that are not ignored as well\n> as the ignored files? If so, was that intentional? I'm fully on\n> board with removing the gitignore'd files, but I'm worried removing\n> other untracked files is dangerous.\n\nI believe that 'git sparse-checkout (set|add|reapply)' will fail before\nreaching this method if there are untracked files that could potentially\nbe removed. I will double-check to ensure this is the case. It is\ndefinitely my intention to protect any untracked, non-ignored files in\nthese directories by failing the sparse-checkout modification.\n\n> My implementation of this concept (in an external tool) was more along\n> the lines of\n> \n> * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`\n> output and finding common fully-sparse directories\n> * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES\n\nI initially was running 'git clean -dfx -- ...' but that also\nrequires parsing and expanding the index (or being very careful with\nthe sparse index).\n\n> \n>> + }\n>> + }\n>> +\n>> + strbuf_release(&path);\n>> +\n>> + /*\n>> + * This is temporary: the sparse-checkout builtin is not\n>> + * integrated with the sparse-index yet, so we need to keep\n>> + * it full during the process.\n>> + */\n>> + ensure_full_index(r->index);\n>> +}\n>> +\n>> static int update_working_directory(struct pattern_list *pl)\n>> {\n>> enum update_sparsity_result result;\n>> @@ -141,6 +207,8 @@ static int update_working_directory(struct pattern_list *pl)\n>> else\n>> rollback_lock_file(&lock_file);\n>>\n>> + clean_tracked_sparse_directories(r);\n>> +\n>> r->index->sparse_checkout_patterns = NULL;\n>> return result;\n>> }\n> \n> (Adding a comment here just to separate the two blocks below.)\n> \n>> @@ -540,8 +608,11 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n>> {\n>> int result;\n>> int changed_config = 0;\n>> + struct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));\n>> struct pattern_list *pl = xcalloc(1, sizeof(*pl));\n>>\n>> + get_sparse_checkout_patterns(old_pl);\n>> +\n>> switch (m) {\n>> case ADD:\n>> if (core_sparse_checkout_cone)\n>> @@ -567,7 +638,9 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n>> set_config(MODE_NO_PATTERNS);\n>>\n>> clear_pattern_list(pl);\n>> + clear_pattern_list(old_pl);\n>> free(pl);\n>> + free(old_pl);\n>> return result;\n>> }\n> \n> You create an old_pl, populate it with get_sparse_checkout_patterns(),\n> do nothing with it, then clear and free it? I'm confused by the above\n> two blocks.\n\nSorry that this is also stale. I should read my own patches more carefully.\n\nI was originally going to focus only on the directories that were \"leaving\"\nthe sparse-checkout definition, but that did not catch the 'reapply' case\nor cases where the user created the directories by other means.\n\nWill delete these bits.\n>> + git -C repo sparse-checkout reapply &&\n>> + test_path_is_missing repo/folder1 &&\n>> + test_path_is_missing repo/deep/deeper2 &&\n>> + test_path_is_dir repo/obj &&\n>> + test_path_is_file repo/file.o &&\n>> +\n>> + git -C repo status --porcelain=v2 >out &&\n>> + test_must_be_empty out &&\n>> +\n>> + git -C repo sparse-checkout set deep/deeper2 &&\n>> + test_path_is_missing repo/deep/deeper1 &&\n>> + test_path_is_dir repo/deep/deeper2 &&\n> \n> What's the expectation for repo/obj?\n\nI will add\n\n\ttest_path_is_dir repo/obj\n\nbecause this ignored directory should not be removed. This is\nsimulating the build artifacts that might appear in ignored\ndirectories whose parents are in the sparse-checkout definition.\n\nThanks,\n-Stolee\n"},{"id":"431705","messageId":"CABPp-BE1aKAsp6yKAzM-djRHuiP=GHC_Q7xZxMWPJ=f28bOeUg@mail.gmail.com","threadId":"56179","inReplyTo":"76639e16-204d-7812-d4c5-56c70e280bed@gmail.com","subject":"Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-02T16:17:17Z","receivedAt":"2021-08-02T16:17:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee wrote:\n>\n> On 7/30/2021 9:52 AM, Elijah Newren wrote:\n> > On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n> > wrote:\n...\n> >> + */\n> >> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n> >> + repo_file_exists(r, ce->name)) {\n> >> + strbuf_setlen(&path, pathlen);\n> >> + strbuf_addstr(&path, ce->name);\n> >> +\n> >> + /*\n> >> + * Removal is \"best effort\". If something blocks\n> >> + * the deletion, then continue with a warning.\n> >> + */\n> >> + if (remove_dir_recursively(&path, 0))\n> >> + warning(_(\"failed to remove directory '%s'\"), path.buf);\n> >\n> > Um, doesn't this delete untracked files that are not ignored as well\n> > as the ignored files? If so, was that intentional? I'm fully on\n> > board with removing the gitignore'd files, but I'm worried removing\n> > other untracked files is dangerous.\n>\n> I believe that 'git sparse-checkout (set|add|reapply)' will fail before\n> reaching this method if there are untracked files that could potentially\n> be removed. I will double-check to ensure this is the case. It is\n> definitely my intention to protect any untracked, non-ignored files in\n> these directories by failing the sparse-checkout modification.\n>\n> > My implementation of this concept (in an external tool) was more along\n> > the lines of\n> >\n> > * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`\n> > output and finding common fully-sparse directories\n> > * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES\n>\n> I initially was running 'git clean -dfx -- ...' but that also\n> requires parsing and expanding the index (or being very careful with\n> the sparse index).\n\n`git clean -dfx -- ...` could also be very dangerous because\nit'd delete untracked non-ignored files. You want -X rather than -x.\nOne of those cases where capitalization is critical.\n"},{"id":"432043","messageId":"53d1b6ac-76a0-034b-109c-df2835c8fad6@gmail.com","threadId":"56179","inReplyTo":"CABPp-BE1aKAsp6yKAzM-djRHuiP=GHC_Q7xZxMWPJ=f28bOeUg@mail.gmail.com","subject":"Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-05T01:55:22Z","receivedAt":"2021-08-05T01:55:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/2/2021 12:17 PM, Elijah Newren wrote:\n> On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee wrote:\n>>\n>> On 7/30/2021 9:52 AM, Elijah Newren wrote:\n>>> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n>>> wrote:\n> ...\n>>>> + */\n>>>> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n>>>> + repo_file_exists(r, ce->name)) {\n>>>> + strbuf_setlen(&path, pathlen);\n>>>> + strbuf_addstr(&path, ce->name);\n>>>> +\n>>>> + /*\n>>>> + * Removal is \"best effort\". If something blocks\n>>>> + * the deletion, then continue with a warning.\n>>>> + */\n>>>> + if (remove_dir_recursively(&path, 0))\n>>>> + warning(_(\"failed to remove directory '%s'\"), path.buf);\n>>>\n>>> Um, doesn't this delete untracked files that are not ignored as well\n>>> as the ignored files? If so, was that intentional? I'm fully on\n>>> board with removing the gitignore'd files, but I'm worried removing\n>>> other untracked files is dangerous.\n>>\n>> I believe that 'git sparse-checkout (set|add|reapply)' will fail before\n>> reaching this method if there are untracked files that could potentially\n>> be removed. I will double-check to ensure this is the case. It is\n>> definitely my intention to protect any untracked, non-ignored files in\n>> these directories by failing the sparse-checkout modification.\n\nThis is _not_ true, and I can document it with a test.\n\nHaving untracked files outside of the sparse cone is just as bad as\nignored files, so I want to ensure that these get cleaned up, too.\n\nThe correct thing would be to prevent the 'git sparse-checkout\n(set|add|reapply)' command from making any changes to the sparse-checkout\ncone or the worktree if there are untracked files that would be deleted.\n(Right? Or is there another solution that I'm missing here?)\n\n>>> My implementation of this concept (in an external tool) was more along\n>>> the lines of\n>>>\n>>> * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`\n>>> output and finding common fully-sparse directories\n>>> * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES\n>>\n>> I initially was running 'git clean -dfx -- ...' but that also\n>> requires parsing and expanding the index (or being very careful with\n>> the sparse index).\n> \n> `git clean -dfx -- ...` could also be very dangerous because\n> it'd delete untracked non-ignored files. You want -X rather than -x.\n> One of those cases where capitalization is critical.\n\nGood point. I'd like to avoid using `git clean` as a subcommand, if\npossible, that way we have one fewer thing to do before integrating\nthe `git sparse-checkout` builtin with the sparse index.\n\nThanks,\n-Stolee\n"},{"id":"432049","messageId":"CABPp-BGi1CBnerhtV87Do+=RyEkSexRFPqG566MSvM3zDWam_A@mail.gmail.com","threadId":"56179","inReplyTo":"53d1b6ac-76a0-034b-109c-df2835c8fad6@gmail.com","subject":"Re: [PATCH 2/2] sparse-checkout: clear tracked sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-05T03:54:19Z","receivedAt":"2021-08-05T03:54:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Aug 4, 2021 at 7:55 PM Derrick Stolee wrote:\n>\n> On 8/2/2021 12:17 PM, Elijah Newren wrote:\n> > On Mon, Aug 2, 2021 at 8:34 AM Derrick Stolee wrote:\n> >>\n> >> On 7/30/2021 9:52 AM, Elijah Newren wrote:\n> >>> On Thu, Jul 29, 2021 at 11:27 AM Derrick Stolee via GitGitGadget\n> >>> wrote:\n> > ...\n> >>>> + */\n> >>>> + if (S_ISSPARSEDIR(ce->ce_mode) &&\n> >>>> + repo_file_exists(r, ce->name)) {\n> >>>> + strbuf_setlen(&path, pathlen);\n> >>>> + strbuf_addstr(&path, ce->name);\n> >>>> +\n> >>>> + /*\n> >>>> + * Removal is \"best effort\". If something blocks\n> >>>> + * the deletion, then continue with a warning.\n> >>>> + */\n> >>>> + if (remove_dir_recursively(&path, 0))\n> >>>> + warning(_(\"failed to remove directory '%s'\"), path.buf);\n> >>>\n> >>> Um, doesn't this delete untracked files that are not ignored as well\n> >>> as the ignored files? If so, was that intentional? I'm fully on\n> >>> board with removing the gitignore'd files, but I'm worried removing\n> >>> other untracked files is dangerous.\n> >>\n> >> I believe that 'git sparse-checkout (set|add|reapply)' will fail before\n> >> reaching this method if there are untracked files that could potentially\n> >> be removed. I will double-check to ensure this is the case. It is\n> >> definitely my intention to protect any untracked, non-ignored files in\n> >> these directories by failing the sparse-checkout modification.\n>\n> This is _not_ true, and I can document it with a test.\n>\n> Having untracked files outside of the sparse cone is just as bad as\n> ignored files, so I want to ensure that these get cleaned up, too.\n>\n> The correct thing would be to prevent the 'git sparse-checkout\n> (set|add|reapply)' command from making any changes to the sparse-checkout\n> cone or the worktree if there are untracked files that would be deleted.\n> (Right? Or is there another solution that I'm missing here?)\n\nWe could sparsify as much as possible and print warnings, much like we\ndo with tracked files that are modified but not staged. In fact, it\nmight feel inconsistent if we sparsify as much as possible for one\ntype of file, and abort if we cannot completely sparsify for a\ndifferent type of file. We could consider changing how we treat\ntracked files that are modified but not staged and have them abort the\nsparse-checkout commands as well, but I worry that might cause\nproblems during conflict resolution in the middle of\nmerges/rebases/cherry-picks/reverts. I don't want users caught where\nthey need to update their sparsity paths to gain new files/directories\nthat will help them resolve some conflicts, but be unable to update\ntheir sparsity paths because they have conflicts.\n\nThat said, the basic idea of aborting sparse-checkout in cone mode\nwhen there are untracked unignored files in the way of removing\ndirectories sounds reasonable, if there's some clever way to avoid or\nameliorate the inconsistency issues mentioned above. Implementing it\nmight require walking all untracked (and tracked?) files twice,\nthough, because if there are untracked unignored files in the way, we\nprobably don't want to abort after first deleting lots of ignored\nfiles. (And there's a small race window in the double walk...)\nHowever, I don't expect people to run sparse-checkout commands all\nthat often, so the double walk is probably a perfectly reasonable\nperformance cost. I just wanted to note it.\n\n> >>> My implementation of this concept (in an external tool) was more along\n> >>> the lines of\n> >>>\n> >>> * Get $LIST_OF_NON_SPARSE_DIRECTORIES by walking `git ls-files -t`\n> >>> output and finding common fully-sparse directories\n> >>> * git clean -fX $LIST_OF_NON_SPARSE_DIRECTORIES\n> >>\n> >> I initially was running 'git clean -dfx -- ...' but that also\n> >> requires parsing and expanding the index (or being very careful with\n> >> the sparse index).\n> >\n> > `git clean -dfx -- ...` could also be very dangerous because\n> > it'd delete untracked non-ignored files. You want -X rather than -x.\n> > One of those cases where capitalization is critical.\n>\n> Good point. I'd like to avoid using `git clean` as a subcommand, if\n> possible, that way we have one fewer thing to do before integrating\n> the `git sparse-checkout` builtin with the sparse index.\n\nOh, I didn't want to invoke a subcommand, I was just pointing out\nwhere similar code might be found in case we wanted to call the same\nfunctions from elsewhere (or maybe even turn some of it into library\nfunctions we could call). But that might be a moot point if we end up\nmaking sparse-checkout fail if there are untracked unignored files\nhanging around in the relevant directories.\n"},{"id":"432408","messageId":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.git.1627579637.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:05Z","receivedAt":"2021-08-10T19:50:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\nand v2.33.0-rc1 because of some changes to dir.h that became important!\n\nWe launched an experimental release [1] of the sparse-index feature to our\ninternal users. We immediately discovered a problem due to the isolated way\nin which we tested the sparse index: we were never building the project and\nchanging our sparse-checkout definition.\n\n[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n\nUsers who ran a build in one project and then moved to another still had\nbuild artifacts in their worktree that lived inside the old directories.\nSince the files are marked by the .gitignore patterns, these files were not\nremoved by the 'git sparse-checkout set' command. However, they make the\nsparse-index unusable because every 'git status' command needs to expand the\nsparse-directory entries in order to see if the files are tracked or not.\nThis made the first experimental release actually slower for all users\nbecause of this cost.\n\nThe solution we shipped to these customers was to change the way our fork\nhandles these ignored files. Specifically, instead of Git completely\nignoring the files, we changed Git to understand that with cone-mode\nsparse-checkout patterns, the users is asking for entire directories to be\nremoved from the worktree. The link [1] included earlier has this change.\n\nI believe that this is a reasonable expectation, though I recognize that it\nmight look like breaking the expectations of how .gitignore files work.\n\nSince feedback demonstrated that this is a desired behavior, v2 includes\nthis behavior for all \"cone mode\" repositories.\n\nI'm interested in the community's thoughts about this change, as it seems\nlike one that we should make carefully and intentionally.\n\nWhile the rewrite of the t7519 test seems unrelated, it is required to avoid\na test failure with this change that deletes files outside of the cone. By\nmoving the test into repositories not at $TRASH_DIRECTORY, we gain more\ncontrol over the repository structure.\n\n\nUpdates in V2\n=============\n\n * This version correctly leaves untracked files alone. If untracked files\n are found, then the directory is left as-is, in case those ignored files\n are important to the user's work resolving those untracked files.\n\n * This behavior is now enabled by core.sparseCheckoutCone=true.\n\n * To use a sparse index as an in-memory data structure even when\n index.sparse is disabled, a new patch is included to modify the prototype\n of convert_to_sparse() to include a flags parameter.\n\n * A few cleanup patches that I was collecting based on feedback from the\n experimental release and intending for my next series were necessary for\n this implementation.\n\n * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n implementation that used run_subcommand().\n\nThanks, -Stolee\n\nDerrick Stolee (8):\n t7519: rewrite sparse index test\n sparse-index: silently return when not using cone-mode patterns\n sparse-index: silently return when cache tree fails\n unpack-trees: fix nested sparse-dir search\n sparse-checkout: create helper methods\n attr: be careful about sparse directories\n sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n sparse-checkout: clear tracked sparse dirs\n\n Documentation/git-sparse-checkout.txt | 8 +++\n attr.c | 14 ++++\n builtin/add.c | 8 +--\n builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n dir.c | 29 ++++++++\n dir.h | 6 ++\n read-cache.c | 4 +-\n sparse-index.c | 70 +++++++++++---------\n sparse-index.h | 3 +-\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n t/t7519-status-fsmonitor.sh | 38 ++++++-----\n unpack-trees.c | 18 +++--\n 12 files changed, 290 insertions(+), 62 deletions(-)\n\n\nbase-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1009\n\nRange-diff vs v1:\n\n 1: 29f1633f6d5 = 1: e66106f7a99 t7519: rewrite sparse index test\n -: ----------- > 2: fb3ff9108bf sparse-index: silently return when not using cone-mode patterns\n -: ----------- > 3: 37198535268 sparse-index: silently return when cache tree fails\n -: ----------- > 4: 10bcadb284e unpack-trees: fix nested sparse-dir search\n -: ----------- > 5: e9ed5cd2830 sparse-checkout: create helper methods\n -: ----------- > 6: 5680df62e1c attr: be careful about sparse directories\n -: ----------- > 7: 1dd73b36eb4 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n 2: 9212bbf4e3c ! 8: f74015a2be9 sparse-checkout: clear tracked sparse dirs\n @@ Commit message\n newer timestamps than the build artifacts, so the artifacts will need to\n be regenerated anyway.\n \n - If users depend on ignored files within the sparse directories, then\n - they have created a bad shape in their repository. Regardless, such\n - shapes would create risk that changing the behavior for all cone mode\n - users might be too risky to take on at the moment. Since this data shape\n - makes it impossible to get performance benefits using the sparse index,\n - we limit the change to only be enabled when the sparse index is enabled.\n - Users can opt out of this behavior by disabline the sparse index.\n -\n - Depending on user feedback or real-world use, we might want to consider\n - expanding the behavior change to all of cone mode. Since we are\n - currently restricting to the sparse index case, we can use the existence\n - of sparse directory entries in the index as indicators of which\n - directories should be removed.\n + Use the sparse-index as a data structure in order to find the sparse\n + directories that can be safely deleted. Re-expand the index to a full\n + one if it was full before.\n \n Signed-off-by: Derrick Stolee \n \n @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n -+When the sparse index is enabled through the `index.sparse` config option,\n -+the cone mode sparse-checkout patterns will also remove ignored files that\n ++The cone mode sparse-checkout patterns will also remove ignored files that\n +are not within the sparse-checkout definition. This is important behavior\n +to preserve the performance of the sparse index, but also matches that\n -+cone mode patterns care about directories, not files.\n ++cone mode patterns care about directories, not files. If there exist files\n ++that are untracked and not ignored, then Git will not delete files within\n ++that directory other than the tracked files that are now out of scope.\n ++These files should be removed manually to ensure Git can behave optimally.\n +\n \n SUBMODULES\n ----------\n \n ## builtin/sparse-checkout.c ##\n -@@\n - #include \"wt-status.h\"\n - #include \"quote.h\"\n - #include \"sparse-index.h\"\n -+#include \"run-command.h\"\n - \n - static const char *empty_base = \"\";\n - \n @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n +static void clean_tracked_sparse_directories(struct repository *r)\n +{\n -+\tint i;\n ++\tint i, was_full = 0;\n +\tstruct strbuf path = STRBUF_INIT;\n +\tsize_t pathlen;\n ++\tstruct string_list_item *item;\n ++\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n +\n +\t/*\n +\t * If we are not using cone mode patterns, then we cannot\n +\t * delete directories outside of the sparse cone.\n +\t */\n -+\tif (!r || !r->index || !r->index->sparse_checkout_patterns ||\n -+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n ++\tif (!r || !r->index || !r->worktree)\n +\t\treturn;\n -+\t/*\n -+\t * NEEDSWORK: For now, only use this behavior when index.sparse\n -+\t * is enabled. We may want this behavior enabled whenever using\n -+\t * cone mode patterns.\n -+\t */\n -+\tprepare_repo_settings(r);\n -+\tif (!r->worktree || !r->settings.sparse_index)\n ++\tinit_sparse_checkout_patterns(r->index);\n ++\tif (!r->index->sparse_checkout_patterns ||\n ++\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n +\t\treturn;\n +\n +\t/*\n -+\t * Since we now depend on the sparse index to enable this\n -+\t * behavior, use it to our advantage. This process is more\n -+\t * complicated without it.\n ++\t * Use the sparse index as a data structure to assist finding\n ++\t * directories that are safe to delete. This conversion to a\n ++\t * sparse index will not delete directories that contain\n ++\t * conflicted entries or submodules.\n +\t */\n -+\tconvert_to_sparse(r->index);\n ++\tif (!r->index->sparse_index) {\n ++\t\t/*\n ++\t\t * If something, such as a merge conflict or other concern,\n ++\t\t * prevents us from converting to a sparse index, then do\n ++\t\t * not try deleting files.\n ++\t\t */\n ++\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n ++\t\t\treturn;\n ++\t\twas_full = 1;\n ++\t}\n +\n +\tstrbuf_addstr(&path, r->worktree);\n +\tstrbuf_complete(&path, '/');\n +\tpathlen = path.len;\n +\n ++\t/*\n ++\t * Collect directories that have gone out of scope but also\n ++\t * exist on disk, so there is some work to be done. We need to\n ++\t * store the entries in a list before exploring, since that might\n ++\t * expand the sparse-index again.\n ++\t */\n +\tfor (i = 0; i < r->index->cache_nr; i++) {\n +\t\tstruct cache_entry *ce = r->index->cache[i];\n +\n -+\t\t/*\n -+\t\t * Is this a sparse directory? If so, then definitely\n -+\t\t * include it. All contained content is outside of the\n -+\t\t * patterns.\n -+\t\t */\n +\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n -+\t\t repo_file_exists(r, ce->name)) {\n -+\t\t\tstrbuf_setlen(&path, pathlen);\n -+\t\t\tstrbuf_addstr(&path, ce->name);\n ++\t\t repo_file_exists(r, ce->name))\n ++\t\t\tstring_list_append(&sparse_dirs, ce->name);\n ++\t}\n ++\n ++\tfor_each_string_list_item(item, &sparse_dirs) {\n ++\t\tstruct dir_struct dir = DIR_INIT;\n ++\t\tstruct pathspec p = { 0 };\n ++\t\tstruct strvec s = STRVEC_INIT;\n ++\n ++\t\tstrbuf_setlen(&path, pathlen);\n ++\t\tstrbuf_addstr(&path, item->string);\n ++\n ++\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n ++\n ++\t\tsetup_standard_excludes(&dir);\n ++\t\tstrvec_push(&s, path.buf);\n +\n ++\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n ++\t\tfill_directory(&dir, r->index, &p);\n ++\n ++\t\tif (dir.nr) {\n ++\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n ++\t\t\t\t \" but is not in the sparse-checkout cone\"),\n ++\t\t\t\titem->string);\n ++\t\t} else if (remove_dir_recursively(&path, 0)) {\n +\t\t\t/*\n +\t\t\t * Removal is \"best effort\". If something blocks\n +\t\t\t * the deletion, then continue with a warning.\n +\t\t\t */\n -+\t\t\tif (remove_dir_recursively(&path, 0))\n -+\t\t\t\twarning(_(\"failed to remove directory '%s'\"), path.buf);\n ++\t\t\twarning(_(\"failed to remove directory '%s'\"),\n ++\t\t\t\titem->string);\n +\t\t}\n ++\n ++\t\tdir_clear(&dir);\n +\t}\n +\n ++\tstring_list_clear(&sparse_dirs, 0);\n +\tstrbuf_release(&path);\n +\n -+\t/*\n -+\t * This is temporary: the sparse-checkout builtin is not\n -+\t * integrated with the sparse-index yet, so we need to keep\n -+\t * it full during the process.\n -+\t */\n -+\tensure_full_index(r->index);\n ++\tif (was_full)\n ++\t\tensure_full_index(r->index);\n +}\n +\n static int update_working_directory(struct pattern_list *pl)\n @@ builtin/sparse-checkout.c: static int update_working_directory(struct pattern_li\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\n -@@ builtin/sparse-checkout.c: static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n - {\n - \tint result;\n - \tint changed_config = 0;\n -+\tstruct pattern_list *old_pl = xcalloc(1, sizeof(*old_pl));\n - \tstruct pattern_list *pl = xcalloc(1, sizeof(*pl));\n - \n -+\tget_sparse_checkout_patterns(old_pl);\n -+\n - \tswitch (m) {\n - \tcase ADD:\n - \t\tif (core_sparse_checkout_cone)\n -@@ builtin/sparse-checkout.c: static int modify_pattern_list(int argc, const char **argv, enum modify_type m)\n - \t\tset_config(MODE_NO_PATTERNS);\n - \n - \tclear_pattern_list(pl);\n -+\tclear_pattern_list(old_pl);\n - \tfree(pl);\n -+\tfree(old_pl);\n - \treturn result;\n - }\n - \n \n ## t/t1091-sparse-checkout-builtin.sh ##\n @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replace\n +test_expect_success 'cone mode clears ignored subdirectories' '\n +\trm repo/.git/info/sparse-checkout &&\n +\n -+\t# NEEDSWORK: --sparse-index is required for now\n -+\tgit -C repo sparse-checkout init --cone --sparse-index &&\n ++\tgit -C repo sparse-checkout init --cone &&\n +\tgit -C repo sparse-checkout set deep/deeper1 &&\n +\n +\tcat >repo/.gitignore <<-\\EOF &&\n @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success MINGW 'cone mode replace\n +\tgit -C repo sparse-checkout set deep/deeper2 &&\n +\ttest_path_is_missing repo/deep/deeper1 &&\n +\ttest_path_is_dir repo/deep/deeper2 &&\n ++\ttest_path_is_dir repo/obj &&\n ++\ttest_path_is_file repo/file.o &&\n ++\n ++\t>repo/deep/deeper2/ignored.o &&\n ++\t>repo/deep/deeper2/untracked &&\n ++\n ++\t# When an untracked file is in the way, all untracked files\n ++\t# (even ignored files) are preserved.\n ++\tgit -C repo sparse-checkout set folder1 2>err &&\n ++\tgrep \"contains untracked files\" err &&\n ++\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n ++\ttest_path_is_file repo/deep/deeper2/untracked &&\n ++\n ++\t# The rest of the cone matches expectation\n ++\ttest_path_is_missing repo/deep/deeper1 &&\n ++\ttest_path_is_dir repo/obj &&\n ++\ttest_path_is_file repo/file.o &&\n +\n +\tgit -C repo status --porcelain=v2 >out &&\n -+\ttest_must_be_empty out\n ++\techo \"? deep/deeper2/untracked\" >expect &&\n ++\ttest_cmp expect out\n +'\n +\n test_done\n\n-- \ngitgitgadget\n"},{"id":"432409","messageId":"e66106f7a99d94145eec983ea5e72b7cf8a8a479.1628625013.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] t7519: rewrite sparse index test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:06Z","receivedAt":"2021-08-10T19:50:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse index is tested with the FS Monitor hook and extension since\nf8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\nwas very fragile because it shared an index across sparse and non-sparse\nbehavior. Since that expansion and contraction could cause the index to\nlose its FS Monitor bitmap and token, behavior is fragile to changes in\n'git sparse-checkout set'.\n\nRewrite the test to use two clones of the original repo: full and\nsparse. This allows us to also keep the test files (actual, expect,\ntrace2.txt) out of the repos we are testing with 'git status'.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex deea88d4431..6f2cf306f66 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n check_sparse_index_behavior () {\n-\tgit status --porcelain=v2 >expect &&\n-\tgit sparse-checkout init --cone --sparse-index &&\n-\tgit sparse-checkout set dir1 dir2 &&\n+\tgit -C full status --porcelain=v2 >expect &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit status --porcelain=v2 >actual &&\n+\t\tgit -C sparse status --porcelain=v2 >actual &&\n \ttest_region $1 index ensure_full_index trace2.txt &&\n \ttest_region fsm_hook query trace2.txt &&\n \ttest_cmp expect actual &&\n-\trm trace2.txt &&\n-\tgit sparse-checkout disable\n+\trm trace2.txt\n }\n \n test_expect_success 'status succeeds with sparse index' '\n-\tgit reset --hard &&\n+\tgit clone . full &&\n+\tgit clone . sparse &&\n+\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n-\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n-\tcheck_sparse_index_behavior ! &&\n-\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \tEOF\n-\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n+\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n \tcheck_sparse_index_behavior ! &&\n \n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1/modified\\0\"\n \tEOF\n \tcheck_sparse_index_behavior ! &&\n \n-\tcp -r dir1 dir1a &&\n-\tgit add dir1a &&\n-\tgit commit -m \"add dir1a\" &&\n+\tgit -C sparse sparse-checkout add dir1a &&\n+\n+\tfor repo in full sparse\n+\tdo\n+\t\tcp -r $repo/dir1 $repo/dir1a &&\n+\t\tgit -C $repo add dir1a &&\n+\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n+\tdone &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n \t# This one modifies outside the sparse-checkout definition\n \t# and hence we expect to expand the sparse-index.\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1a/modified\\0\"\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"432410","messageId":"fb3ff9108bf832d258155d3c6ee95c8546f21ba5.1628625013.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] sparse-index: silently return when not using cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:07Z","receivedAt":"2021-08-10T19:50:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile the sparse-index is only enabled when core.sparseCheckoutCone is\nalso enabled, it is possible for the user to modify the sparse-checkout\nfile manually in a way that does not match cone-mode patterns. In this\ncase, we should refuse to convert an index into a sparse index, since\nthe sparse_checkout_patterns will not be initialized with recursive and\nparent path hashsets.\n\nAlso silently return if there are no cache entries, which is a simple\ncase: there are no paths to make sparse!\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex c6b4feec413..bc5900eae35 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -130,7 +130,8 @@ static int index_has_unmerged_entries(struct index_state *istate)\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index ||\n+\n+\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n@@ -158,10 +159,16 @@ int convert_to_sparse(struct index_state *istate)\n \t\t\treturn 0;\n \t}\n \n-\tif (!istate->sparse_checkout_patterns->use_cone_patterns) {\n-\t\twarning(_(\"attempting to use sparse-index without cone mode\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * We need cone-mode patterns to use sparse-index. If a user edits\n+\t * their sparse-checkout file manually, then we can detect during\n+\t * parsing that they are not actually using cone-mode patterns and\n+\t * hence we need to abort this conversion _without error_. Warnings\n+\t * already exist in the pattern parsing to inform the user of their\n+\t * bad patterns.\n+\t */\n+\tif (!istate->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn 0;\n \n \t/*\n \t * NEEDSWORK: If we have unmerged entries, then stay full.\n-- \ngitgitgadget\n\n"},{"id":"432411","messageId":"371985352680a767dfacb5477aa77e92e04008ee.1628625013.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] sparse-index: silently return when cache tree fails","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:08Z","receivedAt":"2021-08-10T19:50:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf cache_tree_update() returns a non-zero value, then it could not\ncreate the cache tree. This is likely due to a path having a merge\nconflict. Since we are already returning early, let's return silently to\navoid making it seem like we failed to write the index at all.\n\nIf we remove our dependence on the cache tree within\nconvert_to_sparse(), then we could still recover from this scenario and\nhave a sparse index.\n\nWhen constructing the cache-tree extension in convert_to_sparse(), it is\npossible that we construct a tree object that is new to the object\ndatabase. Without the WRITE_TREE_MISSING_OK flag, this results in an\nerror that halts our conversion to a sparse index. Add this flag to\nremove this limitation.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex bc5900eae35..b6e90417556 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)\n \n \t/* Clear and recompute the cache-tree */\n \tcache_tree_free(&istate->cache_tree);\n-\tif (cache_tree_update(istate, 0)) {\n-\t\twarning(_(\"unable to update cache-tree, staying full\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * Silently return if there is a problem with the cache tree update,\n+\t * which might just be due to a conflict state in some entry.\n+\t *\n+\t * This might create new tree objects, so be sure to use\n+\t * WRITE_TREE_MISSING_OK.\n+\t */\n+\tif (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n+\t\treturn 0;\n \n \tremove_fsmonitor(istate);\n \n-- \ngitgitgadget\n\n"},{"id":"432412","messageId":"10bcadb284e49419f9b4baf75e05f719ec395d98.1628625013.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:09Z","receivedAt":"2021-08-10T19:50:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe iterated search in find_cache_entry() was recently modified to\ninclude a loop that searches backwards for a sparse directory entry that\nmatches the given traverse_info and name_entry. However, the string\ncomparison failed to actually concatenate those two strings, so this\nfailed to find a sparse directory when it was not a top-level directory.\n\nThis caused some errors in rare cases where a 'git checkout' spanned a\ndiff that modified files within the sparse directory entry, but we could\nnot correctly find the entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..df1f4437723 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n-\tstruct cache_entry *ce;\n+\tstruct cache_entry *ce = NULL;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n+\tstruct strbuf full_path = STRBUF_INIT;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n@@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n \n+\tstrbuf_addstr(&full_path, info->traverse_path);\n+\tstrbuf_add(&full_path, p->path, p->pathlen);\n+\tstrbuf_addch(&full_path, '/');\n+\n \t/*\n \t * Due to lexicographic sorting and sparse directory\n \t * entries ending with a trailing slash, our path as a\n@@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \twhile (pos >= 0) {\n \t\tce = o->src_index->cache[pos];\n \n-\t\tif (strncmp(ce->name, p->path, p->pathlen))\n-\t\t\treturn NULL;\n+\t\tif (strncmp(ce->name, full_path.buf, full_path.len)) {\n+\t\t\tce = NULL;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n \t\t sparse_dir_matches_path(ce, info, p))\n-\t\t\treturn ce;\n+\t\t\tbreak;\n \n \t\tpos--;\n \t}\n \n-\treturn NULL;\n+\tstrbuf_release(&full_path);\n+\treturn ce;\n }\n \n static void debug_path(struct traverse_info *info)\n-- \ngitgitgadget\n\n"},{"id":"432413","messageId":"e9ed5cd283087b8f23112f6f3ecbb5a282c4c717.1628625014.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] sparse-checkout: create helper methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:10Z","receivedAt":"2021-08-10T19:50:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we integrate the sparse index into more builtins, we occasionally\nneed to check the sparse-checkout patterns to see if a path is within\nthe sparse-checkout cone. Create some helper methods that help\ninitialize the patterns and check for pattern matching to make this\neasier.\n\nThe existing callers of commands like get_sparse_checkout_patterns() use\na custom 'struct pattern_list' that is not necessarily the one in the\n'struct index_state', so there are not many previous uses that could\nadopt these helpers. There are just two in builtin/add.c and\nsparse-index.c that can use path_in_sparse_checkout().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 8 ++------\n dir.c | 29 +++++++++++++++++++++++++++++\n dir.h | 6 ++++++\n sparse-index.c | 12 +++---------\n 4 files changed, 40 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 17528e8f922..f675bdeae4a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \tint flags = REFRESH_IGNORE_SKIP_WORKTREE |\n \t\t (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);\n-\tstruct pattern_list pl = { 0 };\n-\tint sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);\n \n \tseen = xcalloc(pathspec->nr, 1);\n \trefresh_index(&the_index, flags, pathspec, seen,\n@@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tfor (i = 0; i < pathspec->nr; i++) {\n \t\tif (!seen[i]) {\n \t\t\tconst char *path = pathspec->items[i].original;\n-\t\t\tint dtype = DT_REG;\n \n \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n-\t\t\t (sparse_checkout_enabled &&\n-\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n-\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n+\t\t\t (core_apply_sparse_checkout &&\n+\t\t\t path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec->items[i].original);\n \t\t\t} else {\ndiff --git a/dir.c b/dir.c\nindex 03c4d212672..7027bdfa068 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1439,6 +1439,35 @@ done:\n \treturn result;\n }\n \n+int init_sparse_checkout_patterns(struct index_state *istate)\n+{\n+\tif (istate->sparse_checkout_patterns)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n+\n+\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {\n+\t\tFREE_AND_NULL(istate->sparse_checkout_patterns);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+enum pattern_match_result path_in_sparse_checkout(const char *path,\n+\t\t\t\t\t\t struct index_state *istate)\n+{\n+\tint dtype = DT_REG;\n+\tinit_sparse_checkout_patterns(istate);\n+\n+\tif (!istate->sparse_checkout_patterns)\n+\t\treturn MATCHED;\n+\n+\treturn path_matches_pattern_list(path, strlen(path), NULL, &dtype,\n+\t\t\t\t\t istate->sparse_checkout_patterns,\n+\t\t\t\t\t istate);\n+}\n+\n static struct path_pattern *last_matching_pattern_from_lists(\n \t\tstruct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int pathlen,\ndiff --git a/dir.h b/dir.h\nindex b3e1a54a971..9af2e8c4ba4 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,\n \t\t\t\tconst char *basename, int *dtype,\n \t\t\t\tstruct pattern_list *pl,\n \t\t\t\tstruct index_state *istate);\n+\n+int init_sparse_checkout_patterns(struct index_state *state);\n+\n+enum pattern_match_result path_in_sparse_checkout(const char *path,\n+\t\t\t\t\t\t struct index_state *istate);\n+\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n \t\t\t\t const char *pathname, int len);\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b6e90417556..2efc9fd4910 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,\n \tint i, can_convert = 1;\n \tint start_converted = num_converted;\n \tenum pattern_match_result match;\n-\tint dtype = DT_UNKNOWN;\n \tstruct strbuf child_path = STRBUF_INIT;\n-\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n \n \t/*\n \t * Is the current path outside of the sparse cone?\n \t * Then check if the region can be replaced by a sparse\n \t * directory entry (everything is sparse and merged).\n \t */\n-\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n-\t\t\t\t\t NULL, &dtype, pl, istate);\n+\tmatch = path_in_sparse_checkout(ct_path, istate);\n \tif (match != NOT_MATCHED)\n \t\tcan_convert = 0;\n \n@@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo->settings.sparse_index)\n \t\treturn 0;\n \n-\tif (!istate->sparse_checkout_patterns) {\n-\t\tistate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));\n-\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n-\t\t\treturn 0;\n-\t}\n+\tif (init_sparse_checkout_patterns(istate) < 0)\n+\t\treturn 0;\n \n \t/*\n \t * We need cone-mode patterns to use sparse-index. If a user edits\n-- \ngitgitgadget\n\n"},{"id":"432414","messageId":"5680df62e1cd56714d5b6a61081aed381749ac54.1628625014.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] attr: be careful about sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:11Z","receivedAt":"2021-08-10T19:50:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/attr.c b/attr.c\nindex d029e681f28..a1009f78029 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -14,6 +14,7 @@\n #include \"utf8.h\"\n #include \"quote.h\"\n #include \"thread-utils.h\"\n+#include \"dir.h\"\n \n const char git_attr__true[] = \"(builtin)true\";\n const char git_attr__false[] = \"\\0(builtin)false\";\n@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \tif (!istate)\n \t\treturn NULL;\n \n+\t/*\n+\t * In the case of cone-mode sparse-checkout, getting the\n+\t * .gitattributes file from a directory is meaningless: all\n+\t * contained paths will be sparse if the .gitattributes is also\n+\t * sparse. In the case of a sparse index, it is critical that we\n+\t * don't go looking for one as it will expand the index.\n+\t */\n+\tinit_sparse_checkout_patterns(istate);\n+\tif (istate->sparse_checkout_patterns &&\n+\t istate->sparse_checkout_patterns->use_cone_patterns &&\n+\t path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n+\t\treturn NULL;\n+\n \tbuf = read_blob_data_from_index(istate, path, NULL);\n \tif (!buf)\n \t\treturn NULL;\n-- \ngitgitgadget\n\n"},{"id":"432415","messageId":"1dd73b36eb493fbd3f7a7c964e57179a4ae26523.1628625014.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:12Z","receivedAt":"2021-08-10T19:50:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\nenvironment variable or the \"index.sparse\" config setting before\nconverting the index to a sparse one. This is for ease of use since all\ncurrent consumers are preparing to compress the index before writing it\nto disk. If these settings are not enabled, then convert_to_sparse()\nsilently returns without doing anything.\n\nWe will add a consumer in the next change that wants to use the sparse\nindex as an in-memory data structure, regardless of whether the on-disk\nformat should be sparse.\n\nTo that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip\nthese config checks when enabled. All current consumers are modified to\npass '0' in the new 'flags' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++--\n sparse-index.c | 30 ++++++++++++++++--------------\n sparse-index.h | 3 ++-\n 3 files changed, 20 insertions(+), 17 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9048ef9e905..f5d4385c408 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \tint ret;\n \tint was_full = !istate->sparse_index;\n \n-\tret = convert_to_sparse(istate);\n+\tret = convert_to_sparse(istate, 0);\n \n \tif (ret) {\n \t\twarning(_(\"failed to convert to a sparse-index\"));\n@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,\n \tint ret, was_full = !istate->sparse_index;\n \n \tmove_cache_to_base_index(istate);\n-\tconvert_to_sparse(istate);\n+\tconvert_to_sparse(istate, 0);\n \n \ttrace2_region_enter_printf(\"index\", \"shared/do_write_index\",\n \t\t\t\t the_repository, \"%s\", get_tempfile_path(*temp));\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2efc9fd4910..532fd11787e 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -124,7 +124,7 @@ static int index_has_unmerged_entries(struct index_state *istate)\n \treturn 0;\n }\n \n-int convert_to_sparse(struct index_state *istate)\n+int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n \n@@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n-\t/*\n-\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n-\t * index.sparse config variable to be on.\n-\t */\n-\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n-\tif (test_env >= 0)\n-\t\tset_sparse_index_config(istate->repo, test_env);\n+\tif (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n+\t\t/*\n+\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n+\t\t * index.sparse config variable to be on.\n+\t\t */\n+\t\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n+\t\tif (test_env >= 0)\n+\t\t\tset_sparse_index_config(istate->repo, test_env);\n \n-\t/*\n-\t * Only convert to sparse if index.sparse is set.\n-\t */\n-\tprepare_repo_settings(istate->repo);\n-\tif (!istate->repo->settings.sparse_index)\n-\t\treturn 0;\n+\t\t/*\n+\t\t * Only convert to sparse if index.sparse is set.\n+\t\t */\n+\t\tprepare_repo_settings(istate->repo);\n+\t\tif (!istate->repo->settings.sparse_index)\n+\t\t\treturn 0;\n+\t}\n \n \tif (init_sparse_checkout_patterns(istate) < 0)\n \t\treturn 0;\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 1115a0d7dd9..475f4f0f8da 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,8 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-int convert_to_sparse(struct index_state *istate);\n+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)\n+int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n * Some places in the codebase expect to search for a specific path.\n-- \ngitgitgadget\n\n"},{"id":"432416","messageId":"f74015a2be97eff51ade75a2af95a6e13309a8f2.1628625014.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-10T19:50:13Z","receivedAt":"2021-08-10T19:50:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen changing the scope of a sparse-checkout using cone mode, we might\nhave some tracked directories go out of scope. The current logic removes\nthe tracked files from within those directories, but leaves the ignored\nfiles within those directories. This is a bit unexpected to users who\nhave given input to Git saying they don't need those directories\nanymore.\n\nThis is something that is new to the cone mode pattern type: the user\nhas explicitly said \"I want these directories and _not_ those\ndirectories.\" The typical sparse-checkout patterns more generally apply\nto \"I want files with with these patterns\" so it is natural to leave\nignored files as they are. This focus on directories in cone mode\nprovides us an opportunity to change the behavior.\n\nLeaving these ignored files in the sparse directories makes it\nimpossible to gain performance benefits in the sparse index. When we\ntrack into these directories, we need to know if the files are ignored\nor not, which might depend on the _tracked_ .gitignore file(s) within\nthe sparse directory. This depends on the indexed version of the file,\nso the sparse directory must be expanded.\n\nBy deleting the sparse directories when changing scope (or running 'git\nsparse-checkout reapply') we regain these performance benefits as if the\nrepository was in a clean state.\n\nSince these ignored files are frequently build output or helper files\nfrom IDEs, the users should not need the files now that the tracked\nfiles are removed. If the tracked files reappear, then they will have\nnewer timestamps than the build artifacts, so the artifacts will need to\nbe regenerated anyway.\n\nUse the sparse-index as a data structure in order to find the sparse\ndirectories that can be safely deleted. Re-expand the index to a full\none if it was full before.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 8 +++\n builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n 3 files changed, 162 insertions(+)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex fdcf43f87cb..f9022b9d555 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n+The cone mode sparse-checkout patterns will also remove ignored files that\n+are not within the sparse-checkout definition. This is important behavior\n+to preserve the performance of the sparse index, but also matches that\n+cone mode patterns care about directories, not files. If there exist files\n+that are untracked and not ignored, then Git will not delete files within\n+that directory other than the tracked files that are now out of scope.\n+These files should be removed manually to ensure Git can behave optimally.\n+\n \n SUBMODULES\n ----------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 8ba9f13787b..b06c8f885ac 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void clean_tracked_sparse_directories(struct repository *r)\n+{\n+\tint i, was_full = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t pathlen;\n+\tstruct string_list_item *item;\n+\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n+\n+\t/*\n+\t * If we are not using cone mode patterns, then we cannot\n+\t * delete directories outside of the sparse cone.\n+\t */\n+\tif (!r || !r->index || !r->worktree)\n+\t\treturn;\n+\tinit_sparse_checkout_patterns(r->index);\n+\tif (!r->index->sparse_checkout_patterns ||\n+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn;\n+\n+\t/*\n+\t * Use the sparse index as a data structure to assist finding\n+\t * directories that are safe to delete. This conversion to a\n+\t * sparse index will not delete directories that contain\n+\t * conflicted entries or submodules.\n+\t */\n+\tif (!r->index->sparse_index) {\n+\t\t/*\n+\t\t * If something, such as a merge conflict or other concern,\n+\t\t * prevents us from converting to a sparse index, then do\n+\t\t * not try deleting files.\n+\t\t */\n+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n+\t\t\treturn;\n+\t\twas_full = 1;\n+\t}\n+\n+\tstrbuf_addstr(&path, r->worktree);\n+\tstrbuf_complete(&path, '/');\n+\tpathlen = path.len;\n+\n+\t/*\n+\t * Collect directories that have gone out of scope but also\n+\t * exist on disk, so there is some work to be done. We need to\n+\t * store the entries in a list before exploring, since that might\n+\t * expand the sparse-index again.\n+\t */\n+\tfor (i = 0; i < r->index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = r->index->cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t repo_file_exists(r, ce->name))\n+\t\t\tstring_list_append(&sparse_dirs, ce->name);\n+\t}\n+\n+\tfor_each_string_list_item(item, &sparse_dirs) {\n+\t\tstruct dir_struct dir = DIR_INIT;\n+\t\tstruct pathspec p = { 0 };\n+\t\tstruct strvec s = STRVEC_INIT;\n+\n+\t\tstrbuf_setlen(&path, pathlen);\n+\t\tstrbuf_addstr(&path, item->string);\n+\n+\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n+\n+\t\tsetup_standard_excludes(&dir);\n+\t\tstrvec_push(&s, path.buf);\n+\n+\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n+\t\tfill_directory(&dir, r->index, &p);\n+\n+\t\tif (dir.nr) {\n+\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n+\t\t\t\t \" but is not in the sparse-checkout cone\"),\n+\t\t\t\titem->string);\n+\t\t} else if (remove_dir_recursively(&path, 0)) {\n+\t\t\t/*\n+\t\t\t * Removal is \"best effort\". If something blocks\n+\t\t\t * the deletion, then continue with a warning.\n+\t\t\t */\n+\t\t\twarning(_(\"failed to remove directory '%s'\"),\n+\t\t\t\titem->string);\n+\t\t}\n+\n+\t\tdir_clear(&dir);\n+\t}\n+\n+\tstring_list_clear(&sparse_dirs, 0);\n+\tstrbuf_release(&path);\n+\n+\tif (was_full)\n+\t\tensure_full_index(r->index);\n+}\n+\n static int update_working_directory(struct pattern_list *pl)\n {\n \tenum update_sparsity_result result;\n@@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)\n \telse\n \t\trollback_lock_file(&lock_file);\n \n+\tclean_tracked_sparse_directories(r);\n+\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 38fc8340f5c..71236981e64 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n \tcheck_files repo/deep a deeper1\n '\n \n+test_expect_success 'cone mode clears ignored subdirectories' '\n+\trm repo/.git/info/sparse-checkout &&\n+\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set deep/deeper1 &&\n+\n+\tcat >repo/.gitignore <<-\\EOF &&\n+\tobj/\n+\t*.o\n+\tEOF\n+\n+\tgit -C repo add .gitignore &&\n+\tgit -C repo commit -m \".gitignore\" &&\n+\n+\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n+\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n+\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n+\tdo\n+\t\techo ignored >repo/$file || return 1\n+\tdone &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout reapply &&\n+\ttest_path_is_missing repo/folder1 &&\n+\ttest_path_is_missing repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper2 &&\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\t>repo/deep/deeper2/ignored.o &&\n+\t>repo/deep/deeper2/untracked &&\n+\n+\t# When an untracked file is in the way, all untracked files\n+\t# (even ignored files) are preserved.\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\tgrep \"contains untracked files\" err &&\n+\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n+\ttest_path_is_file repo/deep/deeper2/untracked &&\n+\n+\t# The rest of the cone matches expectation\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\techo \"? deep/deeper2/untracked\" >expect &&\n+\ttest_cmp expect out\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"432586","messageId":"bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com","threadId":"56179","inReplyTo":"e9ed5cd283087b8f23112f6f3ecbb5a282c4c717.1628625014.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/8] sparse-checkout: create helper methods","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-12T17:29:08Z","receivedAt":"2021-08-12T17:29:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/10/2021 3:50 PM, Derrick Stolee via GitGitGadget wrote:\n...\n> +enum pattern_match_result path_in_sparse_checkout(const char *path,\n> +\t\t\t\t\t\t struct index_state *istate)\n> +{\n> +\tint dtype = DT_REG;\n> +\tinit_sparse_checkout_patterns(istate);\n> +\n> +\tif (!istate->sparse_checkout_patterns)\n> +\t\treturn MATCHED;\n> +\n> +\treturn path_matches_pattern_list(path, strlen(path), NULL, &dtype,\n> +\t\t\t\t\t istate->sparse_checkout_patterns,\n> +\t\t\t\t\t istate);\n\nWhile expanding on this work to fix behavior in 'git (add|rm|mv)' around\nsparse entries, I noticed a problem with this method, specifically with\nnon-cone-mode patterns:\n\n1. The NULL here should be the \"basename\" of the path, not NULL. This doesn't\n matter for cone mode, but _does_ matter for more general patterns.\n\n2. The return type here can be UNDECIDED with general patterns, which really\n means \"not matched\" but is distinct from NOT_MATCHED because of the recursive\n assumptions when a directory is returned with NOT_MATCHED. Since the usage\n pattern for path_in_sparse_checkout() is to get a boolean result, the\n return type should switch to 'int' and we should return\n \"path_matches_pattern_list(...) > 0\".\n\nI'm still doing some more testing to ensure I've got the necessary tweaks in\nplace to work with the other changes I'm going for. Plan on me sending a v3\nwith the appropriate changes here.\n\n> \t/*\n> \t * Is the current path outside of the sparse cone?\n> \t * Then check if the region can be replaced by a sparse\n> \t * directory entry (everything is sparse and merged).\n> \t */\n> -\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n> -\t\t\t\t\t NULL, &dtype, pl, istate);\n> +\tmatch = path_in_sparse_checkout(ct_path, istate);\n> \tif (match != NOT_MATCHED)\n> \t\tcan_convert = 0;\n\nWe could remove the use of \"match\" here and use the boolean result of\npath_in_sparse_checkout() instead.\n\nThanks,\n-Stolee\n"},{"id":"432949","messageId":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v2.git.1628625013.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:14Z","receivedAt":"2021-08-17T13:23:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\nand v2.33.0-rc1 because of some changes to dir.h that became important!\n\nWe launched an experimental release [1] of the sparse-index feature to our\ninternal users. We immediately discovered a problem due to the isolated way\nin which we tested the sparse index: we were never building the project and\nchanging our sparse-checkout definition.\n\n[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n\nUsers who ran a build in one project and then moved to another still had\nbuild artifacts in their worktree that lived inside the old directories.\nSince the files are marked by the .gitignore patterns, these files were not\nremoved by the 'git sparse-checkout set' command. However, they make the\nsparse-index unusable because every 'git status' command needs to expand the\nsparse-directory entries in order to see if the files are tracked or not.\nThis made the first experimental release actually slower for all users\nbecause of this cost.\n\nThe solution we shipped to these customers was to change the way our fork\nhandles these ignored files. Specifically, instead of Git completely\nignoring the files, we changed Git to understand that with cone-mode\nsparse-checkout patterns, the users is asking for entire directories to be\nremoved from the worktree. The link [1] included earlier has this change.\n\nI believe that this is a reasonable expectation, though I recognize that it\nmight look like breaking the expectations of how .gitignore files work.\n\nSince feedback demonstrated that this is a desired behavior, v2 includes\nthis behavior for all \"cone mode\" repositories.\n\nI'm interested in the community's thoughts about this change, as it seems\nlike one that we should make carefully and intentionally.\n\nWhile the rewrite of the t7519 test seems unrelated, it is required to avoid\na test failure with this change that deletes files outside of the cone. By\nmoving the test into repositories not at $TRASH_DIRECTORY, we gain more\ncontrol over the repository structure.\n\n\nUpdate in V3\n============\n\n * As promised [2], the helper methods are fixed to work with non-cone-mode\n patterns. A later series will use them to their fullest potential\n (changing git add, git rm, and git mv when interacting with sparse\n entries).\n\n[2]\nhttps://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/\n\n\nUpdates in V2\n=============\n\n * This version correctly leaves untracked files alone. If untracked files\n are found, then the directory is left as-is, in case those ignored files\n are important to the user's work resolving those untracked files.\n\n * This behavior is now enabled by core.sparseCheckoutCone=true.\n\n * To use a sparse index as an in-memory data structure even when\n index.sparse is disabled, a new patch is included to modify the prototype\n of convert_to_sparse() to include a flags parameter.\n\n * A few cleanup patches that I was collecting based on feedback from the\n experimental release and intending for my next series were necessary for\n this implementation.\n\n * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n implementation that used run_subcommand().\n\nThanks, -Stolee\n\nDerrick Stolee (8):\n t7519: rewrite sparse index test\n sparse-index: silently return when not using cone-mode patterns\n sparse-index: silently return when cache tree fails\n unpack-trees: fix nested sparse-dir search\n sparse-checkout: create helper methods\n attr: be careful about sparse directories\n sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n sparse-checkout: clear tracked sparse dirs\n\n Documentation/git-sparse-checkout.txt | 8 +++\n attr.c | 14 ++++\n builtin/add.c | 8 +--\n builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n dir.c | 33 ++++++++++\n dir.h | 6 ++\n read-cache.c | 4 +-\n sparse-index.c | 70 +++++++++++---------\n sparse-index.h | 3 +-\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n t/t7519-status-fsmonitor.sh | 38 ++++++-----\n unpack-trees.c | 18 +++--\n 12 files changed, 294 insertions(+), 62 deletions(-)\n\n\nbase-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1009\n\nRange-diff vs v2:\n\n 1: e66106f7a99 = 1: e66106f7a99 t7519: rewrite sparse index test\n 2: fb3ff9108bf = 2: fb3ff9108bf sparse-index: silently return when not using cone-mode patterns\n 3: 37198535268 = 3: 37198535268 sparse-index: silently return when cache tree fails\n 4: 10bcadb284e = 4: 10bcadb284e unpack-trees: fix nested sparse-dir search\n 5: e9ed5cd2830 ! 5: 5d28570c82a sparse-checkout: create helper methods\n @@ dir.c: done:\n \n +int init_sparse_checkout_patterns(struct index_state *istate)\n +{\n -+\tif (istate->sparse_checkout_patterns)\n ++\tif (!core_apply_sparse_checkout ||\n ++\t istate->sparse_checkout_patterns)\n +\t\treturn 0;\n +\n +\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n @@ dir.c: done:\n +\treturn 0;\n +}\n +\n -+enum pattern_match_result path_in_sparse_checkout(const char *path,\n -+\t\t\t\t\t\t struct index_state *istate)\n ++int path_in_sparse_checkout(const char *path,\n ++\t\t\t struct index_state *istate)\n +{\n ++\tconst char *base;\n +\tint dtype = DT_REG;\n +\tinit_sparse_checkout_patterns(istate);\n +\n +\tif (!istate->sparse_checkout_patterns)\n +\t\treturn MATCHED;\n +\n -+\treturn path_matches_pattern_list(path, strlen(path), NULL, &dtype,\n ++\tbase = strrchr(path, '/');\n ++\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n ++\t\t\t\t\t &dtype,\n +\t\t\t\t\t istate->sparse_checkout_patterns,\n -+\t\t\t\t\t istate);\n ++\t\t\t\t\t istate) > 0;\n +}\n +\n static struct path_pattern *last_matching_pattern_from_lists(\n @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n +\n +int init_sparse_checkout_patterns(struct index_state *state);\n +\n -+enum pattern_match_result path_in_sparse_checkout(const char *path,\n -+\t\t\t\t\t\t struct index_state *istate);\n ++int path_in_sparse_checkout(const char *path,\n ++\t\t\t struct index_state *istate);\n +\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n 6: 5680df62e1c = 6: c9e100e68f8 attr: be careful about sparse directories\n 7: 1dd73b36eb4 = 7: b0ece4b7dcc sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n 8: f74015a2be9 = 8: febef675f05 sparse-checkout: clear tracked sparse dirs\n\n-- \ngitgitgadget\n"},{"id":"432947","messageId":"fb3ff9108bf832d258155d3c6ee95c8546f21ba5.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] sparse-index: silently return when not using cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:16Z","receivedAt":"2021-08-17T13:23:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile the sparse-index is only enabled when core.sparseCheckoutCone is\nalso enabled, it is possible for the user to modify the sparse-checkout\nfile manually in a way that does not match cone-mode patterns. In this\ncase, we should refuse to convert an index into a sparse index, since\nthe sparse_checkout_patterns will not be initialized with recursive and\nparent path hashsets.\n\nAlso silently return if there are no cache entries, which is a simple\ncase: there are no paths to make sparse!\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 17 ++++++++++++-----\n 1 file changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex c6b4feec413..bc5900eae35 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -130,7 +130,8 @@ static int index_has_unmerged_entries(struct index_state *istate)\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index ||\n+\n+\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n@@ -158,10 +159,16 @@ int convert_to_sparse(struct index_state *istate)\n \t\t\treturn 0;\n \t}\n \n-\tif (!istate->sparse_checkout_patterns->use_cone_patterns) {\n-\t\twarning(_(\"attempting to use sparse-index without cone mode\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * We need cone-mode patterns to use sparse-index. If a user edits\n+\t * their sparse-checkout file manually, then we can detect during\n+\t * parsing that they are not actually using cone-mode patterns and\n+\t * hence we need to abort this conversion _without error_. Warnings\n+\t * already exist in the pattern parsing to inform the user of their\n+\t * bad patterns.\n+\t */\n+\tif (!istate->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn 0;\n \n \t/*\n \t * NEEDSWORK: If we have unmerged entries, then stay full.\n-- \ngitgitgadget\n\n"},{"id":"432948","messageId":"e66106f7a99d94145eec983ea5e72b7cf8a8a479.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] t7519: rewrite sparse index test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:15Z","receivedAt":"2021-08-17T13:23:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse index is tested with the FS Monitor hook and extension since\nf8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\nwas very fragile because it shared an index across sparse and non-sparse\nbehavior. Since that expansion and contraction could cause the index to\nlose its FS Monitor bitmap and token, behavior is fragile to changes in\n'git sparse-checkout set'.\n\nRewrite the test to use two clones of the original repo: full and\nsparse. This allows us to also keep the test files (actual, expect,\ntrace2.txt) out of the repos we are testing with 'git status'.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex deea88d4431..6f2cf306f66 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n check_sparse_index_behavior () {\n-\tgit status --porcelain=v2 >expect &&\n-\tgit sparse-checkout init --cone --sparse-index &&\n-\tgit sparse-checkout set dir1 dir2 &&\n+\tgit -C full status --porcelain=v2 >expect &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit status --porcelain=v2 >actual &&\n+\t\tgit -C sparse status --porcelain=v2 >actual &&\n \ttest_region $1 index ensure_full_index trace2.txt &&\n \ttest_region fsm_hook query trace2.txt &&\n \ttest_cmp expect actual &&\n-\trm trace2.txt &&\n-\tgit sparse-checkout disable\n+\trm trace2.txt\n }\n \n test_expect_success 'status succeeds with sparse index' '\n-\tgit reset --hard &&\n+\tgit clone . full &&\n+\tgit clone . sparse &&\n+\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n-\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n-\tcheck_sparse_index_behavior ! &&\n-\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \tEOF\n-\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n+\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n \tcheck_sparse_index_behavior ! &&\n \n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1/modified\\0\"\n \tEOF\n \tcheck_sparse_index_behavior ! &&\n \n-\tcp -r dir1 dir1a &&\n-\tgit add dir1a &&\n-\tgit commit -m \"add dir1a\" &&\n+\tgit -C sparse sparse-checkout add dir1a &&\n+\n+\tfor repo in full sparse\n+\tdo\n+\t\tcp -r $repo/dir1 $repo/dir1a &&\n+\t\tgit -C $repo add dir1a &&\n+\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n+\tdone &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n \t# This one modifies outside the sparse-checkout definition\n \t# and hence we expect to expand the sparse-index.\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1a/modified\\0\"\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"432950","messageId":"371985352680a767dfacb5477aa77e92e04008ee.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] sparse-index: silently return when cache tree fails","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:17Z","receivedAt":"2021-08-17T13:23:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf cache_tree_update() returns a non-zero value, then it could not\ncreate the cache tree. This is likely due to a path having a merge\nconflict. Since we are already returning early, let's return silently to\navoid making it seem like we failed to write the index at all.\n\nIf we remove our dependence on the cache tree within\nconvert_to_sparse(), then we could still recover from this scenario and\nhave a sparse index.\n\nWhen constructing the cache-tree extension in convert_to_sparse(), it is\npossible that we construct a tree object that is new to the object\ndatabase. Without the WRITE_TREE_MISSING_OK flag, this results in an\nerror that halts our conversion to a sparse index. Add this flag to\nremove this limitation.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 13 +++++++++----\n 1 file changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex bc5900eae35..b6e90417556 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)\n \n \t/* Clear and recompute the cache-tree */\n \tcache_tree_free(&istate->cache_tree);\n-\tif (cache_tree_update(istate, 0)) {\n-\t\twarning(_(\"unable to update cache-tree, staying full\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * Silently return if there is a problem with the cache tree update,\n+\t * which might just be due to a conflict state in some entry.\n+\t *\n+\t * This might create new tree objects, so be sure to use\n+\t * WRITE_TREE_MISSING_OK.\n+\t */\n+\tif (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n+\t\treturn 0;\n \n \tremove_fsmonitor(istate);\n \n-- \ngitgitgadget\n\n"},{"id":"432951","messageId":"5d28570c82af19b4bda4253e72ace3760dfe2606.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] sparse-checkout: create helper methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:19Z","receivedAt":"2021-08-17T13:23:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we integrate the sparse index into more builtins, we occasionally\nneed to check the sparse-checkout patterns to see if a path is within\nthe sparse-checkout cone. Create some helper methods that help\ninitialize the patterns and check for pattern matching to make this\neasier.\n\nThe existing callers of commands like get_sparse_checkout_patterns() use\na custom 'struct pattern_list' that is not necessarily the one in the\n'struct index_state', so there are not many previous uses that could\nadopt these helpers. There are just two in builtin/add.c and\nsparse-index.c that can use path_in_sparse_checkout().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 8 ++------\n dir.c | 33 +++++++++++++++++++++++++++++++++\n dir.h | 6 ++++++\n sparse-index.c | 12 +++---------\n 4 files changed, 44 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 17528e8f922..f675bdeae4a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \tint flags = REFRESH_IGNORE_SKIP_WORKTREE |\n \t\t (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);\n-\tstruct pattern_list pl = { 0 };\n-\tint sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);\n \n \tseen = xcalloc(pathspec->nr, 1);\n \trefresh_index(&the_index, flags, pathspec, seen,\n@@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tfor (i = 0; i < pathspec->nr; i++) {\n \t\tif (!seen[i]) {\n \t\t\tconst char *path = pathspec->items[i].original;\n-\t\t\tint dtype = DT_REG;\n \n \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n-\t\t\t (sparse_checkout_enabled &&\n-\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n-\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n+\t\t\t (core_apply_sparse_checkout &&\n+\t\t\t path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec->items[i].original);\n \t\t\t} else {\ndiff --git a/dir.c b/dir.c\nindex 03c4d212672..6fd4f2e0f27 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1439,6 +1439,39 @@ done:\n \treturn result;\n }\n \n+int init_sparse_checkout_patterns(struct index_state *istate)\n+{\n+\tif (!core_apply_sparse_checkout ||\n+\t istate->sparse_checkout_patterns)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n+\n+\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {\n+\t\tFREE_AND_NULL(istate->sparse_checkout_patterns);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate)\n+{\n+\tconst char *base;\n+\tint dtype = DT_REG;\n+\tinit_sparse_checkout_patterns(istate);\n+\n+\tif (!istate->sparse_checkout_patterns)\n+\t\treturn MATCHED;\n+\n+\tbase = strrchr(path, '/');\n+\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n+\t\t\t\t\t &dtype,\n+\t\t\t\t\t istate->sparse_checkout_patterns,\n+\t\t\t\t\t istate) > 0;\n+}\n+\n static struct path_pattern *last_matching_pattern_from_lists(\n \t\tstruct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int pathlen,\ndiff --git a/dir.h b/dir.h\nindex b3e1a54a971..b899ee43d81 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,\n \t\t\t\tconst char *basename, int *dtype,\n \t\t\t\tstruct pattern_list *pl,\n \t\t\t\tstruct index_state *istate);\n+\n+int init_sparse_checkout_patterns(struct index_state *state);\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate);\n+\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n \t\t\t\t const char *pathname, int len);\ndiff --git a/sparse-index.c b/sparse-index.c\nindex b6e90417556..2efc9fd4910 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,\n \tint i, can_convert = 1;\n \tint start_converted = num_converted;\n \tenum pattern_match_result match;\n-\tint dtype = DT_UNKNOWN;\n \tstruct strbuf child_path = STRBUF_INIT;\n-\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n \n \t/*\n \t * Is the current path outside of the sparse cone?\n \t * Then check if the region can be replaced by a sparse\n \t * directory entry (everything is sparse and merged).\n \t */\n-\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n-\t\t\t\t\t NULL, &dtype, pl, istate);\n+\tmatch = path_in_sparse_checkout(ct_path, istate);\n \tif (match != NOT_MATCHED)\n \t\tcan_convert = 0;\n \n@@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo->settings.sparse_index)\n \t\treturn 0;\n \n-\tif (!istate->sparse_checkout_patterns) {\n-\t\tistate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));\n-\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n-\t\t\treturn 0;\n-\t}\n+\tif (init_sparse_checkout_patterns(istate) < 0)\n+\t\treturn 0;\n \n \t/*\n \t * We need cone-mode patterns to use sparse-index. If a user edits\n-- \ngitgitgadget\n\n"},{"id":"432952","messageId":"10bcadb284e49419f9b4baf75e05f719ec395d98.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:18Z","receivedAt":"2021-08-17T13:23:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe iterated search in find_cache_entry() was recently modified to\ninclude a loop that searches backwards for a sparse directory entry that\nmatches the given traverse_info and name_entry. However, the string\ncomparison failed to actually concatenate those two strings, so this\nfailed to find a sparse directory when it was not a top-level directory.\n\nThis caused some errors in rare cases where a 'git checkout' spanned a\ndiff that modified files within the sparse directory entry, but we could\nnot correctly find the entry.\n\nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..df1f4437723 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n-\tstruct cache_entry *ce;\n+\tstruct cache_entry *ce = NULL;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n+\tstruct strbuf full_path = STRBUF_INIT;\n \n \tif (0 <= pos)\n \t\treturn o->src_index->cache[pos];\n@@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \tif (pos < 0 || pos >= o->src_index->cache_nr)\n \t\treturn NULL;\n \n+\tstrbuf_addstr(&full_path, info->traverse_path);\n+\tstrbuf_add(&full_path, p->path, p->pathlen);\n+\tstrbuf_addch(&full_path, '/');\n+\n \t/*\n \t * Due to lexicographic sorting and sparse directory\n \t * entries ending with a trailing slash, our path as a\n@@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \twhile (pos >= 0) {\n \t\tce = o->src_index->cache[pos];\n \n-\t\tif (strncmp(ce->name, p->path, p->pathlen))\n-\t\t\treturn NULL;\n+\t\tif (strncmp(ce->name, full_path.buf, full_path.len)) {\n+\t\t\tce = NULL;\n+\t\t\tbreak;\n+\t\t}\n \n \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n \t\t sparse_dir_matches_path(ce, info, p))\n-\t\t\treturn ce;\n+\t\t\tbreak;\n \n \t\tpos--;\n \t}\n \n-\treturn NULL;\n+\tstrbuf_release(&full_path);\n+\treturn ce;\n }\n \n static void debug_path(struct traverse_info *info)\n-- \ngitgitgadget\n\n"},{"id":"432953","messageId":"febef675f051eb08896751bb5661b6deb5579ead.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:22Z","receivedAt":"2021-08-17T13:23:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen changing the scope of a sparse-checkout using cone mode, we might\nhave some tracked directories go out of scope. The current logic removes\nthe tracked files from within those directories, but leaves the ignored\nfiles within those directories. This is a bit unexpected to users who\nhave given input to Git saying they don't need those directories\nanymore.\n\nThis is something that is new to the cone mode pattern type: the user\nhas explicitly said \"I want these directories and _not_ those\ndirectories.\" The typical sparse-checkout patterns more generally apply\nto \"I want files with with these patterns\" so it is natural to leave\nignored files as they are. This focus on directories in cone mode\nprovides us an opportunity to change the behavior.\n\nLeaving these ignored files in the sparse directories makes it\nimpossible to gain performance benefits in the sparse index. When we\ntrack into these directories, we need to know if the files are ignored\nor not, which might depend on the _tracked_ .gitignore file(s) within\nthe sparse directory. This depends on the indexed version of the file,\nso the sparse directory must be expanded.\n\nBy deleting the sparse directories when changing scope (or running 'git\nsparse-checkout reapply') we regain these performance benefits as if the\nrepository was in a clean state.\n\nSince these ignored files are frequently build output or helper files\nfrom IDEs, the users should not need the files now that the tracked\nfiles are removed. If the tracked files reappear, then they will have\nnewer timestamps than the build artifacts, so the artifacts will need to\nbe regenerated anyway.\n\nUse the sparse-index as a data structure in order to find the sparse\ndirectories that can be safely deleted. Re-expand the index to a full\none if it was full before.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 8 +++\n builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n 3 files changed, 162 insertions(+)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex fdcf43f87cb..f9022b9d555 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n+The cone mode sparse-checkout patterns will also remove ignored files that\n+are not within the sparse-checkout definition. This is important behavior\n+to preserve the performance of the sparse index, but also matches that\n+cone mode patterns care about directories, not files. If there exist files\n+that are untracked and not ignored, then Git will not delete files within\n+that directory other than the tracked files that are now out of scope.\n+These files should be removed manually to ensure Git can behave optimally.\n+\n \n SUBMODULES\n ----------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 8ba9f13787b..b06c8f885ac 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void clean_tracked_sparse_directories(struct repository *r)\n+{\n+\tint i, was_full = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t pathlen;\n+\tstruct string_list_item *item;\n+\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n+\n+\t/*\n+\t * If we are not using cone mode patterns, then we cannot\n+\t * delete directories outside of the sparse cone.\n+\t */\n+\tif (!r || !r->index || !r->worktree)\n+\t\treturn;\n+\tinit_sparse_checkout_patterns(r->index);\n+\tif (!r->index->sparse_checkout_patterns ||\n+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn;\n+\n+\t/*\n+\t * Use the sparse index as a data structure to assist finding\n+\t * directories that are safe to delete. This conversion to a\n+\t * sparse index will not delete directories that contain\n+\t * conflicted entries or submodules.\n+\t */\n+\tif (!r->index->sparse_index) {\n+\t\t/*\n+\t\t * If something, such as a merge conflict or other concern,\n+\t\t * prevents us from converting to a sparse index, then do\n+\t\t * not try deleting files.\n+\t\t */\n+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n+\t\t\treturn;\n+\t\twas_full = 1;\n+\t}\n+\n+\tstrbuf_addstr(&path, r->worktree);\n+\tstrbuf_complete(&path, '/');\n+\tpathlen = path.len;\n+\n+\t/*\n+\t * Collect directories that have gone out of scope but also\n+\t * exist on disk, so there is some work to be done. We need to\n+\t * store the entries in a list before exploring, since that might\n+\t * expand the sparse-index again.\n+\t */\n+\tfor (i = 0; i < r->index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = r->index->cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t repo_file_exists(r, ce->name))\n+\t\t\tstring_list_append(&sparse_dirs, ce->name);\n+\t}\n+\n+\tfor_each_string_list_item(item, &sparse_dirs) {\n+\t\tstruct dir_struct dir = DIR_INIT;\n+\t\tstruct pathspec p = { 0 };\n+\t\tstruct strvec s = STRVEC_INIT;\n+\n+\t\tstrbuf_setlen(&path, pathlen);\n+\t\tstrbuf_addstr(&path, item->string);\n+\n+\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n+\n+\t\tsetup_standard_excludes(&dir);\n+\t\tstrvec_push(&s, path.buf);\n+\n+\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n+\t\tfill_directory(&dir, r->index, &p);\n+\n+\t\tif (dir.nr) {\n+\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n+\t\t\t\t \" but is not in the sparse-checkout cone\"),\n+\t\t\t\titem->string);\n+\t\t} else if (remove_dir_recursively(&path, 0)) {\n+\t\t\t/*\n+\t\t\t * Removal is \"best effort\". If something blocks\n+\t\t\t * the deletion, then continue with a warning.\n+\t\t\t */\n+\t\t\twarning(_(\"failed to remove directory '%s'\"),\n+\t\t\t\titem->string);\n+\t\t}\n+\n+\t\tdir_clear(&dir);\n+\t}\n+\n+\tstring_list_clear(&sparse_dirs, 0);\n+\tstrbuf_release(&path);\n+\n+\tif (was_full)\n+\t\tensure_full_index(r->index);\n+}\n+\n static int update_working_directory(struct pattern_list *pl)\n {\n \tenum update_sparsity_result result;\n@@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)\n \telse\n \t\trollback_lock_file(&lock_file);\n \n+\tclean_tracked_sparse_directories(r);\n+\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 38fc8340f5c..71236981e64 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n \tcheck_files repo/deep a deeper1\n '\n \n+test_expect_success 'cone mode clears ignored subdirectories' '\n+\trm repo/.git/info/sparse-checkout &&\n+\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set deep/deeper1 &&\n+\n+\tcat >repo/.gitignore <<-\\EOF &&\n+\tobj/\n+\t*.o\n+\tEOF\n+\n+\tgit -C repo add .gitignore &&\n+\tgit -C repo commit -m \".gitignore\" &&\n+\n+\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n+\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n+\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n+\tdo\n+\t\techo ignored >repo/$file || return 1\n+\tdone &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout reapply &&\n+\ttest_path_is_missing repo/folder1 &&\n+\ttest_path_is_missing repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper2 &&\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\t>repo/deep/deeper2/ignored.o &&\n+\t>repo/deep/deeper2/untracked &&\n+\n+\t# When an untracked file is in the way, all untracked files\n+\t# (even ignored files) are preserved.\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\tgrep \"contains untracked files\" err &&\n+\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n+\ttest_path_is_file repo/deep/deeper2/untracked &&\n+\n+\t# The rest of the cone matches expectation\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\techo \"? deep/deeper2/untracked\" >expect &&\n+\ttest_cmp expect out\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"432954","messageId":"b0ece4b7dccf0cbe0477b3a6d238ea1362aed24d.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:21Z","receivedAt":"2021-08-17T13:23:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\nenvironment variable or the \"index.sparse\" config setting before\nconverting the index to a sparse one. This is for ease of use since all\ncurrent consumers are preparing to compress the index before writing it\nto disk. If these settings are not enabled, then convert_to_sparse()\nsilently returns without doing anything.\n\nWe will add a consumer in the next change that wants to use the sparse\nindex as an in-memory data structure, regardless of whether the on-disk\nformat should be sparse.\n\nTo that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip\nthese config checks when enabled. All current consumers are modified to\npass '0' in the new 'flags' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++--\n sparse-index.c | 30 ++++++++++++++++--------------\n sparse-index.h | 3 ++-\n 3 files changed, 20 insertions(+), 17 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9048ef9e905..f5d4385c408 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \tint ret;\n \tint was_full = !istate->sparse_index;\n \n-\tret = convert_to_sparse(istate);\n+\tret = convert_to_sparse(istate, 0);\n \n \tif (ret) {\n \t\twarning(_(\"failed to convert to a sparse-index\"));\n@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,\n \tint ret, was_full = !istate->sparse_index;\n \n \tmove_cache_to_base_index(istate);\n-\tconvert_to_sparse(istate);\n+\tconvert_to_sparse(istate, 0);\n \n \ttrace2_region_enter_printf(\"index\", \"shared/do_write_index\",\n \t\t\t\t the_repository, \"%s\", get_tempfile_path(*temp));\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 2efc9fd4910..532fd11787e 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -124,7 +124,7 @@ static int index_has_unmerged_entries(struct index_state *istate)\n \treturn 0;\n }\n \n-int convert_to_sparse(struct index_state *istate)\n+int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n \n@@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n-\t/*\n-\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n-\t * index.sparse config variable to be on.\n-\t */\n-\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n-\tif (test_env >= 0)\n-\t\tset_sparse_index_config(istate->repo, test_env);\n+\tif (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n+\t\t/*\n+\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n+\t\t * index.sparse config variable to be on.\n+\t\t */\n+\t\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n+\t\tif (test_env >= 0)\n+\t\t\tset_sparse_index_config(istate->repo, test_env);\n \n-\t/*\n-\t * Only convert to sparse if index.sparse is set.\n-\t */\n-\tprepare_repo_settings(istate->repo);\n-\tif (!istate->repo->settings.sparse_index)\n-\t\treturn 0;\n+\t\t/*\n+\t\t * Only convert to sparse if index.sparse is set.\n+\t\t */\n+\t\tprepare_repo_settings(istate->repo);\n+\t\tif (!istate->repo->settings.sparse_index)\n+\t\t\treturn 0;\n+\t}\n \n \tif (init_sparse_checkout_patterns(istate) < 0)\n \t\treturn 0;\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 1115a0d7dd9..475f4f0f8da 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,8 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-int convert_to_sparse(struct index_state *istate);\n+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)\n+int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n * Some places in the codebase expect to search for a specific path.\n-- \ngitgitgadget\n\n"},{"id":"432955","messageId":"c9e100e68f80196a35a37b5d0aad74e8e1174766.1629206603.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-17T13:23:20Z","receivedAt":"2021-08-17T13:23:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/attr.c b/attr.c\nindex d029e681f28..a1009f78029 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -14,6 +14,7 @@\n #include \"utf8.h\"\n #include \"quote.h\"\n #include \"thread-utils.h\"\n+#include \"dir.h\"\n \n const char git_attr__true[] = \"(builtin)true\";\n const char git_attr__false[] = \"\\0(builtin)false\";\n@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \tif (!istate)\n \t\treturn NULL;\n \n+\t/*\n+\t * In the case of cone-mode sparse-checkout, getting the\n+\t * .gitattributes file from a directory is meaningless: all\n+\t * contained paths will be sparse if the .gitattributes is also\n+\t * sparse. In the case of a sparse index, it is critical that we\n+\t * don't go looking for one as it will expand the index.\n+\t */\n+\tinit_sparse_checkout_patterns(istate);\n+\tif (istate->sparse_checkout_patterns &&\n+\t istate->sparse_checkout_patterns->use_cone_patterns &&\n+\t path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n+\t\treturn NULL;\n+\n \tbuf = read_blob_data_from_index(istate, path, NULL);\n \tif (!buf)\n \t\treturn NULL;\n-- \ngitgitgadget\n\n"},{"id":"432984","messageId":"CABPp-BHea67AMvhPAq0SHv_YmM_bV2KoCZW2eHoUm9pxwJEq7A@mail.gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/8] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-17T14:09:42Z","receivedAt":"2021-08-17T14:09:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\n> and v2.33.0-rc1 because of some changes to dir.h that became important!\n>\n> We launched an experimental release [1] of the sparse-index feature to our\n> internal users. We immediately discovered a problem due to the isolated way\n> in which we tested the sparse index: we were never building the project and\n> changing our sparse-checkout definition.\n>\n> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n>\n> Users who ran a build in one project and then moved to another still had\n> build artifacts in their worktree that lived inside the old directories.\n> Since the files are marked by the .gitignore patterns, these files were not\n> removed by the 'git sparse-checkout set' command. However, they make the\n> sparse-index unusable because every 'git status' command needs to expand the\n> sparse-directory entries in order to see if the files are tracked or not.\n> This made the first experimental release actually slower for all users\n> because of this cost.\n>\n> The solution we shipped to these customers was to change the way our fork\n> handles these ignored files. Specifically, instead of Git completely\n> ignoring the files, we changed Git to understand that with cone-mode\n> sparse-checkout patterns, the users is asking for entire directories to be\n> removed from the worktree. The link [1] included earlier has this change.\n>\n> I believe that this is a reasonable expectation, though I recognize that it\n> might look like breaking the expectations of how .gitignore files work.\n>\n> Since feedback demonstrated that this is a desired behavior, v2 includes\n> this behavior for all \"cone mode\" repositories.\n>\n> I'm interested in the community's thoughts about this change, as it seems\n> like one that we should make carefully and intentionally.\n>\n> While the rewrite of the t7519 test seems unrelated, it is required to avoid\n> a test failure with this change that deletes files outside of the cone. By\n> moving the test into repositories not at $TRASH_DIRECTORY, we gain more\n> control over the repository structure.\n>\n>\n> Update in V3\n> ============\n>\n> * As promised [2], the helper methods are fixed to work with non-cone-mode\n> patterns. A later series will use them to their fullest potential\n> (changing git add, git rm, and git mv when interacting with sparse\n> entries).\n\nSorry I didn't review V2. I'll find some time in the next few days to\nreview V3.\n\n>\n> [2]\n> https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/\n>\n>\n> Updates in V2\n> =============\n>\n> * This version correctly leaves untracked files alone. If untracked files\n> are found, then the directory is left as-is, in case those ignored files\n> are important to the user's work resolving those untracked files.\n>\n> * This behavior is now enabled by core.sparseCheckoutCone=true.\n>\n> * To use a sparse index as an in-memory data structure even when\n> index.sparse is disabled, a new patch is included to modify the prototype\n> of convert_to_sparse() to include a flags parameter.\n>\n> * A few cleanup patches that I was collecting based on feedback from the\n> experimental release and intending for my next series were necessary for\n> this implementation.\n>\n> * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n> implementation that used run_subcommand().\n>\n> Thanks, -Stolee\n>\n> Derrick Stolee (8):\n> t7519: rewrite sparse index test\n> sparse-index: silently return when not using cone-mode patterns\n> sparse-index: silently return when cache tree fails\n> unpack-trees: fix nested sparse-dir search\n> sparse-checkout: create helper methods\n> attr: be careful about sparse directories\n> sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n> sparse-checkout: clear tracked sparse dirs\n>\n> Documentation/git-sparse-checkout.txt | 8 +++\n> attr.c | 14 ++++\n> builtin/add.c | 8 +--\n> builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n> dir.c | 33 ++++++++++\n> dir.h | 6 ++\n> read-cache.c | 4 +-\n> sparse-index.c | 70 +++++++++++---------\n> sparse-index.h | 3 +-\n> t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n> t/t7519-status-fsmonitor.sh | 38 ++++++-----\n> unpack-trees.c | 18 +++--\n> 12 files changed, 294 insertions(+), 62 deletions(-)\n>\n>\n> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v3\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v3\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1009\n>\n> Range-diff vs v2:\n>\n> 1: e66106f7a99 = 1: e66106f7a99 t7519: rewrite sparse index test\n> 2: fb3ff9108bf = 2: fb3ff9108bf sparse-index: silently return when not using cone-mode patterns\n> 3: 37198535268 = 3: 37198535268 sparse-index: silently return when cache tree fails\n> 4: 10bcadb284e = 4: 10bcadb284e unpack-trees: fix nested sparse-dir search\n> 5: e9ed5cd2830 ! 5: 5d28570c82a sparse-checkout: create helper methods\n> @@ dir.c: done:\n>\n> +int init_sparse_checkout_patterns(struct index_state *istate)\n> +{\n> -+ if (istate->sparse_checkout_patterns)\n> ++ if (!core_apply_sparse_checkout ||\n> ++ istate->sparse_checkout_patterns)\n> + return 0;\n> +\n> + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n> @@ dir.c: done:\n> + return 0;\n> +}\n> +\n> -+enum pattern_match_result path_in_sparse_checkout(const char *path,\n> -+ struct index_state *istate)\n> ++int path_in_sparse_checkout(const char *path,\n> ++ struct index_state *istate)\n> +{\n> ++ const char *base;\n> + int dtype = DT_REG;\n> + init_sparse_checkout_patterns(istate);\n> +\n> + if (!istate->sparse_checkout_patterns)\n> + return MATCHED;\n> +\n> -+ return path_matches_pattern_list(path, strlen(path), NULL, &dtype,\n> ++ base = strrchr(path, '/');\n> ++ return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n> ++ &dtype,\n> + istate->sparse_checkout_patterns,\n> -+ istate);\n> ++ istate) > 0;\n> +}\n> +\n> static struct path_pattern *last_matching_pattern_from_lists(\n> @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n> +\n> +int init_sparse_checkout_patterns(struct index_state *state);\n> +\n> -+enum pattern_match_result path_in_sparse_checkout(const char *path,\n> -+ struct index_state *istate);\n> ++int path_in_sparse_checkout(const char *path,\n> ++ struct index_state *istate);\n> +\n> struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n> struct index_state *istate,\n> 6: 5680df62e1c = 6: c9e100e68f8 attr: be careful about sparse directories\n> 7: 1dd73b36eb4 = 7: b0ece4b7dcc sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n> 8: f74015a2be9 = 8: febef675f05 sparse-checkout: clear tracked sparse dirs\n>\n> --\n> gitgitgadget\n"},{"id":"433079","messageId":"16cb1bd2-c284-6529-4aa3-b6fc7b37b407@gmail.com","threadId":"56179","inReplyTo":"b0ece4b7dccf0cbe0477b3a6d238ea1362aed24d.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/8] sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-18T18:59:21Z","receivedAt":"2021-08-18T18:59:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/17/2021 9:23 AM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\n> environment variable or the \"index.sparse\" config setting before\n> converting the index to a sparse one. This is for ease of use since all\n> current consumers are preparing to compress the index before writing it\n> to disk. If these settings are not enabled, then convert_to_sparse()\n> silently returns without doing anything.\n> \n> We will add a consumer in the next change that wants to use the sparse\n> index as an in-memory data structure, regardless of whether the on-disk\n> format should be sparse.\n> \n> To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip\n> these config checks when enabled. All current consumers are modified to\n> pass '0' in the new 'flags' parameter.\n\n...\n\n> -int convert_to_sparse(struct index_state *istate)\n> +int convert_to_sparse(struct index_state *istate, int flags)\n> {\n> \tint test_env;\n> \n> @@ -135,20 +135,22 @@ int convert_to_sparse(struct index_state *istate)\n> \tif (!istate->repo)\n> \t\tistate->repo = the_repository;\n> \n\nAbove this hunk, we fail automatically if the index has a split index.\n\nThe purpose of this flag is instead to say \"convert to sparse for the\npurpose of in-memory computations, not for writing to disk\". For such\na case, we could move the split index check to be within the hunk\nbelow. It would be appropriate to rename the flag to something like\nSPARSE_INDEX_IN_MEMORY or SPARSE_INDEX_NO_DISK_WRITE to make the\nintention more clear.\n\nThanks to SZEDER for pointing out this failure. I will fix it in the\nnext version.\n\n> -\t/*\n> -\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n> -\t * index.sparse config variable to be on.\n> -\t */\n> -\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n> -\tif (test_env >= 0)\n> -\t\tset_sparse_index_config(istate->repo, test_env);\n> +\tif (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n> +\t\t/*\n> +\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n> +\t\t * index.sparse config variable to be on.\n> +\t\t */\n> +\t\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n> +\t\tif (test_env >= 0)\n> +\t\t\tset_sparse_index_config(istate->repo, test_env);\n> \n> -\t/*\n> -\t * Only convert to sparse if index.sparse is set.\n> -\t */\n> -\tprepare_repo_settings(istate->repo);\n> -\tif (!istate->repo->settings.sparse_index)\n> -\t\treturn 0;\n> +\t\t/*\n> +\t\t * Only convert to sparse if index.sparse is set.\n> +\t\t */\n> +\t\tprepare_repo_settings(istate->repo);\n> +\t\tif (!istate->repo->settings.sparse_index)\n> +\t\t\treturn 0;\n> +\t}\n\nIf you want to try this, here is a diff that can help:\n\n--- >8 ---\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b06c8f885ac..c6a512a2107 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -131,7 +131,7 @@ static void clean_tracked_sparse_directories(struct repository *r)\n \t\t * prevents us from converting to a sparse index, then do\n \t\t * not try deleting files.\n \t\t */\n-\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))\n \t\t\treturn;\n \t\twas_full = 1;\n \t}\ndiff --git a/sparse-index.c b/sparse-index.c\nindex e0a854f9fc3..267503b21fa 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -128,14 +128,20 @@ int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n \n-\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n+\tif (istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n-\tif (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n+\tif (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {\n+\t\t/*\n+\t\t * The sparse index is not integrated with a split index.\n+\t\t */\n+\t\tif (istate->sparse_index)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n \t\t * index.sparse config variable to be on.\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 475f4f0f8da..9f3d7bc7faf 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,7 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)\n+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n\n\n--- >8 ---\n\nThanks,\n-Stolee\n"},{"id":"433112","messageId":"nycvar.QRO.7.76.6.2108190933210.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"e66106f7a99d94145eec983ea5e72b7cf8a8a479.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] t7519: rewrite sparse index test","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-19T07:45:57Z","receivedAt":"2021-08-19T07:46:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stolee,\n\nOn Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The sparse index is tested with the FS Monitor hook and extension since\n> f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\n> was very fragile because it shared an index across sparse and non-sparse\n> behavior. Since that expansion and contraction could cause the index to\n> lose its FS Monitor bitmap and token, behavior is fragile to changes in\n> 'git sparse-checkout set'.\n>\n> Rewrite the test to use two clones of the original repo: full and\n> sparse. This allows us to also keep the test files (actual, expect,\n> trace2.txt) out of the repos we are testing with 'git status'.\n\nMakes sense.\n\nIt also should accelerate the test, as it does not have to convert between\nsparse and full index all the time.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n> 1 file changed, 21 insertions(+), 17 deletions(-)\n>\n> diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\n> index deea88d4431..6f2cf306f66 100755\n> --- a/t/t7519-status-fsmonitor.sh\n> +++ b/t/t7519-status-fsmonitor.sh\n> @@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n> # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n> # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n> check_sparse_index_behavior () {\n> -\tgit status --porcelain=v2 >expect &&\n> -\tgit sparse-checkout init --cone --sparse-index &&\n> -\tgit sparse-checkout set dir1 dir2 &&\n> +\tgit -C full status --porcelain=v2 >expect &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> -\t\tgit status --porcelain=v2 >actual &&\n> +\t\tgit -C sparse status --porcelain=v2 >actual &&\n> \ttest_region $1 index ensure_full_index trace2.txt &&\n> \ttest_region fsm_hook query trace2.txt &&\n> \ttest_cmp expect actual &&\n> -\trm trace2.txt &&\n> -\tgit sparse-checkout disable\n> +\trm trace2.txt\n> }\n>\n> test_expect_success 'status succeeds with sparse index' '\n> -\tgit reset --hard &&\n> +\tgit clone . full &&\n> +\tgit clone . sparse &&\n> +\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n\nWould it make sense to call `git clone --sparse . sparse`? I see that\nthere is no support for `--sparse=cone`, which makes me wonder whether we\nwant that at some stage. In any case, cloning with `--sparse` and then\nsetting up the cone mode should result in a little less work, right?\n\n> +\tgit -C sparse sparse-checkout set dir1 dir2 &&\n>\n> -\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n> -\tcheck_sparse_index_behavior ! &&\n> -\n> -\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> +\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n> \t\tprintf \"last_update_token\\0\"\n\nTechnically, the backslash needs to be escaped because it is within double\nquotes and we do not want the shell to interpolate the `\\0`, but `printf`.\nPractically, all the shells I tried handle this as expected.\n\nAlso, I have a slight preference for:\n\n\t\tprintf \"%s\\\\0\" last_update_token\n\nand later\n\n\t\tprintf \"%s\\\\0\" last_update_token dir1/modified\n\nWhat do your taste buds say about this?\n\nThese are only minor nits, of course. I do like the overall patch and\nwould be fine with it as-is.\n\nCiao,\nDscho\n\n> \tEOF\n> -\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n> +\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n> +\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n> \tcheck_sparse_index_behavior ! &&\n>\n> -\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> +\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n> \t\tprintf \"last_update_token\\0\"\n> \t\tprintf \"dir1/modified\\0\"\n> \tEOF\n> \tcheck_sparse_index_behavior ! &&\n>\n> -\tcp -r dir1 dir1a &&\n> -\tgit add dir1a &&\n> -\tgit commit -m \"add dir1a\" &&\n> +\tgit -C sparse sparse-checkout add dir1a &&\n> +\n> +\tfor repo in full sparse\n> +\tdo\n> +\t\tcp -r $repo/dir1 $repo/dir1a &&\n> +\t\tgit -C $repo add dir1a &&\n> +\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n> +\tdone &&\n> +\tgit -C sparse sparse-checkout set dir1 dir2 &&\n>\n> \t# This one modifies outside the sparse-checkout definition\n> \t# and hence we expect to expand the sparse-index.\n> -\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n> +\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n> \t\tprintf \"last_update_token\\0\"\n> \t\tprintf \"dir1a/modified\\0\"\n> \tEOF\n> --\n> gitgitgadget\n>\n>\n"},{"id":"433114","messageId":"nycvar.QRO.7.76.6.2108191005560.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"5d28570c82af19b4bda4253e72ace3760dfe2606.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/8] sparse-checkout: create helper methods","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-19T08:07:46Z","receivedAt":"2021-08-19T08:07:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stolee,\n\nOn Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> As we integrate the sparse index into more builtins, we occasionally\n> need to check the sparse-checkout patterns to see if a path is within\n> the sparse-checkout cone. Create some helper methods that help\n> initialize the patterns and check for pattern matching to make this\n> easier.\n>\n> The existing callers of commands like get_sparse_checkout_patterns() use\n> a custom 'struct pattern_list' that is not necessarily the one in the\n> 'struct index_state', so there are not many previous uses that could\n> adopt these helpers. There are just two in builtin/add.c and\n> sparse-index.c that can use path_in_sparse_checkout().\n\nVery good!\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/add.c | 8 ++------\n> dir.c | 33 +++++++++++++++++++++++++++++++++\n> dir.h | 6 ++++++\n> sparse-index.c | 12 +++---------\n> 4 files changed, 44 insertions(+), 15 deletions(-)\n>\n> diff --git a/builtin/add.c b/builtin/add.c\n> index 17528e8f922..f675bdeae4a 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n> \tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n> \tint flags = REFRESH_IGNORE_SKIP_WORKTREE |\n> \t\t (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);\n> -\tstruct pattern_list pl = { 0 };\n> -\tint sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);\n>\n> \tseen = xcalloc(pathspec->nr, 1);\n> \trefresh_index(&the_index, flags, pathspec, seen,\n> @@ -199,12 +197,10 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n> \tfor (i = 0; i < pathspec->nr; i++) {\n> \t\tif (!seen[i]) {\n> \t\t\tconst char *path = pathspec->items[i].original;\n> -\t\t\tint dtype = DT_REG;\n>\n> \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n> -\t\t\t (sparse_checkout_enabled &&\n> -\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n> -\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n> +\t\t\t (core_apply_sparse_checkout &&\n\nDo we need to test for `core_apply_sparse_checkout` here? Or does the `if\n(!istate->sparse_checkout_patterns) return MATCHED;` early return in\n`path_in_sparse_checkout()` suffice to catch this?\n\nThe remainder of the patch looks good to me.\n\nThank you,\nDscho\n\n> +\t\t\t path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n> \t\t\t\tstring_list_append(&only_match_skip_worktree,\n> \t\t\t\t\t\t pathspec->items[i].original);\n> \t\t\t} else {\n> diff --git a/dir.c b/dir.c\n> index 03c4d212672..6fd4f2e0f27 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1439,6 +1439,39 @@ done:\n> \treturn result;\n> }\n>\n> +int init_sparse_checkout_patterns(struct index_state *istate)\n> +{\n> +\tif (!core_apply_sparse_checkout ||\n> +\t istate->sparse_checkout_patterns)\n> +\t\treturn 0;\n> +\n> +\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n> +\n> +\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {\n> +\t\tFREE_AND_NULL(istate->sparse_checkout_patterns);\n> +\t\treturn -1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> +int path_in_sparse_checkout(const char *path,\n> +\t\t\t struct index_state *istate)\n> +{\n> +\tconst char *base;\n> +\tint dtype = DT_REG;\n> +\tinit_sparse_checkout_patterns(istate);\n> +\n> +\tif (!istate->sparse_checkout_patterns)\n> +\t\treturn MATCHED;\n> +\n> +\tbase = strrchr(path, '/');\n> +\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n> +\t\t\t\t\t &dtype,\n> +\t\t\t\t\t istate->sparse_checkout_patterns,\n> +\t\t\t\t\t istate) > 0;\n> +}\n> +\n> static struct path_pattern *last_matching_pattern_from_lists(\n> \t\tstruct dir_struct *dir, struct index_state *istate,\n> \t\tconst char *pathname, int pathlen,\n> diff --git a/dir.h b/dir.h\n> index b3e1a54a971..b899ee43d81 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -394,6 +394,12 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,\n> \t\t\t\tconst char *basename, int *dtype,\n> \t\t\t\tstruct pattern_list *pl,\n> \t\t\t\tstruct index_state *istate);\n> +\n> +int init_sparse_checkout_patterns(struct index_state *state);\n> +\n> +int path_in_sparse_checkout(const char *path,\n> +\t\t\t struct index_state *istate);\n> +\n> struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n> \t\t\t\t struct index_state *istate,\n> \t\t\t\t const char *pathname, int len);\n> diff --git a/sparse-index.c b/sparse-index.c\n> index b6e90417556..2efc9fd4910 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -34,17 +34,14 @@ static int convert_to_sparse_rec(struct index_state *istate,\n> \tint i, can_convert = 1;\n> \tint start_converted = num_converted;\n> \tenum pattern_match_result match;\n> -\tint dtype = DT_UNKNOWN;\n> \tstruct strbuf child_path = STRBUF_INIT;\n> -\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n>\n> \t/*\n> \t * Is the current path outside of the sparse cone?\n> \t * Then check if the region can be replaced by a sparse\n> \t * directory entry (everything is sparse and merged).\n> \t */\n> -\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n> -\t\t\t\t\t NULL, &dtype, pl, istate);\n> +\tmatch = path_in_sparse_checkout(ct_path, istate);\n> \tif (match != NOT_MATCHED)\n> \t\tcan_convert = 0;\n>\n> @@ -153,11 +150,8 @@ int convert_to_sparse(struct index_state *istate)\n> \tif (!istate->repo->settings.sparse_index)\n> \t\treturn 0;\n>\n> -\tif (!istate->sparse_checkout_patterns) {\n> -\t\tistate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));\n> -\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n> -\t\t\treturn 0;\n> -\t}\n> +\tif (init_sparse_checkout_patterns(istate) < 0)\n> +\t\treturn 0;\n>\n> \t/*\n> \t * We need cone-mode patterns to use sparse-index. If a user edits\n> --\n> gitgitgadget\n>\n>\n"},{"id":"433115","messageId":"nycvar.QRO.7.76.6.2108190950540.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"10bcadb284e49419f9b4baf75e05f719ec395d98.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-19T08:01:20Z","receivedAt":"2021-08-19T08:09:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stolee,\n\nOn Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The iterated search in find_cache_entry() was recently modified to\n> include a loop that searches backwards for a sparse directory entry that\n> matches the given traverse_info and name_entry. However, the string\n> comparison failed to actually concatenate those two strings, so this\n> failed to find a sparse directory when it was not a top-level directory.\n>\n> This caused some errors in rare cases where a 'git checkout' spanned a\n> diff that modified files within the sparse directory entry, but we could\n> not correctly find the entry.\n\nGood explanation.\n\nI wonder a bit about the performance impact. How \"hot\" is this function?\nI.e. how often is it called, on average?\n\nI ask because I see opportunities to optimize in both directions: it could\nbe written more concisely (if speed does not matter as much), and it could\nbe made faster (if speed matters a lot). See below for more.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 18 +++++++++++++-----\n> 1 file changed, 13 insertions(+), 5 deletions(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 5786645f315..df1f4437723 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n> static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> \t\t\t\t\t const struct name_entry *p)\n> {\n> -\tstruct cache_entry *ce;\n> +\tstruct cache_entry *ce = NULL;\n\nMakes sense: since you need to release the allocated memory, you can no\nlonger `return NULL` early, but have to break out of the loop and return\n`ce`.\n\n> \tint pos = find_cache_pos(info, p->path, p->pathlen);\n> \tstruct unpack_trees_options *o = info->data;\n> +\tstruct strbuf full_path = STRBUF_INIT;\n>\n> \tif (0 <= pos)\n> \t\treturn o->src_index->cache[pos];\n> @@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> \tif (pos < 0 || pos >= o->src_index->cache_nr)\n> \t\treturn NULL;\n>\n> +\tstrbuf_addstr(&full_path, info->traverse_path);\n> +\tstrbuf_add(&full_path, p->path, p->pathlen);\n> +\tstrbuf_addch(&full_path, '/');\n\nThis could be reduced to:\n\n\tstrbuf_addf(&full_path, \"%s%.*s/\",\n\t\t info->traverse_path, (int)p->pathlen, p->path);\n\nBut if speed matters, we probably need something more like this:\n\n\tsize_t full_path_len;\n\tconst char *full_path;\n\tchar *full_path_1 = NULL;\n\n\tif (!*info->traverse_path) {\n\t\tfull_path = p->path;\n\t\tfull_path_len = p->pathlen;\n\t} else {\n\t\tsize_t len = strlen(info->traverse_path);\n\n\t\tfull_path_len = len + p->pathlen + 1;\n\t\tfull_path = full_path_1 = xmalloc(full_path_len + 1);\n\t\tmemcpy(full_path_1, info->traverse_path, len);\n\t\tmemcpy(full_path_1 + len, p->path, p->pathlen);\n\t\tfull_path_1[full_path_len - 1] = '/';\n\t\tfull_path_1[full_path_len] = '\\0';\n\t}\n\n\t[...]\n\n\tfree(full_path_1);\n\nIt would obviously be much nicer if we did not have to go for that ugly\nlong version...\n\n> +\n> \t/*\n> \t * Due to lexicographic sorting and sparse directory\n> \t * entries ending with a trailing slash, our path as a\n> @@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> \twhile (pos >= 0) {\n> \t\tce = o->src_index->cache[pos];\n>\n> -\t\tif (strncmp(ce->name, p->path, p->pathlen))\n> -\t\t\treturn NULL;\n> +\t\tif (strncmp(ce->name, full_path.buf, full_path.len)) {\n> +\t\t\tce = NULL;\n> +\t\t\tbreak;\n> +\t\t}\n>\n> \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n> \t\t sparse_dir_matches_path(ce, info, p))\n> -\t\t\treturn ce;\n> +\t\t\tbreak;\n>\n> \t\tpos--;\n> \t}\n>\n> -\treturn NULL;\n> +\tstrbuf_release(&full_path);\n> +\treturn ce;\n> }\n>\n> static void debug_path(struct traverse_info *info)\n> --\n> gitgitgadget\n>\n>\n"},{"id":"433116","messageId":"nycvar.QRO.7.76.6.2108191008160.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"c9e100e68f80196a35a37b5d0aad74e8e1174766.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-19T08:11:27Z","receivedAt":"2021-08-19T08:11:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stolee,\n\nOn Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Signed-off-by: Derrick Stolee \n> ---\n> attr.c | 14 ++++++++++++++\n> 1 file changed, 14 insertions(+)\n>\n> diff --git a/attr.c b/attr.c\n> index d029e681f28..a1009f78029 100644\n> --- a/attr.c\n> +++ b/attr.c\n> @@ -14,6 +14,7 @@\n> #include \"utf8.h\"\n> #include \"quote.h\"\n> #include \"thread-utils.h\"\n> +#include \"dir.h\"\n>\n> const char git_attr__true[] = \"(builtin)true\";\n> const char git_attr__false[] = \"\\0(builtin)false\";\n> @@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n> \tif (!istate)\n> \t\treturn NULL;\n>\n> +\t/*\n> +\t * In the case of cone-mode sparse-checkout, getting the\n> +\t * .gitattributes file from a directory is meaningless: all\n> +\t * contained paths will be sparse if the .gitattributes is also\n> +\t * sparse. In the case of a sparse index, it is critical that we\n> +\t * don't go looking for one as it will expand the index.\n> +\t */\n> +\tinit_sparse_checkout_patterns(istate);\nAt first I thought that `init_sparse_checkout_patterns()` is called by\n`path_in_sparse_checkout()` below, and therefore would not be necessary.\n\nBut it is! Without it, we have no way to test whether `use_cone_patterns`\nis set, because, well, it gets set by `init_sparse_checkout_patterns()`.\n\nWould it therefore make sense to refactor the code to have a\n`path_in_sparse_checkout_cone()` function? Or add a flag\n`only_in_cone_mode` as function parameter to `path_in_sparse_checkout()`?\n\nCiao,\nDscho\n\n> +\tif (istate->sparse_checkout_patterns &&\n> +\t istate->sparse_checkout_patterns->use_cone_patterns &&\n> +\t path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n\n> +\t\treturn NULL;\n> +\n> \tbuf = read_blob_data_from_index(istate, path, NULL);\n> \tif (!buf)\n> \t\treturn NULL;\n> --\n> gitgitgadget\n>\n>\n"},{"id":"433118","messageId":"nycvar.QRO.7.76.6.2108191015260.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"febef675f051eb08896751bb5661b6deb5579ead.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-19T08:48:30Z","receivedAt":"2021-08-19T08:49:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Stolee,\n\nOn Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> When changing the scope of a sparse-checkout using cone mode, we might\n> have some tracked directories go out of scope. The current logic removes\n> the tracked files from within those directories, but leaves the ignored\n> files within those directories. This is a bit unexpected to users who\n> have given input to Git saying they don't need those directories\n> anymore.\n>\n> This is something that is new to the cone mode pattern type: the user\n> has explicitly said \"I want these directories and _not_ those\n> directories.\" The typical sparse-checkout patterns more generally apply\n> to \"I want files with with these patterns\" so it is natural to leave\n> ignored files as they are. This focus on directories in cone mode\n> provides us an opportunity to change the behavior.\n>\n> Leaving these ignored files in the sparse directories makes it\n> impossible to gain performance benefits in the sparse index. When we\n> track into these directories, we need to know if the files are ignored\n> or not, which might depend on the _tracked_ .gitignore file(s) within\n> the sparse directory. This depends on the indexed version of the file,\n> so the sparse directory must be expanded.\n>\n> By deleting the sparse directories when changing scope (or running 'git\n> sparse-checkout reapply') we regain these performance benefits as if the\n> repository was in a clean state.\n>\n> Since these ignored files are frequently build output or helper files\n> from IDEs, the users should not need the files now that the tracked\n> files are removed. If the tracked files reappear, then they will have\n> newer timestamps than the build artifacts, so the artifacts will need to\n> be regenerated anyway.\n>\n> Use the sparse-index as a data structure in order to find the sparse\n> directories that can be safely deleted. Re-expand the index to a full\n> one if it was full before.\n\nThis description makes sense, and is easy to explain.\n\nIt does not cover the case where untracked files are found and the\ndirectory is _not_ removed as a consequence, though. I would like to ask\nto add this to the commit message, because it is kind of important.\n\nThe implementation of this behavior looks fine to me.\n\nAbout this behavior itself: in my experience, the more tricky a feature is\nto explain, the more likely the design should have been adjusted in the\nfirst place. And I find myself struggling a little bit to explain what\nfiles `git switch` touches in cone mode in addition to tracked files.\n\nSo I wonder whether an easier-to-explain behavior would be the following:\nignored files in directories that fell out of the sparse-checkout cone are\ndeleted. (Even if there are untracked, unignored files in the same\ndirectory tree.)\n\nThis is different than what this patch implements: we would now have to\ndelete the ignored and out-of-cone files _also_ when there are untracked\nfiles in the same directory, i.e. we could no longer use the sweet\n`remove_dir_recursively()` call. Therefore, the implementation of what I\nsuggested would be much more complicated: you would have to enumerate the\nignored files and remove them individually.\n\nHaving said that, even after mulling over this behavior and sleeping over\nit, I am unsure what the best way forward would be. Just because it is\neasy to explain does not make it right.\n\nIt is tricky to decide mostly because \"ignored\" files are definitely not\nalways build output. Apart from VIM's temporary files, users like me\nfrequently write other files and/or directories that we simply do not want\nto see tracked in Git. For example, I often test things in an `a1.c` file\nthat -- for convenience -- lives in the current worktree. Obviously I\ndon't want Git to track it, but I also don't want it to be deleted, so I\noften add corresponding lines to `.git/info/exclude`. Likewise, I\nsometimes download additional information related to what I am\nimplementing, and that also lives in the current worktree (but then, I\nusually am too lazy to add an entry to `.git/info/exclude` in those\ncases).\n\nNow, I don't want to over-index on my own habits. There are so many users\nout there, and most of them have different preferences from mine.\n\nWhich leaves me to wonder whether we need at least a flag to turn this\nbehavior on and off? Something like\n`core.ignoredFilesInSparseConesArePrecious = true` (obviously with a\nbetter, shorter name).\n\nCiao,\nDscho\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 8 +++\n> builtin/sparse-checkout.c | 95 +++++++++++++++++++++++++++\n> t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n> 3 files changed, 162 insertions(+)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index fdcf43f87cb..f9022b9d555 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -210,6 +210,14 @@ case-insensitive check. This corrects for case mismatched filenames in the\n> 'git sparse-checkout set' command to reflect the expected cone in the working\n> directory.\n>\n> +The cone mode sparse-checkout patterns will also remove ignored files that\n> +are not within the sparse-checkout definition. This is important behavior\n> +to preserve the performance of the sparse index, but also matches that\n> +cone mode patterns care about directories, not files. If there exist files\n> +that are untracked and not ignored, then Git will not delete files within\n> +that directory other than the tracked files that are now out of scope.\n> +These files should be removed manually to ensure Git can behave optimally.\n> +\n>\n> SUBMODULES\n> ----------\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 8ba9f13787b..b06c8f885ac 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -100,6 +100,99 @@ static int sparse_checkout_list(int argc, const char **argv)\n> \treturn 0;\n> }\n>\n> +static void clean_tracked_sparse_directories(struct repository *r)\n> +{\n> +\tint i, was_full = 0;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tsize_t pathlen;\n> +\tstruct string_list_item *item;\n> +\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n> +\n> +\t/*\n> +\t * If we are not using cone mode patterns, then we cannot\n> +\t * delete directories outside of the sparse cone.\n> +\t */\n> +\tif (!r || !r->index || !r->worktree)\n> +\t\treturn;\n> +\tinit_sparse_checkout_patterns(r->index);\n> +\tif (!r->index->sparse_checkout_patterns ||\n> +\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n> +\t\treturn;\n> +\n> +\t/*\n> +\t * Use the sparse index as a data structure to assist finding\n> +\t * directories that are safe to delete. This conversion to a\n> +\t * sparse index will not delete directories that contain\n> +\t * conflicted entries or submodules.\n> +\t */\n> +\tif (!r->index->sparse_index) {\n> +\t\t/*\n> +\t\t * If something, such as a merge conflict or other concern,\n> +\t\t * prevents us from converting to a sparse index, then do\n> +\t\t * not try deleting files.\n> +\t\t */\n> +\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n> +\t\t\treturn;\n> +\t\twas_full = 1;\n> +\t}\n> +\n> +\tstrbuf_addstr(&path, r->worktree);\n> +\tstrbuf_complete(&path, '/');\n> +\tpathlen = path.len;\n> +\n> +\t/*\n> +\t * Collect directories that have gone out of scope but also\n> +\t * exist on disk, so there is some work to be done. We need to\n> +\t * store the entries in a list before exploring, since that might\n> +\t * expand the sparse-index again.\n> +\t */\n> +\tfor (i = 0; i < r->index->cache_nr; i++) {\n> +\t\tstruct cache_entry *ce = r->index->cache[i];\n> +\n> +\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n> +\t\t repo_file_exists(r, ce->name))\n> +\t\t\tstring_list_append(&sparse_dirs, ce->name);\n> +\t}\n> +\n> +\tfor_each_string_list_item(item, &sparse_dirs) {\n> +\t\tstruct dir_struct dir = DIR_INIT;\n> +\t\tstruct pathspec p = { 0 };\n> +\t\tstruct strvec s = STRVEC_INIT;\n> +\n> +\t\tstrbuf_setlen(&path, pathlen);\n> +\t\tstrbuf_addstr(&path, item->string);\n> +\n> +\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n> +\n> +\t\tsetup_standard_excludes(&dir);\n> +\t\tstrvec_push(&s, path.buf);\n> +\n> +\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n> +\t\tfill_directory(&dir, r->index, &p);\n> +\n> +\t\tif (dir.nr) {\n> +\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n> +\t\t\t\t \" but is not in the sparse-checkout cone\"),\n> +\t\t\t\titem->string);\n> +\t\t} else if (remove_dir_recursively(&path, 0)) {\n> +\t\t\t/*\n> +\t\t\t * Removal is \"best effort\". If something blocks\n> +\t\t\t * the deletion, then continue with a warning.\n> +\t\t\t */\n> +\t\t\twarning(_(\"failed to remove directory '%s'\"),\n> +\t\t\t\titem->string);\n> +\t\t}\n> +\n> +\t\tdir_clear(&dir);\n> +\t}\n> +\n> +\tstring_list_clear(&sparse_dirs, 0);\n> +\tstrbuf_release(&path);\n> +\n> +\tif (was_full)\n> +\t\tensure_full_index(r->index);\n> +}\n> +\n> static int update_working_directory(struct pattern_list *pl)\n> {\n> \tenum update_sparsity_result result;\n> @@ -141,6 +234,8 @@ static int update_working_directory(struct pattern_list *pl)\n> \telse\n> \t\trollback_lock_file(&lock_file);\n>\n> +\tclean_tracked_sparse_directories(r);\n> +\n> \tr->index->sparse_checkout_patterns = NULL;\n> \treturn result;\n> }\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 38fc8340f5c..71236981e64 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n> \tcheck_files repo/deep a deeper1\n> '\n>\n> +test_expect_success 'cone mode clears ignored subdirectories' '\n> +\trm repo/.git/info/sparse-checkout &&\n> +\n> +\tgit -C repo sparse-checkout init --cone &&\n> +\tgit -C repo sparse-checkout set deep/deeper1 &&\n> +\n> +\tcat >repo/.gitignore <<-\\EOF &&\n> +\tobj/\n> +\t*.o\n> +\tEOF\n> +\n> +\tgit -C repo add .gitignore &&\n> +\tgit -C repo commit -m \".gitignore\" &&\n> +\n> +\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n> +\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n> +\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n> +\tdo\n> +\t\techo ignored >repo/$file || return 1\n> +\tdone &&\n> +\n> +\tgit -C repo status --porcelain=v2 >out &&\n> +\ttest_must_be_empty out &&\n> +\n> +\tgit -C repo sparse-checkout reapply &&\n> +\ttest_path_is_missing repo/folder1 &&\n> +\ttest_path_is_missing repo/deep/deeper2 &&\n> +\ttest_path_is_dir repo/obj &&\n> +\ttest_path_is_file repo/file.o &&\n> +\n> +\tgit -C repo status --porcelain=v2 >out &&\n> +\ttest_must_be_empty out &&\n> +\n> +\tgit -C repo sparse-checkout set deep/deeper2 &&\n> +\ttest_path_is_missing repo/deep/deeper1 &&\n> +\ttest_path_is_dir repo/deep/deeper2 &&\n> +\ttest_path_is_dir repo/obj &&\n> +\ttest_path_is_file repo/file.o &&\n> +\n> +\t>repo/deep/deeper2/ignored.o &&\n> +\t>repo/deep/deeper2/untracked &&\n> +\n> +\t# When an untracked file is in the way, all untracked files\n> +\t# (even ignored files) are preserved.\n> +\tgit -C repo sparse-checkout set folder1 2>err &&\n> +\tgrep \"contains untracked files\" err &&\n> +\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n> +\ttest_path_is_file repo/deep/deeper2/untracked &&\n> +\n> +\t# The rest of the cone matches expectation\n> +\ttest_path_is_missing repo/deep/deeper1 &&\n> +\ttest_path_is_dir repo/obj &&\n> +\ttest_path_is_file repo/file.o &&\n> +\n> +\tgit -C repo status --porcelain=v2 >out &&\n> +\techo \"? deep/deeper2/untracked\" >expect &&\n> +\ttest_cmp expect out\n> +'\n> +\n> test_done\n> --\n> gitgitgadget\n>\n"},{"id":"433145","messageId":"CABPp-BFYHZUtiBy1B1NqvKD-S7cT-L138H0rF4-6J_sEpPXYAQ@mail.gmail.com","threadId":"56179","inReplyTo":"371985352680a767dfacb5477aa77e92e04008ee.1628625013.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/8] sparse-index: silently return when cache tree fails","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-19T18:24:17Z","receivedAt":"2021-08-19T18:24:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 10, 2021 at 12:50 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> If cache_tree_update() returns a non-zero value, then it could not\n> create the cache tree. This is likely due to a path having a merge\n> conflict. Since we are already returning early, let's return silently to\n> avoid making it seem like we failed to write the index at all.\n>\n> If we remove our dependence on the cache tree within\n> convert_to_sparse(), then we could still recover from this scenario and\n> have a sparse index.\n>\n> When constructing the cache-tree extension in convert_to_sparse(), it is\n> possible that we construct a tree object that is new to the object\n> database. Without the WRITE_TREE_MISSING_OK flag, this results in an\n> error that halts our conversion to a sparse index. Add this flag to\n> remove this limitation.\n\nWould this only happen when the user has staged but uncommitted\nchanges outside the sparsity paths, and tries to sparsify while in\nthat state? (Notably, this is a much different condition than the\nabove mentioned merge conflict case that would case\ncache_tree_update() to just fail.)\n\nI think it might be nicer to split this commit in two, just to make it\neasier to understand for future readers. This seems like two logical\nchanges and trying to understand them and why would I think be easier\nif the two were split. I'd be tempted to put the\nWRITE_TREE_MISSING_OK first.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> sparse-index.c | 13 +++++++++----\n> 1 file changed, 9 insertions(+), 4 deletions(-)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index bc5900eae35..b6e90417556 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -179,10 +179,15 @@ int convert_to_sparse(struct index_state *istate)\n>\n> /* Clear and recompute the cache-tree */\n> cache_tree_free(&istate->cache_tree);\n> - if (cache_tree_update(istate, 0)) {\n> - warning(_(\"unable to update cache-tree, staying full\"));\n> - return -1;\n> - }\n> + /*\n> + * Silently return if there is a problem with the cache tree update,\n> + * which might just be due to a conflict state in some entry.\n> + *\n> + * This might create new tree objects, so be sure to use\n> + * WRITE_TREE_MISSING_OK.\n> + */\n> + if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n> + return 0;\n>\n> remove_fsmonitor(istate);\n>\n> --\n> gitgitgadget\n\nThese feel like cases where it would be nice to have a testcase\ndemonstrating the change in behavior. Perhaps just splitting the\ncommit would be enough, but it took a bit of time to try to understand\nwhat would change and why, despite the simple changes.\n"},{"id":"433147","messageId":"CABPp-BG4gdX7G-a=jthfGVR_TJ=OxC210Ed6++rMkDwUe2QbiQ@mail.gmail.com","threadId":"56179","inReplyTo":"10bcadb284e49419f9b4baf75e05f719ec395d98.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-19T18:29:04Z","receivedAt":"2021-08-19T18:29:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The iterated search in find_cache_entry() was recently modified to\n> include a loop that searches backwards for a sparse directory entry that\n> matches the given traverse_info and name_entry. However, the string\n> comparison failed to actually concatenate those two strings, so this\n> failed to find a sparse directory when it was not a top-level directory.\n>\n> This caused some errors in rare cases where a 'git checkout' spanned a\n> diff that modified files within the sparse directory entry, but we could\n> not correctly find the entry.\n\nDoh, sorry for missing that in my review of the earlier series.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 18 +++++++++++++-----\n> 1 file changed, 13 insertions(+), 5 deletions(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 5786645f315..df1f4437723 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1255,9 +1255,10 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n> static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> const struct name_entry *p)\n> {\n> - struct cache_entry *ce;\n> + struct cache_entry *ce = NULL;\n> int pos = find_cache_pos(info, p->path, p->pathlen);\n> struct unpack_trees_options *o = info->data;\n> + struct strbuf full_path = STRBUF_INIT;\n>\n> if (0 <= pos)\n> return o->src_index->cache[pos];\n> @@ -1273,6 +1274,10 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> if (pos < 0 || pos >= o->src_index->cache_nr)\n> return NULL;\n>\n> + strbuf_addstr(&full_path, info->traverse_path);\n> + strbuf_add(&full_path, p->path, p->pathlen);\n> + strbuf_addch(&full_path, '/');\n> +\n> /*\n> * Due to lexicographic sorting and sparse directory\n> * entries ending with a trailing slash, our path as a\n> @@ -1283,17 +1288,20 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> while (pos >= 0) {\n> ce = o->src_index->cache[pos];\n>\n> - if (strncmp(ce->name, p->path, p->pathlen))\n> - return NULL;\n> + if (strncmp(ce->name, full_path.buf, full_path.len)) {\n> + ce = NULL;\n> + break;\n> + }\n>\n> if (S_ISSPARSEDIR(ce->ce_mode) &&\n> sparse_dir_matches_path(ce, info, p))\n> - return ce;\n> + break;\n>\n> pos--;\n> }\n>\n> - return NULL;\n> + strbuf_release(&full_path);\n> + return ce;\n> }\n>\n> static void debug_path(struct traverse_info *info)\n> --\n> gitgitgadget\n\nMakes sense.\n"},{"id":"433170","messageId":"CABPp-BHqTSv7MkS8-nq3Qg3CA3pLxSD9TOVvL_8R1-eF3Rn7pQ@mail.gmail.com","threadId":"56179","inReplyTo":"c9e100e68f80196a35a37b5d0aad74e8e1174766.1629206603.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-19T20:53:51Z","receivedAt":"2021-08-19T20:54:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Signed-off-by: Derrick Stolee \n> ---\n> attr.c | 14 ++++++++++++++\n> 1 file changed, 14 insertions(+)\n>\n> diff --git a/attr.c b/attr.c\n> index d029e681f28..a1009f78029 100644\n> --- a/attr.c\n> +++ b/attr.c\n> @@ -14,6 +14,7 @@\n> #include \"utf8.h\"\n> #include \"quote.h\"\n> #include \"thread-utils.h\"\n> +#include \"dir.h\"\n>\n> const char git_attr__true[] = \"(builtin)true\";\n> const char git_attr__false[] = \"\\0(builtin)false\";\n> @@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n> if (!istate)\n> return NULL;\n>\n> + /*\n> + * In the case of cone-mode sparse-checkout, getting the\n> + * .gitattributes file from a directory is meaningless: all\n> + * contained paths will be sparse if the .gitattributes is also\n> + * sparse. In the case of a sparse index, it is critical that we\n> + * don't go looking for one as it will expand the index.\n> + */\n\n\"all contained paths will be sparse if the .gitattributes is also sparse\"?\n\nDo you mean something more like \"the .gitattributes only applies for\nfiles under the given directory, and if the directory is sparse, then\nneither the .gitattributes file or any other file under that directory\nwill be present\" ?\n\nAlso, out of curiosity, I was suggesting in the past we do something\nlike this for .gitignore files, for the same reason. Do we have such\nlogic in place, or is that another area of the code that hasn't been\nhandled yet?\n\n> + init_sparse_checkout_patterns(istate);\n> + if (istate->sparse_checkout_patterns &&\n> + istate->sparse_checkout_patterns->use_cone_patterns &&\n> + path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n> + return NULL;\n> +\n> buf = read_blob_data_from_index(istate, path, NULL);\n> if (!buf)\n> return NULL;\n> --\n> gitgitgadget\n\nThough the code appears correct, I too am curious about the questions\nDscho asked about the code in this patch.\n"},{"id":"433225","messageId":"4ec57b02-fcec-cc7a-69c4-5d593fbeb648@gmail.com","threadId":"56179","inReplyTo":"CABPp-BFYHZUtiBy1B1NqvKD-S7cT-L138H0rF4-6J_sEpPXYAQ@mail.gmail.com","subject":"Re: [PATCH v2 3/8] sparse-index: silently return when cache tree fails","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:04:52Z","receivedAt":"2021-08-20T15:04:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 2:24 PM, Elijah Newren wrote:\n> On Tue, Aug 10, 2021 at 12:50 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> If cache_tree_update() returns a non-zero value, then it could not\n>> create the cache tree. This is likely due to a path having a merge\n>> conflict. Since we are already returning early, let's return silently to\n>> avoid making it seem like we failed to write the index at all.\n>>\n>> If we remove our dependence on the cache tree within\n>> convert_to_sparse(), then we could still recover from this scenario and\n>> have a sparse index.\n>>\n>> When constructing the cache-tree extension in convert_to_sparse(), it is\n>> possible that we construct a tree object that is new to the object\n>> database. Without the WRITE_TREE_MISSING_OK flag, this results in an\n>> error that halts our conversion to a sparse index. Add this flag to\n>> remove this limitation.\n> \n> Would this only happen when the user has staged but uncommitted\n> changes outside the sparsity paths, and tries to sparsify while in\n> that state? (Notably, this is a much different condition than the\n> above mentioned merge conflict case that would case\n> cache_tree_update() to just fail.)\n> \n> I think it might be nicer to split this commit in two, just to make it\n> easier to understand for future readers. This seems like two logical\n> changes and trying to understand them and why would I think be easier\n> if the two were split. I'd be tempted to put the\n> WRITE_TREE_MISSING_OK first.\n\nIronically, I _had_ this as two commits because I discovered the\nproblems independently. It wasn't until I was organizing things that\nI realized I was editing the same 'if' twice and thought it better\nto merge patches. But I also don't feel strongly about that, so I\ncan split them.\n\n>> + /*\n>> + * Silently return if there is a problem with the cache tree update,\n>> + * which might just be due to a conflict state in some entry.\n>> + *\n>> + * This might create new tree objects, so be sure to use\n>> + * WRITE_TREE_MISSING_OK.\n>> + */\n>> + if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n>> + return 0;\n...\n> These feel like cases where it would be nice to have a testcase\n> demonstrating the change in behavior. Perhaps just splitting the\n> commit would be enough, but it took a bit of time to try to understand\n> what would change and why, despite the simple changes.\n\nI found these were required in the Scalar functional tests, so I bet\nthat if I remove this change I can create a test case from that. Thanks.\n\n-Stolee\n"},{"id":"433226","messageId":"24b1b932-caf9-b709-baf4-a2f8669deb88@gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108190933210.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 1/8] t7519: rewrite sparse index test","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:09:38Z","receivedAt":"2021-08-20T15:09:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 3:45 AM, Johannes Schindelin wrote:> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n>> test_expect_success 'status succeeds with sparse index' '\n>> -\tgit reset --hard &&\n>> +\tgit clone . full &&\n>> +\tgit clone . sparse &&\n>> +\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n> \n> Would it make sense to call `git clone --sparse . sparse`? I see that\n> there is no support for `--sparse=cone`, which makes me wonder whether we\n> want that at some stage. In any case, cloning with `--sparse` and then\n> setting up the cone mode should result in a little less work, right?\n\nThe amount of work saved is miniscule, but I can understand wanting to\nshave what we can from the cost of tests.\n\n>> +\tgit -C sparse sparse-checkout set dir1 dir2 &&\n>>\n>> -\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n>> -\tcheck_sparse_index_behavior ! &&\n>> -\n>> -\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n>> +\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n>> \t\tprintf \"last_update_token\\0\"\n> \n> Technically, the backslash needs to be escaped because it is within double\n> quotes and we do not want the shell to interpolate the `\\0`, but `printf`.\n> Practically, all the shells I tried handle this as expected.\n> \n> Also, I have a slight preference for:\n> \n> \t\tprintf \"%s\\\\0\" last_update_token\n> \n> and later\n> \n> \t\tprintf \"%s\\\\0\" last_update_token dir1/modified\n> \n> What do your taste buds say about this?\n\nI have no opinions on this one way or another. I will just point out that\nthe pattern used in this test is also used throughout the test script, so\nany change to that format should be applied universally. The test has been\noperating without complaint since it was introduced in 5c8cdcf (fsmonitor:\nadd test cases for fsmonitor extension, 2017-09-22), so compatibility is\nlikely not a problem.\n\nThanks,\n-Stolee\n"},{"id":"433227","messageId":"b7bd8e73-de86-c563-8b7d-405310ce6c57@gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108190950540.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:18:18Z","receivedAt":"2021-08-20T15:18:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 4:01 AM, Johannes Schindelin wrote:\n> Hi Stolee,\n> \n> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> The iterated search in find_cache_entry() was recently modified to\n>> include a loop that searches backwards for a sparse directory entry that\n>> matches the given traverse_info and name_entry. However, the string\n>> comparison failed to actually concatenate those two strings, so this\n>> failed to find a sparse directory when it was not a top-level directory.\n>>\n>> This caused some errors in rare cases where a 'git checkout' spanned a\n>> diff that modified files within the sparse directory entry, but we could\n>> not correctly find the entry.\n> \n> Good explanation.\n> \n> I wonder a bit about the performance impact. How \"hot\" is this function?\n> I.e. how often is it called, on average?\n> \n> I ask because I see opportunities to optimize in both directions: it could\n> be written more concisely (if speed does not matter as much), and it could\n> be made faster (if speed matters a lot). See below for more.\n\nI would definitely optimize for speed here. This can be a very hot path,\nI believe.\n\n>> +\tstrbuf_addstr(&full_path, info->traverse_path);\n>> +\tstrbuf_add(&full_path, p->path, p->pathlen);\n>> +\tstrbuf_addch(&full_path, '/');\n> \n> This could be reduced to:\n> \n> \tstrbuf_addf(&full_path, \"%s%.*s/\",\n> \t\t info->traverse_path, (int)p->pathlen, p->path);\n\nWe should definitely avoid formatted strings here, if possible.\n\n> But if speed matters, we probably need something more like this:\n> \n> \tsize_t full_path_len;\n> \tconst char *full_path;\n> \tchar *full_path_1 = NULL;\n> \n> \tif (!*info->traverse_path) {\n> \t\tfull_path = p->path;\n> \t\tfull_path_len = p->pathlen;\n> \t} else {\n> \t\tsize_t len = strlen(info->traverse_path);\n> \n> \t\tfull_path_len = len + p->pathlen + 1;\n> \t\tfull_path = full_path_1 = xmalloc(full_path_len + 1);\n> \t\tmemcpy(full_path_1, info->traverse_path, len);\n> \t\tmemcpy(full_path_1 + len, p->path, p->pathlen);\n> \t\tfull_path_1[full_path_len - 1] = '/';\n> \t\tfull_path_1[full_path_len] = '\\0';\n> \t}\n\nThe critical benefit here is that we do not need to allocate a\nbuffer if the traverse_path does not exist. That might be a\nworthwhile investment. That leads to justifying the use of\nbare 'char *'s instead of 'struct strbuf'.\n\nIf the traverse_path is usually non-null, then we could continue using\nstrbufs as a helper and get the planned performance gains by using\nstrbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()\n(instead of strbuf_addstr()). That would make this code a bit less\nugly with the only real overhead being the extra insertions of '\\0'\ncharacters as we add the strings to the strbuf().\n\nI will need to investigate so see which one is the best.\n\nThanks,\n-Stolee\n"},{"id":"433228","messageId":"f9c451e4-c1b2-1680-944a-e3355248cc44@gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108191005560.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 5/8] sparse-checkout: create helper methods","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:30:40Z","receivedAt":"2021-08-20T15:30:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 4:07 AM, Johannes Schindelin wrote:\n...\n>> \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n>> -\t\t\t (sparse_checkout_enabled &&\n>> -\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n>> -\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n>> +\t\t\t (core_apply_sparse_checkout &&\n> \n> Do we need to test for `core_apply_sparse_checkout` here? Or does the `if\n> (!istate->sparse_checkout_patterns) return MATCHED;` early return in\n> `path_in_sparse_checkout()` suffice to catch this?\n> \n> The remainder of the patch looks good to me.\n> \n> Thank you,\n> Dscho\n> \n>> +\t\t\t path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n\nThank you for pointing out this. This is actually a stale change from an earlier\nversion where path_in_sparse_checkout returned an 'enum pattern_match_result'\nbut now casts down to an 'int', meaning '0' is not in the sparse-checkout and '1'\nis that it _is_ in the sparse-checkout.\n\n>> +int path_in_sparse_checkout(const char *path,\n>> +\t\t\t struct index_state *istate)\n>> +{\n>> +\tconst char *base;\n>> +\tint dtype = DT_REG;\n>> +\tinit_sparse_checkout_patterns(istate);\n>> +\n>> +\tif (!istate->sparse_checkout_patterns)\n>> +\t\treturn MATCHED;\n\nSo here, if we do not have sparse-checkout patterns (for example, if\ncore_apply_sparse_checkout is false), then this returns MATCHED (== 1).\n\nTo be extra clear, this should just be 'return 1;'.\n\n>> +\tbase = strrchr(path, '/');\n>> +\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n>> +\t\t\t\t\t &dtype,\n>> +\t\t\t\t\t istate->sparse_checkout_patterns,\n>> +\t\t\t\t\t istate) > 0;\n\nHere, we are selecting the portion of 'enum pattern_match_result' that\nwe care about (MATCHED and MATCHED_RECURSIVE). The UNMATCHED (0) and\nUNDECIDED (-1) are the other possibilities, but file paths will not\nreturn UNDECIDED, that is instead for directories in non-cone mode\npatterns.\n\nThanks,\n-Stolee\n"},{"id":"433229","messageId":"b101db8d-0156-c90a-1324-e916c0a3b979@gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108191008160.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:36:51Z","receivedAt":"2021-08-20T15:37:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 4:11 AM, Johannes Schindelin wrote:\n> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n>> +\t/*\n>> +\t * In the case of cone-mode sparse-checkout, getting the\n>> +\t * .gitattributes file from a directory is meaningless: all\n>> +\t * contained paths will be sparse if the .gitattributes is also\n>> +\t * sparse. In the case of a sparse index, it is critical that we\n>> +\t * don't go looking for one as it will expand the index.\n>> +\t */\n>> +\tinit_sparse_checkout_patterns(istate);\n> At first I thought that `init_sparse_checkout_patterns()` is called by\n> `path_in_sparse_checkout()` below, and therefore would not be necessary.\n> \n> But it is! Without it, we have no way to test whether `use_cone_patterns`\n> is set, because, well, it gets set by `init_sparse_checkout_patterns()`.\n> \n> Would it therefore make sense to refactor the code to have a\n> `path_in_sparse_checkout_cone()` function? Or add a flag\n> `only_in_cone_mode` as function parameter to `path_in_sparse_checkout()`?\n\nOne way to clean this up as-is would be to replace\n \n>> +\tif (istate->sparse_checkout_patterns &&\n>> +\t istate->sparse_checkout_patterns->use_cone_patterns &&\n>> +\t path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n\nwith\n\n\tif (!path_in_sparse_checkout(path, istate) &&\n\t istate->sparse_checkout_patterns->use_cone_patterns)\n\nto get a similar effect. However, it creates wasted pattern-match\nchecks when not in cone-mode, so you are probably right that a\npath_in_cone_mode_sparse_checkout() would be best to short-circuit\nin the non-cone-mode case.\n\nThis special casing should be rare, so I don't think it is worth\nadding a flag to path_in_sparse_checkout() and making those call\nsites more complicated.\n\nThanks,\n-Stolee\n"},{"id":"433231","messageId":"07cc5a17-024a-910c-35c5-0dc468172f5e@gmail.com","threadId":"56179","inReplyTo":"CABPp-BHqTSv7MkS8-nq3Qg3CA3pLxSD9TOVvL_8R1-eF3Rn7pQ@mail.gmail.com","subject":"Re: [PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:39:10Z","receivedAt":"2021-08-20T15:39:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 4:53 PM, Elijah Newren wrote:\n> On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> + /*\n>> + * In the case of cone-mode sparse-checkout, getting the\n>> + * .gitattributes file from a directory is meaningless: all\n>> + * contained paths will be sparse if the .gitattributes is also\n>> + * sparse. In the case of a sparse index, it is critical that we\n>> + * don't go looking for one as it will expand the index.\n>> + */\n> \n> \"all contained paths will be sparse if the .gitattributes is also sparse\"?\n> \n> Do you mean something more like \"the .gitattributes only applies for\n> files under the given directory, and if the directory is sparse, then\n> neither the .gitattributes file or any other file under that directory\n> will be present\" ?\n\nYes, you understand correctly and explain it better. Thanks.\n \n> Also, out of curiosity, I was suggesting in the past we do something\n> like this for .gitignore files, for the same reason. Do we have such\n> logic in place, or is that another area of the code that hasn't been\n> handled yet?\n\nI don't believe this has been handled. It definitely is less obvious\nwhat to do there, because the point of .gitignore is to skip files that\nexist in the working tree even if Git didn't put them there. Meanwhile,\n.gitattributes is about how Git writes tracked files, but Git doesn't\nwrite sparse tracked files.\n\n> Though the code appears correct, I too am curious about the questions\n> Dscho asked about the code in this patch.\n \nThanks,\n-Stolee\n"},{"id":"433237","messageId":"643d67a9-054f-bc22-cf30-763fc082e9d3@gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108191015260.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-20T15:49:58Z","receivedAt":"2021-08-20T15:50:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2021 4:48 AM, Johannes Schindelin wrote:\n> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n> This description makes sense, and is easy to explain.\n> \n> It does not cover the case where untracked files are found and the\n> directory is _not_ removed as a consequence, though. I would like to ask\n> to add this to the commit message, because it is kind of important.\n\nRight. I should have modified the message from my earlier version when\nthe issues with untracked files came up.\n \n> The implementation of this behavior looks fine to me.\n> \n> About this behavior itself: in my experience, the more tricky a feature is\n> to explain, the more likely the design should have been adjusted in the\n> first place. And I find myself struggling a little bit to explain what\n> files `git switch` touches in cone mode in addition to tracked files.\n\nKeep in mind that 'git switch' does not change the sparse-checkout cone,\nand the activity being described is something that would happen within\n'git sparse-checkout set' or 'git sparse-checkout reapply'.\n\n> So I wonder whether an easier-to-explain behavior would be the following:\n> ignored files in directories that fell out of the sparse-checkout cone are\n> deleted. (Even if there are untracked, unignored files in the same\n> directory tree.)\n> \n> This is different than what this patch implements: we would now have to\n> delete the ignored and out-of-cone files _also_ when there are untracked\n> files in the same directory, i.e. we could no longer use the sweet\n> `remove_dir_recursively()` call. Therefore, the implementation of what I\n> suggested would be much more complicated: you would have to enumerate the\n> ignored files and remove them individually.\n\nOutside of \"it's harder to write that feature\", perhaps I could convince\nyou that it is better to do nothing in the presence of untracked files.\n\nIf a user has an untracked file within a directory that is leaving the\nsparse cone, then that means they were doing something in that space and\nperhaps has unfinished work. By leaving the files on-disk, they have an\nopportunity to revert the change to the sparse-checkout cone and continue\ntheir work interrupted. This includes keeping things like build artifacts\n(that are ignored) so they can incrementally build from that position.\n\nThe general thought I have here is: having untracked, not-ignored files\nin a directory that is leaving the sparse-checkout cone is an unexpected\nbehavior, so we should do as little as possible in that scenario.\n\nIt also makes it more clear to the user what happened: \"You had untracked\nfiles here, so we left them alone\" is easier to understand than \"You had\nuntracked files here, so we deleted the ones that were ignored and kept\nthe rest.\"\n\n> Having said that, even after mulling over this behavior and sleeping over\n> it, I am unsure what the best way forward would be. Just because it is\n> easy to explain does not make it right.\n\nOf course, you already have a retort to my claim that \"simpler is better\",\nbut I'll just focus on the point that \"simpler for the user to understand\"\nis a different point than \"simpler to implement\".\n\n> Which leaves me to wonder whether we need at least a flag to turn this\n> behavior on and off? Something like\n> `core.ignoredFilesInSparseConesArePrecious = true` (obviously with a\n> better, shorter name).\n\nYou are right that there are a lot of users out there, and they have\nvery different habits. Those habits are shaped by the way Git has worked\nfor a long time, so adding a way to go back to the previous behavior\nwould be good to have.\n\nThanks,\n-Stolee\n"},{"id":"433239","messageId":"CABPp-BFUoPQUQDRaRj3Rq2gShOD8WPv6Oh-Wdj4m-7sf=okY6Q@mail.gmail.com","threadId":"56179","inReplyTo":"nycvar.QRO.7.76.6.2108191015260.55@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-20T15:56:26Z","receivedAt":"2021-08-20T15:56:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Aug 19, 2021 at 1:48 AM Johannes Schindelin\n wrote:\n>\n> Hi Stolee,\n>\n> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n>\n> > From: Derrick Stolee \n> >\n> > When changing the scope of a sparse-checkout using cone mode, we might\n> > have some tracked directories go out of scope. The current logic removes\n> > the tracked files from within those directories, but leaves the ignored\n> > files within those directories. This is a bit unexpected to users who\n> > have given input to Git saying they don't need those directories\n> > anymore.\n> >\n> > This is something that is new to the cone mode pattern type: the user\n> > has explicitly said \"I want these directories and _not_ those\n> > directories.\" The typical sparse-checkout patterns more generally apply\n> > to \"I want files with with these patterns\" so it is natural to leave\n> > ignored files as they are. This focus on directories in cone mode\n> > provides us an opportunity to change the behavior.\n> >\n> > Leaving these ignored files in the sparse directories makes it\n> > impossible to gain performance benefits in the sparse index. When we\n> > track into these directories, we need to know if the files are ignored\n> > or not, which might depend on the _tracked_ .gitignore file(s) within\n> > the sparse directory. This depends on the indexed version of the file,\n> > so the sparse directory must be expanded.\n> >\n> > By deleting the sparse directories when changing scope (or running 'git\n> > sparse-checkout reapply') we regain these performance benefits as if the\n> > repository was in a clean state.\n> >\n> > Since these ignored files are frequently build output or helper files\n> > from IDEs, the users should not need the files now that the tracked\n> > files are removed. If the tracked files reappear, then they will have\n> > newer timestamps than the build artifacts, so the artifacts will need to\n> > be regenerated anyway.\n> >\n> > Use the sparse-index as a data structure in order to find the sparse\n> > directories that can be safely deleted. Re-expand the index to a full\n> > one if it was full before.\n>\n> This description makes sense, and is easy to explain.\n>\n> It does not cover the case where untracked files are found and the\n> directory is _not_ removed as a consequence, though. I would like to ask\n> to add this to the commit message, because it is kind of important.\n>\n> The implementation of this behavior looks fine to me.\n>\n> About this behavior itself: in my experience, the more tricky a feature is\n> to explain, the more likely the design should have been adjusted in the\n> first place. And I find myself struggling a little bit to explain what\n> files `git switch` touches in cone mode in addition to tracked files.\n\nI share this concern.\n\n> So I wonder whether an easier-to-explain behavior would be the following:\n> ignored files in directories that fell out of the sparse-checkout cone are\n> deleted. (Even if there are untracked, unignored files in the same\n> directory tree.)\n>\n> This is different than what this patch implements: we would now have to\n> delete the ignored and out-of-cone files _also_ when there are untracked\n> files in the same directory, i.e. we could no longer use the sweet\n> `remove_dir_recursively()` call. Therefore, the implementation of what I\n> suggested would be much more complicated: you would have to enumerate the\n> ignored files and remove them individually.\n\nThe ignored files are already enumerated by the fill_directory call;\nyou just need to iterate over the dir->ignored_nr entries in\ndir->ignored.\n\n> Having said that, even after mulling over this behavior and sleeping over\n> it, I am unsure what the best way forward would be. Just because it is\n> easy to explain does not make it right.\n>\n> It is tricky to decide mostly because \"ignored\" files are definitely not\n> always build output. Apart from VIM's temporary files, users like me\n> frequently write other files and/or directories that we simply do not want\n> to see tracked in Git. For example, I often test things in an `a1.c` file\n> that -- for convenience -- lives in the current worktree. Obviously I\n> don't want Git to track it, but I also don't want it to be deleted, so I\n> often add corresponding lines to `.git/info/exclude`. Likewise, I\n> sometimes download additional information related to what I am\n> implementing, and that also lives in the current worktree (but then, I\n> usually am too lazy to add an entry to `.git/info/exclude` in those\n> cases).\n\nI do the same thing, and I know other users that do as well...but I\ndon't put such files in directories that are irrelevant to me. I\ncreate cruft files near other files that I'm working on, or in a\nspecial directory of its own, but not under some directory that is\nirrelevant to the areas I'm working on.\n\nFor reference, we implemented something like this in our `sparsify`\nwrapper we have internally, where 'git clean -fdX ` is executed whenever folks sparsify. (We have our own\ntool and don't have users use sparse-checkout directly, because our\ntool computes dependencies to determine which directories are needed.)\n I was really hesitant to add that cleaning behavior by default, and\njust made it an option. My colleagues tired of all the bug reports\nabout left-around directories and made it the default, waiting to hear\ncomplaints. We never got one. It's been over a year.\n\n> Now, I don't want to over-index on my own habits. There are so many users\n> out there, and most of them have different preferences from mine.\n>\n> Which leaves me to wonder whether we need at least a flag to turn this\n> behavior on and off? Something like\n> `core.ignoredFilesInSparseConesArePrecious = true` (obviously with a\n> better, shorter name).\n\nThat seems fine, but I suspect you're projecting the same concerns I\nhad, and it turns out much less useful than you'd expect (perhaps even\ntotally useless).\n"},{"id":"433241","messageId":"CABPp-BEf0JEc=VgjFVcpF9uZq8aD492Pkf-OzQC_us6RXx6drg@mail.gmail.com","threadId":"56179","inReplyTo":"07cc5a17-024a-910c-35c5-0dc468172f5e@gmail.com","subject":"Re: [PATCH v3 6/8] attr: be careful about sparse directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-20T16:05:05Z","receivedAt":"2021-08-20T16:11:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Aug 20, 2021 at 8:39 AM Derrick Stolee wrote:\n>\n> On 8/19/2021 4:53 PM, Elijah Newren wrote:\n> > On Tue, Aug 17, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >> + /*\n> >> + * In the case of cone-mode sparse-checkout, getting the\n> >> + * .gitattributes file from a directory is meaningless: all\n> >> + * contained paths will be sparse if the .gitattributes is also\n> >> + * sparse. In the case of a sparse index, it is critical that we\n> >> + * don't go looking for one as it will expand the index.\n> >> + */\n> >\n> > \"all contained paths will be sparse if the .gitattributes is also sparse\"?\n> >\n> > Do you mean something more like \"the .gitattributes only applies for\n> > files under the given directory, and if the directory is sparse, then\n> > neither the .gitattributes file or any other file under that directory\n> > will be present\" ?\n>\n> Yes, you understand correctly and explain it better. Thanks.\n>\n> > Also, out of curiosity, I was suggesting in the past we do something\n> > like this for .gitignore files, for the same reason. Do we have such\n> > logic in place, or is that another area of the code that hasn't been\n> > handled yet?\n>\n> I don't believe this has been handled. It definitely is less obvious\n> what to do there, because the point of .gitignore is to skip files that\n> exist in the working tree even if Git didn't put them there. Meanwhile,\n> .gitattributes is about how Git writes tracked files, but Git doesn't\n> write sparse tracked files.\n\nWell, one advantage of deleting ignored files in sparse directories\nwhen we sparsify, is that we know if any files are left, they are all\nuntracked and not ignored. So we don't need to load the .gitignore\nfile for those sparse directories.\n\nSure, there's a small edge case of users adding new untracked files\nthat would have matched the .gitignore file, but it's also clear that\nthey removed the .gitignore file when they sparsified, so I don't see\na problem in reporting it as untracked while that directory was\nas-sparsified-away-as-possible.\n"},{"id":"433243","messageId":"CABPp-BE0V3LQdCYH5D2Xv8HnubWAnYxHSCUo4JTpgGHkAoOLGw@mail.gmail.com","threadId":"56179","inReplyTo":"643d67a9-054f-bc22-cf30-763fc082e9d3@gmail.com","subject":"Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-20T16:15:47Z","receivedAt":"2021-08-20T16:16:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Aug 20, 2021 at 8:50 AM Derrick Stolee wrote:\n>\n> On 8/19/2021 4:48 AM, Johannes Schindelin wrote:\n> > On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n> > This description makes sense, and is easy to explain.\n> >\n> > It does not cover the case where untracked files are found and the\n> > directory is _not_ removed as a consequence, though. I would like to ask\n> > to add this to the commit message, because it is kind of important.\n>\n> Right. I should have modified the message from my earlier version when\n> the issues with untracked files came up.\n\n:+1:\n\n> > The implementation of this behavior looks fine to me.\n> >\n> > About this behavior itself: in my experience, the more tricky a feature is\n> > to explain, the more likely the design should have been adjusted in the\n> > first place. And I find myself struggling a little bit to explain what\n> > files `git switch` touches in cone mode in addition to tracked files.\n>\n> Keep in mind that 'git switch' does not change the sparse-checkout cone,\n> and the activity being described is something that would happen within\n> 'git sparse-checkout set' or 'git sparse-checkout reapply'.\n\nGood points.\n\n> > So I wonder whether an easier-to-explain behavior would be the following:\n> > ignored files in directories that fell out of the sparse-checkout cone are\n> > deleted. (Even if there are untracked, unignored files in the same\n> > directory tree.)\n> >\n> > This is different than what this patch implements: we would now have to\n> > delete the ignored and out-of-cone files _also_ when there are untracked\n> > files in the same directory, i.e. we could no longer use the sweet\n> > `remove_dir_recursively()` call. Therefore, the implementation of what I\n> > suggested would be much more complicated: you would have to enumerate the\n> > ignored files and remove them individually.\n>\n> Outside of \"it's harder to write that feature\"\n\nI don't think it is; see my other email.\n\n> perhaps I could convince\n> you that it is better to do nothing in the presence of untracked files.\n>\n> If a user has an untracked file within a directory that is leaving the\n> sparse cone, then that means they were doing something in that space and\n> perhaps has unfinished work. By leaving the files on-disk, they have an\n> opportunity to revert the change to the sparse-checkout cone and continue\n> their work interrupted. This includes keeping things like build artifacts\n> (that are ignored) so they can incrementally build from that position.\n>\n> The general thought I have here is: having untracked, not-ignored files\n> in a directory that is leaving the sparse-checkout cone is an unexpected\n> behavior, so we should do as little as possible in that scenario.\n>\n> It also makes it more clear to the user what happened: \"You had untracked\n> files here, so we left them alone\" is easier to understand than \"You had\n> untracked files here, so we deleted the ones that were ignored and kept\n> the rest.\"\n\nThis explanation seems reasonable, but it certainly belongs in the\ncommit message.\n\nHowever, doesn't it defeat the point of your removing the ignored\nfiles? Not only do you have directories full of files, but you won't\nbe able to have sparse directory entries due to the need to have\n.gitignore files from underneath them.\n\nPerhaps this is okay because we expect this to be an unusual case.\nBut, if so, do we want a more verbose warning (not with every\ndirectory that fails to be deleted, but just at the end if there were\nany), suggesting to the user that they clean up the relevant\ndirectories and do a sparse-checkout reapply? And, ultimately, once\nthe directory is gone, is that good enough to allow us to keep our\nindexes sparse and fast, or are we looking at a huge amount of effort\nspent on .gitignore files underneath sparse directories?\n\n> > Having said that, even after mulling over this behavior and sleeping over\n> > it, I am unsure what the best way forward would be. Just because it is\n> > easy to explain does not make it right.\n>\n> Of course, you already have a retort to my claim that \"simpler is better\",\n> but I'll just focus on the point that \"simpler for the user to understand\"\n> is a different point than \"simpler to implement\".\n\nI'm confused now. Which behavior are you arguing is simpler for the\nuser to understand?\n"},{"id":"433245","messageId":"CAPig+cQM35GXQ_2ghi+c7zDOaNx3_ryCOCuXq7ib4Ej+fUeMEA@mail.gmail.com","threadId":"56179","inReplyTo":"24b1b932-caf9-b709-baf4-a2f8669deb88@gmail.com","subject":"Re: [PATCH v3 1/8] t7519: rewrite sparse index test","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-08-20T16:40:44Z","receivedAt":"2021-08-20T16:40:58Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Aug 20, 2021 at 11:09 AM Derrick Stolee wrote:\n> On 8/19/2021 3:45 AM, Johannes Schindelin wrote:> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n> >> + write_script .git/hooks/fsmonitor-test <<-\\EOF &&\n> >> printf \"last_update_token\\0\"\n> >\n> > Technically, the backslash needs to be escaped because it is within double\n> > quotes and we do not want the shell to interpolate the `\\0`, but `printf`.\n> > Practically, all the shells I tried handle this as expected.\n\nTechnically, for a POSIX-conforming shell, `\\0` is correct, and the\nbackslash does not need to be escaped. Of course, it doesn't hurt to\nwrite `\\\\0` since it resolves to the same thing, but it isn't\nnecessary. See POSIX section \"2.2.3 Double-Quotes\":\nhttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_03\n\n> I have no opinions on this one way or another. I will just point out that\n> the pattern used in this test is also used throughout the test script, so\n> any change to that format should be applied universally. The test has been\n> operating without complaint since it was introduced in 5c8cdcf (fsmonitor:\n> add test cases for fsmonitor extension, 2017-09-22), so compatibility is\n> likely not a problem.\n\nSince POSIX says that `\\0` is correct and since we haven't encountered\nany shells which handle this incorrectly, there seems little reason to\nchange it.\n"},{"id":"433273","messageId":"6d3844e7-bcda-490b-ba08-57c9e8058c4b@web.de","threadId":"56179","inReplyTo":"b7bd8e73-de86-c563-8b7d-405310ce6c57@gmail.com","subject":"Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-20T19:35:53Z","receivedAt":"2021-08-20T19:36:04Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 20.08.21 um 17:18 schrieb Derrick Stolee:\n> On 8/19/2021 4:01 AM, Johannes Schindelin wrote:\n>> Hi Stolee,\n>>\n>> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> From: Derrick Stolee \n>>>\n>>> The iterated search in find_cache_entry() was recently modified to\n>>> include a loop that searches backwards for a sparse directory entry that\n>>> matches the given traverse_info and name_entry. However, the string\n>>> comparison failed to actually concatenate those two strings, so this\n>>> failed to find a sparse directory when it was not a top-level directory.\n>>>\n>>> This caused some errors in rare cases where a 'git checkout' spanned a\n>>> diff that modified files within the sparse directory entry, but we could\n>>> not correctly find the entry.\n>>\n>> Good explanation.\n>>\n>> I wonder a bit about the performance impact. How \"hot\" is this function?\n>> I.e. how often is it called, on average?\n>>\n>> I ask because I see opportunities to optimize in both directions: it could\n>> be written more concisely (if speed does not matter as much), and it could\n>> be made faster (if speed matters a lot). See below for more.\n>\n> I would definitely optimize for speed here. This can be a very hot path,\n> I believe.\n>\n>>> +\tstrbuf_addstr(&full_path, info->traverse_path);\n>>> +\tstrbuf_add(&full_path, p->path, p->pathlen);\n>>> +\tstrbuf_addch(&full_path, '/');\n>>\n>> This could be reduced to:\n>>\n>> \tstrbuf_addf(&full_path, \"%s%.*s/\",\n>> \t\t info->traverse_path, (int)p->pathlen, p->path);\n>\n> We should definitely avoid formatted strings here, if possible.\n>\n>> But if speed matters, we probably need something more like this:\n>>\n>> \tsize_t full_path_len;\n>> \tconst char *full_path;\n>> \tchar *full_path_1 = NULL;\n>>\n>> \tif (!*info->traverse_path) {\n>> \t\tfull_path = p->path;\n>> \t\tfull_path_len = p->pathlen;\n>> \t} else {\n>> \t\tsize_t len = strlen(info->traverse_path);\n>>\n>> \t\tfull_path_len = len + p->pathlen + 1;\n>> \t\tfull_path = full_path_1 = xmalloc(full_path_len + 1);\n>> \t\tmemcpy(full_path_1, info->traverse_path, len);\n>> \t\tmemcpy(full_path_1 + len, p->path, p->pathlen);\n>> \t\tfull_path_1[full_path_len - 1] = '/';\n>> \t\tfull_path_1[full_path_len] = '\\0';\n>> \t}\n>\n> The critical benefit here is that we do not need to allocate a\n> buffer if the traverse_path does not exist. That might be a\n> worthwhile investment. That leads to justifying the use of\n> bare 'char *'s instead of 'struct strbuf'.\n>\n> If the traverse_path is usually non-null, then we could continue using\n> strbufs as a helper and get the planned performance gains by using\n> strbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()\n> (instead of strbuf_addstr()). That would make this code a bit less\n> ugly with the only real overhead being the extra insertions of '\\0'\n> characters as we add the strings to the strbuf().\n\nYou create full_path only to compare it to another string. You can\ncompare the pieces directly, without allocating and copying:\n\n\tconst char *path;\n\n\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n\t strncmp(path, p->path, p->pathlen) ||\n\t strcmp(path + p->pathlen, \"/\"))\n\t\treturn NULL;\n\nA test would be nice to demonstrate the fixed issue.\n\nRené\n"},{"id":"433277","messageId":"15624e06-d5cd-d83d-f894-f8ffe3809db0@web.de","threadId":"56179","inReplyTo":"6d3844e7-bcda-490b-ba08-57c9e8058c4b@web.de","subject":"Re: [PATCH v3 4/8] unpack-trees: fix nested sparse-dir search","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-20T20:22:41Z","receivedAt":"2021-08-20T20:22:51Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 20.08.21 um 21:35 schrieb René Scharfe:\n> Am 20.08.21 um 17:18 schrieb Derrick Stolee:\n>> On 8/19/2021 4:01 AM, Johannes Schindelin wrote:\n>>> Hi Stolee,\n>>>\n>>> On Tue, 17 Aug 2021, Derrick Stolee via GitGitGadget wrote:\n>>>\n>>>> From: Derrick Stolee \n>>>>\n>>>> The iterated search in find_cache_entry() was recently modified to\n>>>> include a loop that searches backwards for a sparse directory entry that\n>>>> matches the given traverse_info and name_entry. However, the string\n>>>> comparison failed to actually concatenate those two strings, so this\n>>>> failed to find a sparse directory when it was not a top-level directory.\n>>>>\n>>>> This caused some errors in rare cases where a 'git checkout' spanned a\n>>>> diff that modified files within the sparse directory entry, but we could\n>>>> not correctly find the entry.\n>>>\n>>> Good explanation.\n>>>\n>>> I wonder a bit about the performance impact. How \"hot\" is this function?\n>>> I.e. how often is it called, on average?\n>>>\n>>> I ask because I see opportunities to optimize in both directions: it could\n>>> be written more concisely (if speed does not matter as much), and it could\n>>> be made faster (if speed matters a lot). See below for more.\n>>\n>> I would definitely optimize for speed here. This can be a very hot path,\n>> I believe.\n>>\n>>>> +\tstrbuf_addstr(&full_path, info->traverse_path);\n>>>> +\tstrbuf_add(&full_path, p->path, p->pathlen);\n>>>> +\tstrbuf_addch(&full_path, '/');\n>>>\n>>> This could be reduced to:\n>>>\n>>> \tstrbuf_addf(&full_path, \"%s%.*s/\",\n>>> \t\t info->traverse_path, (int)p->pathlen, p->path);\n>>\n>> We should definitely avoid formatted strings here, if possible.\n>>\n>>> But if speed matters, we probably need something more like this:\n>>>\n>>> \tsize_t full_path_len;\n>>> \tconst char *full_path;\n>>> \tchar *full_path_1 = NULL;\n>>>\n>>> \tif (!*info->traverse_path) {\n>>> \t\tfull_path = p->path;\n>>> \t\tfull_path_len = p->pathlen;\n>>> \t} else {\n>>> \t\tsize_t len = strlen(info->traverse_path);\n>>>\n>>> \t\tfull_path_len = len + p->pathlen + 1;\n>>> \t\tfull_path = full_path_1 = xmalloc(full_path_len + 1);\n>>> \t\tmemcpy(full_path_1, info->traverse_path, len);\n>>> \t\tmemcpy(full_path_1 + len, p->path, p->pathlen);\n>>> \t\tfull_path_1[full_path_len - 1] = '/';\n>>> \t\tfull_path_1[full_path_len] = '\\0';\n>>> \t}\n>>\n>> The critical benefit here is that we do not need to allocate a\n>> buffer if the traverse_path does not exist. That might be a\n>> worthwhile investment. That leads to justifying the use of\n>> bare 'char *'s instead of 'struct strbuf'.\n>>\n>> If the traverse_path is usually non-null, then we could continue using\n>> strbufs as a helper and get the planned performance gains by using\n>> strbuf_grow(&full_path, full_path_len + 1) followed by strbuf_add()\n>> (instead of strbuf_addstr()). That would make this code a bit less\n>> ugly with the only real overhead being the extra insertions of '\\0'\n>> characters as we add the strings to the strbuf().\n>\n> You create full_path only to compare it to another string. You can\n> compare the pieces directly, without allocating and copying:\n>\n> \tconst char *path;\n>\n> \tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n> \t strncmp(path, p->path, p->pathlen) ||\n> \t strcmp(path + p->pathlen, \"/\"))\n\nThe strcmp line is wrong (should be path[p->pathlen] != '/'), but\nyou get the idea..\n\n> \t\treturn NULL;\n>\n> A test would be nice to demonstrate the fixed issue.\n>\n> René\n>\n"},{"id":"433486","messageId":"nycvar.QRO.7.76.6.2108232158420.55@tvgsbejvaqbjf.bet","threadId":"56179","inReplyTo":"CABPp-BFUoPQUQDRaRj3Rq2gShOD8WPv6Oh-Wdj4m-7sf=okY6Q@mail.gmail.com","subject":"Re: [PATCH v3 8/8] sparse-checkout: clear tracked sparse dirs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-08-23T20:00:00Z","receivedAt":"2021-08-23T20:00:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Fri, 20 Aug 2021, Elijah Newren wrote:\n\n> On Thu, Aug 19, 2021 at 1:48 AM Johannes Schindelin\n> wrote:\n> >\n> > Having said that, even after mulling over this behavior and sleeping\n> > over it, I am unsure what the best way forward would be. Just because\n> > it is easy to explain does not make it right.\n> >\n> > It is tricky to decide mostly because \"ignored\" files are definitely\n> > not always build output. Apart from VIM's temporary files, users like\n> > me frequently write other files and/or directories that we simply do\n> > not want to see tracked in Git. For example, I often test things in an\n> > `a1.c` file that -- for convenience -- lives in the current worktree.\n> > Obviously I don't want Git to track it, but I also don't want it to be\n> > deleted, so I often add corresponding lines to `.git/info/exclude`.\n> > Likewise, I sometimes download additional information related to what\n> > I am implementing, and that also lives in the current worktree (but\n> > then, I usually am too lazy to add an entry to `.git/info/exclude` in\n> > those cases).\n>\n> I do the same thing, and I know other users that do as well...but I\n> don't put such files in directories that are irrelevant to me. I create\n> cruft files near other files that I'm working on, or in a special\n> directory of its own, but not under some directory that is irrelevant to\n> the areas I'm working on.\n>\n> For reference, we implemented something like this in our `sparsify`\n> wrapper we have internally, where 'git clean -fdX directories>` is executed whenever folks sparsify. (We have our own\n> tool and don't have users use sparse-checkout directly, because our tool\n> computes dependencies to determine which directories are needed.) I was\n> really hesitant to add that cleaning behavior by default, and just made\n> it an option. My colleagues tired of all the bug reports about\n> left-around directories and made it the default, waiting to hear\n> complaints. We never got one. It's been over a year.\n\nIt is really nice to hear that you have evidence from users using this in\npractice. I find that very convincing, and it weighs much more than all of\nmy (very theoretical) considerations.\n\nThank you,\nDscho\n"},{"id":"433635","messageId":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v3.git.1629206602.gitgitgadget@gmail.com","subject":"[PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:34Z","receivedAt":"2021-08-24T21:51:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\nand v2.33.0-rc1 because of some changes to dir.h that became important!\n\nWe launched an experimental release [1] of the sparse-index feature to our\ninternal users. We immediately discovered a problem due to the isolated way\nin which we tested the sparse index: we were never building the project and\nchanging our sparse-checkout definition.\n\n[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n\nUsers who ran a build in one project and then moved to another still had\nbuild artifacts in their worktree that lived inside the old directories.\nSince the files are marked by the .gitignore patterns, these files were not\nremoved by the 'git sparse-checkout set' command. However, they make the\nsparse-index unusable because every 'git status' command needs to expand the\nsparse-directory entries in order to see if the files are tracked or not.\nThis made the first experimental release actually slower for all users\nbecause of this cost.\n\nThe solution we shipped to these customers was to change the way our fork\nhandles these ignored files. Specifically, instead of Git completely\nignoring the files, we changed Git to understand that with cone-mode\nsparse-checkout patterns, the users is asking for entire directories to be\nremoved from the worktree. The link [1] included earlier has this change.\n\nI believe that this is a reasonable expectation, though I recognize that it\nmight look like breaking the expectations of how .gitignore files work.\n\nSince feedback demonstrated that this is a desired behavior, v2 includes\nthis behavior for all \"cone mode\" repositories.\n\nI'm interested in the community's thoughts about this change, as it seems\nlike one that we should make carefully and intentionally.\n\nWhile the rewrite of the t7519 test seems unrelated, it is required to avoid\na test failure with this change that deletes files outside of the cone. By\nmoving the test into repositories not at $TRASH_DIRECTORY, we gain more\ncontrol over the repository structure.\n\n\nUpdates in V4\n=============\n\n * Fixed an issue with the split index.\n\n * The helper methods are used more consistently.\n\n * The helper method path_in_cone_mode_sparse_checkout() is introduced.\n\n * Commit messages are edited for clarity.\n\n * A new config option is added to disable the behavior being added in this\n series.\n\n * I split the commit that involves cache_tree_update(). I have not yet\n succeeded in creating tests to demonstrate why this is required outside\n of needing it in the Scalar functional tests, which includes a version of\n partial clone. I will continue to investigate recreating this scenario in\n the Git test suite, but I wanted to send this version out to get feedback\n on the things that have changed.\n\n\nUpdate in V3\n============\n\n * As promised [2], the helper methods are fixed to work with non-cone-mode\n patterns. A later series will use them to their fullest potential\n (changing git add, git rm, and git mv when interacting with sparse\n entries).\n\n[2]\nhttps://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/\n\n\nUpdates in V2\n=============\n\n * This version correctly leaves untracked files alone. If untracked files\n are found, then the directory is left as-is, in case those ignored files\n are important to the user's work resolving those untracked files.\n\n * This behavior is now enabled by core.sparseCheckoutCone=true.\n\n * To use a sparse index as an in-memory data structure even when\n index.sparse is disabled, a new patch is included to modify the prototype\n of convert_to_sparse() to include a flags parameter.\n\n * A few cleanup patches that I was collecting based on feedback from the\n experimental release and intending for my next series were necessary for\n this implementation.\n\n * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n implementation that used run_subcommand().\n\nThanks, -Stolee\n\nDerrick Stolee (10):\n t7519: rewrite sparse index test\n sparse-index: silently return when not using cone-mode patterns\n sparse-index: silently return when cache tree fails\n sparse-index: use WRITE_TREE_MISSING_OK\n unpack-trees: fix nested sparse-dir search\n sparse-checkout: create helper methods\n attr: be careful about sparse directories\n sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n sparse-checkout: clear tracked sparse dirs\n sparse-checkout: add config to disable deleting dirs\n\n Documentation/config/index.txt | 6 ++\n Documentation/git-sparse-checkout.txt | 10 +++\n attr.c | 14 ++++\n builtin/add.c | 7 +-\n builtin/sparse-checkout.c | 101 ++++++++++++++++++++++++++\n dir.c | 52 +++++++++++++\n dir.h | 8 ++\n read-cache.c | 4 +-\n sparse-index.c | 76 ++++++++++---------\n sparse-index.h | 3 +-\n t/t1091-sparse-checkout-builtin.sh | 63 ++++++++++++++++\n t/t7519-status-fsmonitor.sh | 38 +++++-----\n unpack-trees.c | 7 +-\n 13 files changed, 328 insertions(+), 61 deletions(-)\n\n\nbase-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1009\n\nRange-diff vs v3:\n\n 1: e66106f7a99 ! 1: c407b2cb346 t7519: rewrite sparse index test\n @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/\n test_expect_success 'status succeeds with sparse index' '\n -\tgit reset --hard &&\n +\tgit clone . full &&\n -+\tgit clone . sparse &&\n ++\tgit clone --sparse . sparse &&\n +\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n +\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n 2: fb3ff9108bf ! 2: 8660877ba7a sparse-index: silently return when not using cone-mode patterns\n @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate\n {\n \tint test_env;\n -\tif (istate->split_index || istate->sparse_index ||\n -+\n +\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n 3: 37198535268 ! 3: a669740af9a sparse-index: silently return when cache tree fails\n @@ Commit message\n convert_to_sparse(), then we could still recover from this scenario and\n have a sparse index.\n \n - When constructing the cache-tree extension in convert_to_sparse(), it is\n - possible that we construct a tree object that is new to the object\n - database. Without the WRITE_TREE_MISSING_OK flag, this results in an\n - error that halts our conversion to a sparse index. Add this flag to\n - remove this limitation.\n -\n Signed-off-by: Derrick Stolee \n \n ## sparse-index.c ##\n @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n +\t/*\n +\t * Silently return if there is a problem with the cache tree update,\n +\t * which might just be due to a conflict state in some entry.\n -+\t *\n -+\t * This might create new tree objects, so be sure to use\n -+\t * WRITE_TREE_MISSING_OK.\n +\t */\n -+\tif (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n ++\tif (cache_tree_update(istate, 0))\n +\t\treturn 0;\n \n \tremove_fsmonitor(istate);\n -: ----------- > 4: b379b8fc61a sparse-index: use WRITE_TREE_MISSING_OK\n 4: 10bcadb284e ! 5: acdded0f762 unpack-trees: fix nested sparse-dir search\n @@ Commit message\n diff that modified files within the sparse directory entry, but we could\n not correctly find the entry.\n \n + Helped-by: Johannes Schindelin \n + Helped-by: René Scharfe \n Signed-off-by: Derrick Stolee \n \n ## unpack-trees.c ##\n @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,\n \t\t\t\t\t const struct name_entry *p)\n {\n -\tstruct cache_entry *ce;\n ++\tconst char *path;\n +\tstruct cache_entry *ce = NULL;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n -+\tstruct strbuf full_path = STRBUF_INIT;\n \n - \tif (0 <= pos)\n - \t\treturn o->src_index->cache[pos];\n -@@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n - \tif (pos < 0 || pos >= o->src_index->cache_nr)\n - \t\treturn NULL;\n - \n -+\tstrbuf_addstr(&full_path, info->traverse_path);\n -+\tstrbuf_add(&full_path, p->path, p->pathlen);\n -+\tstrbuf_addch(&full_path, '/');\n -+\n - \t/*\n - \t * Due to lexicographic sorting and sparse directory\n - \t * entries ending with a trailing slash, our path as a\n @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \twhile (pos >= 0) {\n \t\tce = o->src_index->cache[pos];\n \n -\t\tif (strncmp(ce->name, p->path, p->pathlen))\n --\t\t\treturn NULL;\n -+\t\tif (strncmp(ce->name, full_path.buf, full_path.len)) {\n -+\t\t\tce = NULL;\n -+\t\t\tbreak;\n -+\t\t}\n ++\t\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n ++\t\t strncmp(path, p->path, p->pathlen) ||\n ++\t\t path[p->pathlen] != '/')\n + \t\t\treturn NULL;\n \n \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n - \t\t sparse_dir_matches_path(ce, info, p))\n --\t\t\treturn ce;\n -+\t\t\tbreak;\n - \n - \t\tpos--;\n - \t}\n - \n --\treturn NULL;\n -+\tstrbuf_release(&full_path);\n -+\treturn ce;\n - }\n - \n - static void debug_path(struct traverse_info *info)\n 5: 5d28570c82a ! 6: 1958751aa0e sparse-checkout: create helper methods\n @@ Commit message\n adopt these helpers. There are just two in builtin/add.c and\n sparse-index.c that can use path_in_sparse_checkout().\n \n + We add a path_in_cone_mode_sparse_checkout() as well that will only\n + return false if the path is outside of the sparse-checkout definition\n + _and_ the sparse-checkout patterns are in cone mode.\n +\n Signed-off-by: Derrick Stolee \n \n ## builtin/add.c ##\n @@ builtin/add.c: static int refresh(int verbose, const struct pathspec *pathspec)\n -\t\t\t (sparse_checkout_enabled &&\n -\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n -\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n -+\t\t\t (core_apply_sparse_checkout &&\n -+\t\t\t path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n ++\t\t\t !path_in_sparse_checkout(path, &the_index)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec->items[i].original);\n \t\t\t} else {\n @@ dir.c: done:\n \n +int init_sparse_checkout_patterns(struct index_state *istate)\n +{\n -+\tif (!core_apply_sparse_checkout ||\n -+\t istate->sparse_checkout_patterns)\n ++\tif (!core_apply_sparse_checkout)\n ++\t\treturn 1;\n ++\tif (istate->sparse_checkout_patterns)\n +\t\treturn 0;\n +\n +\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n @@ dir.c: done:\n +\treturn 0;\n +}\n +\n -+int path_in_sparse_checkout(const char *path,\n -+\t\t\t struct index_state *istate)\n ++static int path_in_sparse_checkout_1(const char *path,\n ++\t\t\t\t struct index_state *istate,\n ++\t\t\t\t int require_cone_mode)\n +{\n +\tconst char *base;\n +\tint dtype = DT_REG;\n -+\tinit_sparse_checkout_patterns(istate);\n +\n -+\tif (!istate->sparse_checkout_patterns)\n -+\t\treturn MATCHED;\n ++\t/*\n ++\t * We default to accepting a path if there are no patterns or\n ++\t * they are of the wrong type.\n ++\t */\n ++\tif (init_sparse_checkout_patterns(istate) ||\n ++\t (require_cone_mode &&\n ++\t !istate->sparse_checkout_patterns->use_cone_patterns))\n ++\t\treturn 1;\n +\n +\tbase = strrchr(path, '/');\n +\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n @@ dir.c: done:\n +\t\t\t\t\t istate->sparse_checkout_patterns,\n +\t\t\t\t\t istate) > 0;\n +}\n ++\n ++int path_in_sparse_checkout(const char *path,\n ++\t\t\t struct index_state *istate)\n ++{\n ++\treturn path_in_sparse_checkout_1(path, istate, 0);\n ++}\n ++\n ++int path_in_cone_mode_sparse_checkout(const char *path,\n ++\t\t\t\t struct index_state *istate)\n ++{\n ++\treturn path_in_sparse_checkout_1(path, istate, 1);\n ++}\n +\n static struct path_pattern *last_matching_pattern_from_lists(\n \t\tstruct dir_struct *dir, struct index_state *istate,\n @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n +\n +int path_in_sparse_checkout(const char *path,\n +\t\t\t struct index_state *istate);\n ++int path_in_cone_mode_sparse_checkout(const char *path,\n ++\t\t\t\t struct index_state *istate);\n +\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n \n ## sparse-index.c ##\n @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,\n + {\n \tint i, can_convert = 1;\n \tint start_converted = num_converted;\n - \tenum pattern_match_result match;\n +-\tenum pattern_match_result match;\n -\tint dtype = DT_UNKNOWN;\n \tstruct strbuf child_path = STRBUF_INIT;\n -\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,\n \t */\n -\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n -\t\t\t\t\t NULL, &dtype, pl, istate);\n -+\tmatch = path_in_sparse_checkout(ct_path, istate);\n - \tif (match != NOT_MATCHED)\n +-\tif (match != NOT_MATCHED)\n ++\tif (path_in_sparse_checkout(ct_path, istate))\n \t\tcan_convert = 0;\n \n + \tfor (i = start; can_convert && i < end; i++) {\n @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo->settings.sparse_index)\n \t\treturn 0;\n @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n -\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n -\t\t\treturn 0;\n -\t}\n -+\tif (init_sparse_checkout_patterns(istate) < 0)\n ++\tif (init_sparse_checkout_patterns(istate))\n +\t\treturn 0;\n \n \t/*\n 6: c9e100e68f8 ! 7: e496f3cee66 attr: be careful about sparse directories\n @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat\n \t\treturn NULL;\n \n +\t/*\n -+\t * In the case of cone-mode sparse-checkout, getting the\n -+\t * .gitattributes file from a directory is meaningless: all\n -+\t * contained paths will be sparse if the .gitattributes is also\n -+\t * sparse. In the case of a sparse index, it is critical that we\n -+\t * don't go looking for one as it will expand the index.\n ++\t * The .gitattributes file only applies to files within its\n ++\t * parent directory. In the case of cone-mode sparse-checkout,\n ++\t * the .gitattributes file is sparse if and only if all paths\n ++\t * within that directory are also sparse. Thus, don't load the\n ++\t * .gitattributes file since it will not matter.\n ++\t *\n ++\t * In the case of a sparse index, it is critical that we don't go\n ++\t * looking for a .gitattributes file, as the index will expand.\n +\t */\n -+\tinit_sparse_checkout_patterns(istate);\n -+\tif (istate->sparse_checkout_patterns &&\n -+\t istate->sparse_checkout_patterns->use_cone_patterns &&\n -+\t path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n ++\tif (!path_in_cone_mode_sparse_checkout(path, istate))\n +\t\treturn NULL;\n +\n \tbuf = read_blob_data_from_index(istate, path, NULL);\n 7: b0ece4b7dcc ! 8: cab9360b1e9 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n + sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n \n The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\n environment variable or the \"index.sparse\" config setting before\n @@ Commit message\n index as an in-memory data structure, regardless of whether the on-disk\n format should be sparse.\n \n - To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip\n + To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip\n these config checks when enabled. All current consumers are modified to\n pass '0' in the new 'flags' parameter.\n \n @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate\n +int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n +-\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n ++\tif (istate->sparse_index || !istate->cache_nr ||\n + \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n + \t\treturn 0;\n \n -@@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n -\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n -\tif (test_env >= 0)\n -\t\tset_sparse_index_config(istate->repo, test_env);\n -+\tif (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n ++\tif (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {\n ++\t\t/*\n ++\t\t * The sparse index is not (yet) integrated with a split index.\n ++\t\t */\n ++\t\tif (istate->split_index)\n ++\t\t\treturn 0;\n +\t\t/*\n +\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n +\t\t * index.sparse config variable to be on.\n @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n +\t\t\treturn 0;\n +\t}\n \n - \tif (init_sparse_checkout_patterns(istate) < 0)\n + \tif (init_sparse_checkout_patterns(istate))\n \t\treturn 0;\n \n ## sparse-index.h ##\n @@ sparse-index.h\n \n struct index_state;\n -int convert_to_sparse(struct index_state *istate);\n -+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)\n ++#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n +int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n @@ Commit message\n the sparse directory. This depends on the indexed version of the file,\n so the sparse directory must be expanded.\n \n + We must take special care to look for untracked, non-ignored files in\n + these directories before deleting them. We do not want to delete any\n + meaningful work that the users were doing in those directories and\n + perhaps forgot to add and commit before switching sparse-checkout\n + definitions. Since those untracked files might be code files that\n + generated ignored build output, also do not delete any ignored files\n + from these directories in that case. The users can recover their state\n + by resetting their sparse-checkout definition to include that directory\n + and continue. Alternatively, they can see the warning that is presented\n + and delete the directory themselves to regain the performance they\n + expect.\n +\n By deleting the sparse directories when changing scope (or running 'git\n sparse-checkout reapply') we regain these performance benefits as if the\n repository was in a clean state.\n @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n -+The cone mode sparse-checkout patterns will also remove ignored files that\n -+are not within the sparse-checkout definition. This is important behavior\n -+to preserve the performance of the sparse index, but also matches that\n -+cone mode patterns care about directories, not files. If there exist files\n -+that are untracked and not ignored, then Git will not delete files within\n -+that directory other than the tracked files that are now out of scope.\n -+These files should be removed manually to ensure Git can behave optimally.\n ++When changing the sparse-checkout patterns in cone mode, Git will inspect each\n ++tracked directory that is not within the sparse-checkout cone to see if it\n ++contains any untracked files. If all of those files are ignored due to the\n ++`.gitignore` patterns, then the directory will be deleted. If any of the\n ++untracked files within that directory is not ignored, then no deletions will\n ++occur within that directory and a warning message will appear. If these files\n ++are important, then reset your sparse-checkout definition so they are included,\n ++use `git add` and `git commit` to store them, then remove any remaining files\n ++manually to ensure Git can behave optimally.\n +\n \n SUBMODULES\n @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char\n +\t */\n +\tif (!r || !r->index || !r->worktree)\n +\t\treturn;\n -+\tinit_sparse_checkout_patterns(r->index);\n -+\tif (!r->index->sparse_checkout_patterns ||\n ++\tif (init_sparse_checkout_patterns(r->index) ||\n +\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n +\t\treturn;\n +\n @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char\n +\t\t * prevents us from converting to a sparse index, then do\n +\t\t * not try deleting files.\n +\t\t */\n -+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n ++\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))\n +\t\t\treturn;\n +\t\twas_full = 1;\n +\t}\n -: ----------- > 10: 8d55a6ba2fd sparse-checkout: add config to disable deleting dirs\n\n-- \ngitgitgadget\n"},{"id":"433634","messageId":"c407b2cb346b32cdf95c6017d16b93dd1483c627.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 01/10] t7519: rewrite sparse index test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:35Z","receivedAt":"2021-08-24T21:51:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse index is tested with the FS Monitor hook and extension since\nf8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\nwas very fragile because it shared an index across sparse and non-sparse\nbehavior. Since that expansion and contraction could cause the index to\nlose its FS Monitor bitmap and token, behavior is fragile to changes in\n'git sparse-checkout set'.\n\nRewrite the test to use two clones of the original repo: full and\nsparse. This allows us to also keep the test files (actual, expect,\ntrace2.txt) out of the repos we are testing with 'git status'.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex deea88d4431..f1463197b99 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n check_sparse_index_behavior () {\n-\tgit status --porcelain=v2 >expect &&\n-\tgit sparse-checkout init --cone --sparse-index &&\n-\tgit sparse-checkout set dir1 dir2 &&\n+\tgit -C full status --porcelain=v2 >expect &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit status --porcelain=v2 >actual &&\n+\t\tgit -C sparse status --porcelain=v2 >actual &&\n \ttest_region $1 index ensure_full_index trace2.txt &&\n \ttest_region fsm_hook query trace2.txt &&\n \ttest_cmp expect actual &&\n-\trm trace2.txt &&\n-\tgit sparse-checkout disable\n+\trm trace2.txt\n }\n \n test_expect_success 'status succeeds with sparse index' '\n-\tgit reset --hard &&\n+\tgit clone . full &&\n+\tgit clone --sparse . sparse &&\n+\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n-\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n-\tcheck_sparse_index_behavior ! &&\n-\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \tEOF\n-\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n+\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n \tcheck_sparse_index_behavior ! &&\n \n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1/modified\\0\"\n \tEOF\n \tcheck_sparse_index_behavior ! &&\n \n-\tcp -r dir1 dir1a &&\n-\tgit add dir1a &&\n-\tgit commit -m \"add dir1a\" &&\n+\tgit -C sparse sparse-checkout add dir1a &&\n+\n+\tfor repo in full sparse\n+\tdo\n+\t\tcp -r $repo/dir1 $repo/dir1a &&\n+\t\tgit -C $repo add dir1a &&\n+\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n+\tdone &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n \t# This one modifies outside the sparse-checkout definition\n \t# and hence we expect to expand the sparse-index.\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1a/modified\\0\"\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"433636","messageId":"8660877ba7a06971bb5c443bbc1e07825e950a60.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 02/10] sparse-index: silently return when not using cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:36Z","receivedAt":"2021-08-24T21:51:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile the sparse-index is only enabled when core.sparseCheckoutCone is\nalso enabled, it is possible for the user to modify the sparse-checkout\nfile manually in a way that does not match cone-mode patterns. In this\ncase, we should refuse to convert an index into a sparse index, since\nthe sparse_checkout_patterns will not be initialized with recursive and\nparent path hashsets.\n\nAlso silently return if there are no cache entries, which is a simple\ncase: there are no paths to make sparse!\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex c6b4feec413..cd6e0d5f408 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate)\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index ||\n+\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n@@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate)\n \t\t\treturn 0;\n \t}\n \n-\tif (!istate->sparse_checkout_patterns->use_cone_patterns) {\n-\t\twarning(_(\"attempting to use sparse-index without cone mode\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * We need cone-mode patterns to use sparse-index. If a user edits\n+\t * their sparse-checkout file manually, then we can detect during\n+\t * parsing that they are not actually using cone-mode patterns and\n+\t * hence we need to abort this conversion _without error_. Warnings\n+\t * already exist in the pattern parsing to inform the user of their\n+\t * bad patterns.\n+\t */\n+\tif (!istate->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn 0;\n \n \t/*\n \t * NEEDSWORK: If we have unmerged entries, then stay full.\n-- \ngitgitgadget\n\n"},{"id":"433638","messageId":"a669740af9ad7edbc6067644e50a927a284c732d.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 03/10] sparse-index: silently return when cache tree fails","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:37Z","receivedAt":"2021-08-24T21:51:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf cache_tree_update() returns a non-zero value, then it could not\ncreate the cache tree. This is likely due to a path having a merge\nconflict. Since we are already returning early, let's return silently to\navoid making it seem like we failed to write the index at all.\n\nIf we remove our dependence on the cache tree within\nconvert_to_sparse(), then we could still recover from this scenario and\nhave a sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex cd6e0d5f408..d9b07695953 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate)\n \n \t/* Clear and recompute the cache-tree */\n \tcache_tree_free(&istate->cache_tree);\n-\tif (cache_tree_update(istate, 0)) {\n-\t\twarning(_(\"unable to update cache-tree, staying full\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * Silently return if there is a problem with the cache tree update,\n+\t * which might just be due to a conflict state in some entry.\n+\t */\n+\tif (cache_tree_update(istate, 0))\n+\t\treturn 0;\n \n \tremove_fsmonitor(istate);\n \n-- \ngitgitgadget\n\n"},{"id":"433637","messageId":"b379b8fc61af8a8c39ff8b73aae03ad4999a456c.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:38Z","receivedAt":"2021-08-24T21:51:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen updating the cache tree in convert_to_sparse(), the\nWRITE_TREE_MISSING_OK flag indicates that trees might be computed that\ndo not already exist within the object database. This happens in cases\nsuch as 'git add' creating new trees that it wants to store in\nanticipation of a following 'git commit'. If this flag is not specified,\nthen it might trigger a promisor fetch or a failure due to the object\nnot existing locally.\n\nUse WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these\npossible reasons for the cache_tree_update() to fail.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex d9b07695953..880c5f72338 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)\n \t/*\n \t * Silently return if there is a problem with the cache tree update,\n \t * which might just be due to a conflict state in some entry.\n+\t *\n+\t * This might create new tree objects, so be sure to use\n+\t * WRITE_TREE_MISSING_OK.\n \t */\n-\tif (cache_tree_update(istate, 0))\n+\tif (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n \t\treturn 0;\n \n \tremove_fsmonitor(istate);\n-- \ngitgitgadget\n\n"},{"id":"433639","messageId":"acdded0f762800392515c83aec2b276c62ad691c.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 05/10] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:39Z","receivedAt":"2021-08-24T21:52:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe iterated search in find_cache_entry() was recently modified to\ninclude a loop that searches backwards for a sparse directory entry that\nmatches the given traverse_info and name_entry. However, the string\ncomparison failed to actually concatenate those two strings, so this\nfailed to find a sparse directory when it was not a top-level directory.\n\nThis caused some errors in rare cases where a 'git checkout' spanned a\ndiff that modified files within the sparse directory entry, but we could\nnot correctly find the entry.\n\nHelped-by: Johannes Schindelin \nHelped-by: René Scharfe \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..47ef0cf4c53 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1255,7 +1255,8 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n-\tstruct cache_entry *ce;\n+\tconst char *path;\n+\tstruct cache_entry *ce = NULL;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n@@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \twhile (pos >= 0) {\n \t\tce = o->src_index->cache[pos];\n \n-\t\tif (strncmp(ce->name, p->path, p->pathlen))\n+\t\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n+\t\t strncmp(path, p->path, p->pathlen) ||\n+\t\t path[p->pathlen] != '/')\n \t\t\treturn NULL;\n \n \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n-- \ngitgitgadget\n\n"},{"id":"433640","messageId":"1958751aa0ef84ec18ddc7363d1f0c9e6c3f8d4d.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 06/10] sparse-checkout: create helper methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:40Z","receivedAt":"2021-08-24T21:52:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we integrate the sparse index into more builtins, we occasionally\nneed to check the sparse-checkout patterns to see if a path is within\nthe sparse-checkout cone. Create some helper methods that help\ninitialize the patterns and check for pattern matching to make this\neasier.\n\nThe existing callers of commands like get_sparse_checkout_patterns() use\na custom 'struct pattern_list' that is not necessarily the one in the\n'struct index_state', so there are not many previous uses that could\nadopt these helpers. There are just two in builtin/add.c and\nsparse-index.c that can use path_in_sparse_checkout().\n\nWe add a path_in_cone_mode_sparse_checkout() as well that will only\nreturn false if the path is outside of the sparse-checkout definition\n_and_ the sparse-checkout patterns are in cone mode.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 7 +------\n dir.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++\n dir.h | 8 ++++++++\n sparse-index.c | 14 +++-----------\n 4 files changed, 64 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 17528e8f922..88a6c0c69fb 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \tint flags = REFRESH_IGNORE_SKIP_WORKTREE |\n \t\t (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);\n-\tstruct pattern_list pl = { 0 };\n-\tint sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);\n \n \tseen = xcalloc(pathspec->nr, 1);\n \trefresh_index(&the_index, flags, pathspec, seen,\n@@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tfor (i = 0; i < pathspec->nr; i++) {\n \t\tif (!seen[i]) {\n \t\t\tconst char *path = pathspec->items[i].original;\n-\t\t\tint dtype = DT_REG;\n \n \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n-\t\t\t (sparse_checkout_enabled &&\n-\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n-\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n+\t\t\t !path_in_sparse_checkout(path, &the_index)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec->items[i].original);\n \t\t\t} else {\ndiff --git a/dir.c b/dir.c\nindex 03c4d212672..86afa2eae00 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1439,6 +1439,58 @@ done:\n \treturn result;\n }\n \n+int init_sparse_checkout_patterns(struct index_state *istate)\n+{\n+\tif (!core_apply_sparse_checkout)\n+\t\treturn 1;\n+\tif (istate->sparse_checkout_patterns)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n+\n+\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {\n+\t\tFREE_AND_NULL(istate->sparse_checkout_patterns);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int path_in_sparse_checkout_1(const char *path,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t int require_cone_mode)\n+{\n+\tconst char *base;\n+\tint dtype = DT_REG;\n+\n+\t/*\n+\t * We default to accepting a path if there are no patterns or\n+\t * they are of the wrong type.\n+\t */\n+\tif (init_sparse_checkout_patterns(istate) ||\n+\t (require_cone_mode &&\n+\t !istate->sparse_checkout_patterns->use_cone_patterns))\n+\t\treturn 1;\n+\n+\tbase = strrchr(path, '/');\n+\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n+\t\t\t\t\t &dtype,\n+\t\t\t\t\t istate->sparse_checkout_patterns,\n+\t\t\t\t\t istate) > 0;\n+}\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate)\n+{\n+\treturn path_in_sparse_checkout_1(path, istate, 0);\n+}\n+\n+int path_in_cone_mode_sparse_checkout(const char *path,\n+\t\t\t\t struct index_state *istate)\n+{\n+\treturn path_in_sparse_checkout_1(path, istate, 1);\n+}\n+\n static struct path_pattern *last_matching_pattern_from_lists(\n \t\tstruct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int pathlen,\ndiff --git a/dir.h b/dir.h\nindex b3e1a54a971..6823312521e 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,\n \t\t\t\tconst char *basename, int *dtype,\n \t\t\t\tstruct pattern_list *pl,\n \t\t\t\tstruct index_state *istate);\n+\n+int init_sparse_checkout_patterns(struct index_state *state);\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate);\n+int path_in_cone_mode_sparse_checkout(const char *path,\n+\t\t\t\t struct index_state *istate);\n+\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n \t\t\t\t const char *pathname, int len);\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 880c5f72338..23f7c3bd361 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate,\n {\n \tint i, can_convert = 1;\n \tint start_converted = num_converted;\n-\tenum pattern_match_result match;\n-\tint dtype = DT_UNKNOWN;\n \tstruct strbuf child_path = STRBUF_INIT;\n-\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n \n \t/*\n \t * Is the current path outside of the sparse cone?\n \t * Then check if the region can be replaced by a sparse\n \t * directory entry (everything is sparse and merged).\n \t */\n-\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n-\t\t\t\t\t NULL, &dtype, pl, istate);\n-\tif (match != NOT_MATCHED)\n+\tif (path_in_sparse_checkout(ct_path, istate))\n \t\tcan_convert = 0;\n \n \tfor (i = start; can_convert && i < end; i++) {\n@@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo->settings.sparse_index)\n \t\treturn 0;\n \n-\tif (!istate->sparse_checkout_patterns) {\n-\t\tistate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));\n-\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n-\t\t\treturn 0;\n-\t}\n+\tif (init_sparse_checkout_patterns(istate))\n+\t\treturn 0;\n \n \t/*\n \t * We need cone-mode patterns to use sparse-index. If a user edits\n-- \ngitgitgadget\n\n"},{"id":"433641","messageId":"e496f3cee66d176a8776a3bde5d04716a4b52f26.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 07/10] attr: be careful about sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:41Z","receivedAt":"2021-08-24T21:52:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/attr.c b/attr.c\nindex d029e681f28..62db4aed443 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -14,6 +14,7 @@\n #include \"utf8.h\"\n #include \"quote.h\"\n #include \"thread-utils.h\"\n+#include \"dir.h\"\n \n const char git_attr__true[] = \"(builtin)true\";\n const char git_attr__false[] = \"\\0(builtin)false\";\n@@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \tif (!istate)\n \t\treturn NULL;\n \n+\t/*\n+\t * The .gitattributes file only applies to files within its\n+\t * parent directory. In the case of cone-mode sparse-checkout,\n+\t * the .gitattributes file is sparse if and only if all paths\n+\t * within that directory are also sparse. Thus, don't load the\n+\t * .gitattributes file since it will not matter.\n+\t *\n+\t * In the case of a sparse index, it is critical that we don't go\n+\t * looking for a .gitattributes file, as the index will expand.\n+\t */\n+\tif (!path_in_cone_mode_sparse_checkout(path, istate))\n+\t\treturn NULL;\n+\n \tbuf = read_blob_data_from_index(istate, path, NULL);\n \tif (!buf)\n \t\treturn NULL;\n-- \ngitgitgadget\n\n"},{"id":"433642","messageId":"cab9360b1e9405e62482e3ccf417a8f8d5b2c7d1.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 08/10] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:42Z","receivedAt":"2021-08-24T21:52:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\nenvironment variable or the \"index.sparse\" config setting before\nconverting the index to a sparse one. This is for ease of use since all\ncurrent consumers are preparing to compress the index before writing it\nto disk. If these settings are not enabled, then convert_to_sparse()\nsilently returns without doing anything.\n\nWe will add a consumer in the next change that wants to use the sparse\nindex as an in-memory data structure, regardless of whether the on-disk\nformat should be sparse.\n\nTo that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip\nthese config checks when enabled. All current consumers are modified to\npass '0' in the new 'flags' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++--\n sparse-index.c | 37 ++++++++++++++++++++++---------------\n sparse-index.h | 3 ++-\n 3 files changed, 26 insertions(+), 18 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9048ef9e905..f5d4385c408 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \tint ret;\n \tint was_full = !istate->sparse_index;\n \n-\tret = convert_to_sparse(istate);\n+\tret = convert_to_sparse(istate, 0);\n \n \tif (ret) {\n \t\twarning(_(\"failed to convert to a sparse-index\"));\n@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,\n \tint ret, was_full = !istate->sparse_index;\n \n \tmove_cache_to_base_index(istate);\n-\tconvert_to_sparse(istate);\n+\tconvert_to_sparse(istate, 0);\n \n \ttrace2_region_enter_printf(\"index\", \"shared/do_write_index\",\n \t\t\t\t the_repository, \"%s\", get_tempfile_path(*temp));\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 23f7c3bd361..0bc45f60ac5 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate)\n \treturn 0;\n }\n \n-int convert_to_sparse(struct index_state *istate)\n+int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n+\tif (istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n-\t/*\n-\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n-\t * index.sparse config variable to be on.\n-\t */\n-\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n-\tif (test_env >= 0)\n-\t\tset_sparse_index_config(istate->repo, test_env);\n+\tif (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {\n+\t\t/*\n+\t\t * The sparse index is not (yet) integrated with a split index.\n+\t\t */\n+\t\tif (istate->split_index)\n+\t\t\treturn 0;\n+\t\t/*\n+\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n+\t\t * index.sparse config variable to be on.\n+\t\t */\n+\t\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n+\t\tif (test_env >= 0)\n+\t\t\tset_sparse_index_config(istate->repo, test_env);\n \n-\t/*\n-\t * Only convert to sparse if index.sparse is set.\n-\t */\n-\tprepare_repo_settings(istate->repo);\n-\tif (!istate->repo->settings.sparse_index)\n-\t\treturn 0;\n+\t\t/*\n+\t\t * Only convert to sparse if index.sparse is set.\n+\t\t */\n+\t\tprepare_repo_settings(istate->repo);\n+\t\tif (!istate->repo->settings.sparse_index)\n+\t\t\treturn 0;\n+\t}\n \n \tif (init_sparse_checkout_patterns(istate))\n \t\treturn 0;\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 1115a0d7dd9..9f3d7bc7faf 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,8 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-int convert_to_sparse(struct index_state *istate);\n+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n+int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n * Some places in the codebase expect to search for a specific path.\n-- \ngitgitgadget\n\n"},{"id":"433643","messageId":"c19d93ec5d7f2ff1eeba40d75aad872250073586.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 09/10] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:43Z","receivedAt":"2021-08-24T21:52:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen changing the scope of a sparse-checkout using cone mode, we might\nhave some tracked directories go out of scope. The current logic removes\nthe tracked files from within those directories, but leaves the ignored\nfiles within those directories. This is a bit unexpected to users who\nhave given input to Git saying they don't need those directories\nanymore.\n\nThis is something that is new to the cone mode pattern type: the user\nhas explicitly said \"I want these directories and _not_ those\ndirectories.\" The typical sparse-checkout patterns more generally apply\nto \"I want files with with these patterns\" so it is natural to leave\nignored files as they are. This focus on directories in cone mode\nprovides us an opportunity to change the behavior.\n\nLeaving these ignored files in the sparse directories makes it\nimpossible to gain performance benefits in the sparse index. When we\ntrack into these directories, we need to know if the files are ignored\nor not, which might depend on the _tracked_ .gitignore file(s) within\nthe sparse directory. This depends on the indexed version of the file,\nso the sparse directory must be expanded.\n\nWe must take special care to look for untracked, non-ignored files in\nthese directories before deleting them. We do not want to delete any\nmeaningful work that the users were doing in those directories and\nperhaps forgot to add and commit before switching sparse-checkout\ndefinitions. Since those untracked files might be code files that\ngenerated ignored build output, also do not delete any ignored files\nfrom these directories in that case. The users can recover their state\nby resetting their sparse-checkout definition to include that directory\nand continue. Alternatively, they can see the warning that is presented\nand delete the directory themselves to regain the performance they\nexpect.\n\nBy deleting the sparse directories when changing scope (or running 'git\nsparse-checkout reapply') we regain these performance benefits as if the\nrepository was in a clean state.\n\nSince these ignored files are frequently build output or helper files\nfrom IDEs, the users should not need the files now that the tracked\nfiles are removed. If the tracked files reappear, then they will have\nnewer timestamps than the build artifacts, so the artifacts will need to\nbe regenerated anyway.\n\nUse the sparse-index as a data structure in order to find the sparse\ndirectories that can be safely deleted. Re-expand the index to a full\none if it was full before.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 10 +++\n builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n 3 files changed, 163 insertions(+)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex fdcf43f87cb..42056ee9ff9 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n+When changing the sparse-checkout patterns in cone mode, Git will inspect each\n+tracked directory that is not within the sparse-checkout cone to see if it\n+contains any untracked files. If all of those files are ignored due to the\n+`.gitignore` patterns, then the directory will be deleted. If any of the\n+untracked files within that directory is not ignored, then no deletions will\n+occur within that directory and a warning message will appear. If these files\n+are important, then reset your sparse-checkout definition so they are included,\n+use `git add` and `git commit` to store them, then remove any remaining files\n+manually to ensure Git can behave optimally.\n+\n \n SUBMODULES\n ----------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 8ba9f13787b..d0f5c4702be 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void clean_tracked_sparse_directories(struct repository *r)\n+{\n+\tint i, was_full = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t pathlen;\n+\tstruct string_list_item *item;\n+\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n+\n+\t/*\n+\t * If we are not using cone mode patterns, then we cannot\n+\t * delete directories outside of the sparse cone.\n+\t */\n+\tif (!r || !r->index || !r->worktree)\n+\t\treturn;\n+\tif (init_sparse_checkout_patterns(r->index) ||\n+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn;\n+\n+\t/*\n+\t * Use the sparse index as a data structure to assist finding\n+\t * directories that are safe to delete. This conversion to a\n+\t * sparse index will not delete directories that contain\n+\t * conflicted entries or submodules.\n+\t */\n+\tif (!r->index->sparse_index) {\n+\t\t/*\n+\t\t * If something, such as a merge conflict or other concern,\n+\t\t * prevents us from converting to a sparse index, then do\n+\t\t * not try deleting files.\n+\t\t */\n+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))\n+\t\t\treturn;\n+\t\twas_full = 1;\n+\t}\n+\n+\tstrbuf_addstr(&path, r->worktree);\n+\tstrbuf_complete(&path, '/');\n+\tpathlen = path.len;\n+\n+\t/*\n+\t * Collect directories that have gone out of scope but also\n+\t * exist on disk, so there is some work to be done. We need to\n+\t * store the entries in a list before exploring, since that might\n+\t * expand the sparse-index again.\n+\t */\n+\tfor (i = 0; i < r->index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = r->index->cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t repo_file_exists(r, ce->name))\n+\t\t\tstring_list_append(&sparse_dirs, ce->name);\n+\t}\n+\n+\tfor_each_string_list_item(item, &sparse_dirs) {\n+\t\tstruct dir_struct dir = DIR_INIT;\n+\t\tstruct pathspec p = { 0 };\n+\t\tstruct strvec s = STRVEC_INIT;\n+\n+\t\tstrbuf_setlen(&path, pathlen);\n+\t\tstrbuf_addstr(&path, item->string);\n+\n+\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n+\n+\t\tsetup_standard_excludes(&dir);\n+\t\tstrvec_push(&s, path.buf);\n+\n+\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n+\t\tfill_directory(&dir, r->index, &p);\n+\n+\t\tif (dir.nr) {\n+\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n+\t\t\t\t \" but is not in the sparse-checkout cone\"),\n+\t\t\t\titem->string);\n+\t\t} else if (remove_dir_recursively(&path, 0)) {\n+\t\t\t/*\n+\t\t\t * Removal is \"best effort\". If something blocks\n+\t\t\t * the deletion, then continue with a warning.\n+\t\t\t */\n+\t\t\twarning(_(\"failed to remove directory '%s'\"),\n+\t\t\t\titem->string);\n+\t\t}\n+\n+\t\tdir_clear(&dir);\n+\t}\n+\n+\tstring_list_clear(&sparse_dirs, 0);\n+\tstrbuf_release(&path);\n+\n+\tif (was_full)\n+\t\tensure_full_index(r->index);\n+}\n+\n static int update_working_directory(struct pattern_list *pl)\n {\n \tenum update_sparsity_result result;\n@@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl)\n \telse\n \t\trollback_lock_file(&lock_file);\n \n+\tclean_tracked_sparse_directories(r);\n+\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 38fc8340f5c..71236981e64 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n \tcheck_files repo/deep a deeper1\n '\n \n+test_expect_success 'cone mode clears ignored subdirectories' '\n+\trm repo/.git/info/sparse-checkout &&\n+\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set deep/deeper1 &&\n+\n+\tcat >repo/.gitignore <<-\\EOF &&\n+\tobj/\n+\t*.o\n+\tEOF\n+\n+\tgit -C repo add .gitignore &&\n+\tgit -C repo commit -m \".gitignore\" &&\n+\n+\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n+\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n+\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n+\tdo\n+\t\techo ignored >repo/$file || return 1\n+\tdone &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout reapply &&\n+\ttest_path_is_missing repo/folder1 &&\n+\ttest_path_is_missing repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper2 &&\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\t>repo/deep/deeper2/ignored.o &&\n+\t>repo/deep/deeper2/untracked &&\n+\n+\t# When an untracked file is in the way, all untracked files\n+\t# (even ignored files) are preserved.\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\tgrep \"contains untracked files\" err &&\n+\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n+\ttest_path_is_file repo/deep/deeper2/untracked &&\n+\n+\t# The rest of the cone matches expectation\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\techo \"? deep/deeper2/untracked\" >expect &&\n+\ttest_cmp expect out\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"433644","messageId":"8d55a6ba2fdf64cee4eb51f3cb6f9808bd0b7505.1629841904.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:51:44Z","receivedAt":"2021-08-24T21:52:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe clean_tracked_sparse_directories() method deletes the tracked\ndirectories that go out of scope when the sparse-checkout cone changes,\nat least in cone mode. This is new behavior, but is recommended based on\nour understanding of how users are interacting with the feature in most\ncases.\n\nIt is possible that some users will object to the new behavior, so\ncreate a new configuration option 'index.deleteSparseDirectories' that\ncan be set to 'false' to make clean_tracked_sparse_directories() do\nnothing. This will keep all untracked files in the working tree and\ncause performance problems with the sparse index, but those trade-offs\nare for the user to decide.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/index.txt | 6 ++++++\n builtin/sparse-checkout.c | 9 ++++++++-\n t/t1091-sparse-checkout-builtin.sh | 4 ++++\n 3 files changed, 18 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/index.txt b/Documentation/config/index.txt\nindex 75f3a2d1054..c65da20a931 100644\n--- a/Documentation/config/index.txt\n+++ b/Documentation/config/index.txt\n@@ -1,3 +1,9 @@\n+index.deleteSparseDirectories::\n+\tWhen enabled, the cone mode sparse-checkout feature will delete\n+\tdirectories that are outside of the sparse-checkout cone, unless\n+\tsuch a directory contains an untracked, non-ignored file. Defaults\n+\tto true.\n+\n index.recordEndOfIndexEntries::\n \tSpecifies whether the index file should include an \"End Of Index\n \tEntry\" section. This reduces index load time on multiprocessor\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex d0f5c4702be..33ec729d9de 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -102,7 +102,7 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n static void clean_tracked_sparse_directories(struct repository *r)\n {\n-\tint i, was_full = 0;\n+\tint i, value, was_full = 0;\n \tstruct strbuf path = STRBUF_INIT;\n \tsize_t pathlen;\n \tstruct string_list_item *item;\n@@ -118,6 +118,13 @@ static void clean_tracked_sparse_directories(struct repository *r)\n \t !r->index->sparse_checkout_patterns->use_cone_patterns)\n \t\treturn;\n \n+\t/*\n+\t * Users can disable this behavior.\n+\t */\n+\tif (!repo_config_get_bool(r, \"index.deletesparsedirectories\", &value) &&\n+\t !value)\n+\t\treturn;\n+\n \t/*\n \t * Use the sparse index as a data structure to assist finding\n \t * directories that are safe to delete. This conversion to a\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 71236981e64..e0f31186d89 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -666,6 +666,10 @@ test_expect_success 'cone mode clears ignored subdirectories' '\n \tgit -C repo status --porcelain=v2 >out &&\n \ttest_must_be_empty out &&\n \n+\tgit -C repo -c index.deleteSparseDirectories=false sparse-checkout reapply &&\n+\ttest_path_is_dir repo/folder1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\n \tgit -C repo sparse-checkout reapply &&\n \ttest_path_is_missing repo/folder1 &&\n \ttest_path_is_missing repo/deep/deeper2 &&\n-- \ngitgitgadget\n"},{"id":"433672","messageId":"51c11512-fb71-277e-790b-88ae3d5a358f@web.de","threadId":"56179","inReplyTo":"acdded0f762800392515c83aec2b276c62ad691c.1629841904.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 05/10] unpack-trees: fix nested sparse-dir search","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-08-24T22:21:36Z","receivedAt":"2021-08-24T22:21:59Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 24.08.21 um 23:51 schrieb Derrick Stolee via GitGitGadget:\n> From: Derrick Stolee \n>\n> The iterated search in find_cache_entry() was recently modified to\n> include a loop that searches backwards for a sparse directory entry that\n> matches the given traverse_info and name_entry. However, the string\n> comparison failed to actually concatenate those two strings, so this\n> failed to find a sparse directory when it was not a top-level directory.\n>\n> This caused some errors in rare cases where a 'git checkout' spanned a\n> diff that modified files within the sparse directory entry, but we could\n> not correctly find the entry.\n>\n> Helped-by: Johannes Schindelin \n> Helped-by: René Scharfe \n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 7 +++++--\n> 1 file changed, 5 insertions(+), 2 deletions(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 5786645f315..47ef0cf4c53 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1255,7 +1255,8 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n> static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> \t\t\t\t\t const struct name_entry *p)\n> {\n> -\tstruct cache_entry *ce;\n> +\tconst char *path;\n> +\tstruct cache_entry *ce = NULL;\n ^^^^^^^\nThis added initialization doesn't seem to be needed...\n\n> \tint pos = find_cache_pos(info, p->path, p->pathlen);\n> \tstruct unpack_trees_options *o = info->data;\n>\n> @@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> \twhile (pos >= 0) {\n> \t\tce = o->src_index->cache[pos];\n\n... because ce is set here before it's used.\n\n>\n> -\t\tif (strncmp(ce->name, p->path, p->pathlen))\n> +\t\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n> +\t\t strncmp(path, p->path, p->pathlen) ||\n> +\t\t path[p->pathlen] != '/')\n> \t\t\treturn NULL;\n>\n> \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n>\n"},{"id":"433686","messageId":"249acb56-99bc-4a2a-b3d8-6b697643cf4d@gmail.com","threadId":"56179","inReplyTo":"51c11512-fb71-277e-790b-88ae3d5a358f@web.de","subject":"Re: [PATCH v4 05/10] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-25T01:09:17Z","receivedAt":"2021-08-25T01:09:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/24/2021 6:21 PM, René Scharfe wrote:\n> Am 24.08.21 um 23:51 schrieb Derrick Stolee via GitGitGadget:\n>> -\tstruct cache_entry *ce;\n>> +\tconst char *path;\n>> +\tstruct cache_entry *ce = NULL;\n> ^^^^^^^\n> This added initialization doesn't seem to be needed...\n> \n>> \tint pos = find_cache_pos(info, p->path, p->pathlen);\n>> \tstruct unpack_trees_options *o = info->data;\n>>\n>> @@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n>> \twhile (pos >= 0) {\n>> \t\tce = o->src_index->cache[pos];\n> \n> ... because ce is set here before it's used.\n\nBetter yet, 'ce' is only used within the while loop, so its\ndeclaration can be moved to this line instead.\n\nThanks,\n-Stolee\n"},{"id":"433954","messageId":"CABPp-BGeJJBF0ZmS7ZHcqFmXwXmgVH1Uox_6gNHzcacG2vRjVw@mail.gmail.com","threadId":"56179","inReplyTo":"8d55a6ba2fdf64cee4eb51f3cb6f9808bd0b7505.1629841904.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-27T20:58:17Z","receivedAt":"2021-08-27T20:58:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The clean_tracked_sparse_directories() method deletes the tracked\n> directories that go out of scope when the sparse-checkout cone changes,\n> at least in cone mode. This is new behavior, but is recommended based on\n> our understanding of how users are interacting with the feature in most\n> cases.\n>\n> It is possible that some users will object to the new behavior, so\n> create a new configuration option 'index.deleteSparseDirectories' that\n> can be set to 'false' to make clean_tracked_sparse_directories() do\n> nothing. This will keep all untracked files in the working tree and\n> cause performance problems with the sparse index, but those trade-offs\n> are for the user to decide.\n\nI'm not sold that we need anything here, and it sounds like this is\nall being added based on a theoretical concern. I might not object\ntoo much normally to trying to address theoretical conerns with new\nfeatures, but:\n\n* I'm a little concerned that we're adding a configuration option\n(which live forever and are harder to work with\nbackward-compatibility-wise) rather than a command line flag.\n\n* It's not clear to me that the option name is what we want. For\nexample, git checkout has a --[no-]overwrite-ignored for overwriting\nignored files (or not) when switching to a different branch. git\nmerge has the same flags (though only the fast-forwarding backend\nheeds it currently). This behavior is quite similar to that flag, and\nhas the same default of treating ignored files as expendable. Should\nthe naming be more similar?\n\n* It's not clear to me that the option has the right level of\nimplementation granularity. If someone wants ignored files to be\ntreated as important, should they need to set a\nmerge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a\nindex.deleteSparseDirectories AND other names, or have one high-level\noption that sets all of these behaviors?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/index.txt | 6 ++++++\n> builtin/sparse-checkout.c | 9 ++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 4 ++++\n> 3 files changed, 18 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt\n> index 75f3a2d1054..c65da20a931 100644\n> --- a/Documentation/config/index.txt\n> +++ b/Documentation/config/index.txt\n> @@ -1,3 +1,9 @@\n> +index.deleteSparseDirectories::\n> + When enabled, the cone mode sparse-checkout feature will delete\n> + directories that are outside of the sparse-checkout cone, unless\n> + such a directory contains an untracked, non-ignored file. Defaults\n> + to true.\n> +\n> index.recordEndOfIndexEntries::\n> Specifies whether the index file should include an \"End Of Index\n> Entry\" section. This reduces index load time on multiprocessor\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index d0f5c4702be..33ec729d9de 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -102,7 +102,7 @@ static int sparse_checkout_list(int argc, const char **argv)\n>\n> static void clean_tracked_sparse_directories(struct repository *r)\n> {\n> - int i, was_full = 0;\n> + int i, value, was_full = 0;\n> struct strbuf path = STRBUF_INIT;\n> size_t pathlen;\n> struct string_list_item *item;\n> @@ -118,6 +118,13 @@ static void clean_tracked_sparse_directories(struct repository *r)\n> !r->index->sparse_checkout_patterns->use_cone_patterns)\n> return;\n>\n> + /*\n> + * Users can disable this behavior.\n> + */\n> + if (!repo_config_get_bool(r, \"index.deletesparsedirectories\", &value) &&\n> + !value)\n> + return;\n> +\n> /*\n> * Use the sparse index as a data structure to assist finding\n> * directories that are safe to delete. This conversion to a\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 71236981e64..e0f31186d89 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -666,6 +666,10 @@ test_expect_success 'cone mode clears ignored subdirectories' '\n> git -C repo status --porcelain=v2 >out &&\n> test_must_be_empty out &&\n>\n> + git -C repo -c index.deleteSparseDirectories=false sparse-checkout reapply &&\n> + test_path_is_dir repo/folder1 &&\n> + test_path_is_dir repo/deep/deeper2 &&\n> +\n> git -C repo sparse-checkout reapply &&\n> test_path_is_missing repo/folder1 &&\n> test_path_is_missing repo/deep/deeper2 &&\n> --\n> gitgitgadget\n"},{"id":"433968","messageId":"CABPp-BFcySug2kSvxT7YdJ1Oorza5AfxKPJgMBE2wN8qOw+=Eg@mail.gmail.com","threadId":"56179","inReplyTo":"b379b8fc61af8a8c39ff8b73aae03ad4999a456c.1629841904.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-27T21:33:48Z","receivedAt":"2021-08-27T21:34:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When updating the cache tree in convert_to_sparse(), the\n> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that\n> do not already exist within the object database.\n\nOkay.\n\n> This happens in cases\n> such as 'git add' creating new trees that it wants to store in\n> anticipation of a following 'git commit'.\n\nThis doesn't make any sense to me. Does 'git add' call\nconvert_to_sparse()? I don't see why it would; wouldn't the calls to\nconvert_to_sparse() come via sparse-checkout init/set commands? If\nI'm correct on that, and 'git add' wants to create new trees, then by\nthe time convert_to_sparse() is called in some subsequent git\noperation, then convert_to_sparse would already have the trees it\nneeds.\n\n\nI thought the reason you would need this is someone modified and\nstaged a change to a file underneath a directory that will be\nsparsified away; at the time of convert_to_sparse(), a tree object may\nnot have yet been written for the new tree with the newly modified\nfile (because those tend to be written at commit time), but you'd need\nit at the time you sparsified.\n\n> If this flag is not specified,\n> then it might trigger a promisor fetch or a failure due to the object\n> not existing locally.\n\nGood point.\n\n> Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these\n> possible reasons for the cache_tree_update() to fail.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> sparse-index.c | 5 ++++-\n> 1 file changed, 4 insertions(+), 1 deletion(-)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index d9b07695953..880c5f72338 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)\n> /*\n> * Silently return if there is a problem with the cache tree update,\n> * which might just be due to a conflict state in some entry.\n> + *\n> + * This might create new tree objects, so be sure to use\n> + * WRITE_TREE_MISSING_OK.\n> */\n> - if (cache_tree_update(istate, 0))\n> + if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n> return 0;\n>\n> remove_fsmonitor(istate);\n> --\n> gitgitgadget\n"},{"id":"433970","messageId":"CABPp-BG01tXnd_UhYJDX4Tm_EcXGju9CkB8BXU0XS-YAWEuaeQ@mail.gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-27T21:56:18Z","receivedAt":"2021-08-27T21:56:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\n> and v2.33.0-rc1 because of some changes to dir.h that became important!\n\nIsn't this stale (i.e. just copied from v2/v3)?\n\n...\n> Updates in V4\n> =============\n>\n> * Fixed an issue with the split index.\n>\n> * The helper methods are used more consistently.\n>\n> * The helper method path_in_cone_mode_sparse_checkout() is introduced.\n>\n> * Commit messages are edited for clarity.\n>\n> * A new config option is added to disable the behavior being added in this\n> series.\n>\n> * I split the commit that involves cache_tree_update(). I have not yet\n> succeeded in creating tests to demonstrate why this is required outside\n> of needing it in the Scalar functional tests, which includes a version of\n> partial clone. I will continue to investigate recreating this scenario in\n> the Git test suite, but I wanted to send this version out to get feedback\n> on the things that have changed.\n\nAre you talking about patch 3 or patch 4?\n\nFor patch 4, as I commented on that patch, my guess would be that you\nstart with a full tree, pick some directory that you will sparsify\naway, and before sparsifying it away, change some file under that\ndirectory and stage the change. Then when you sparsify the directory\naway with cone mode, I think the cache_tree_update() will need to\nwrite the tree. At least, some quick greps of cache_tree_update\nsuggest there aren't that many codepaths that attempt to update it, so\nI don't think anything else would have written it yet at that point.\n\nFor patch 3, since it comes before patch 5, it would be triggerable\nwith the same case. Is it possible you originally found both patches\nbased off similar initial conditions that just resulted in slightly\ndifferent behavioral reports, and thus that patch 4 is good enough\nwithout patch 3? Trying to ignore that possibility and looking at the\ncache_tree_update() code, the only other reasons for failure of\ncache_tree_update() that I can see are (a) someone having invalid\nhashes recorded in the index (e.g. someone using `git update-index\n--index-info` with some invalid hashes, or someone manually deleting\nsome loose blobs from under their .git/ directory), or (b) disk being\nfull or read-only so that writing new tree objects to the object store\nfail.\n\n> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1009\n\nA super micro nit:\n\nIt appears this is based on your other branches that have since been\nincorporated into master. Not a big deal, but rebasing on master\nmight make it easier for others to take a look at the series (I was\nconfused at first why this series depended on other unmerged series,\nbefore realizing those other series had since merged down). I know we\nnormally don't like rebasing series, but I suspect it wouldn't change\nthe range-diff (as I don't think others are touching these areas of\nthe code), and thus be transparent to Junio.\n\n\n\nI commented on some individual patches, but have some small comments\nthat are easier to post in response to the range-diff:\n\n> Range-diff vs v3:\n>\n> 1: e66106f7a99 ! 1: c407b2cb346 t7519: rewrite sparse index test\n> @@ t/t7519-status-fsmonitor.sh: test_expect_success 'status succeeds after staging/\n> test_expect_success 'status succeeds with sparse index' '\n> - git reset --hard &&\n> + git clone . full &&\n> -+ git clone . sparse &&\n> ++ git clone --sparse . sparse &&\n> + git -C sparse sparse-checkout init --cone --sparse-index &&\n> + git -C sparse sparse-checkout set dir1 dir2 &&\n>\n> 2: fb3ff9108bf ! 2: 8660877ba7a sparse-index: silently return when not using cone-mode patterns\n> @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate\n> {\n> int test_env;\n> - if (istate->split_index || istate->sparse_index ||\n> -+\n> + if (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n> !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n> return 0;\n> 3: 37198535268 ! 3: a669740af9a sparse-index: silently return when cache tree fails\n> @@ Commit message\n> convert_to_sparse(), then we could still recover from this scenario and\n> have a sparse index.\n>\n> - When constructing the cache-tree extension in convert_to_sparse(), it is\n> - possible that we construct a tree object that is new to the object\n> - database. Without the WRITE_TREE_MISSING_OK flag, this results in an\n> - error that halts our conversion to a sparse index. Add this flag to\n> - remove this limitation.\n> -\n> Signed-off-by: Derrick Stolee \n>\n> ## sparse-index.c ##\n> @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> + /*\n> + * Silently return if there is a problem with the cache tree update,\n> + * which might just be due to a conflict state in some entry.\n> -+ *\n> -+ * This might create new tree objects, so be sure to use\n> -+ * WRITE_TREE_MISSING_OK.\n> + */\n> -+ if (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n> ++ if (cache_tree_update(istate, 0))\n> + return 0;\n>\n> remove_fsmonitor(istate);\n> -: ----------- > 4: b379b8fc61a sparse-index: use WRITE_TREE_MISSING_OK\n\nThanks for splitting patch 3 & 4. I left some comments on patch 5.\nFor patch 4, my comments above about testcases might be of interest.\n\n> 4: 10bcadb284e ! 5: acdded0f762 unpack-trees: fix nested sparse-dir search\n> @@ Commit message\n> diff that modified files within the sparse directory entry, but we could\n> not correctly find the entry.\n>\n> + Helped-by: Johannes Schindelin \n> + Helped-by: René Scharfe \n> Signed-off-by: Derrick Stolee \n>\n> ## unpack-trees.c ##\n> @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,\n> const struct name_entry *p)\n> {\n> - struct cache_entry *ce;\n> ++ const char *path;\n> + struct cache_entry *ce = NULL;\n> int pos = find_cache_pos(info, p->path, p->pathlen);\n> struct unpack_trees_options *o = info->data;\n> -+ struct strbuf full_path = STRBUF_INIT;\n>\n> - if (0 <= pos)\n> - return o->src_index->cache[pos];\n> -@@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> - if (pos < 0 || pos >= o->src_index->cache_nr)\n> - return NULL;\n> -\n> -+ strbuf_addstr(&full_path, info->traverse_path);\n> -+ strbuf_add(&full_path, p->path, p->pathlen);\n> -+ strbuf_addch(&full_path, '/');\n> -+\n> - /*\n> - * Due to lexicographic sorting and sparse directory\n> - * entries ending with a trailing slash, our path as a\n> @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> while (pos >= 0) {\n> ce = o->src_index->cache[pos];\n>\n> - if (strncmp(ce->name, p->path, p->pathlen))\n> -- return NULL;\n> -+ if (strncmp(ce->name, full_path.buf, full_path.len)) {\n> -+ ce = NULL;\n> -+ break;\n> -+ }\n> ++ if (!skip_prefix(ce->name, info->traverse_path, &path) ||\n> ++ strncmp(path, p->path, p->pathlen) ||\n> ++ path[p->pathlen] != '/')\n> + return NULL;\n>\n> if (S_ISSPARSEDIR(ce->ce_mode) &&\n> - sparse_dir_matches_path(ce, info, p))\n> -- return ce;\n> -+ break;\n> -\n> - pos--;\n> - }\n> -\n> -- return NULL;\n> -+ strbuf_release(&full_path);\n> -+ return ce;\n> - }\n> -\n> - static void debug_path(struct traverse_info *info)\n\nRené's suggestion definitely made this patch simpler (which sticks out\nmore when you look at the patch rather than the range-diff).\n\n> 5: 5d28570c82a ! 6: 1958751aa0e sparse-checkout: create helper methods\n> @@ Commit message\n> adopt these helpers. There are just two in builtin/add.c and\n> sparse-index.c that can use path_in_sparse_checkout().\n>\n> + We add a path_in_cone_mode_sparse_checkout() as well that will only\n> + return false if the path is outside of the sparse-checkout definition\n> + _and_ the sparse-checkout patterns are in cone mode.\n> +\n> Signed-off-by: Derrick Stolee \n>\n> ## builtin/add.c ##\n> @@ builtin/add.c: static int refresh(int verbose, const struct pathspec *pathspec)\n> - (sparse_checkout_enabled &&\n> - !path_matches_pattern_list(path, strlen(path), NULL,\n> - &dtype, &pl, &the_index))) {\n> -+ (core_apply_sparse_checkout &&\n> -+ path_in_sparse_checkout(path, &the_index) == NOT_MATCHED)) {\n> ++ !path_in_sparse_checkout(path, &the_index)) {\n> string_list_append(&only_match_skip_worktree,\n> pathspec->items[i].original);\n> } else {\n> @@ dir.c: done:\n>\n> +int init_sparse_checkout_patterns(struct index_state *istate)\n> +{\n> -+ if (!core_apply_sparse_checkout ||\n> -+ istate->sparse_checkout_patterns)\n> ++ if (!core_apply_sparse_checkout)\n> ++ return 1;\n> ++ if (istate->sparse_checkout_patterns)\n> + return 0;\n> +\n> + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n> @@ dir.c: done:\n> + return 0;\n> +}\n> +\n> -+int path_in_sparse_checkout(const char *path,\n> -+ struct index_state *istate)\n> ++static int path_in_sparse_checkout_1(const char *path,\n> ++ struct index_state *istate,\n> ++ int require_cone_mode)\n> +{\n> + const char *base;\n> + int dtype = DT_REG;\n> -+ init_sparse_checkout_patterns(istate);\n> +\n> -+ if (!istate->sparse_checkout_patterns)\n> -+ return MATCHED;\n> ++ /*\n> ++ * We default to accepting a path if there are no patterns or\n> ++ * they are of the wrong type.\n> ++ */\n> ++ if (init_sparse_checkout_patterns(istate) ||\n> ++ (require_cone_mode &&\n> ++ !istate->sparse_checkout_patterns->use_cone_patterns))\n> ++ return 1;\n> +\n> + base = strrchr(path, '/');\n> + return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n> @@ dir.c: done:\n> + istate->sparse_checkout_patterns,\n> + istate) > 0;\n> +}\n> ++\n> ++int path_in_sparse_checkout(const char *path,\n> ++ struct index_state *istate)\n> ++{\n> ++ return path_in_sparse_checkout_1(path, istate, 0);\n> ++}\n> ++\n> ++int path_in_cone_mode_sparse_checkout(const char *path,\n> ++ struct index_state *istate)\n> ++{\n> ++ return path_in_sparse_checkout_1(path, istate, 1);\n> ++}\n> +\n> static struct path_pattern *last_matching_pattern_from_lists(\n> struct dir_struct *dir, struct index_state *istate,\n> @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n> +\n> +int path_in_sparse_checkout(const char *path,\n> + struct index_state *istate);\n> ++int path_in_cone_mode_sparse_checkout(const char *path,\n> ++ struct index_state *istate);\n> +\n> struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n> struct index_state *istate,\n> @@ dir.h: enum pattern_match_result path_matches_pattern_list(const char *pathname,\n>\n> ## sparse-index.c ##\n> @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,\n> + {\n> int i, can_convert = 1;\n> int start_converted = num_converted;\n> - enum pattern_match_result match;\n> +- enum pattern_match_result match;\n> - int dtype = DT_UNKNOWN;\n> struct strbuf child_path = STRBUF_INIT;\n> - struct pattern_list *pl = istate->sparse_checkout_patterns;\n> @@ sparse-index.c: static int convert_to_sparse_rec(struct index_state *istate,\n> */\n> - match = path_matches_pattern_list(ct_path, ct_pathlen,\n> - NULL, &dtype, pl, istate);\n> -+ match = path_in_sparse_checkout(ct_path, istate);\n> - if (match != NOT_MATCHED)\n> +- if (match != NOT_MATCHED)\n> ++ if (path_in_sparse_checkout(ct_path, istate))\n> can_convert = 0;\n>\n> + for (i = start; can_convert && i < end; i++) {\n> @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> if (!istate->repo->settings.sparse_index)\n> return 0;\n> @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> - if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n> - return 0;\n> - }\n> -+ if (init_sparse_checkout_patterns(istate) < 0)\n> ++ if (init_sparse_checkout_patterns(istate))\n> + return 0;\n>\n> /*\n> 6: c9e100e68f8 ! 7: e496f3cee66 attr: be careful about sparse directories\n> @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat\n> return NULL;\n>\n> + /*\n> -+ * In the case of cone-mode sparse-checkout, getting the\n> -+ * .gitattributes file from a directory is meaningless: all\n> -+ * contained paths will be sparse if the .gitattributes is also\n> -+ * sparse. In the case of a sparse index, it is critical that we\n> -+ * don't go looking for one as it will expand the index.\n> ++ * The .gitattributes file only applies to files within its\n> ++ * parent directory. In the case of cone-mode sparse-checkout,\n> ++ * the .gitattributes file is sparse if and only if all paths\n> ++ * within that directory are also sparse. Thus, don't load the\n> ++ * .gitattributes file since it will not matter.\n> ++ *\n> ++ * In the case of a sparse index, it is critical that we don't go\n> ++ * looking for a .gitattributes file, as the index will expand.\n\nThis comment is better, and I appreciate the thought behind adding the\nsecond paragraph, but the wording seems fuzzy and potentially\nconfusing to future readers. Perhaps changing the wording to:\n\n...looking for a .gitattributes file, as doing so would cause the\nindex to expand.\n\n> + */\n> -+ init_sparse_checkout_patterns(istate);\n> -+ if (istate->sparse_checkout_patterns &&\n> -+ istate->sparse_checkout_patterns->use_cone_patterns &&\n> -+ path_in_sparse_checkout(path, istate) == NOT_MATCHED)\n> ++ if (!path_in_cone_mode_sparse_checkout(path, istate))\n> + return NULL;\n> +\n> buf = read_blob_data_from_index(istate, path, NULL);\n> 7: b0ece4b7dcc ! 8: cab9360b1e9 sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n> @@ Metadata\n> Author: Derrick Stolee \n>\n> ## Commit message ##\n> - sparse-index: add SPARSE_INDEX_IGNORE_CONFIG flag\n> + sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n>\n> The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\n> environment variable or the \"index.sparse\" config setting before\n> @@ Commit message\n> index as an in-memory data structure, regardless of whether the on-disk\n> format should be sparse.\n>\n> - To that end, create the SPARSE_INDEX_IGNORE_CONFIG flag that will skip\n> + To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip\n> these config checks when enabled. All current consumers are modified to\n> pass '0' in the new 'flags' parameter.\n>\n> @@ sparse-index.c: static int index_has_unmerged_entries(struct index_state *istate\n> +int convert_to_sparse(struct index_state *istate, int flags)\n> {\n> int test_env;\n> +- if (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n> ++ if (istate->sparse_index || !istate->cache_nr ||\n> + !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n> + return 0;\n>\n> -@@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> if (!istate->repo)\n> istate->repo = the_repository;\n>\n> @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> - test_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n> - if (test_env >= 0)\n> - set_sparse_index_config(istate->repo, test_env);\n> -+ if (!(flags & SPARSE_INDEX_IGNORE_CONFIG)) {\n> ++ if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {\n> ++ /*\n> ++ * The sparse index is not (yet) integrated with a split index.\n> ++ */\n> ++ if (istate->split_index)\n> ++ return 0;\n> + /*\n> + * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n> + * index.sparse config variable to be on.\n> @@ sparse-index.c: int convert_to_sparse(struct index_state *istate)\n> + return 0;\n> + }\n>\n> - if (init_sparse_checkout_patterns(istate) < 0)\n> + if (init_sparse_checkout_patterns(istate))\n> return 0;\n>\n> ## sparse-index.h ##\n> @@ sparse-index.h\n>\n> struct index_state;\n> -int convert_to_sparse(struct index_state *istate);\n> -+#define SPARSE_INDEX_IGNORE_CONFIG (1 << 0)\n> ++#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n> +int convert_to_sparse(struct index_state *istate, int flags);\n>\n> /*\n> 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n> @@ Commit message\n> the sparse directory. This depends on the indexed version of the file,\n> so the sparse directory must be expanded.\n>\n> + We must take special care to look for untracked, non-ignored files in\n> + these directories before deleting them. We do not want to delete any\n> + meaningful work that the users were doing in those directories and\n> + perhaps forgot to add and commit before switching sparse-checkout\n> + definitions. Since those untracked files might be code files that\n> + generated ignored build output, also do not delete any ignored files\n> + from these directories in that case. The users can recover their state\n> + by resetting their sparse-checkout definition to include that directory\n> + and continue. Alternatively, they can see the warning that is presented\n> + and delete the directory themselves to regain the performance they\n> + expect.\n> +\n> By deleting the sparse directories when changing scope (or running 'git\n> sparse-checkout reapply') we regain these performance benefits as if the\n> repository was in a clean state.\n> @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for\n> 'git sparse-checkout set' command to reflect the expected cone in the working\n> directory.\n>\n> -+The cone mode sparse-checkout patterns will also remove ignored files that\n> -+are not within the sparse-checkout definition. This is important behavior\n> -+to preserve the performance of the sparse index, but also matches that\n> -+cone mode patterns care about directories, not files. If there exist files\n> -+that are untracked and not ignored, then Git will not delete files within\n> -+that directory other than the tracked files that are now out of scope.\n> -+These files should be removed manually to ensure Git can behave optimally.\n> ++When changing the sparse-checkout patterns in cone mode, Git will inspect each\n> ++tracked directory that is not within the sparse-checkout cone to see if it\n> ++contains any untracked files. If all of those files are ignored due to the\n> ++`.gitignore` patterns, then the directory will be deleted. If any of the\n> ++untracked files within that directory is not ignored, then no deletions will\n> ++occur within that directory and a warning message will appear. If these files\n> ++are important, then reset your sparse-checkout definition so they are included,\n> ++use `git add` and `git commit` to store them, then remove any remaining files\n> ++manually to ensure Git can behave optimally.\n> +\n>\n> SUBMODULES\n> @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char\n> + */\n> + if (!r || !r->index || !r->worktree)\n> + return;\n> -+ init_sparse_checkout_patterns(r->index);\n> -+ if (!r->index->sparse_checkout_patterns ||\n> ++ if (init_sparse_checkout_patterns(r->index) ||\n> + !r->index->sparse_checkout_patterns->use_cone_patterns)\n> + return;\n> +\n> @@ builtin/sparse-checkout.c: static int sparse_checkout_list(int argc, const char\n> + * prevents us from converting to a sparse index, then do\n> + * not try deleting files.\n> + */\n> -+ if (convert_to_sparse(r->index, SPARSE_INDEX_IGNORE_CONFIG))\n> ++ if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))\n> + return;\n> + was_full = 1;\n> + }\n\nThanks for the extra explanations; those help.\n\nYou haven't yet addressed how this choice will interact with\n.gitignore files, though (as I commented on v3). You alluded in patch\n7 to how it was critical to avoid looking for .gitattributes files\nbecause doing so would require the index to expand, possibly\nrecursively all the way down. Don't .gitignore files have the same\nproblem? Nuking all the ignored files so that you can delete extra\ndirectories and thus not need to walk into them during operations\nhelps, but does the fact that we sometimes avoid removing directories\nmean we have to walk into them anyway? Do we only need to walk into\nsuch directories when they are (still? again?) present in the working\ncopy? Does there need to be some code added somewhere that checks for\nthe presence of a directory in the working copy and unsparsifies\nsparse directory entries in the index when such directories are found?\n\nAlso, does non-cone mode sparsity paths have weird bugs around letting\nusers specify patterns that would happen to also exclude .gitignore\nfiles from the working copy, but keep other sibling files within the\nsame directory? In particular, if the .gitignore files aren't\npresent, does git add -A add a bunch of supposed-to-be-ignored files?\n(If so, that's an issue that's probably tangential to this series\nsince it's not about cone mode or the sparse index, but it's certainly\nan interesting thought.)\n\nMaybe the whole .gitignore situation with cone mode has a simple\nanswer, but since this whole series is about gaining performance by\navoiding recursing into directories corresponding to sparse directory\nentries, it feels the answers to how .gitignore will be handled should\nat least be addressed.\n\n> -: ----------- > 10: 8d55a6ba2fd sparse-checkout: add config to disable deleting dirs\n\nI left some comments.\n\n\nThanks, as always, for all your hard work on this!\n"},{"id":"433971","messageId":"CABPp-BH2+Vq4gLnOyRg4AdNhZgZuFf5+XBW6JMmSzGxoppEctA@mail.gmail.com","threadId":"56179","inReplyTo":"CABPp-BG01tXnd_UhYJDX4Tm_EcXGju9CkB8BXU0XS-YAWEuaeQ@mail.gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-27T22:01:30Z","receivedAt":"2021-08-27T22:01:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry, one more thing...\n\nOn Fri, Aug 27, 2021 at 2:56 PM Elijah Newren wrote:\n>\n> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> wrote:\n\n> > 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n> > @@ Commit message\n> > the sparse directory. This depends on the indexed version of the file,\n> > so the sparse directory must be expanded.\n> >\n> > + We must take special care to look for untracked, non-ignored files in\n> > + these directories before deleting them. We do not want to delete any\n> > + meaningful work that the users were doing in those directories and\n> > + perhaps forgot to add and commit before switching sparse-checkout\n> > + definitions. Since those untracked files might be code files that\n> > + generated ignored build output, also do not delete any ignored files\n> > + from these directories in that case. The users can recover their state\n> > + by resetting their sparse-checkout definition to include that directory\n> > + and continue. Alternatively, they can see the warning that is presented\n> > + and delete the directory themselves to regain the performance they\n> > + expect.\n\nWill deleting the directory regain the performance they expect, or is\nanother step needed? In other worse, will the sparsification of the\npaths under the no-longer-needed directory into a sparse directory\nentry just happen automatically as part of some other command like\ncommit/add, or do they need to manually run `git sparse-checkout\nreapply`?\n\n> > +\n> > By deleting the sparse directories when changing scope (or running 'git\n> > sparse-checkout reapply') we regain these performance benefits as if the\n> > repository was in a clean state.\n> > @@ Documentation/git-sparse-checkout.txt: case-insensitive check. This corrects for\n> > 'git sparse-checkout set' command to reflect the expected cone in the working\n> > directory.\n> >\n> > -+The cone mode sparse-checkout patterns will also remove ignored files that\n> > -+are not within the sparse-checkout definition. This is important behavior\n> > -+to preserve the performance of the sparse index, but also matches that\n> > -+cone mode patterns care about directories, not files. If there exist files\n> > -+that are untracked and not ignored, then Git will not delete files within\n> > -+that directory other than the tracked files that are now out of scope.\n> > -+These files should be removed manually to ensure Git can behave optimally.\n> > ++When changing the sparse-checkout patterns in cone mode, Git will inspect each\n> > ++tracked directory that is not within the sparse-checkout cone to see if it\n> > ++contains any untracked files. If all of those files are ignored due to the\n> > ++`.gitignore` patterns, then the directory will be deleted. If any of the\n> > ++untracked files within that directory is not ignored, then no deletions will\n> > ++occur within that directory and a warning message will appear. If these files\n> > ++are important, then reset your sparse-checkout definition so they are included,\n> > ++use `git add` and `git commit` to store them, then remove any remaining files\n> > ++manually to ensure Git can behave optimally.\n"},{"id":"434071","messageId":"2870fcb8-a356-c2c2-d084-b560326e1ad4@gmail.com","threadId":"56179","inReplyTo":"CABPp-BFcySug2kSvxT7YdJ1Oorza5AfxKPJgMBE2wN8qOw+=Eg@mail.gmail.com","subject":"Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-30T13:19:15Z","receivedAt":"2021-08-30T13:19:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:33 PM, Elijah Newren wrote:\n> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When updating the cache tree in convert_to_sparse(), the\n>> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that\n>> do not already exist within the object database.\n> \n> Okay.\n> \n>> This happens in cases\n>> such as 'git add' creating new trees that it wants to store in\n>> anticipation of a following 'git commit'.\n> \n> This doesn't make any sense to me. Does 'git add' call\n> convert_to_sparse()? I don't see why it would; wouldn't the calls to\n> convert_to_sparse() come via sparse-checkout init/set commands? If\n> I'm correct on that, and 'git add' wants to create new trees, then by\n> the time convert_to_sparse() is called in some subsequent git\n> operation, then convert_to_sparse would already have the trees it\n> needs.\n\nIf someone adds a change outside the sparse-checkout cone, then the\nindex is expanded in-memory, then is converted to sparse when the\nindex is written again.\n\n> I thought the reason you would need this is someone modified and\n> staged a change to a file underneath a directory that will be\n> sparsified away; at the time of convert_to_sparse(), a tree object may\n> not have yet been written for the new tree with the newly modified\n> file (because those tend to be written at commit time), but you'd need\n> it at the time you sparsified.\n\nYes. I think we are trying to say the same thing.\n\nThanks,\n-Stolee\n"},{"id":"434073","messageId":"d63d5d18-bdb3-e13b-ae36-e7e6cf10be39@gmail.com","threadId":"56179","inReplyTo":"CABPp-BGeJJBF0ZmS7ZHcqFmXwXmgVH1Uox_6gNHzcacG2vRjVw@mail.gmail.com","subject":"Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-30T13:30:48Z","receivedAt":"2021-08-30T13:30:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 4:58 PM, Elijah Newren wrote:\n> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The clean_tracked_sparse_directories() method deletes the tracked\n>> directories that go out of scope when the sparse-checkout cone changes,\n>> at least in cone mode. This is new behavior, but is recommended based on\n>> our understanding of how users are interacting with the feature in most\n>> cases.\n>>\n>> It is possible that some users will object to the new behavior, so\n>> create a new configuration option 'index.deleteSparseDirectories' that\n>> can be set to 'false' to make clean_tracked_sparse_directories() do\n>> nothing. This will keep all untracked files in the working tree and\n>> cause performance problems with the sparse index, but those trade-offs\n>> are for the user to decide.\n> \n> I'm not sold that we need anything here, and it sounds like this is\n> all being added based on a theoretical concern. I might not object\n> too much normally to trying to address theoretical conerns with new\n> features, but:\n> \n> * I'm a little concerned that we're adding a configuration option\n> (which live forever and are harder to work with\n> backward-compatibility-wise) rather than a command line flag.\n\nThe issue with a command-line flag is that it would need to be added\nto all the commands that reapply the sparse-checkout definition.\nMaybe that's just the 'git sparse-checkout (init|set|add|reapply)'\nand 'git read-tree' commands, but are there other places where this\nlogic might be applied in the future?\n\nAlso, as more third-party tools integrate with sparse-checkout, I\ndon't expect users to be directly involved in changing their cone,\nso any use of a command-line flag would need to be integrated into\nthose tools. A config option applies this logic universally, when\nneeded.\n\n> * It's not clear to me that the option name is what we want. For\n> example, git checkout has a --[no-]overwrite-ignored for overwriting\n> ignored files (or not) when switching to a different branch. git\n> merge has the same flags (though only the fast-forwarding backend\n> heeds it currently). This behavior is quite similar to that flag, and\n> has the same default of treating ignored files as expendable. Should\n> the naming be more similar?\n\nI'm open to changing the name to more closely match existing naming\nconventions, once we've decided if this should be included at all.\n\n> * It's not clear to me that the option has the right level of\n> implementation granularity. If someone wants ignored files to be\n> treated as important, should they need to set a\n> merge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a\n> index.deleteSparseDirectories AND other names, or have one high-level\n> option that sets all of these behaviors?\n\nThese cases have different meanings for why an ignored file is\nimportant. For merge and checkout, the argument is saying \"don't\noverwrite ignored files if they are to be replaced by a tracked\nfile\". For sparse-checkout, the config is saying \"don't delete\nignored files that are leaving the sparse-checkout scope\". I think\nit is meaningful to say that files that match the .gitignore\npatterns but are still tracked are deleted, because sparse-checkout\nremoves all tracked files.\n\nI'm less convinced that there needs to be a meta-setting that covers\nall of these cases simultaneously.\n\n---\n\nAs for moving forward, I'm fine skipping this patch if there is no\nsupport for it. I just wanted to demonstrate that it could be done.\nPerhaps we wait for the theoretical concern to be a real one, as\nrequested by a user with a legitimate reason to care?\n\nThanks,\n-Stolee\n"},{"id":"434075","messageId":"9b4adc13-d362-455f-60f6-fe7d16608ae6@gmail.com","threadId":"56179","inReplyTo":"CABPp-BH2+Vq4gLnOyRg4AdNhZgZuFf5+XBW6JMmSzGxoppEctA@mail.gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-30T13:34:10Z","receivedAt":"2021-08-30T13:34:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 6:01 PM, Elijah Newren wrote:\n> Sorry, one more thing...\n> \n> On Fri, Aug 27, 2021 at 2:56 PM Elijah Newren wrote:\n>>\n>> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n>> wrote:\n> \n>>> 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n>>> @@ Commit message\n>>> the sparse directory. This depends on the indexed version of the file,\n>>> so the sparse directory must be expanded.\n>>>\n>>> + We must take special care to look for untracked, non-ignored files in\n>>> + these directories before deleting them. We do not want to delete any\n>>> + meaningful work that the users were doing in those directories and\n>>> + perhaps forgot to add and commit before switching sparse-checkout\n>>> + definitions. Since those untracked files might be code files that\n>>> + generated ignored build output, also do not delete any ignored files\n>>> + from these directories in that case. The users can recover their state\n>>> + by resetting their sparse-checkout definition to include that directory\n>>> + and continue. Alternatively, they can see the warning that is presented\n>>> + and delete the directory themselves to regain the performance they\n>>> + expect.\n> \n> Will deleting the directory regain the performance they expect, or is\n> another step needed? In other worse, will the sparsification of the\n> paths under the no-longer-needed directory into a sparse directory\n> entry just happen automatically as part of some other command like\n> commit/add, or do they need to manually run `git sparse-checkout\n> reapply`?\n\nIf the directory is gone, then their performance returns to as they\nexpect, since the next Git command will not need to expand the index\nto detect the difference between tracked, untracked, and ignored files\nwithin the sparse directory.\n\n(Recall that the index is stored as sparse even if it expands to a\nfull one in-memory during execution.)\n\nThanks,\n-Stolee\n"},{"id":"434079","messageId":"d55363df-c2e9-3847-d98c-3d07e22d8a5d@gmail.com","threadId":"56179","inReplyTo":"CABPp-BG01tXnd_UhYJDX4Tm_EcXGju9CkB8BXU0XS-YAWEuaeQ@mail.gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-08-30T13:54:07Z","receivedAt":"2021-08-30T13:54:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:56 PM, Elijah Newren wrote:\n> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> UPDATE: I had to base this version on a merge of ds/add-with-sparse-index\n>> and v2.33.0-rc1 because of some changes to dir.h that became important!\n> \n> Isn't this stale (i.e. just copied from v2/v3)?\n\nSorry, yes. It was an update to the original cover letter, but was not\nupdated in later versions.\n\n>> * I split the commit that involves cache_tree_update(). I have not yet\n>> succeeded in creating tests to demonstrate why this is required outside\n>> of needing it in the Scalar functional tests, which includes a version of\n>> partial clone. I will continue to investigate recreating this scenario in\n>> the Git test suite, but I wanted to send this version out to get feedback\n>> on the things that have changed.\n> \n> Are you talking about patch 3 or patch 4?\n\nI'm talking about patches 3 and 4, which (at the time that I created\nthem a few months ago, before later merging them) they each had solved\nsome problems in the Scalar functional tests. The issue I'm seeing now\nis that since so many commands have integrated with the sparse index,\nthose functional tests are not testing convert_to_sparse() as heavily,\nso simply reverting each does not show the problem any more.\n\n> For patch 4, as I commented on that patch, my guess would be that you\n> start with a full tree, pick some directory that you will sparsify\n> away, and before sparsifying it away, change some file under that\n> directory and stage the change. Then when you sparsify the directory\n> away with cone mode, I think the cache_tree_update() will need to\n> write the tree. At least, some quick greps of cache_tree_update\n> suggest there aren't that many codepaths that attempt to update it, so\n> I don't think anything else would have written it yet at that point.\n> \n> For patch 3, since it comes before patch 5, it would be triggerable\n> with the same case. Is it possible you originally found both patches\n> based off similar initial conditions that just resulted in slightly\n> different behavioral reports, and thus that patch 4 is good enough\n> without patch 3? Trying to ignore that possibility and looking at the\n> cache_tree_update() code, the only other reasons for failure of\n> cache_tree_update() that I can see are (a) someone having invalid\n> hashes recorded in the index (e.g. someone using `git update-index\n> --index-info` with some invalid hashes, or someone manually deleting\n> some loose blobs from under their .git/ directory), or (b) disk being\n> full or read-only so that writing new tree objects to the object store\n> fail.\n\nJust for full context:\n\n* Patch 3 removes a warning and \"return -1\" when cache_tree_update()\n fails. This makes convert_to_sparse() resilient to any failure that\n might happen in cache_tree_update(), whatever it might be.\n\n* Patch 4 adds the WRITE_TREE_MISSING_OK, which removes one possible\n reason that cache_tree_update() could fail (and removes an attempted\n promisor fetch in the case of partial clone).\n\nSo I could see that either of the two patches would be sufficient to\nmake tests pass in all normally-testable scenarios. However, patch 3\nis valuable for possibly more complicated failure scenarios such as\ncorrupted data, while patch 4 is valuable for performance reasons in\na very specific environment.\n\n>> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\n>> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v4\n>> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v4\n>> Pull-Request: https://github.com/gitgitgadget/git/pull/1009\n> \n> A super micro nit:\n> \n> It appears this is based on your other branches that have since been\n> incorporated into master. Not a big deal, but rebasing on master\n> might make it easier for others to take a look at the series (I was\n> confused at first why this series depended on other unmerged series,\n> before realizing those other series had since merged down). I know we\n> normally don't like rebasing series, but I suspect it wouldn't change\n> the range-diff (as I don't think others are touching these areas of\n> the code), and thus be transparent to Junio.\n\nThe 'git add' integration only recently hit 'master', so I have not\nhad a chance to base on a version of that branch with the necessary\ndependencies.\n\n>> 6: c9e100e68f8 ! 7: e496f3cee66 attr: be careful about sparse directories\n>> @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat\n>> return NULL;\n>>\n>> + /*\n>> -+ * In the case of cone-mode sparse-checkout, getting the\n>> -+ * .gitattributes file from a directory is meaningless: all\n>> -+ * contained paths will be sparse if the .gitattributes is also\n>> -+ * sparse. In the case of a sparse index, it is critical that we\n>> -+ * don't go looking for one as it will expand the index.\n>> ++ * The .gitattributes file only applies to files within its\n>> ++ * parent directory. In the case of cone-mode sparse-checkout,\n>> ++ * the .gitattributes file is sparse if and only if all paths\n>> ++ * within that directory are also sparse. Thus, don't load the\n>> ++ * .gitattributes file since it will not matter.\n>> ++ *\n>> ++ * In the case of a sparse index, it is critical that we don't go\n>> ++ * looking for a .gitattributes file, as the index will expand.\n> \n> This comment is better, and I appreciate the thought behind adding the\n> second paragraph, but the wording seems fuzzy and potentially\n> confusing to future readers. Perhaps changing the wording to:\n> \n> ...looking for a .gitattributes file, as doing so would cause the\n> index to expand.\n\nThanks.\n\n>> 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n>> @@ Commit message\n>> the sparse directory. This depends on the indexed version of the file,\n>> so the sparse directory must be expanded.\n>>\n>> + We must take special care to look for untracked, non-ignored files in\n>> + these directories before deleting them. We do not want to delete any\n>> + meaningful work that the users were doing in those directories and\n>> + perhaps forgot to add and commit before switching sparse-checkout\n>> + definitions. Since those untracked files might be code files that\n>> + generated ignored build output, also do not delete any ignored files\n>> + from these directories in that case. The users can recover their state\n>> + by resetting their sparse-checkout definition to include that directory\n>> + and continue. Alternatively, they can see the warning that is presented\n>> + and delete the directory themselves to regain the performance they\n>> + expect.\n>> +\n...\n>> ++When changing the sparse-checkout patterns in cone mode, Git will inspect each\n>> ++tracked directory that is not within the sparse-checkout cone to see if it\n>> ++contains any untracked files. If all of those files are ignored due to the\n>> ++`.gitignore` patterns, then the directory will be deleted. If any of the\n>> ++untracked files within that directory is not ignored, then no deletions will\n>> ++occur within that directory and a warning message will appear. If these files\n>> ++are important, then reset your sparse-checkout definition so they are included,\n>> ++use `git add` and `git commit` to store them, then remove any remaining files\n>> ++manually to ensure Git can behave optimally.\n...\n> Thanks for the extra explanations; those help.\n> \n> You haven't yet addressed how this choice will interact with\n> .gitignore files, though (as I commented on v3). You alluded in patch\n> 7 to how it was critical to avoid looking for .gitattributes files\n> because doing so would require the index to expand, possibly\n> recursively all the way down. Don't .gitignore files have the same\n> problem? Nuking all the ignored files so that you can delete extra\n> directories and thus not need to walk into them during operations\n> helps, but does the fact that we sometimes avoid removing directories\n> mean we have to walk into them anyway? Do we only need to walk into\n> such directories when they are (still? again?) present in the working\n> copy? Does there need to be some code added somewhere that checks for\n> the presence of a directory in the working copy and unsparsifies\n> sparse directory entries in the index when such directories are found?\n> \n> Also, does non-cone mode sparsity paths have weird bugs around letting\n> users specify patterns that would happen to also exclude .gitignore\n> files from the working copy, but keep other sibling files within the\n> same directory? In particular, if the .gitignore files aren't\n> present, does git add -A add a bunch of supposed-to-be-ignored files?\n> (If so, that's an issue that's probably tangential to this series\n> since it's not about cone mode or the sparse index, but it's certainly\n> an interesting thought.)\n> \n> Maybe the whole .gitignore situation with cone mode has a simple\n> answer, but since this whole series is about gaining performance by\n> avoiding recursing into directories corresponding to sparse directory\n> entries, it feels the answers to how .gitignore will be handled should\n> at least be addressed.\n\nThe goal of this series is to allow Git to operate as normal (respecting\n.gitattributes and .gitignore files as part of that) without needing to\nexpand a sparse index. The way it achieves this is by deleting the\ndirectories that leave the sparse-checkout cone. If those directories\ndon't exist, then we don't care about the .gitattributes or .gitignore\nfiles in those locations. When the directory _does_ exist, then we need\nto look for the .gitignore file which expands the index to full. This\nis done naturally by Git using index_name_pos() for \"/.gitignore\"\nwhich finds that \"\" (or one of its parent directories) is a\nsparse directory, then expands the index and searches for\n\"/.gitignore\" again, returning the position if it exists in the\nfully-expanded index.\n\nThe difference between the .gitattributes and .gitignore files is that\nwe don't need to do anything with the .gitignore files to prevent Git\nfrom accessing them with their contained directories don't exist. The\n.gitattributes change in attr.c is reachable from some code path even\nwhen the directory doesn't exist, so it needs special logic to avoid\nthat expansion.\n\nWhen we don't succeed in deleting sparse directories, performance\nsuffers. That's the reason behind having a very-visible warnings to\nusers that something is blocking the removal. We will be monitoring\nperformance of users where it is _very clear_ when this happens. From\nthat experience, we might discover new ways to assist them to get\ninto a better state (perhaps a warning during \"git status\" would help?).\n\nThanks,\n-Stolee\n"},{"id":"434154","messageId":"CABPp-BF6-1_AVmuT9bE4wRqqh1p4RAjfK1EqyyRW98sQPYNDyw@mail.gmail.com","threadId":"56179","inReplyTo":"2870fcb8-a356-c2c2-d084-b560326e1ad4@gmail.com","subject":"Re: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-30T20:08:39Z","receivedAt":"2021-08-30T20:08:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 30, 2021 at 6:19 AM Derrick Stolee wrote:\n>\n> On 8/27/2021 5:33 PM, Elijah Newren wrote:\n> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n> >> From: Derrick Stolee \n> >>\n> >> When updating the cache tree in convert_to_sparse(), the\n> >> WRITE_TREE_MISSING_OK flag indicates that trees might be computed that\n> >> do not already exist within the object database.\n> >\n> > Okay.\n> >\n> >> This happens in cases\n> >> such as 'git add' creating new trees that it wants to store in\n> >> anticipation of a following 'git commit'.\n> >\n> > This doesn't make any sense to me. Does 'git add' call\n> > convert_to_sparse()? I don't see why it would; wouldn't the calls to\n> > convert_to_sparse() come via sparse-checkout init/set commands? If\n> > I'm correct on that, and 'git add' wants to create new trees, then by\n> > the time convert_to_sparse() is called in some subsequent git\n> > operation, then convert_to_sparse would already have the trees it\n> > needs.\n>\n> If someone adds a change outside the sparse-checkout cone, then the\n> index is expanded in-memory, then is converted to sparse when the\n> index is written again.\n\nAh, I missed that convert_to_sparse() is called from write_index(),\nand that we are dealing with a single operation that does a\nsparse->full (in memory)->sparse roundtrip.\n\nThanks for clearing that up.\n\n> > I thought the reason you would need this is someone modified and\n> > staged a change to a file underneath a directory that will be\n> > sparsified away; at the time of convert_to_sparse(), a tree object may\n> > not have yet been written for the new tree with the newly modified\n> > file (because those tend to be written at commit time), but you'd need\n> > it at the time you sparsified.\n>\n> Yes. I think we are trying to say the same thing.\n\nWhile writing this, I was thinking in terms of the `git add` being\ndone when the index was full (both in memory and on disk), and then a\nlater `git sparse-checkout ...` command would invoke the\nconvert_to_sparse(). I didn't fully catch the nuances here.\n\nThanks for bringing me up to speed.\n"},{"id":"434155","messageId":"CABPp-BFT31Gj3WkhYQzaT2_efzr2c2viReLNq63_MAQiWAc4Fw@mail.gmail.com","threadId":"56179","inReplyTo":"d63d5d18-bdb3-e13b-ae36-e7e6cf10be39@gmail.com","subject":"Re: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-30T20:11:28Z","receivedAt":"2021-08-30T20:11:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 30, 2021 at 6:30 AM Derrick Stolee wrote:\n>\n> On 8/27/2021 4:58 PM, Elijah Newren wrote:\n> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n> >> From: Derrick Stolee \n> >>\n> >> The clean_tracked_sparse_directories() method deletes the tracked\n> >> directories that go out of scope when the sparse-checkout cone changes,\n> >> at least in cone mode. This is new behavior, but is recommended based on\n> >> our understanding of how users are interacting with the feature in most\n> >> cases.\n> >>\n> >> It is possible that some users will object to the new behavior, so\n> >> create a new configuration option 'index.deleteSparseDirectories' that\n> >> can be set to 'false' to make clean_tracked_sparse_directories() do\n> >> nothing. This will keep all untracked files in the working tree and\n> >> cause performance problems with the sparse index, but those trade-offs\n> >> are for the user to decide.\n> >\n> > I'm not sold that we need anything here, and it sounds like this is\n> > all being added based on a theoretical concern. I might not object\n> > too much normally to trying to address theoretical conerns with new\n> > features, but:\n> >\n> > * I'm a little concerned that we're adding a configuration option\n> > (which live forever and are harder to work with\n> > backward-compatibility-wise) rather than a command line flag.\n>\n> The issue with a command-line flag is that it would need to be added\n> to all the commands that reapply the sparse-checkout definition.\n> Maybe that's just the 'git sparse-checkout (init|set|add|reapply)'\n> and 'git read-tree' commands, but are there other places where this\n> logic might be applied in the future?\n>\n> Also, as more third-party tools integrate with sparse-checkout, I\n> don't expect users to be directly involved in changing their cone,\n> so any use of a command-line flag would need to be integrated into\n> those tools. A config option applies this logic universally, when\n> needed.\n>\n> > * It's not clear to me that the option name is what we want. For\n> > example, git checkout has a --[no-]overwrite-ignored for overwriting\n> > ignored files (or not) when switching to a different branch. git\n> > merge has the same flags (though only the fast-forwarding backend\n> > heeds it currently). This behavior is quite similar to that flag, and\n> > has the same default of treating ignored files as expendable. Should\n> > the naming be more similar?\n>\n> I'm open to changing the name to more closely match existing naming\n> conventions, once we've decided if this should be included at all.\n>\n> > * It's not clear to me that the option has the right level of\n> > implementation granularity. If someone wants ignored files to be\n> > treated as important, should they need to set a\n> > merge.no_overwrite_ignored AND a checkout.no_overwrite_ignored AND a\n> > index.deleteSparseDirectories AND other names, or have one high-level\n> > option that sets all of these behaviors?\n>\n> These cases have different meanings for why an ignored file is\n> important. For merge and checkout, the argument is saying \"don't\n> overwrite ignored files if they are to be replaced by a tracked\n> file\". For sparse-checkout, the config is saying \"don't delete\n> ignored files that are leaving the sparse-checkout scope\". I think\n> it is meaningful to say that files that match the .gitignore\n> patterns but are still tracked are deleted, because sparse-checkout\n> removes all tracked files.\n>\n> I'm less convinced that there needs to be a meta-setting that covers\n> all of these cases simultaneously.\n>\n> ---\n>\n> As for moving forward, I'm fine skipping this patch if there is no\n> support for it. I just wanted to demonstrate that it could be done.\n\nThat's fair, and you certainly achieved that.\n\n> Perhaps we wait for the theoretical concern to be a real one, as\n> requested by a user with a legitimate reason to care?\n\nSounds good to me. Then we'll have a better frame of reference for\njudging how the feature should be tailored as well.\n"},{"id":"434156","messageId":"CABPp-BGq7CccMVC4fCtH6wTz2Mcms3UxAk5VTr4RSP0RugTVmg@mail.gmail.com","threadId":"56179","inReplyTo":"9b4adc13-d362-455f-60f6-fe7d16608ae6@gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-30T20:14:39Z","receivedAt":"2021-08-30T20:14:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 30, 2021 at 6:34 AM Derrick Stolee wrote:\n>\n> On 8/27/2021 6:01 PM, Elijah Newren wrote:\n> > Sorry, one more thing...\n> >\n> > On Fri, Aug 27, 2021 at 2:56 PM Elijah Newren wrote:\n> >>\n> >> On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> >> wrote:\n> >\n> >>> 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n> >>> @@ Commit message\n> >>> the sparse directory. This depends on the indexed version of the file,\n> >>> so the sparse directory must be expanded.\n> >>>\n> >>> + We must take special care to look for untracked, non-ignored files in\n> >>> + these directories before deleting them. We do not want to delete any\n> >>> + meaningful work that the users were doing in those directories and\n> >>> + perhaps forgot to add and commit before switching sparse-checkout\n> >>> + definitions. Since those untracked files might be code files that\n> >>> + generated ignored build output, also do not delete any ignored files\n> >>> + from these directories in that case. The users can recover their state\n> >>> + by resetting their sparse-checkout definition to include that directory\n> >>> + and continue. Alternatively, they can see the warning that is presented\n> >>> + and delete the directory themselves to regain the performance they\n> >>> + expect.\n> >\n> > Will deleting the directory regain the performance they expect, or is\n> > another step needed? In other worse, will the sparsification of the\n> > paths under the no-longer-needed directory into a sparse directory\n> > entry just happen automatically as part of some other command like\n> > commit/add, or do they need to manually run `git sparse-checkout\n> > reapply`?\n>\n> If the directory is gone, then their performance returns to as they\n> expect, since the next Git command will not need to expand the index\n> to detect the difference between tracked, untracked, and ignored files\n> within the sparse directory.\n>\n> (Recall that the index is stored as sparse even if it expands to a\n> full one in-memory during execution.)\n\nRight, as with the other email, this was the piece that hadn't quite\nstuck or registered for me. Thanks for clarifying.\n"},{"id":"434158","messageId":"CABPp-BFUnxp3sCOQWRa8C_gda5B19duFiOvHKCRCppS2Git_jg@mail.gmail.com","threadId":"56179","inReplyTo":"d55363df-c2e9-3847-d98c-3d07e22d8a5d@gmail.com","subject":"Re: [PATCH v4 00/10] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-08-30T20:23:44Z","receivedAt":"2021-08-30T20:23:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 30, 2021 at 6:54 AM Derrick Stolee wrote:\n>\n> On 8/27/2021 5:56 PM, Elijah Newren wrote:\n> > On Tue, Aug 24, 2021 at 2:51 PM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n...\n> >> 8: febef675f05 ! 9: c19d93ec5d7 sparse-checkout: clear tracked sparse dirs\n> >> @@ Commit message\n> >> the sparse directory. This depends on the indexed version of the file,\n> >> so the sparse directory must be expanded.\n> >>\n> >> + We must take special care to look for untracked, non-ignored files in\n> >> + these directories before deleting them. We do not want to delete any\n> >> + meaningful work that the users were doing in those directories and\n> >> + perhaps forgot to add and commit before switching sparse-checkout\n> >> + definitions. Since those untracked files might be code files that\n> >> + generated ignored build output, also do not delete any ignored files\n> >> + from these directories in that case. The users can recover their state\n> >> + by resetting their sparse-checkout definition to include that directory\n> >> + and continue. Alternatively, they can see the warning that is presented\n> >> + and delete the directory themselves to regain the performance they\n> >> + expect.\n> >> +\n> ...\n> >> ++When changing the sparse-checkout patterns in cone mode, Git will inspect each\n> >> ++tracked directory that is not within the sparse-checkout cone to see if it\n> >> ++contains any untracked files. If all of those files are ignored due to the\n> >> ++`.gitignore` patterns, then the directory will be deleted. If any of the\n> >> ++untracked files within that directory is not ignored, then no deletions will\n> >> ++occur within that directory and a warning message will appear. If these files\n> >> ++are important, then reset your sparse-checkout definition so they are included,\n> >> ++use `git add` and `git commit` to store them, then remove any remaining files\n> >> ++manually to ensure Git can behave optimally.\n> ...\n> > Thanks for the extra explanations; those help.\n> >\n> > You haven't yet addressed how this choice will interact with\n> > .gitignore files, though (as I commented on v3).\n...\n> The difference between the .gitattributes and .gitignore files is that\n> we don't need to do anything with the .gitignore files to prevent Git\n> from accessing them with their contained directories don't exist. The\n> .gitattributes change in attr.c is reachable from some code path even\n> when the directory doesn't exist, so it needs special logic to avoid\n> that expansion.\n\nAh, this is what I was looking for and couldn't quite work out --\nwhether .gitignore files might need to be loaded (and whether our\nother UI decisions were affecting that). Sounds like you have it\ncovered.\n\nThanks for patiently explaining these various items I was not\nunderstanding or missing. Sorry I missed a couple key things despite\nreading the earlier rounds. Anyway, looks like we're in pretty good\nshape and there's only a few minor things left, mostly stuff you\nbrought up yourself that you wanted to tweak for v5.\n"},{"id":"434990","messageId":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v4.git.1629841904.gitgitgadget@gmail.com","subject":"[PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:24Z","receivedAt":"2021-09-08T01:42:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"We launched an experimental release [1] of the sparse-index feature to our\ninternal users. We immediately discovered a problem due to the isolated way\nin which we tested the sparse index: we were never building the project and\nchanging our sparse-checkout definition.\n\n[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n\nUsers who ran a build in one project and then moved to another still had\nbuild artifacts in their worktree that lived inside the old directories.\nSince the files are marked by the .gitignore patterns, these files were not\nremoved by the 'git sparse-checkout set' command. However, they make the\nsparse-index unusable because every 'git status' command needs to expand the\nsparse-directory entries in order to see if the files are tracked or not.\nThis made the first experimental release actually slower for all users\nbecause of this cost.\n\nThe solution we shipped to these customers was to change the way our fork\nhandles these ignored files. Specifically, instead of Git completely\nignoring the files, we changed Git to understand that with cone-mode\nsparse-checkout patterns, the users is asking for entire directories to be\nremoved from the worktree. The link [1] included earlier has this change.\n\nI believe that this is a reasonable expectation, though I recognize that it\nmight look like breaking the expectations of how .gitignore files work.\n\nSince feedback demonstrated that this is a desired behavior, v2 includes\nthis behavior for all \"cone mode\" repositories.\n\nI'm interested in the community's thoughts about this change, as it seems\nlike one that we should make carefully and intentionally.\n\nWhile the rewrite of the t7519 test seems unrelated, it is required to avoid\na test failure with this change that deletes files outside of the cone. By\nmoving the test into repositories not at $TRASH_DIRECTORY, we gain more\ncontrol over the repository structure.\n\n\nUpdates in V5\n=============\n\n * Updated the locality of a cache_entry pointer.\n\n * Rephrased a comment.\n\n * Removed the patch adding a config option.\n\n * I tried, but failed, to create a scenario where the call to\n cache_tree_update() causes a test to fail. I still think this is valuable\n as defensive programming for the reasons mentioned in the patch, which is\n why I didn't remove them here.\n\n\nUpdates in V4\n=============\n\n * Fixed an issue with the split index.\n\n * The helper methods are used more consistently.\n\n * The helper method path_in_cone_mode_sparse_checkout() is introduced.\n\n * Commit messages are edited for clarity.\n\n * A new config option is added to disable the behavior being added in this\n series.\n\n * I split the commit that involves cache_tree_update(). I have not yet\n succeeded in creating tests to demonstrate why this is required outside\n of needing it in the Scalar functional tests, which includes a version of\n partial clone. I will continue to investigate recreating this scenario in\n the Git test suite, but I wanted to send this version out to get feedback\n on the things that have changed.\n\n\nUpdate in V3\n============\n\n * As promised [2], the helper methods are fixed to work with non-cone-mode\n patterns. A later series will use them to their fullest potential\n (changing git add, git rm, and git mv when interacting with sparse\n entries).\n\n[2]\nhttps://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/\n\n\nUpdates in V2\n=============\n\n * This version correctly leaves untracked files alone. If untracked files\n are found, then the directory is left as-is, in case those ignored files\n are important to the user's work resolving those untracked files.\n\n * This behavior is now enabled by core.sparseCheckoutCone=true.\n\n * To use a sparse index as an in-memory data structure even when\n index.sparse is disabled, a new patch is included to modify the prototype\n of convert_to_sparse() to include a flags parameter.\n\n * A few cleanup patches that I was collecting based on feedback from the\n experimental release and intending for my next series were necessary for\n this implementation.\n\n * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n implementation that used run_subcommand().\n\nThanks, -Stolee\n\nDerrick Stolee (9):\n t7519: rewrite sparse index test\n sparse-index: silently return when not using cone-mode patterns\n unpack-trees: fix nested sparse-dir search\n sparse-index: silently return when cache tree fails\n sparse-index: use WRITE_TREE_MISSING_OK\n sparse-checkout: create helper methods\n attr: be careful about sparse directories\n sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n sparse-checkout: clear tracked sparse dirs\n\n Documentation/git-sparse-checkout.txt | 10 +++\n attr.c | 15 +++++\n builtin/add.c | 7 +-\n builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++\n dir.c | 52 +++++++++++++++\n dir.h | 8 +++\n read-cache.c | 4 +-\n sparse-index.c | 76 ++++++++++++----------\n sparse-index.h | 3 +-\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n t/t7519-status-fsmonitor.sh | 38 ++++++-----\n unpack-trees.c | 8 ++-\n 12 files changed, 312 insertions(+), 62 deletions(-)\n\n\nbase-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1009\n\nRange-diff vs v4:\n\n 1: c407b2cb346 = 1: c407b2cb346 t7519: rewrite sparse index test\n 2: 8660877ba7a = 2: 8660877ba7a sparse-index: silently return when not using cone-mode patterns\n 5: acdded0f762 ! 3: edb00d3f9aa unpack-trees: fix nested sparse-dir search\n @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,\n {\n -\tstruct cache_entry *ce;\n +\tconst char *path;\n -+\tstruct cache_entry *ce = NULL;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n + \t * paths (e.g. \"subdir-\").\n + \t */\n \twhile (pos >= 0) {\n - \t\tce = o->src_index->cache[pos];\n +-\t\tce = o->src_index->cache[pos];\n ++\t\tstruct cache_entry *ce = o->src_index->cache[pos];\n \n -\t\tif (strncmp(ce->name, p->path, p->pathlen))\n +\t\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n 3: a669740af9a = 4: c8620de61eb sparse-index: silently return when cache tree fails\n 4: b379b8fc61a = 5: 37171612424 sparse-index: use WRITE_TREE_MISSING_OK\n 6: 1958751aa0e = 6: 98b4cae2297 sparse-checkout: create helper methods\n 7: e496f3cee66 ! 7: 6ec3cb2042e attr: be careful about sparse directories\n @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat\n +\t * .gitattributes file since it will not matter.\n +\t *\n +\t * In the case of a sparse index, it is critical that we don't go\n -+\t * looking for a .gitattributes file, as the index will expand.\n ++\t * looking for a .gitattributes file, as doing so would cause the\n ++\t * index to expand.\n +\t */\n +\tif (!path_in_cone_mode_sparse_checkout(path, istate))\n +\t\treturn NULL;\n 8: cab9360b1e9 = 8: d57f48c445c sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n 9: c19d93ec5d7 = 9: 91b53f20109 sparse-checkout: clear tracked sparse dirs\n 10: 8d55a6ba2fd < -: ----------- sparse-checkout: add config to disable deleting dirs\n\n-- \ngitgitgadget\n"},{"id":"434991","messageId":"c407b2cb346b32cdf95c6017d16b93dd1483c627.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 1/9] t7519: rewrite sparse index test","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:25Z","receivedAt":"2021-09-08T01:42:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse index is tested with the FS Monitor hook and extension since\nf8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test\nwas very fragile because it shared an index across sparse and non-sparse\nbehavior. Since that expansion and contraction could cause the index to\nlose its FS Monitor bitmap and token, behavior is fragile to changes in\n'git sparse-checkout set'.\n\nRewrite the test to use two clones of the original repo: full and\nsparse. This allows us to also keep the test files (actual, expect,\ntrace2.txt) out of the repos we are testing with 'git status'.\n\nSigned-off-by: Derrick Stolee \n---\n t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++-----------------\n 1 file changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex deea88d4431..f1463197b99 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' '\n # If \"!\" is supplied, then we verify that we do not call ensure_full_index\n # during a call to 'git status'. Otherwise, we verify that we _do_ call it.\n check_sparse_index_behavior () {\n-\tgit status --porcelain=v2 >expect &&\n-\tgit sparse-checkout init --cone --sparse-index &&\n-\tgit sparse-checkout set dir1 dir2 &&\n+\tgit -C full status --porcelain=v2 >expect &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n-\t\tgit status --porcelain=v2 >actual &&\n+\t\tgit -C sparse status --porcelain=v2 >actual &&\n \ttest_region $1 index ensure_full_index trace2.txt &&\n \ttest_region fsm_hook query trace2.txt &&\n \ttest_cmp expect actual &&\n-\trm trace2.txt &&\n-\tgit sparse-checkout disable\n+\trm trace2.txt\n }\n \n test_expect_success 'status succeeds with sparse index' '\n-\tgit reset --hard &&\n+\tgit clone . full &&\n+\tgit clone --sparse . sparse &&\n+\tgit -C sparse sparse-checkout init --cone --sparse-index &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n-\ttest_config core.fsmonitor \"$TEST_DIRECTORY/t7519/fsmonitor-all\" &&\n-\tcheck_sparse_index_behavior ! &&\n-\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \tEOF\n-\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\tgit -C full config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n+\tgit -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test &&\n \tcheck_sparse_index_behavior ! &&\n \n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1/modified\\0\"\n \tEOF\n \tcheck_sparse_index_behavior ! &&\n \n-\tcp -r dir1 dir1a &&\n-\tgit add dir1a &&\n-\tgit commit -m \"add dir1a\" &&\n+\tgit -C sparse sparse-checkout add dir1a &&\n+\n+\tfor repo in full sparse\n+\tdo\n+\t\tcp -r $repo/dir1 $repo/dir1a &&\n+\t\tgit -C $repo add dir1a &&\n+\t\tgit -C $repo commit -m \"add dir1a\" || return 1\n+\tdone &&\n+\tgit -C sparse sparse-checkout set dir1 dir2 &&\n \n \t# This one modifies outside the sparse-checkout definition\n \t# and hence we expect to expand the sparse-index.\n-\twrite_script .git/hooks/fsmonitor-test<<-\\EOF &&\n+\twrite_script .git/hooks/fsmonitor-test <<-\\EOF &&\n \t\tprintf \"last_update_token\\0\"\n \t\tprintf \"dir1a/modified\\0\"\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"434992","messageId":"8660877ba7a06971bb5c443bbc1e07825e950a60.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 2/9] sparse-index: silently return when not using cone-mode patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:26Z","receivedAt":"2021-09-08T01:42:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhile the sparse-index is only enabled when core.sparseCheckoutCone is\nalso enabled, it is possible for the user to modify the sparse-checkout\nfile manually in a way that does not match cone-mode patterns. In this\ncase, we should refuse to convert an index into a sparse index, since\nthe sparse_checkout_patterns will not be initialized with recursive and\nparent path hashsets.\n\nAlso silently return if there are no cache entries, which is a simple\ncase: there are no paths to make sparse!\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 16 +++++++++++-----\n 1 file changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex c6b4feec413..cd6e0d5f408 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate)\n int convert_to_sparse(struct index_state *istate)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index ||\n+\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n@@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate)\n \t\t\treturn 0;\n \t}\n \n-\tif (!istate->sparse_checkout_patterns->use_cone_patterns) {\n-\t\twarning(_(\"attempting to use sparse-index without cone mode\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * We need cone-mode patterns to use sparse-index. If a user edits\n+\t * their sparse-checkout file manually, then we can detect during\n+\t * parsing that they are not actually using cone-mode patterns and\n+\t * hence we need to abort this conversion _without error_. Warnings\n+\t * already exist in the pattern parsing to inform the user of their\n+\t * bad patterns.\n+\t */\n+\tif (!istate->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn 0;\n \n \t/*\n \t * NEEDSWORK: If we have unmerged entries, then stay full.\n-- \ngitgitgadget\n\n"},{"id":"434993","messageId":"edb00d3f9aa68f5021c50a72c8be4bc54dd64ea2.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 3/9] unpack-trees: fix nested sparse-dir search","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:27Z","receivedAt":"2021-09-08T01:42:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe iterated search in find_cache_entry() was recently modified to\ninclude a loop that searches backwards for a sparse directory entry that\nmatches the given traverse_info and name_entry. However, the string\ncomparison failed to actually concatenate those two strings, so this\nfailed to find a sparse directory when it was not a top-level directory.\n\nThis caused some errors in rare cases where a 'git checkout' spanned a\ndiff that modified files within the sparse directory entry, but we could\nnot correctly find the entry.\n\nHelped-by: Johannes Schindelin \nHelped-by: René Scharfe \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 8 +++++---\n 1 file changed, 5 insertions(+), 3 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5786645f315..b78d7ee6c09 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1255,7 +1255,7 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t\t\t\t\t const struct name_entry *p)\n {\n-\tstruct cache_entry *ce;\n+\tconst char *path;\n \tint pos = find_cache_pos(info, p->path, p->pathlen);\n \tstruct unpack_trees_options *o = info->data;\n \n@@ -1281,9 +1281,11 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info,\n \t * paths (e.g. \"subdir-\").\n \t */\n \twhile (pos >= 0) {\n-\t\tce = o->src_index->cache[pos];\n+\t\tstruct cache_entry *ce = o->src_index->cache[pos];\n \n-\t\tif (strncmp(ce->name, p->path, p->pathlen))\n+\t\tif (!skip_prefix(ce->name, info->traverse_path, &path) ||\n+\t\t strncmp(path, p->path, p->pathlen) ||\n+\t\t path[p->pathlen] != '/')\n \t\t\treturn NULL;\n \n \t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n-- \ngitgitgadget\n\n"},{"id":"434994","messageId":"371716124247cf89069c1344e77c08853a1f76fb.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 5/9] sparse-index: use WRITE_TREE_MISSING_OK","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:29Z","receivedAt":"2021-09-08T01:42:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen updating the cache tree in convert_to_sparse(), the\nWRITE_TREE_MISSING_OK flag indicates that trees might be computed that\ndo not already exist within the object database. This happens in cases\nsuch as 'git add' creating new trees that it wants to store in\nanticipation of a following 'git commit'. If this flag is not specified,\nthen it might trigger a promisor fetch or a failure due to the object\nnot existing locally.\n\nUse WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these\npossible reasons for the cache_tree_update() to fail.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex d9b07695953..880c5f72338 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate)\n \t/*\n \t * Silently return if there is a problem with the cache tree update,\n \t * which might just be due to a conflict state in some entry.\n+\t *\n+\t * This might create new tree objects, so be sure to use\n+\t * WRITE_TREE_MISSING_OK.\n \t */\n-\tif (cache_tree_update(istate, 0))\n+\tif (cache_tree_update(istate, WRITE_TREE_MISSING_OK))\n \t\treturn 0;\n \n \tremove_fsmonitor(istate);\n-- \ngitgitgadget\n\n"},{"id":"434995","messageId":"c8620de61eb459c48919fbd4d122bc82cf8ed50a.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 4/9] sparse-index: silently return when cache tree fails","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:28Z","receivedAt":"2021-09-08T01:42:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf cache_tree_update() returns a non-zero value, then it could not\ncreate the cache tree. This is likely due to a path having a merge\nconflict. Since we are already returning early, let's return silently to\navoid making it seem like we failed to write the index at all.\n\nIf we remove our dependence on the cache tree within\nconvert_to_sparse(), then we could still recover from this scenario and\nhave a sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex cd6e0d5f408..d9b07695953 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate)\n \n \t/* Clear and recompute the cache-tree */\n \tcache_tree_free(&istate->cache_tree);\n-\tif (cache_tree_update(istate, 0)) {\n-\t\twarning(_(\"unable to update cache-tree, staying full\"));\n-\t\treturn -1;\n-\t}\n+\t/*\n+\t * Silently return if there is a problem with the cache tree update,\n+\t * which might just be due to a conflict state in some entry.\n+\t */\n+\tif (cache_tree_update(istate, 0))\n+\t\treturn 0;\n \n \tremove_fsmonitor(istate);\n \n-- \ngitgitgadget\n\n"},{"id":"434996","messageId":"98b4cae2297cc93abe83a573360859c16806a6fd.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 6/9] sparse-checkout: create helper methods","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:30Z","receivedAt":"2021-09-08T01:42:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we integrate the sparse index into more builtins, we occasionally\nneed to check the sparse-checkout patterns to see if a path is within\nthe sparse-checkout cone. Create some helper methods that help\ninitialize the patterns and check for pattern matching to make this\neasier.\n\nThe existing callers of commands like get_sparse_checkout_patterns() use\na custom 'struct pattern_list' that is not necessarily the one in the\n'struct index_state', so there are not many previous uses that could\nadopt these helpers. There are just two in builtin/add.c and\nsparse-index.c that can use path_in_sparse_checkout().\n\nWe add a path_in_cone_mode_sparse_checkout() as well that will only\nreturn false if the path is outside of the sparse-checkout definition\n_and_ the sparse-checkout patterns are in cone mode.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 7 +------\n dir.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++\n dir.h | 8 ++++++++\n sparse-index.c | 14 +++-----------\n 4 files changed, 64 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 17528e8f922..88a6c0c69fb 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \tint flags = REFRESH_IGNORE_SKIP_WORKTREE |\n \t\t (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET);\n-\tstruct pattern_list pl = { 0 };\n-\tint sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl);\n \n \tseen = xcalloc(pathspec->nr, 1);\n \trefresh_index(&the_index, flags, pathspec, seen,\n@@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec)\n \tfor (i = 0; i < pathspec->nr; i++) {\n \t\tif (!seen[i]) {\n \t\t\tconst char *path = pathspec->items[i].original;\n-\t\t\tint dtype = DT_REG;\n \n \t\t\tif (matches_skip_worktree(pathspec, i, &skip_worktree_seen) ||\n-\t\t\t (sparse_checkout_enabled &&\n-\t\t\t !path_matches_pattern_list(path, strlen(path), NULL,\n-\t\t\t\t\t\t\t&dtype, &pl, &the_index))) {\n+\t\t\t !path_in_sparse_checkout(path, &the_index)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec->items[i].original);\n \t\t\t} else {\ndiff --git a/dir.c b/dir.c\nindex 03c4d212672..86afa2eae00 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1439,6 +1439,58 @@ done:\n \treturn result;\n }\n \n+int init_sparse_checkout_patterns(struct index_state *istate)\n+{\n+\tif (!core_apply_sparse_checkout)\n+\t\treturn 1;\n+\tif (istate->sparse_checkout_patterns)\n+\t\treturn 0;\n+\n+\tCALLOC_ARRAY(istate->sparse_checkout_patterns, 1);\n+\n+\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) {\n+\t\tFREE_AND_NULL(istate->sparse_checkout_patterns);\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int path_in_sparse_checkout_1(const char *path,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t int require_cone_mode)\n+{\n+\tconst char *base;\n+\tint dtype = DT_REG;\n+\n+\t/*\n+\t * We default to accepting a path if there are no patterns or\n+\t * they are of the wrong type.\n+\t */\n+\tif (init_sparse_checkout_patterns(istate) ||\n+\t (require_cone_mode &&\n+\t !istate->sparse_checkout_patterns->use_cone_patterns))\n+\t\treturn 1;\n+\n+\tbase = strrchr(path, '/');\n+\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n+\t\t\t\t\t &dtype,\n+\t\t\t\t\t istate->sparse_checkout_patterns,\n+\t\t\t\t\t istate) > 0;\n+}\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate)\n+{\n+\treturn path_in_sparse_checkout_1(path, istate, 0);\n+}\n+\n+int path_in_cone_mode_sparse_checkout(const char *path,\n+\t\t\t\t struct index_state *istate)\n+{\n+\treturn path_in_sparse_checkout_1(path, istate, 1);\n+}\n+\n static struct path_pattern *last_matching_pattern_from_lists(\n \t\tstruct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int pathlen,\ndiff --git a/dir.h b/dir.h\nindex b3e1a54a971..6823312521e 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname,\n \t\t\t\tconst char *basename, int *dtype,\n \t\t\t\tstruct pattern_list *pl,\n \t\t\t\tstruct index_state *istate);\n+\n+int init_sparse_checkout_patterns(struct index_state *state);\n+\n+int path_in_sparse_checkout(const char *path,\n+\t\t\t struct index_state *istate);\n+int path_in_cone_mode_sparse_checkout(const char *path,\n+\t\t\t\t struct index_state *istate);\n+\n struct dir_entry *dir_add_ignored(struct dir_struct *dir,\n \t\t\t\t struct index_state *istate,\n \t\t\t\t const char *pathname, int len);\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 880c5f72338..23f7c3bd361 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate,\n {\n \tint i, can_convert = 1;\n \tint start_converted = num_converted;\n-\tenum pattern_match_result match;\n-\tint dtype = DT_UNKNOWN;\n \tstruct strbuf child_path = STRBUF_INIT;\n-\tstruct pattern_list *pl = istate->sparse_checkout_patterns;\n \n \t/*\n \t * Is the current path outside of the sparse cone?\n \t * Then check if the region can be replaced by a sparse\n \t * directory entry (everything is sparse and merged).\n \t */\n-\tmatch = path_matches_pattern_list(ct_path, ct_pathlen,\n-\t\t\t\t\t NULL, &dtype, pl, istate);\n-\tif (match != NOT_MATCHED)\n+\tif (path_in_sparse_checkout(ct_path, istate))\n \t\tcan_convert = 0;\n \n \tfor (i = start; can_convert && i < end; i++) {\n@@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate)\n \tif (!istate->repo->settings.sparse_index)\n \t\treturn 0;\n \n-\tif (!istate->sparse_checkout_patterns) {\n-\t\tistate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list));\n-\t\tif (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0)\n-\t\t\treturn 0;\n-\t}\n+\tif (init_sparse_checkout_patterns(istate))\n+\t\treturn 0;\n \n \t/*\n \t * We need cone-mode patterns to use sparse-index. If a user edits\n-- \ngitgitgadget\n\n"},{"id":"434997","messageId":"6ec3cb2042e1064387dba410c9494a6dc8be0e54.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 7/9] attr: be careful about sparse directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:31Z","receivedAt":"2021-09-08T01:42:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/attr.c b/attr.c\nindex d029e681f28..79adaa50ea1 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -14,6 +14,7 @@\n #include \"utf8.h\"\n #include \"quote.h\"\n #include \"thread-utils.h\"\n+#include \"dir.h\"\n \n const char git_attr__true[] = \"(builtin)true\";\n const char git_attr__false[] = \"\\0(builtin)false\";\n@@ -744,6 +745,20 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \tif (!istate)\n \t\treturn NULL;\n \n+\t/*\n+\t * The .gitattributes file only applies to files within its\n+\t * parent directory. In the case of cone-mode sparse-checkout,\n+\t * the .gitattributes file is sparse if and only if all paths\n+\t * within that directory are also sparse. Thus, don't load the\n+\t * .gitattributes file since it will not matter.\n+\t *\n+\t * In the case of a sparse index, it is critical that we don't go\n+\t * looking for a .gitattributes file, as doing so would cause the\n+\t * index to expand.\n+\t */\n+\tif (!path_in_cone_mode_sparse_checkout(path, istate))\n+\t\treturn NULL;\n+\n \tbuf = read_blob_data_from_index(istate, path, NULL);\n \tif (!buf)\n \t\treturn NULL;\n-- \ngitgitgadget\n\n"},{"id":"434998","messageId":"d57f48c445cc148a7070fca799a3e57aa3dace76.1631065353.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 8/9] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:32Z","receivedAt":"2021-09-08T01:42:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX\nenvironment variable or the \"index.sparse\" config setting before\nconverting the index to a sparse one. This is for ease of use since all\ncurrent consumers are preparing to compress the index before writing it\nto disk. If these settings are not enabled, then convert_to_sparse()\nsilently returns without doing anything.\n\nWe will add a consumer in the next change that wants to use the sparse\nindex as an in-memory data structure, regardless of whether the on-disk\nformat should be sparse.\n\nTo that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip\nthese config checks when enabled. All current consumers are modified to\npass '0' in the new 'flags' parameter.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++--\n sparse-index.c | 37 ++++++++++++++++++++++---------------\n sparse-index.h | 3 ++-\n 3 files changed, 26 insertions(+), 18 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9048ef9e905..f5d4385c408 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l\n \tint ret;\n \tint was_full = !istate->sparse_index;\n \n-\tret = convert_to_sparse(istate);\n+\tret = convert_to_sparse(istate, 0);\n \n \tif (ret) {\n \t\twarning(_(\"failed to convert to a sparse-index\"));\n@@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate,\n \tint ret, was_full = !istate->sparse_index;\n \n \tmove_cache_to_base_index(istate);\n-\tconvert_to_sparse(istate);\n+\tconvert_to_sparse(istate, 0);\n \n \ttrace2_region_enter_printf(\"index\", \"shared/do_write_index\",\n \t\t\t\t the_repository, \"%s\", get_tempfile_path(*temp));\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 23f7c3bd361..0bc45f60ac5 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate)\n \treturn 0;\n }\n \n-int convert_to_sparse(struct index_state *istate)\n+int convert_to_sparse(struct index_state *istate, int flags)\n {\n \tint test_env;\n-\tif (istate->split_index || istate->sparse_index || !istate->cache_nr ||\n+\tif (istate->sparse_index || !istate->cache_nr ||\n \t !core_apply_sparse_checkout || !core_sparse_checkout_cone)\n \t\treturn 0;\n \n \tif (!istate->repo)\n \t\tistate->repo = the_repository;\n \n-\t/*\n-\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n-\t * index.sparse config variable to be on.\n-\t */\n-\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n-\tif (test_env >= 0)\n-\t\tset_sparse_index_config(istate->repo, test_env);\n+\tif (!(flags & SPARSE_INDEX_MEMORY_ONLY)) {\n+\t\t/*\n+\t\t * The sparse index is not (yet) integrated with a split index.\n+\t\t */\n+\t\tif (istate->split_index)\n+\t\t\treturn 0;\n+\t\t/*\n+\t\t * The GIT_TEST_SPARSE_INDEX environment variable triggers the\n+\t\t * index.sparse config variable to be on.\n+\t\t */\n+\t\ttest_env = git_env_bool(\"GIT_TEST_SPARSE_INDEX\", -1);\n+\t\tif (test_env >= 0)\n+\t\t\tset_sparse_index_config(istate->repo, test_env);\n \n-\t/*\n-\t * Only convert to sparse if index.sparse is set.\n-\t */\n-\tprepare_repo_settings(istate->repo);\n-\tif (!istate->repo->settings.sparse_index)\n-\t\treturn 0;\n+\t\t/*\n+\t\t * Only convert to sparse if index.sparse is set.\n+\t\t */\n+\t\tprepare_repo_settings(istate->repo);\n+\t\tif (!istate->repo->settings.sparse_index)\n+\t\t\treturn 0;\n+\t}\n \n \tif (init_sparse_checkout_patterns(istate))\n \t\treturn 0;\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 1115a0d7dd9..9f3d7bc7faf 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,8 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-int convert_to_sparse(struct index_state *istate);\n+#define SPARSE_INDEX_MEMORY_ONLY (1 << 0)\n+int convert_to_sparse(struct index_state *istate, int flags);\n \n /*\n * Some places in the codebase expect to search for a specific path.\n-- \ngitgitgadget\n\n"},{"id":"434999","messageId":"91b53f20109fe55635b1815f87afd5d5da68a182.1631065354.git.gitgitgadget@gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"[PATCH v5 9/9] sparse-checkout: clear tracked sparse dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-08T01:42:33Z","receivedAt":"2021-09-08T01:42:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen changing the scope of a sparse-checkout using cone mode, we might\nhave some tracked directories go out of scope. The current logic removes\nthe tracked files from within those directories, but leaves the ignored\nfiles within those directories. This is a bit unexpected to users who\nhave given input to Git saying they don't need those directories\nanymore.\n\nThis is something that is new to the cone mode pattern type: the user\nhas explicitly said \"I want these directories and _not_ those\ndirectories.\" The typical sparse-checkout patterns more generally apply\nto \"I want files with with these patterns\" so it is natural to leave\nignored files as they are. This focus on directories in cone mode\nprovides us an opportunity to change the behavior.\n\nLeaving these ignored files in the sparse directories makes it\nimpossible to gain performance benefits in the sparse index. When we\ntrack into these directories, we need to know if the files are ignored\nor not, which might depend on the _tracked_ .gitignore file(s) within\nthe sparse directory. This depends on the indexed version of the file,\nso the sparse directory must be expanded.\n\nWe must take special care to look for untracked, non-ignored files in\nthese directories before deleting them. We do not want to delete any\nmeaningful work that the users were doing in those directories and\nperhaps forgot to add and commit before switching sparse-checkout\ndefinitions. Since those untracked files might be code files that\ngenerated ignored build output, also do not delete any ignored files\nfrom these directories in that case. The users can recover their state\nby resetting their sparse-checkout definition to include that directory\nand continue. Alternatively, they can see the warning that is presented\nand delete the directory themselves to regain the performance they\nexpect.\n\nBy deleting the sparse directories when changing scope (or running 'git\nsparse-checkout reapply') we regain these performance benefits as if the\nrepository was in a clean state.\n\nSince these ignored files are frequently build output or helper files\nfrom IDEs, the users should not need the files now that the tracked\nfiles are removed. If the tracked files reappear, then they will have\nnewer timestamps than the build artifacts, so the artifacts will need to\nbe regenerated anyway.\n\nUse the sparse-index as a data structure in order to find the sparse\ndirectories that can be safely deleted. Re-expand the index to a full\none if it was full before.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 10 +++\n builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n 3 files changed, 163 insertions(+)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex fdcf43f87cb..42056ee9ff9 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the\n 'git sparse-checkout set' command to reflect the expected cone in the working\n directory.\n \n+When changing the sparse-checkout patterns in cone mode, Git will inspect each\n+tracked directory that is not within the sparse-checkout cone to see if it\n+contains any untracked files. If all of those files are ignored due to the\n+`.gitignore` patterns, then the directory will be deleted. If any of the\n+untracked files within that directory is not ignored, then no deletions will\n+occur within that directory and a warning message will appear. If these files\n+are important, then reset your sparse-checkout definition so they are included,\n+use `git add` and `git commit` to store them, then remove any remaining files\n+manually to ensure Git can behave optimally.\n+\n \n SUBMODULES\n ----------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 8ba9f13787b..d0f5c4702be 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void clean_tracked_sparse_directories(struct repository *r)\n+{\n+\tint i, was_full = 0;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t pathlen;\n+\tstruct string_list_item *item;\n+\tstruct string_list sparse_dirs = STRING_LIST_INIT_DUP;\n+\n+\t/*\n+\t * If we are not using cone mode patterns, then we cannot\n+\t * delete directories outside of the sparse cone.\n+\t */\n+\tif (!r || !r->index || !r->worktree)\n+\t\treturn;\n+\tif (init_sparse_checkout_patterns(r->index) ||\n+\t !r->index->sparse_checkout_patterns->use_cone_patterns)\n+\t\treturn;\n+\n+\t/*\n+\t * Use the sparse index as a data structure to assist finding\n+\t * directories that are safe to delete. This conversion to a\n+\t * sparse index will not delete directories that contain\n+\t * conflicted entries or submodules.\n+\t */\n+\tif (!r->index->sparse_index) {\n+\t\t/*\n+\t\t * If something, such as a merge conflict or other concern,\n+\t\t * prevents us from converting to a sparse index, then do\n+\t\t * not try deleting files.\n+\t\t */\n+\t\tif (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY))\n+\t\t\treturn;\n+\t\twas_full = 1;\n+\t}\n+\n+\tstrbuf_addstr(&path, r->worktree);\n+\tstrbuf_complete(&path, '/');\n+\tpathlen = path.len;\n+\n+\t/*\n+\t * Collect directories that have gone out of scope but also\n+\t * exist on disk, so there is some work to be done. We need to\n+\t * store the entries in a list before exploring, since that might\n+\t * expand the sparse-index again.\n+\t */\n+\tfor (i = 0; i < r->index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = r->index->cache[i];\n+\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t repo_file_exists(r, ce->name))\n+\t\t\tstring_list_append(&sparse_dirs, ce->name);\n+\t}\n+\n+\tfor_each_string_list_item(item, &sparse_dirs) {\n+\t\tstruct dir_struct dir = DIR_INIT;\n+\t\tstruct pathspec p = { 0 };\n+\t\tstruct strvec s = STRVEC_INIT;\n+\n+\t\tstrbuf_setlen(&path, pathlen);\n+\t\tstrbuf_addstr(&path, item->string);\n+\n+\t\tdir.flags |= DIR_SHOW_IGNORED_TOO;\n+\n+\t\tsetup_standard_excludes(&dir);\n+\t\tstrvec_push(&s, path.buf);\n+\n+\t\tparse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v);\n+\t\tfill_directory(&dir, r->index, &p);\n+\n+\t\tif (dir.nr) {\n+\t\t\twarning(_(\"directory '%s' contains untracked files,\"\n+\t\t\t\t \" but is not in the sparse-checkout cone\"),\n+\t\t\t\titem->string);\n+\t\t} else if (remove_dir_recursively(&path, 0)) {\n+\t\t\t/*\n+\t\t\t * Removal is \"best effort\". If something blocks\n+\t\t\t * the deletion, then continue with a warning.\n+\t\t\t */\n+\t\t\twarning(_(\"failed to remove directory '%s'\"),\n+\t\t\t\titem->string);\n+\t\t}\n+\n+\t\tdir_clear(&dir);\n+\t}\n+\n+\tstring_list_clear(&sparse_dirs, 0);\n+\tstrbuf_release(&path);\n+\n+\tif (was_full)\n+\t\tensure_full_index(r->index);\n+}\n+\n static int update_working_directory(struct pattern_list *pl)\n {\n \tenum update_sparsity_result result;\n@@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl)\n \telse\n \t\trollback_lock_file(&lock_file);\n \n+\tclean_tracked_sparse_directories(r);\n+\n \tr->index->sparse_checkout_patterns = NULL;\n \treturn result;\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 38fc8340f5c..71236981e64 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' '\n \tcheck_files repo/deep a deeper1\n '\n \n+test_expect_success 'cone mode clears ignored subdirectories' '\n+\trm repo/.git/info/sparse-checkout &&\n+\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set deep/deeper1 &&\n+\n+\tcat >repo/.gitignore <<-\\EOF &&\n+\tobj/\n+\t*.o\n+\tEOF\n+\n+\tgit -C repo add .gitignore &&\n+\tgit -C repo commit -m \".gitignore\" &&\n+\n+\tmkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj &&\n+\tfor file in folder1/obj/a obj/a folder1/file.o folder1.o \\\n+\t\t deep/deeper2/obj/a deep/deeper2/file.o file.o\n+\tdo\n+\t\techo ignored >repo/$file || return 1\n+\tdone &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout reapply &&\n+\ttest_path_is_missing repo/folder1 &&\n+\ttest_path_is_missing repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\ttest_must_be_empty out &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper2 &&\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/deep/deeper2 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\t>repo/deep/deeper2/ignored.o &&\n+\t>repo/deep/deeper2/untracked &&\n+\n+\t# When an untracked file is in the way, all untracked files\n+\t# (even ignored files) are preserved.\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\tgrep \"contains untracked files\" err &&\n+\ttest_path_is_file repo/deep/deeper2/ignored.o &&\n+\ttest_path_is_file repo/deep/deeper2/untracked &&\n+\n+\t# The rest of the cone matches expectation\n+\ttest_path_is_missing repo/deep/deeper1 &&\n+\ttest_path_is_dir repo/obj &&\n+\ttest_path_is_file repo/file.o &&\n+\n+\tgit -C repo status --porcelain=v2 >out &&\n+\techo \"? deep/deeper2/untracked\" >expect &&\n+\ttest_cmp expect out\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"435024","messageId":"xmqqwnnr7icz.fsf@gitster.g","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-08T05:21:16Z","receivedAt":"2021-09-08T05:21:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Updates in V5\n> =============\n>\n> * Updated the locality of a cache_entry pointer.\n>\n> * Rephrased a comment.\n>\n> * Removed the patch adding a config option.\n>\n> * I tried, but failed, to create a scenario where the call to\n> cache_tree_update() causes a test to fail. I still think this is valuable\n> as defensive programming for the reasons mentioned in the patch, which is\n> why I didn't remove them here.\n\nOK. \n\nThe 'sg/test-split-index-fix' topic that has been queued in 'seen'\ntextually depended on the earlier iteration of this topic, so I'll\ndiscard it while replacing this topic.\n\n"},{"id":"435026","messageId":"CABPp-BG+VMSL9hXHGA6HdZTNAiZM3wCQEK36-YFr17YU=T921g@mail.gmail.com","threadId":"56179","inReplyTo":"pull.1009.v5.git.1631065353.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-08T05:30:38Z","receivedAt":"2021-09-08T05:30:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 7, 2021 at 6:42 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> We launched an experimental release [1] of the sparse-index feature to our\n> internal users. We immediately discovered a problem due to the isolated way\n> in which we tested the sparse index: we were never building the project and\n> changing our sparse-checkout definition.\n>\n> [1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp\n>\n> Users who ran a build in one project and then moved to another still had\n> build artifacts in their worktree that lived inside the old directories.\n> Since the files are marked by the .gitignore patterns, these files were not\n> removed by the 'git sparse-checkout set' command. However, they make the\n> sparse-index unusable because every 'git status' command needs to expand the\n> sparse-directory entries in order to see if the files are tracked or not.\n> This made the first experimental release actually slower for all users\n> because of this cost.\n>\n> The solution we shipped to these customers was to change the way our fork\n> handles these ignored files. Specifically, instead of Git completely\n> ignoring the files, we changed Git to understand that with cone-mode\n> sparse-checkout patterns, the users is asking for entire directories to be\n> removed from the worktree. The link [1] included earlier has this change.\n>\n> I believe that this is a reasonable expectation, though I recognize that it\n> might look like breaking the expectations of how .gitignore files work.\n>\n> Since feedback demonstrated that this is a desired behavior, v2 includes\n> this behavior for all \"cone mode\" repositories.\n>\n> I'm interested in the community's thoughts about this change, as it seems\n> like one that we should make carefully and intentionally.\n>\n> While the rewrite of the t7519 test seems unrelated, it is required to avoid\n> a test failure with this change that deletes files outside of the cone. By\n> moving the test into repositories not at $TRASH_DIRECTORY, we gain more\n> control over the repository structure.\n>\n>\n> Updates in V5\n> =============\n>\n> * Updated the locality of a cache_entry pointer.\n>\n> * Rephrased a comment.\n>\n> * Removed the patch adding a config option.\n>\n> * I tried, but failed, to create a scenario where the call to\n> cache_tree_update() causes a test to fail. I still think this is valuable\n> as defensive programming for the reasons mentioned in the patch, which is\n> why I didn't remove them here.\n>\n>\n> Updates in V4\n> =============\n>\n> * Fixed an issue with the split index.\n>\n> * The helper methods are used more consistently.\n>\n> * The helper method path_in_cone_mode_sparse_checkout() is introduced.\n>\n> * Commit messages are edited for clarity.\n>\n> * A new config option is added to disable the behavior being added in this\n> series.\n>\n> * I split the commit that involves cache_tree_update(). I have not yet\n> succeeded in creating tests to demonstrate why this is required outside\n> of needing it in the Scalar functional tests, which includes a version of\n> partial clone. I will continue to investigate recreating this scenario in\n> the Git test suite, but I wanted to send this version out to get feedback\n> on the things that have changed.\n>\n>\n> Update in V3\n> ============\n>\n> * As promised [2], the helper methods are fixed to work with non-cone-mode\n> patterns. A later series will use them to their fullest potential\n> (changing git add, git rm, and git mv when interacting with sparse\n> entries).\n>\n> [2]\n> https://lore.kernel.org/git/bac76c72-955d-1ade-4ecf-778ffc45f297@gmail.com/\n>\n>\n> Updates in V2\n> =============\n>\n> * This version correctly leaves untracked files alone. If untracked files\n> are found, then the directory is left as-is, in case those ignored files\n> are important to the user's work resolving those untracked files.\n>\n> * This behavior is now enabled by core.sparseCheckoutCone=true.\n>\n> * To use a sparse index as an in-memory data structure even when\n> index.sparse is disabled, a new patch is included to modify the prototype\n> of convert_to_sparse() to include a flags parameter.\n>\n> * A few cleanup patches that I was collecting based on feedback from the\n> experimental release and intending for my next series were necessary for\n> this implementation.\n>\n> * Cleaned up the tests (no NEEDSWORK) and the remainders of a previous\n> implementation that used run_subcommand().\n>\n> Thanks, -Stolee\n>\n> Derrick Stolee (9):\n> t7519: rewrite sparse index test\n> sparse-index: silently return when not using cone-mode patterns\n> unpack-trees: fix nested sparse-dir search\n> sparse-index: silently return when cache tree fails\n> sparse-index: use WRITE_TREE_MISSING_OK\n> sparse-checkout: create helper methods\n> attr: be careful about sparse directories\n> sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n> sparse-checkout: clear tracked sparse dirs\n>\n> Documentation/git-sparse-checkout.txt | 10 +++\n> attr.c | 15 +++++\n> builtin/add.c | 7 +-\n> builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++\n> dir.c | 52 +++++++++++++++\n> dir.h | 8 +++\n> read-cache.c | 4 +-\n> sparse-index.c | 76 ++++++++++++----------\n> sparse-index.h | 3 +-\n> t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++\n> t/t7519-status-fsmonitor.sh | 38 ++++++-----\n> unpack-trees.c | 8 ++-\n> 12 files changed, 312 insertions(+), 62 deletions(-)\n>\n>\n> base-commit: 80b8d6c56b8a5f5db1d5c2a0159fd808e8a7fc4f\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1009%2Fderrickstolee%2Fsparse-index%2Fignored-files-v5\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1009/derrickstolee/sparse-index/ignored-files-v5\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1009\n>\n> Range-diff vs v4:\n>\n> 1: c407b2cb346 = 1: c407b2cb346 t7519: rewrite sparse index test\n> 2: 8660877ba7a = 2: 8660877ba7a sparse-index: silently return when not using cone-mode patterns\n> 5: acdded0f762 ! 3: edb00d3f9aa unpack-trees: fix nested sparse-dir search\n> @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,\n> {\n> - struct cache_entry *ce;\n> + const char *path;\n> -+ struct cache_entry *ce = NULL;\n> int pos = find_cache_pos(info, p->path, p->pathlen);\n> struct unpack_trees_options *o = info->data;\n>\n> @@ unpack-trees.c: static struct cache_entry *find_cache_entry(struct traverse_info *info,\n> + * paths (e.g. \"subdir-\").\n> + */\n> while (pos >= 0) {\n> - ce = o->src_index->cache[pos];\n> +- ce = o->src_index->cache[pos];\n> ++ struct cache_entry *ce = o->src_index->cache[pos];\n>\n> - if (strncmp(ce->name, p->path, p->pathlen))\n> + if (!skip_prefix(ce->name, info->traverse_path, &path) ||\n> 3: a669740af9a = 4: c8620de61eb sparse-index: silently return when cache tree fails\n> 4: b379b8fc61a = 5: 37171612424 sparse-index: use WRITE_TREE_MISSING_OK\n> 6: 1958751aa0e = 6: 98b4cae2297 sparse-checkout: create helper methods\n> 7: e496f3cee66 ! 7: 6ec3cb2042e attr: be careful about sparse directories\n> @@ attr.c: static struct attr_stack *read_attr_from_index(struct index_state *istat\n> + * .gitattributes file since it will not matter.\n> + *\n> + * In the case of a sparse index, it is critical that we don't go\n> -+ * looking for a .gitattributes file, as the index will expand.\n> ++ * looking for a .gitattributes file, as doing so would cause the\n> ++ * index to expand.\n> + */\n> + if (!path_in_cone_mode_sparse_checkout(path, istate))\n> + return NULL;\n> 8: cab9360b1e9 = 8: d57f48c445c sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag\n> 9: c19d93ec5d7 = 9: 91b53f20109 sparse-checkout: clear tracked sparse dirs\n> 10: 8d55a6ba2fd < -: ----------- sparse-checkout: add config to disable deleting dirs\n>\n\nThanks for all your diligence working on this, and your patience in\nexplaining all the details to me in the previous rounds. There wasn't\nmuch left to address after the last round, and this round addresses\nthe few loose ends.\n\nReviewed-by: Elijah Newren \n"},{"id":"435031","messageId":"xmqqczpj7dyk.fsf@gitster.g","threadId":"56179","inReplyTo":"xmqqwnnr7icz.fsf@gitster.g","subject":"Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-08T06:56:19Z","receivedAt":"2021-09-08T06:56:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> Updates in V5\n>> =============\n>>\n>> * Updated the locality of a cache_entry pointer.\n>>\n>> * Rephrased a comment.\n>>\n>> * Removed the patch adding a config option.\n>>\n>> * I tried, but failed, to create a scenario where the call to\n>> cache_tree_update() causes a test to fail. I still think this is valuable\n>> as defensive programming for the reasons mentioned in the patch, which is\n>> why I didn't remove them here.\n>\n> OK. \n>\n> The 'sg/test-split-index-fix' topic that has been queued in 'seen'\n> textually depended on the earlier iteration of this topic, so I'll\n> discard it while replacing this topic.\n\nI rebased the dependent topic myself, but I would not be surprised\nif I weren't careful enough and introduced a new bug. Please double\ncheck when a new 'seen' is pushed out in several hours.\n\nThanks.\n"},{"id":"435077","messageId":"4a0125d2-67d9-b533-4f8b-618c3de3b411@gmail.com","threadId":"56179","inReplyTo":"xmqqczpj7dyk.fsf@gitster.g","subject":"Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T11:39:54Z","receivedAt":"2021-09-08T11:40:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2021 2:56 AM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> Updates in V5\n>>> =============\n>>>\n>>> * Updated the locality of a cache_entry pointer.\n>>>\n>>> * Rephrased a comment.\n>>>\n>>> * Removed the patch adding a config option.\n>>>\n>>> * I tried, but failed, to create a scenario where the call to\n>>> cache_tree_update() causes a test to fail. I still think this is valuable\n>>> as defensive programming for the reasons mentioned in the patch, which is\n>>> why I didn't remove them here.\n>>\n>> OK. \n>>\n>> The 'sg/test-split-index-fix' topic that has been queued in 'seen'\n>> textually depended on the earlier iteration of this topic, so I'll\n>> discard it while replacing this topic.\n> \n> I rebased the dependent topic myself, but I would not be surprised\n> if I weren't careful enough and introduced a new bug. Please double\n> check when a new 'seen' is pushed out in several hours.\n\nDid you have any conflicts? I don't think this version changed any\nof the textual dependencies. Anyway, the test suite passes with\nGIT_TEST_SPARSE_INDEX=1 on my machine at 'seen' (except for an issue\nwith t7900-maintenance.sh that is unrelated).\n\nThanks,\n-Stolee\n"},{"id":"435112","messageId":"xmqqk0jr59o1.fsf@gitster.g","threadId":"56179","inReplyTo":"4a0125d2-67d9-b533-4f8b-618c3de3b411@gmail.com","subject":"Re: [PATCH v5 0/9] Sparse index: delete ignored files outside sparse cone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-08T16:11:58Z","receivedAt":"2021-09-08T16:12:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> I rebased the dependent topic myself, but I would not be surprised\n>> if I weren't careful enough and introduced a new bug. Please double\n>> check when a new 'seen' is pushed out in several hours.\n>\n> Did you have any conflicts? I don't think this version changed any\n> of the textual dependencies. Anyway, the test suite passes with\n> GIT_TEST_SPARSE_INDEX=1 on my machine at 'seen' (except for an issue\n> with t7900-maintenance.sh that is unrelated).\n\nThere was a bit of test refactoring in the other topic, so I\ninitially punted but then I found a bit of energy to look at\nthe conflict and realize it was not all that bad.\n\n"}]}