{"thread":{"id":"56370","subject":"[PATCH 01/13] t1092: behavior for adding sparse files","startedAt":"2021-08-24T21:54:49Z","lastAt":"2021-11-03T17:14:07Z","messageCount":116,"participants":["Derrick Stolee via GitGitGadget","Matheus Tavares Bernardino","Matheus Tavares","Derrick Stolee","Ævar Arnfjörð Bjarmason","Elijah Newren","Junio C Hamano","René Scharfe","Sean Christopherson","Glen Choo"],"isPatch":true,"patchVersion":1,"patchTotal":13},"messages":[{"id":"433654","messageId":"pull.1018.git.1629842085.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":null,"subject":"[PATCH 00/13] [RFC] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:32Z","receivedAt":"2021-08-24T21:54:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"As requested, this series looks to update the behavior of git add, git rm,\nand git mv when they attempt to modify paths outside of the sparse-checkout\ncone. In particular, this care is expanded to not just cache entries with\nthe SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\ndefinition.\n\nThis means that commands that worked before this series can now fail. In\nparticular, if 'git merge' results in a conflict outside of the\nsparse-checkout cone, then 'git add ' will now fail.\n\nIn order to allow users to circumvent these protections, a new '--sparse'\noption is added that ignores the sparse-checkout patterns and the\nSKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\nassist with discovery of this option.\n\nThere is a subtle issue with git mv in that it does not check the index\nuntil it discovers a directory and then uses the index to find the contained\nentries. This means that in non-cone-mode patterns, a pattern such as\n\"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n\nIn order to allow for checking arbitrary paths against the sparse-checkout\npatterns, some changes to the underlying pattern matching code is required.\nIt turns out that there are some bugs in the methods as advertised, but\nthese bugs were never discovered because of the way methods like\nunpack_trees() will check a directory for a pattern match before checking\nits contained paths. Our new \"check patterns on-demand\" approach pokes holes\nin that approach, specifically with patterns that match entire directories.\n\nI'm open to alternative designs here, too.\n\nThe patches are based on v2 of the integration with merge, cherry-pick, and\nrebase. I'm sending the RFC really early in case it needs to be\nsignificantly reworked.\n\nThanks, -Stolee\n\nDerrick Stolee (13):\n t1092: behavior for adding sparse files\n dir: extract directory-matching logic\n dir: select directories correctly\n dir: fix pattern matching on dirs\n add: fail when adding an untracked sparse file\n add: skip paths that are outside sparse-checkout cone\n add: implement the --sparse option\n add: prevent adding sparse conflict files\n rm: add --sparse option\n rm: skip sparse paths with missing SKIP_WORKTREE\n mv: refuse to move sparse paths\n mv: add '--sparse' option to ignore sparse-checkout\n advice: update message to suggest '--sparse'\n\n Documentation/git-add.txt | 9 +-\n Documentation/git-rm.txt | 6 ++\n advice.c | 3 +-\n builtin/add.c | 22 ++++-\n builtin/mv.c | 39 ++++++--\n builtin/rm.c | 10 +-\n dir.c | 54 ++++++++--\n pathspec.c | 5 +-\n t/t1091-sparse-checkout-builtin.sh | 4 +-\n t/t1092-sparse-checkout-compatibility.sh | 47 ++++++---\n t/t3602-rm-sparse-checkout.sh | 27 ++++-\n t/t3705-add-sparse-checkout.sh | 10 +-\n t/t7002-mv-sparse-checkout.sh | 121 +++++++++++++++++++++++\n 13 files changed, 316 insertions(+), 41 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\n\nbase-commit: df4bbec744f5cd4a060082212d95a36b812fa50b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1018\n-- \ngitgitgadget\n"},{"id":"433653","messageId":"69ca54877b4eac5c85195a1061f0d72ba6fca313.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 01/13] t1092: behavior for adding sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:33Z","receivedAt":"2021-08-24T21:54:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests to demonstrate the current behavior around adding files\noutside of the sparse-checkout cone. Currently, untracked files are\nhandled differently from tracked files. A futuer change will make these\ncases be handled the same way.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex bbc6de712c4..23bee918260 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -291,6 +291,18 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+# NEEDSWORK: This documents current behavior, but is not a desirable\n+# behavior (untracked files are handled differently than tracked).\n+test_expect_success 'add outside sparse cone' '\n+\tinit_repos &&\n+\n+\trun_on_sparse mkdir folder1 &&\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_sparse ../edit-contents folder1/newfile &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_sparse_match git add folder1/newfile\n+'\n+\n test_expect_success 'commit including unstaged changes' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"433655","messageId":"8cca5bcf405f220ef899ec44d01fee4d588c7a38.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 02/13] dir: extract directory-matching logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:34Z","receivedAt":"2021-08-24T21:54:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe last_matching_pattern_from_list() logic performs some checks on the\nfiletype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag\nis set. This works great when setting SKIP_WORKTREE bits within\nunpack_trees(), but doesn't work well when passing an arbitrary path\nsuch as a file within a matching directory.\n\nThis change only rearranges the logic but does not change its\nfunctionality.\n\nWe will expand the path_matches_dir_pattern() method in a following\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 22 +++++++++++++++++-----\n 1 file changed, 17 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 86afa2eae00..652135df896 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1303,6 +1303,19 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t\t\t WM_PATHNAME) == 0;\n }\n \n+static int path_matches_dir_pattern(const char *pathname,\n+\t\t\t\t int pathlen,\n+\t\t\t\t int *dtype,\n+\t\t\t\t struct path_pattern *pattern,\n+\t\t\t\t struct index_state *istate)\n+{\n+\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n+\tif (*dtype != DT_DIR)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n /*\n * Scan the given exclude list in reverse to see whether pathname\n * should be ignored. The first match (i.e. the last on the list), if\n@@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n-\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n-\t\t\tif (*dtype != DT_DIR)\n-\t\t\t\tcontinue;\n-\t\t}\n+\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\t\t\t\t dtype, pattern, istate))\n+\t\t\tcontinue;\n \n \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n \t\t\tif (match_basename(basename,\n-- \ngitgitgadget\n\n"},{"id":"433656","messageId":"d47c7a1cf2a3fa8cfdcfc6be1ac800af123e7efc.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 03/13] dir: select directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:35Z","receivedAt":"2021-08-24T21:54:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen matching a path against a list of patterns, the ones that require a\ndirectory match previously did not work when a filename is specified.\nThis was fine when all pattern-matching was done within methods such as\nunpack_trees() that check a directory before recursing into the\ncontained files. However, other commands will start matching individual\nfiles against pattern lists without that recursive approach.\n\nWe modify path_matches_dir_pattern() to take a strbuf 'path_parent' that\nis used to store the parent directory of 'pathname' between multiple\npattern matching tests. This is loaded lazily, only on the first pattern\nit finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n\nIf we find that a path has a parent directory, we start by checking to\nsee if that parent directory matches the pattern. If so, then we do not\nneed to query the index for the type (which can be expensive). If we\nfind that the parent does not match, then we still must check the type\nfrom the index for the given pathname.\n\nNote that this does not affect cone mode pattern matching, but instead\nthe more general -- and slower -- full pattern set. Thus, this does not\naffect the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 652135df896..fe5ee87bb5f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,\n \n static int path_matches_dir_pattern(const char *pathname,\n \t\t\t\t int pathlen,\n+\t\t\t\t struct strbuf *path_parent,\n \t\t\t\t int *dtype,\n \t\t\t\t struct path_pattern *pattern,\n \t\t\t\t struct index_state *istate)\n {\n+\t/*\n+\t * Use 'alloc' as an indicator that the string has not been\n+\t * initialized, in case the parent is the root directory.\n+\t */\n+\tif (!path_parent->alloc) {\n+\t\tchar *slash;\n+\t\tstrbuf_addstr(path_parent, pathname);\n+\t\tslash = find_last_dir_sep(path_parent->buf);\n+\n+\t\tif (slash)\n+\t\t\t*slash = '\\0';\n+\t\telse\n+\t\t\tstrbuf_setlen(path_parent, 0);\n+\t}\n+\n+\t/*\n+\t * If the parent directory matches the pattern, then we do not\n+\t * need to check for dtype.\n+\t */\n+\tif (path_parent->len &&\n+\t match_pathname(path_parent->buf, path_parent->len,\n+\t\t\t pattern->base,\n+\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n+\t\t\t pattern->pattern, pattern->nowildcardlen,\n+\t\t\t pattern->patternlen, pattern->flags))\n+\t\treturn 1;\n+\n \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n \tif (*dtype != DT_DIR)\n \t\treturn 0;\n@@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n \tint i;\n+\tstruct strbuf path_parent = STRBUF_INIT;\n \n \tif (!pl->nr)\n \t\treturn NULL;\t/* undefined */\n@@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n-\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n \t\t\t\t\t dtype, pattern, istate))\n \t\t\tcontinue;\n \n@@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\t\tbreak;\n \t\t}\n \t}\n+\tstrbuf_release(&path_parent);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"433657","messageId":"58e7df44982965234c028174ff108a0abeb9862b.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 04/13] dir: fix pattern matching on dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:36Z","receivedAt":"2021-08-24T21:54:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWithin match_pathname(), one successful matching category happens when\nthe pattern is equal to its non-wildcard prefix. At this point, we have\nchecked that the input 'pathname' matches the pattern up to the prefix\nlength, and then we subtraced that length from both 'patternlen' and\n'namelen'.\n\nIn the case of a directory match, this prefix match should be\nsufficient. However, the success condition only cared about _exact_\nequality here. Instead, we should allow any path that agrees on this\nprefix in the case of PATTERN_FLAG_MUSTBEDIR.\n\nThis case was not tested before because of the way unpack_trees() would\nmatch a parent directory before visiting the contained paths. This\napproach is changing, so we must change this comparison.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex fe5ee87bb5f..80fd0ad2fc0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t * then our prefix match is all we need; we\n \t\t * do not need to call fnmatch at all.\n \t\t */\n-\t\tif (!patternlen && !namelen)\n+\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n \t\t\treturn 1;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"433658","messageId":"e80fcfa932cca394c5c8b349cafadb0754a594dd.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 05/13] add: fail when adding an untracked sparse file","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:37Z","receivedAt":"2021-08-24T21:54:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe add_files() method in builtin/add.c takes a set of untracked files\nthat are being added by the input pathspec and inserts them into the\nindex. If these files are outside of the sparse-checkout cone, then they\ngain the SKIP_WORKTREE bit at some point. However, this was not checked\nbefore inserting into the index, so these files are added even though we\nwant to avoid modifying the index outside of the sparse-checkout cone.\n\nAdd a check within add_files() for these files and write the advice\nabout files outside of the sprase-checkout cone.\n\nThis behavior change modifies some existing tests within t1092. These\ntests intended to document how a user could interact with the existing\nbehavior in place. Many of these tests need to be marked as expecting\nfailure. A future change will allow these tests to pass by adding a flag\nto 'git add' that allows users to modify index entries outside of the\nsparse-checkout cone.\n\nThe 'submodule handling' test is intended to document what happens to\ndirectories that contain a submodule when the sparse index is enabled.\nIt is not trying to say that users should be able to add submodules\noutside of the sparse-checkout cone, so that test can be modified to\navoid that operation.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 14 ++++++++++\n t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++++-------\n 2 files changed, 38 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 88a6c0c69fb..3a109276b74 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n static int add_files(struct dir_struct *dir, int flags)\n {\n \tint i, exit_status = 0;\n+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \n \tif (dir->ignored_nr) {\n \t\tfprintf(stderr, _(ignore_error));\n@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n+\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\t\tstring_list_append(&only_match_skip_worktree,\n+\t\t\t\t\t dir->entries[i]->name);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {\n \t\t\tif (!ignore_add_errors)\n \t\t\t\tdie(_(\"adding files failed\"));\n@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)\n \t\t\tcheck_embedded_repo(dir->entries[i]->name);\n \t\t}\n \t}\n+\n+\tif (only_match_skip_worktree.nr) {\n+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n+\t\texit_status = 1;\n+\t}\n+\n+\tstring_list_clear(&only_match_skip_worktree, 0);\n+\n \treturn exit_status;\n }\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 23bee918260..962bece03e1 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -291,8 +291,6 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-# NEEDSWORK: This documents current behavior, but is not a desirable\n-# behavior (untracked files are handled differently than tracked).\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n@@ -300,7 +298,7 @@ test_expect_success 'add outside sparse cone' '\n \trun_on_sparse ../edit-contents folder1/a &&\n \trun_on_sparse ../edit-contents folder1/newfile &&\n \ttest_sparse_match test_must_fail git add folder1/a &&\n-\ttest_sparse_match git add folder1/newfile\n+\ttest_sparse_match test_must_fail git add folder1/newfile\n '\n \n test_expect_success 'commit including unstaged changes' '\n@@ -331,7 +329,11 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-test_expect_success 'status/add: outside sparse cone' '\n+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n+# attempted here fail for the sparse-checkout and sparse-index repos.\n+# We must enable a way for adding files outside the sparse-checkout\n+# done, even if it is by an optional flag.\n+test_expect_failure 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -352,10 +354,9 @@ test_expect_success 'status/add: outside sparse cone' '\n \t# Adding the path outside of the sparse-checkout cone should fail.\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/new &&\n \n-\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n-\ttest_sparse_match git add folder1/new &&\n-\n+\t# NEEDSWORK: behavior begins to deviate here.\n \ttest_all_match git add . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n@@ -511,7 +512,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n # Right now, users might be using this flow to work through conflicts,\n # so any solution should present advice to users who try this sequence\n # of commands to follow whatever new method we create.\n-test_expect_success 'merge with conflict outside cone' '\n+test_expect_failure 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -525,12 +526,18 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n+\t# NEEDSWORK: Even though the merge conflict removed the\n+\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t# warn that this is a problematic add.\n \ttest_all_match git add folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n+\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t# outside of the sparse-checkout cone and does not match an\n+\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_all_match git add folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n@@ -539,7 +546,7 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -556,11 +563,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 2. Add the file with conflict markers\n+\t\t# NEEDSWORK: Even though the merge conflict removed the\n+\t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t\t# warn that this is a problematic add.\n \t\ttest_all_match git add folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n \t\t# accept conflict markers as resolved content.\n+\t\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t\t# outside of the sparse-checkout cone and does not match an\n+\t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n \t\ttest_all_match git add folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n@@ -638,6 +651,7 @@ test_expect_success 'clean' '\n test_expect_success 'submodule handling' '\n \tinit_repos &&\n \n+\ttest_sparse_match git sparse-checkout add modules &&\n \ttest_all_match mkdir modules &&\n \ttest_all_match touch modules/a &&\n \ttest_all_match git add modules &&\n@@ -647,6 +661,7 @@ test_expect_success 'submodule handling' '\n \ttest_all_match git commit -m \"add submodule\" &&\n \n \t# having a submodule prevents \"modules\" from collapse\n+\ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n \ttest-tool -C sparse-index read-cache --table >cache &&\n \tgrep \"100644 blob .*\tmodules/a\" cache &&\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n-- \ngitgitgadget\n\n"},{"id":"433659","messageId":"eeba97ad492307302637faf33f6bf6ae8965faa3.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 06/13] add: skip paths that are outside sparse-checkout cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:38Z","receivedAt":"2021-08-24T21:55:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen 'git add' adds a tracked file that is outside of the\nsparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file\nexists outside of the sparse-checkout cone. This is usually correct,\nexcept in the case of a merge conflict outside of the cone.\n\nModify add_pathspec_matched_against_index() to be more careful about\npathes by checking the sparse-checkout patterns in addition to the\nSKIP_WORKTREE bit. This causes 'git add' to no longer allow files\noutside of the cone that removed the SKIP_WORKTREE bit due to a merge\nconflict.\n\nWith only this change, users will only be able to add the file after\nadding the file to the sparse-checkout cone. A later change will allow\nusers to force adding even though the file is outside of the\nsparse-checkout cone.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 3 ++-\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++----\n 2 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex 44306fdaca2..0e6e60fdc5a 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\treturn;\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n-\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n+\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n+\t\t (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n \t\t\tcontinue;\n \t\tce_path_match(istate, ce, pathspec, seen);\n \t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 962bece03e1..c2a4eec548d 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -529,7 +529,7 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# NEEDSWORK: Even though the merge conflict removed the\n \t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n \t# warn that this is a problematic add.\n-\ttest_all_match git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n@@ -538,7 +538,7 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# NEEDSWORK: This mode now fails, because folder2/z is\n \t# outside of the sparse-checkout cone and does not match an\n \t# existing index entry with the SKIP_WORKTREE bit cleared.\n-\ttest_all_match git add folder2 &&\n+\ttest_sparse_match test_must_fail git add folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -566,7 +566,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# NEEDSWORK: Even though the merge conflict removed the\n \t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n \t\t# warn that this is a problematic add.\n-\t\ttest_all_match git add folder1/a &&\n+\t\ttest_sparse_match test_must_fail git add folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -575,7 +575,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# outside of the sparse-checkout cone and does not match an\n \t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n-\t\ttest_all_match git add folder2 &&\n+\t\ttest_sparse_match test_must_fail git add folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\n-- \ngitgitgadget\n\n"},{"id":"433663","messageId":"6edadbc2000bd6d2ec76cbeff08b309ae4928b37.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 07/13] add: implement the --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:39Z","receivedAt":"2021-08-24T21:55:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe previously modified 'git add' to refuse updating index entries\noutside of the sparse-checkout cone. This is justified to prevent users\nfrom accidentally getting into a confusing state when Git removes those\nfiles from the working tree at some later point.\n\nUnfortunately, this caused some workflows that were previously possible\nto become impossible, especially around merge conflicts outside of the\nsparse-checkout cone. These were documented in tests within t1092.\n\nWe now re-enable these workflows using a new '--sparse' option to 'git\nadd'. This allows users to signal \"Yes, I do know what I'm doing with\nthese files,\" and accept the consequences of the files leaving the\nworktree later.\n\nWe delay updating the advice message until implementing a similar option\nin 'git rm' and 'git mv'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-add.txt | 9 +++++++-\n builtin/add.c | 10 ++++++---\n t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++----------------\n t/t3705-add-sparse-checkout.sh | 8 +++++++\n 4 files changed, 32 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex be5e3ac54b8..bb79016d2ca 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -9,7 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n-\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n+\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]\n \t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--pathspec-from-file= [--pathspec-file-nul]]\n \t [--] [...]\n@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].\n --force::\n \tAllow adding otherwise ignored files.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git add` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone, since those files might\n+\tbe removed from the working tree without warning. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -i::\n --interactive::\n \tAdd modified contents in the working tree interactively to\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 3a109276b74..68f2de80594 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n static int add_renormalize;\n static int pathspec_file_nul;\n+static int include_sparse;\n static const char *pathspec_from_file;\n static int legacy_stash_p; /* support for the scripted `git stash` */\n \n@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n@@ -379,6 +380,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n \tOPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n \tOPT_BOOL( 0 , \"ignore-missing\", &ignore_missing, N_(\"check if - even missing - files are ignored in dry run\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_STRING(0, \"chmod\", &chmod_arg, \"(+|-)x\",\n \t\t N_(\"override the executable bit of the listed files\")),\n \tOPT_HIDDEN_BOOL(0, \"warn-embedded-repo\", &warn_on_embedded_repo,\n@@ -457,7 +459,8 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n-\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\tif (!include_sparse &&\n+\t\t !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n \t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t dir->entries[i]->name);\n \t\t\tcontinue;\n@@ -642,7 +645,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\tif (seen[i])\n \t\t\t\tcontinue;\n \n-\t\t\tif (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n+\t\t\tif (!include_sparse &&\n+\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec.items[i].original);\n \t\t\t\tcontinue;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex c2a4eec548d..1e7799fd76a 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -329,11 +329,7 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n-# attempted here fail for the sparse-checkout and sparse-index repos.\n-# We must enable a way for adding files outside the sparse-checkout\n-# done, even if it is by an optional flag.\n-test_expect_failure 'status/add: outside sparse cone' '\n+test_expect_success 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -355,15 +351,16 @@ test_expect_failure 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n \ttest_sparse_match test_must_fail git add folder1/new &&\n+\ttest_sparse_match git add --sparse folder1/a &&\n+\ttest_sparse_match git add --sparse folder1/new &&\n \n-\t# NEEDSWORK: behavior begins to deviate here.\n-\ttest_all_match git add . &&\n+\ttest_all_match git add --sparse . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n \ttest_all_match git rev-parse HEAD^{tree} &&\n \n \trun_on_all ../edit-contents folder1/newer &&\n-\ttest_all_match git add folder1/ &&\n+\ttest_all_match git add --sparse folder1/ &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/newer &&\n \ttest_all_match git rev-parse HEAD^{tree}\n@@ -507,12 +504,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n \tdone\n '\n \n-# NEEDSWORK: This test is documenting current behavior, but that\n-# behavior can be confusing to users so there is desire to change it.\n-# Right now, users might be using this flow to work through conflicts,\n-# so any solution should present advice to users who try this sequence\n-# of commands to follow whatever new method we create.\n-test_expect_failure 'merge with conflict outside cone' '\n+test_expect_success 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -528,17 +520,15 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# 2. Add the file with conflict markers\n \t# NEEDSWORK: Even though the merge conflict removed the\n \t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n-\t# warn that this is a problematic add.\n-\ttest_sparse_match test_must_fail git add folder1/a &&\n+\t# warn that this is a problematic add when --sparse is not set.\n+\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n-\t# NEEDSWORK: This mode now fails, because folder2/z is\n-\t# outside of the sparse-checkout cone and does not match an\n-\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n+\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 2b1fd0d0eef..99c27dd9fb7 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -152,4 +152,12 @@ test_expect_success 'add obeys advice.updateSparsePath' '\n \n '\n \n+test_expect_success 'add allows sparse entries with --sparse' '\n+\tgit sparse-checkout set a &&\n+\techo modified >sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\tgit add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"433660","messageId":"f1764f9ed187a2f383bd8269e198192fe0486bda.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 08/13] add: prevent adding sparse conflict files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:40Z","receivedAt":"2021-08-24T21:55:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a merge results in a conflict outside of the sparse-checkout cone,\nthe conflicted file is written to the working tree and the index entry\nloses the SKIP_WORKTREE bit. This allows users to add the file to the\nindex without realizing that the file might leave the working tree in a\nlater Git command.\n\nBlock this behavior, but keep in mind that the user can override the\nfailure using the '--sparse' option.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 4 +++-\n t/t1092-sparse-checkout-compatibility.sh | 6 +++---\n 3 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex 0e6e60fdc5a..ddeeba79114 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -71,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))\n \t\t ce_path_match(istate, ce, pathspec, seen);\n \t}\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e0f31186d89..b6efdb3c52f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat\n \tgit -C unmerged sparse-checkout disable\n '\n \n-test_expect_success 'sparse-checkout reapply' '\n+test_expect_failure 'sparse-checkout reapply' '\n \tgit clone repo tweak &&\n \n \techo dirty >tweak/deep/deeper2/a &&\n@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '\n \ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n \ttest_path_is_file tweak/folder1/a &&\n \n+\t# NEEDSWORK: We are asking to update a file outside of the\n+\t# sparse-checkout cone, but this is no longer allowed.\n \tgit -C tweak add folder1/a &&\n \tgit -C tweak sparse-checkout reapply 2>err &&\n \ttest_must_be_empty err &&\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 1e7799fd76a..65998e664a9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -518,9 +518,8 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n-\t# NEEDSWORK: Even though the merge conflict removed the\n-\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n-\t# warn that this is a problematic add when --sparse is not set.\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n@@ -528,6 +527,7 @@ test_expect_success 'merge with conflict outside cone' '\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n \ttest_sparse_match test_must_fail git add folder2 &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n-- \ngitgitgadget\n\n"},{"id":"433661","messageId":"6c9c986ff43fe7f065c27e61468534007e70d2a7.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 09/13] rm: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:41Z","receivedAt":"2021-08-24T21:55:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-rm.txt | 6 ++++++\n builtin/rm.c | 8 ++++++--\n t/t3602-rm-sparse-checkout.sh | 14 +++++++++++++-\n 3 files changed, 25 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-rm.txt b/Documentation/git-rm.txt\nindex 26e9b284704..81bc23f3cdb 100644\n--- a/Documentation/git-rm.txt\n+++ b/Documentation/git-rm.txt\n@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].\n --ignore-unmatch::\n \tExit with a zero status even if no files matched.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git rm` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -q::\n --quiet::\n \t`git rm` normally outputs one line (in the form of an `rm` command)\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 8a24c715e02..4208f3f9a5f 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)\n \n static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;\n static int ignore_unmatch = 0, pathspec_file_nul;\n+static int include_sparse;\n static char *pathspec_from_file;\n \n static struct option builtin_rm_options[] = {\n@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {\n \tOPT_BOOL('r', NULL, &recursive, N_(\"allow recursive removal\")),\n \tOPT_BOOL( 0 , \"ignore-unmatch\", &ignore_unmatch,\n \t\t\t\tN_(\"exit with a zero status even if nothing matched\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_PATHSPEC_FROM_FILE(&pathspec_from_file),\n \tOPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),\n \tOPT_END(),\n@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\n@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\t\tseen_any = 1;\n \t\t\telse if (ignore_unmatch)\n \t\t\t\tcontinue;\n-\t\t\telse if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n+\t\t\telse if (!include_sparse &&\n+\t\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n \t\t\t\tstring_list_append(&only_match_skip_worktree, original);\n \t\t\telse\n \t\t\t\tdie(_(\"pathspec '%s' did not match any files\"), original);\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex e9e9a15c74c..a34b978bfd8 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -36,13 +36,25 @@ done\n \n test_expect_success 'recursive rm does not remove sparse entries' '\n \tgit reset --hard &&\n-\tgit sparse-checkout set sub/dir &&\n+\tgit sparse-checkout set sub/dir/ &&\n \tgit rm -r sub &&\n \tgit status --porcelain -uno >actual &&\n \techo \"D sub/dir/e\" >expected &&\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'recursive rm --sparse removes sparse entries' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set \"sub/dir\" &&\n+\tgit rm --sparse -r sub &&\n+\tgit status --porcelain -uno >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'rm obeys advice.updateSparsePath' '\n \tgit reset --hard &&\n \tgit sparse-checkout set a &&\n-- \ngitgitgadget\n\n"},{"id":"433662","messageId":"5153accded46c8ced8784ec135dbc77d5bc1a306.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 10/13] rm: skip sparse paths with missing SKIP_WORKTREE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:42Z","receivedAt":"2021-08-24T21:55:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a path does not match the sparse-checkout cone but is somehow missing\nthe SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the\nfile. One reason a user might be in this situation is a merge conflict\noutside of the sparse-checkout cone. Removing such a file might be\nproblematic for users who are not sure what they are doing.\n\nAdd a check to path_in_sparse_checkout() when 'git rm' is checking if a\npath should be considered for deletion. Of course, this check is ignored\nif the '--sparse' option is specified, allowing users who accept the\nrisks to continue with the removal.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 4 +++-\n t/t3602-rm-sparse-checkout.sh | 11 +++++++++++\n 2 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4208f3f9a5f..a6da03da2be 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex a34b978bfd8..44f3e923164 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -87,4 +87,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n \tgit ls-files --error-unmatch b\n '\n \n+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\tgit update-index --no-skip-worktree b &&\n+\ttest_must_fail git rm b 2>stderr &&\n+\ttest_cmp b_error_and_hint stderr &&\n+\tgit rm --sparse b 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"433664","messageId":"d31c641180645ee4059aab9230841ad90f9244fe.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 11/13] mv: refuse to move sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:43Z","receivedAt":"2021-08-24T21:55:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince cmd_mv() does not operate on cache entries and instead directly\nchecks the filesystem, we can only use path_in_sparse_checkout() as a\nmechanism for seeing if a path is sparse or not. Be sure to skip\nreturning a failure if '-k' is specified.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/mv.c | 19 +++++++\n t/t7002-mv-sparse-checkout.sh | 99 +++++++++++++++++++++++++++++++++++\n 2 files changed, 118 insertions(+)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex c2f96c8e895..b58fd4ce5ba 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -133,6 +133,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct cache_entry *ce;\n+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \n \tgit_config(git_default_config, NULL);\n \n@@ -176,10 +177,22 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tint length, src_is_dir;\n \t\tconst char *bad = NULL;\n+\t\tint skip_sparse = 0;\n \n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n+\t\tif (!path_in_sparse_checkout(src, &the_index)) {\n+\t\t\tstring_list_append(&only_match_skip_worktree, src);\n+\t\t\tskip_sparse = 1;\n+\t\t}\n+\t\tif (!path_in_sparse_checkout(dst, &the_index)) {\n+\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n+\t\t\tskip_sparse = 1;\n+\t\t}\n+\t\tif (skip_sparse)\n+\t\t\tcontinue;\n+\n \t\tlength = strlen(src);\n \t\tif (lstat(src, &st) < 0)\n \t\t\tbad = _(\"bad source\");\n@@ -266,6 +279,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (only_match_skip_worktree.nr) {\n+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n+\t\tif (!ignore_errors)\n+\t\t\treturn 1;\n+\t}\n+\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tenum update_mode mode = modes[i];\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..5397c6d07bd\n--- /dev/null\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -0,0 +1,99 @@\n+#!/bin/sh\n+\n+test_description='git mv in sparse working trees'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' \"\n+\tmkdir -p sub/dir sub/dir2 &&\n+\ttouch a b c sub/d sub/dir/e sub/dir2/e &&\n+\tgit add -A &&\n+\tgit commit -m files &&\n+\n+\tcat >sparse_error_header <<-EOF &&\n+\tThe following pathspecs didn't match any eligible path, but they do match index\n+\tentries outside the current sparse checkout:\n+\tEOF\n+\n+\tcat >sparse_hint <<-EOF\n+\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n+\tEOF\n+\"\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse' '\n+\trm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n+\trm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttouch b &&\n+\tgit mv -k b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n+\trm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttest_must_fail git mv a e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n+\trm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a e &&\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_expect_success 'recursive mv refuses to move (possible) sparse' '\n+\trm -f e &&\n+\tgit reset --hard &&\n+\t# Without cone mode, \"sub\" and \"sub2\" do not match\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo sub >>expect &&\n+\techo sub2 >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_expect_success 'recursive mv refuses to move sparse' '\n+\tgit reset --hard &&\n+\t# Use cone mode so \"sub/\" matches the sparse-checkout patterns\n+\tgit sparse-checkout init --cone &&\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo sub/dir2/e >>expect &&\n+\techo sub2/dir2/e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"433665","messageId":"f6c0d4e3a0695c331b8216103bb46bfb4a461d1e.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 12/13] mv: add '--sparse' option to ignore sparse-checkout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:44Z","receivedAt":"2021-08-24T21:55:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nUsers can get into strange situations if 'git mv' allows moving files\ninto, out of, or around the sparse-checkout cone. However, some users\nmay still want to do it. Allow knowledgeable users to do so via a new\n'--sparse' option.\n\nThere are some special cases that occur in this change, such as the case\nof a directory that doesn't match the sparse-checkout cone, but exists\nin the working tree because a subset of its contents do match. We need\nto communicate that index entries with the SKIP_WORKTREE bit are not\nexpected to be in the working directory and hence are not needed when\nmoving the contents. This is only a check for the existence of the\nsource file. The call to rename_cache_entry_at() still changes the index\nappropriately in these cases.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/mv.c | 24 ++++++++++++++----------\n t/t7002-mv-sparse-checkout.sh | 34 ++++++++++++++++++++++++++++------\n 2 files changed, 42 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex b58fd4ce5ba..92ea9f0ca37 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -118,17 +118,18 @@ static int index_range_of_same_dir(const char *src, int length,\n int cmd_mv(int argc, const char **argv, const char *prefix)\n {\n \tint i, flags, gitmodules_modified = 0;\n-\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n+\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n \tstruct option builtin_mv_options[] = {\n \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n \t\tOPT__DRY_RUN(&show_only, N_(\"dry run\")),\n \t\tOPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n+\t\tOPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \t\tOPT_END(),\n \t};\n \tconst char **source, **destination, **dest_path, **submodule_gitfile;\n-\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n+\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n \tstruct stat st;\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n@@ -182,11 +183,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n-\t\tif (!path_in_sparse_checkout(src, &the_index)) {\n+\t\tif (!ignore_sparse && !path_in_sparse_checkout(src, &the_index)) {\n \t\t\tstring_list_append(&only_match_skip_worktree, src);\n \t\t\tskip_sparse = 1;\n \t\t}\n-\t\tif (!path_in_sparse_checkout(dst, &the_index)) {\n+\t\tif (!ignore_sparse && !path_in_sparse_checkout(dst, &the_index)) {\n \t\t\tstring_list_append(&only_match_skip_worktree, dst);\n \t\t\tskip_sparse = 1;\n \t\t}\n@@ -194,9 +195,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \n \t\tlength = strlen(src);\n-\t\tif (lstat(src, &st) < 0)\n-\t\t\tbad = _(\"bad source\");\n-\t\telse if (!strncmp(src, dst, length) &&\n+\t\tif (lstat(src, &st) < 0) {\n+\t\t\t/* only error if existence is expected. */\n+\t\t\tif (modes[i] != SPARSE)\n+\t\t\t\tbad = _(\"bad source\");\n+\t\t} else if (!strncmp(src, dst, length) &&\n \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n \t\t\tbad = _(\"can not move directory into itself\");\n \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n@@ -225,11 +228,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\t\tdst_len = strlen(dst);\n \n \t\t\t\tfor (j = 0; j < last - first; j++) {\n-\t\t\t\t\tconst char *path = active_cache[first + j]->name;\n+\t\t\t\t\tconst struct cache_entry *ce = active_cache[first + j];\n+\t\t\t\t\tconst char *path = ce->name;\n \t\t\t\t\tsource[argc + j] = path;\n \t\t\t\t\tdestination[argc + j] =\n \t\t\t\t\t\tprefix_path(dst, dst_len, path + length + 1);\n-\t\t\t\t\tmodes[argc + j] = INDEX;\n+\t\t\t\t\tmodes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n \t\t\t\t\tsubmodule_gitfile[argc + j] = NULL;\n \t\t\t\t}\n \t\t\t\targc += last - first;\n@@ -293,7 +297,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(_(\"Renaming %s to %s\\n\"), src, dst);\n \t\tif (show_only)\n \t\t\tcontinue;\n-\t\tif (mode != INDEX && rename(src, dst) < 0) {\n+\t\tif (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n \t\t\tif (ignore_errors)\n \t\t\t\tcontinue;\n \t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 5397c6d07bd..517fd587fa8 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -31,7 +31,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse' '\n \techo b >>expect &&\n \techo e >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n '\n \n test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n@@ -44,7 +46,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n \techo b >>expect &&\n \techo e >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse -k b e 2>stderr &&\n+\ttest_must_be_empty stderr\n '\n \n test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n@@ -55,7 +59,9 @@ test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n \tcat sparse_error_header >expect &&\n \techo e >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a e 2>stderr &&\n+\ttest_must_be_empty stderr\n '\n \n test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n@@ -67,7 +73,9 @@ test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n \tcat sparse_error_header >expect &&\n \techo b >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n '\n \n test_expect_success 'recursive mv refuses to move (possible) sparse' '\n@@ -80,7 +88,14 @@ test_expect_success 'recursive mv refuses to move (possible) sparse' '\n \techo sub >>expect &&\n \techo sub2 >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n '\n \n test_expect_success 'recursive mv refuses to move sparse' '\n@@ -93,7 +108,14 @@ test_expect_success 'recursive mv refuses to move sparse' '\n \techo sub/dir2/e >>expect &&\n \techo sub2/dir2/e >>expect &&\n \tcat sparse_hint >>expect &&\n-\ttest_cmp expect stderr\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"433666","messageId":"7749a69a38faeceadc7a8735fb3d0edd359c954c.1629842085.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH 13/13] advice: update message to suggest '--sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-08-24T21:54:45Z","receivedAt":"2021-08-24T21:55:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous changes modified the behavior of 'git add', 'git rm', and\n'git mv' to not adjust paths outside the sparse-checkout cone, even if\nthey exist in the working tree and their cache entries lack the\nSKIP_WORKTREE bit. The intention is to warn users that they are doing\nsomething potentially dangerous. The '--sparse' option was added to each\ncommand to allow careful users the same ability they had before.\n\nTo improve the discoverability of this new functionality, add a message\nto advice.updateSparsePath that mentions the existence of the option.\n\nSigned-off-by: Derrick Stolee \n---\n advice.c | 3 ++-\n t/t3602-rm-sparse-checkout.sh | 2 +-\n t/t3705-add-sparse-checkout.sh | 2 +-\n t/t7002-mv-sparse-checkout.sh | 2 +-\n 4 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex 0b9c89c48ab..695693e5fa8 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -300,7 +300,8 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n-\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n+\t\t\t _(\"Disable or modify the sparsity rules or\"\n+\t\t\t \" use the --sparse option if you intend\"\n \t\t\t \" to update such entries.\"));\n }\n \ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 44f3e923164..22170feef62 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -16,7 +16,7 @@ test_expect_success 'setup' \"\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 99c27dd9fb7..0a8071370d3 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -43,7 +43,7 @@ test_expect_success 'setup' \"\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 517fd587fa8..b6e9ea0b2c8 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -16,7 +16,7 @@ test_expect_success 'setup' \"\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \"\n-- \ngitgitgadget\n"},{"id":"433959","messageId":"CAHd-oW7D1jikE5ByS36AjACfSJoZeekLCKzX2MRRfayKwKv=qQ@mail.gmail.com","threadId":"56370","inReplyTo":"e80fcfa932cca394c5c8b349cafadb0754a594dd.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/13] add: fail when adding an untracked sparse file","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:06:52Z","receivedAt":"2021-08-27T21:07:08Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The add_files() method in builtin/add.c takes a set of untracked files\n> that are being added by the input pathspec and inserts them into the\n> index. If these files are outside of the sparse-checkout cone, then they\n> gain the SKIP_WORKTREE bit at some point. However, this was not checked\n> before inserting into the index, so these files are added even though we\n> want to avoid modifying the index outside of the sparse-checkout cone.\n>\n> Add a check within add_files() for these files and write the advice\n> about files outside of the sprase-checkout cone.\n\ns/sprase/sparse/\n\n> This behavior change modifies some existing tests within t1092. These\n> tests intended to document how a user could interact with the existing\n> behavior in place. Many of these tests need to be marked as expecting\n> failure. A future change will allow these tests to pass by adding a flag\n> to 'git add' that allows users to modify index entries outside of the\n> sparse-checkout cone.\n>\n> The 'submodule handling' test is intended to document what happens to\n> directories that contain a submodule when the sparse index is enabled.\n> It is not trying to say that users should be able to add submodules\n> outside of the sparse-checkout cone, so that test can be modified to\n> avoid that operation.\n\nWhile I was playing with this patch, I did the following:\n\necho a >a\necho b >b\ngit add .\ngit commit -m files\ngit sparse-checkout set a\necho c >c\ngit add c\n\nAnd the last `git add` was successful in adding the untracked `c` file\nwhich is outside the sparse checkout. I'm not sure if I'm doing\nsomething wrong, but it seems that `path_in_sparse_checkout()` returns\nUNDECIDED for `c`. Is it because there was no pattern in the list\nexplicitly excluding it? And if so, should we consider UNDECIDED as\nNOT_MATCHED for `path_in_sparse_checkout()`?\n\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/add.c | 14 ++++++++++\n> t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++++-------\n> 2 files changed, 38 insertions(+), 9 deletions(-)\n>\n> diff --git a/builtin/add.c b/builtin/add.c\n> index 88a6c0c69fb..3a109276b74 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n> static int add_files(struct dir_struct *dir, int flags)\n> {\n> int i, exit_status = 0;\n> + struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n\nI see this reuses the logic from cmd_add() and refresh(). But since we\nare operating on untracked files here, perhaps we could replace\n\"skip_worktree\" by \"sparse_paths\" or something similar?\n\n> if (dir->ignored_nr) {\n> fprintf(stderr, _(ignore_error));\n> @@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)\n> }\n>\n> for (i = 0; i < dir->nr; i++) {\n> + if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n> + string_list_append(&only_match_skip_worktree,\n> + dir->entries[i]->name);\n> + continue;\n> + }\n> if (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {\n> if (!ignore_add_errors)\n> die(_(\"adding files failed\"));\n> @@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)\n> check_embedded_repo(dir->entries[i]->name);\n> }\n> }\n> +\n> + if (only_match_skip_worktree.nr) {\n> + advise_on_updating_sparse_paths(&only_match_skip_worktree);\n\n\nHmm, advise_on_updating_sparse_paths() takes a list of pathspecs that\nonly matched sparse paths, but here we are passing a list of actual\npathnames... Well, these are technically pathspecs too, but the advice\nmessage may be confusing.\n\nFor example, if we ran `git add *.c` on a repo with the untracked\nfiles `d1/file.c` and `d2/file.c`, we will get:\n\nThe following pathspecs didn't match any eligible path, but they do match index\nentries outside the current sparse checkout:\nd1/file.c\nd2/file.c\n\nHowever, `d1/file.c` and `d2/file.c` are neither index entries nor the\npathspecs that the user has given to `git add`. So perhaps we need to\nchange the error/advice message?\n\n\n> + exit_status = 1;\n> + }\n> + string_list_clear(&only_match_skip_worktree, 0);\n> +\n> return exit_status;\n> }\n"},{"id":"433960","messageId":"20210827211310.108999-1-matheus.bernardino@usp.br","threadId":"56370","inReplyTo":"eeba97ad492307302637faf33f6bf6ae8965faa3.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:13:10Z","receivedAt":"2021-08-27T21:13:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget wrote:\n>\n> Subject: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone\n\nPerhaps this could be \"skip _tracked_ paths that ...\" (to help\ndifferentiate the end goal of this patch from the previous one).\n\n> diff --git a/pathspec.c b/pathspec.c\n> index 44306fdaca2..0e6e60fdc5a 100644\n> --- a/pathspec.c\n> +++ b/pathspec.c\n> @@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n> return;\n> for (i = 0; i < istate->cache_nr; i++) {\n> const struct cache_entry *ce = istate->cache[i];\n> - if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n> + if (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n> + (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n\nHmm, even though we skip the sparse paths here, cmd_add() will call\nadd_files_to_cache() at the end and still update these paths in the\nindex. I think there are two ways to fix this. We could either change\nrun_diff_files() to skip these paths (but I don't know how other callers\nof this functions want to handle this, so maybe this needs to hide\nbehind an option flag):\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex f9eadc4fc1..4245d7ead5 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -198,7 +198,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t\tcontinue;\n \t\t}\n \n-\t\tif (ce_uptodate(ce) || ce_skip_worktree(ce))\n+\t\tif (ce_uptodate(ce) || ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, istate))\n \t\t\tcontinue;\n \n \t\t/*\n\nOr we could change the callback in add itself:\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f675bdeae4..3d7762aac2 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n+\n+\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\t\tcontinue;\n+\n \t\tswitch (fix_unmerged_status(p, data)) {\n \t\tdefault:\n \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\n\nI believe we also need to update a few other places to use the\n`(ce_skip_worktree(ce) || !path_in_sparse_checkout())` logic in order to\navoid updating tracked sparse paths: chmod_pathspec() for add's --chmod\noption, renormalize_tracked_files() for --renormalize, and\nread-cache.c:refresh_index() for --refresh.\n\n> continue;\n> ce_path_match(istate, ce, pathspec, seen);\n> }\n\nHmm, don't we also want to update\nfind_pathspecs_matching_skip_worktree() in this patch to use\npath_in_sparse_checkout()? I see you did that in patch 8, but I think\nthis should be together with this current patch as, without it, we stop\nadding tracked sparse paths but we print no error/advice message about\nit.\n"},{"id":"433961","messageId":"CAHd-oW7xvnojJsxi7thukOtQ0=m2eiGdkr9Ne+kj1Lufc_MbZg@mail.gmail.com","threadId":"56370","inReplyTo":"6edadbc2000bd6d2ec76cbeff08b309ae4928b37.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/13] add: implement the --sparse option","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:14:12Z","receivedAt":"2021-08-27T21:14:26Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> diff --git a/builtin/add.c b/builtin/add.c\n> index 3a109276b74..68f2de80594 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n>\n> @@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n> struct cache_entry *ce = active_cache[i];\n> int err;\n>\n> - if (ce_skip_worktree(ce))\n> + if (!include_sparse && ce_skip_worktree(ce))\n> continue;\n\nNot related to this change but just as a reminder: if you agree with\nmy suggestion from the previous patch to use the skipping logic at\nrun_diff_files() or update_callback(), we also need to override it in\nthis patch when include_sparse is true.\n"},{"id":"433962","messageId":"CAHd-oW5ZgG0Rr4THdeyWQtkdNpJGmp-WuCW7RjJA1AiN1nmSKg@mail.gmail.com","threadId":"56370","inReplyTo":"f1764f9ed187a2f383bd8269e198192fe0486bda.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/13] add: prevent adding sparse conflict files","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:16:00Z","receivedAt":"2021-08-27T21:16:15Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When a merge results in a conflict outside of the sparse-checkout cone,\n> the conflicted file is written to the working tree and the index entry\n> loses the SKIP_WORKTREE bit. This allows users to add the file to the\n> index without realizing that the file might leave the working tree in a\n> later Git command.\n>\n> Block this behavior, but keep in mind that the user can override the\n> failure using the '--sparse' option.\n\nHmm, didn't we already block this behavior at patch 6?\n\nNevertheless, as I mentioned there, I think the change to\nfind_pathspecs_matching_skip_worktree() from this patch should be\ntogether with the other changes from 6.\n"},{"id":"433963","messageId":"CAHd-oW5JD56UerW6N18Ap9Dmt9eVyd7dZKUhJR6-hSjzzM0s5A@mail.gmail.com","threadId":"56370","inReplyTo":"6c9c986ff43fe7f065c27e61468534007e70d2a7.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/13] rm: add --sparse option","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:17:46Z","receivedAt":"2021-08-27T21:18:02Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> Subject: [PATCH 09/12] rm: add --sparse option\n\nMaybe mention in the commit message that, for now, rm's --sparse only\naffects entries with the skip_worktree bit set? (Which will be changed\nin the following patch.)\n\n> --- a/builtin/rm.c\n> +++ b/builtin/rm.c\n> @@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n> ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> const struct cache_entry *ce = active_cache[i];\n> - if (ce_skip_worktree(ce))\n> +\n> + if (!include_sparse && ce_skip_worktree(ce))\n> continue;\n\nOK, we no longer ignore the skip_worktree entry if --sparse is given ...\n\n> if (!ce_path_match(&the_index, ce, &pathspec, seen))\n> continue;\n> @@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n> seen_any = 1;\n> else if (ignore_unmatch)\n> continue;\n> - else if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n> + else if (!include_sparse &&\n> + matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n> string_list_append(&only_match_skip_worktree, original);\n\n... and we also don't need to look for matches here as we won't\ndisplay the error/advice match.\n\n> else\n> die(_(\"pathspec '%s' did not match any files\"), original);\n> diff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\n> index e9e9a15c74c..a34b978bfd8 100755\n> --- a/t/t3602-rm-sparse-checkout.sh\n> +++ b/t/t3602-rm-sparse-checkout.sh\n> @@ -36,13 +36,25 @@ done\n>\n> test_expect_success 'recursive rm does not remove sparse entries' '\n> git reset --hard &&\n> - git sparse-checkout set sub/dir &&\n> + git sparse-checkout set sub/dir/ &&\n\nIs this change necessary?\n\n> git rm -r sub &&\n> git status --porcelain -uno >actual &&\n> echo \"D sub/dir/e\" >expected &&\n> test_cmp expected actual\n> '\n>\n> +test_expect_success 'recursive rm --sparse removes sparse entries' '\n> + git reset --hard &&\n> + git sparse-checkout set \"sub/dir\" &&\n> + git rm --sparse -r sub &&\n> + git status --porcelain -uno >actual &&\n> + cat >expected <<-\\EOF &&\n> + D sub/d\n> + D sub/dir/e\n> + EOF\n> + test_cmp expected actual\n> +'\n\nNice!\n"},{"id":"433964","messageId":"CAHd-oW50QSBmtdNhwur=OcgcPnMBgszCp7aPbmwvz7qMPFj0iQ@mail.gmail.com","threadId":"56370","inReplyTo":"5153accded46c8ced8784ec135dbc77d5bc1a306.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/13] rm: skip sparse paths with missing SKIP_WORKTREE","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:18:20Z","receivedAt":"2021-08-27T21:18:35Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> diff --git a/builtin/rm.c b/builtin/rm.c\n> index 4208f3f9a5f..a6da03da2be 100644\n> --- a/builtin/rm.c\n> +++ b/builtin/rm.c\n> @@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n> for (i = 0; i < active_nr; i++) {\n> const struct cache_entry *ce = active_cache[i];\n>\n> - if (!include_sparse && ce_skip_worktree(ce))\n> + if (!include_sparse &&\n> + (ce_skip_worktree(ce) ||\n> + !path_in_sparse_checkout(ce->name, &the_index)))\n> continue;\n\nOK. And we don't need to update the advice reporting code below,\nbecause `matches_skip_worktree()` already uses the\n`(ce_skip_worktree(ce) || !path_in_sparse_checkout())` logic.\n\n> + !path_in_sparse_checkout\"\n\n> if (!ce_path_match(&the_index, ce, &pathspec, seen))\n> continue;\n> diff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\n> index a34b978bfd8..44f3e923164 100755\n> --- a/t/t3602-rm-sparse-checkout.sh\n> +++ b/t/t3602-rm-sparse-checkout.sh\n> @@ -87,4 +87,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n> git ls-files --error-unmatch b\n> '\n>\n> +test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '\n> + git reset --hard &&\n> + git sparse-checkout set a &&\n> + git update-index --no-skip-worktree b &&\n> + test_must_fail git rm b 2>stderr &&\n> + test_cmp b_error_and_hint stderr &&\n> + git rm --sparse b 2>stderr &&\n> + test_must_be_empty stderr &&\n> + test_path_is_missing b\n> +'\n\nOK.\n"},{"id":"433966","messageId":"CAHd-oW5-f2Kb5DR-UTfu1qB1fm63oHf62WYsbGd5ajZueOWHtA@mail.gmail.com","threadId":"56370","inReplyTo":"d31c641180645ee4059aab9230841ad90f9244fe.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 11/13] mv: refuse to move sparse paths","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T21:20:27Z","receivedAt":"2021-08-27T21:20:41Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> diff --git a/builtin/mv.c b/builtin/mv.c\n> index c2f96c8e895..b58fd4ce5ba 100644\n> --- a/builtin/mv.c\n> +++ b/builtin/mv.c\n> @@ -176,10 +177,22 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> const char *src = source[i], *dst = destination[i];\n> int length, src_is_dir;\n> const char *bad = NULL;\n> + int skip_sparse = 0;\n>\n> if (show_only)\n> printf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n>\n> + if (!path_in_sparse_checkout(src, &the_index)) {\n\n`git mv` can only move/rename tracked paths, but since we check\nwhether `src` is sparse before checking if it is in the index, the\nuser will get the sparse error message instead. This is OK, but the\nadvice might be misleading, as it says they can use `--sparse` if they\nreally want to move the file, but repeating the command with\n`--sparse` will now fail for another reason. I wonder if we should\ncheck whether `src` is tracked before checking if it is sparse, or if\nthat is not really an issue we should bother with.\n\n> + string_list_append(&only_match_skip_worktree, src);\n> + skip_sparse = 1;\n> + }\n> + if (!path_in_sparse_checkout(dst, &the_index)) {\n> + string_list_append(&only_match_skip_worktree, dst);\n> + skip_sparse = 1;\n> + }\n> + if (skip_sparse)\n> + continue;\n> +\n> length = strlen(src);\n> if (lstat(src, &st) < 0)\n> bad = _(\"bad source\");\n>\n> diff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\n> new file mode 100755\n> index 00000000000..5397c6d07bd\n> --- /dev/null\n> +++ b/t/t7002-mv-sparse-checkout.sh\n> @@ -0,0 +1,99 @@\n> +#!/bin/sh\n> +\n> +test_description='git mv in sparse working trees'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' \"\n> + mkdir -p sub/dir sub/dir2 &&\n> + touch a b c sub/d sub/dir/e sub/dir2/e &&\n> + git add -A &&\n> + git commit -m files &&\n> +\n> + cat >sparse_error_header <<-EOF &&\n> + The following pathspecs didn't match any eligible path, but they do match index\n> + entries outside the current sparse checkout:\n> + EOF\n> +\n> + cat >sparse_hint <<-EOF\n> + hint: Disable or modify the sparsity rules if you intend to update such entries.\n> + hint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n> + EOF\n> +\"\n> +\n> +test_expect_success 'mv refuses to move sparse-to-sparse' '\n> + rm -f e &&\n\nAt first glance, it confused me a bit that we are removing `e` when\nthe setup didn't create it. But then I realized the test itself might\ncreate `e` if `git mv` succeeds in moving the `b` file. Could perhaps\nthis and the others `rm -f e` be a `test_when_finished`, to make it\nclearer that it is a cleanup?\n\n> + git reset --hard &&\n> + git sparse-checkout set a &&\n> + touch b &&\n> + test_must_fail git mv b e 2>stderr &&\n\nHere we try to move a \"tracked sparse path\" to an \"untracked sparse\npath\". Do we also want to test with a tracked to tracked operation?\n(Although the code path will be the same, of course.)\n\n> + cat sparse_error_header >expect &&\n> + echo b >>expect &&\n> + echo e >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n> +\n> +test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n> + rm -f e &&\n> + git reset --hard &&\n> + git sparse-checkout set a &&\n> + touch b &&\n> + git mv -k b e 2>stderr &&\n\nMaybe also check that `b` is still there, and `e` is missing?\n\n> + cat sparse_error_header >expect &&\n> + echo b >>expect &&\n> + echo e >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n> +\n> +test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n> + rm -f e &&\n> + git reset --hard &&\n> + git sparse-checkout set a &&\n> + test_must_fail git mv a e 2>stderr &&\n> + cat sparse_error_header >expect &&\n> + echo e >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n\nOK.\n\n> +test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n> + rm -f e &&\n> + git reset --hard &&\n> + git sparse-checkout set a e &&\n> + touch b &&\n> + test_must_fail git mv b e 2>stderr &&\n> + cat sparse_error_header >expect &&\n> + echo b >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n\nOK.\n\n> +test_expect_success 'recursive mv refuses to move (possible) sparse' '\n> + rm -f e &&\n> + git reset --hard &&\n> + # Without cone mode, \"sub\" and \"sub2\" do not match\n> + git sparse-checkout set sub/dir sub2/dir &&\n> + test_must_fail git mv sub sub2 2>stderr &&\n> + cat sparse_error_header >expect &&\n> + echo sub >>expect &&\n> + echo sub2 >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n> +\n> +test_expect_success 'recursive mv refuses to move sparse' '\n> + git reset --hard &&\n> + # Use cone mode so \"sub/\" matches the sparse-checkout patterns\n> + git sparse-checkout init --cone &&\n> + git sparse-checkout set sub/dir sub2/dir &&\n> + test_must_fail git mv sub sub2 2>stderr &&\n> + cat sparse_error_header >expect &&\n> + echo sub/dir2/e >>expect &&\n> + echo sub2/dir2/e >>expect &&\n> + cat sparse_hint >>expect &&\n> + test_cmp expect stderr\n> +'\n> +\n\nAh, these last two are very interesting cases!\n"},{"id":"433976","messageId":"CAHd-oW5uzek9Ewdw+jXdw6L9WiwC-xz3O3tRr38HRruB8puDkg@mail.gmail.com","threadId":"56370","inReplyTo":"CAHd-oW7D1jikE5ByS36AjACfSJoZeekLCKzX2MRRfayKwKv=qQ@mail.gmail.com","subject":"Re: [PATCH 05/13] add: fail when adding an untracked sparse file","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T22:50:30Z","receivedAt":"2021-08-27T22:50:45Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Aug 27, 2021 at 6:06 PM Matheus Tavares Bernardino\n wrote:\n>\n> While I was playing with this patch, I did the following:\n>\n> echo a >a\n> echo b >b\n> git add .\n> git commit -m files\n> git sparse-checkout set a\n> echo c >c\n> git add c\n>\n> And the last `git add` was successful in adding the untracked `c` file\n> which is outside the sparse checkout. I'm not sure if I'm doing\n> something wrong, but it seems that `path_in_sparse_checkout()` returns\n> UNDECIDED for `c`. Is it because there was no pattern in the list\n> explicitly excluding it? And if so, should we consider UNDECIDED as\n> NOT_MATCHED for `path_in_sparse_checkout()`?\n\nPlease disconsider this, It was my fault indeed. I had applied the\npatches onto the wrong base. Now I fetched them again but from the GGG\ntag, and my manual test worked as expected.\n"},{"id":"433979","messageId":"CAHd-oW55fWkjf5494OJPsJGtRh59SDJFbfhw=nt2q=To1DQJqA@mail.gmail.com","threadId":"56370","inReplyTo":"CAHd-oW5-f2Kb5DR-UTfu1qB1fm63oHf62WYsbGd5ajZueOWHtA@mail.gmail.com","subject":"Re: [PATCH 11/13] mv: refuse to move sparse paths","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-27T23:44:44Z","receivedAt":"2021-08-27T23:44:58Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Fri, Aug 27, 2021 at 6:20 PM Matheus Tavares Bernardino\n wrote:\n>\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > diff --git a/builtin/mv.c b/builtin/mv.c\n> > index c2f96c8e895..b58fd4ce5ba 100644\n> > --- a/builtin/mv.c\n> > +++ b/builtin/mv.c\n> > @@ -176,10 +177,22 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> > const char *src = source[i], *dst = destination[i];\n> > int length, src_is_dir;\n> > const char *bad = NULL;\n> > + int skip_sparse = 0;\n> >\n> > if (show_only)\n> > printf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n> >\n> > + if (!path_in_sparse_checkout(src, &the_index)) {\n>\n> `git mv` can only move/rename tracked paths, but since we check\n> whether `src` is sparse before checking if it is in the index, the\n> user will get the sparse error message instead. This is OK, but the\n> advice might be misleading, as it says they can use `--sparse` if they\n> really want to move the file, but repeating the command with\n> `--sparse` will now fail for another reason. I wonder if we should\n> check whether `src` is tracked before checking if it is sparse, or if\n> that is not really an issue we should bother with.\n\nAnother problem is that the displayed message will say that the\npathspecs \"match index entries outside sparse checkout\" even when the\npath given to mv doesn't really exist:\n\ngit sparse-checkout set some/dir/\ngit mv nonexistent-file foo\n\nThe following pathspecs didn't match any eligible path, but they do match index\nentries outside the current sparse checkout:\nnonexistent-file\nhint: Disable or modify the sparsity rules if you intend to update such entries.\nhint: Disable this message with \"git config advice.updateSparsePath false\"\n"},{"id":"434019","messageId":"CAHd-oW4ANhC4QF9qWc==3BR1uMWjw-Qb4nNCYdd3s8shWwwT=w@mail.gmail.com","threadId":"56370","inReplyTo":"f6c0d4e3a0695c331b8216103bb46bfb4a461d1e.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 12/13] mv: add '--sparse' option to ignore sparse-checkout","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-08-28T14:18:40Z","receivedAt":"2021-08-28T14:27:29Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> diff --git a/builtin/mv.c b/builtin/mv.c\n> index b58fd4ce5ba..92ea9f0ca37 100644\n> --- a/builtin/mv.c\n> +++ b/builtin/mv.c\n> @@ -118,17 +118,18 @@ static int index_range_of_same_dir(const char *src, int length,\n> int cmd_mv(int argc, const char **argv, const char *prefix)\n> {\n> int i, flags, gitmodules_modified = 0;\n> - int verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n> + int verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n> struct option builtin_mv_options[] = {\n> OPT__VERBOSE(&verbose, N_(\"be verbose\")),\n> OPT__DRY_RUN(&show_only, N_(\"dry run\")),\n> OPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n> PARSE_OPT_NOCOMPLETE),\n> OPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n> + OPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n\nShould this include a doc update too?\n\n> OPT_END(),\n> };\n> const char **source, **destination, **dest_path, **submodule_gitfile;\n> - enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n> + enum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n> struct stat st;\n> struct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n> struct lock_file lock_file = LOCK_INIT;\n> @@ -182,11 +183,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> if (show_only)\n> printf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n>\n> - if (!path_in_sparse_checkout(src, &the_index)) {\n> + if (!ignore_sparse && !path_in_sparse_checkout(src, &the_index)) {\n> string_list_append(&only_match_skip_worktree, src);\n> skip_sparse = 1;\n> }\n> - if (!path_in_sparse_checkout(dst, &the_index)) {\n> + if (!ignore_sparse && !path_in_sparse_checkout(dst, &the_index)) {\n> string_list_append(&only_match_skip_worktree, dst);\n> skip_sparse = 1;\n> }\n> @@ -194,9 +195,11 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> continue;\n>\n> length = strlen(src);\n> - if (lstat(src, &st) < 0)\n> - bad = _(\"bad source\");\n> - else if (!strncmp(src, dst, length) &&\n> + if (lstat(src, &st) < 0) {\n> + /* only error if existence is expected. */\n> + if (modes[i] != SPARSE)\n> + bad = _(\"bad source\");\n\nOK, so this is about the directories which contain sparse entries in\nit, right? In this case, we don't expect such entries to be in the\nworking tree, so we don't error out if they are missing and still let\nthe parent directory be moved.\n\nThis made me wonder about a slightly different case: would it be\ninteresting to also allow `git mv --sparse` to rename a single sparse\nentry even if it's not in the working tree? I mean, something like:\n\necho a >a\necho b >b\ngit add a b\ngit commit -m files\ngit sparse-checkout set a\ngit mv --sparse b c\n\nThis currently wouldn't be allowed because \"b\" is not in the working\ntree (\"fatal: bad source\" error). But, perhaps, would it be\ninteresting to allow the index to be updated anyway?\n\n> + } else if (!strncmp(src, dst, length) &&\n> (dst[length] == 0 || dst[length] == '/')) {\n> bad = _(\"can not move directory into itself\");\n> } else if ((src_is_dir = S_ISDIR(st.st_mode))\n> @@ -225,11 +228,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> dst_len = strlen(dst);\n>\n> for (j = 0; j < last - first; j++) {\n> - const char *path = active_cache[first + j]->name;\n> + const struct cache_entry *ce = active_cache[first + j];\n> + const char *path = ce->name;\n> source[argc + j] = path;\n> destination[argc + j] =\n> prefix_path(dst, dst_len, path + length + 1);\n> - modes[argc + j] = INDEX;\n> + modes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n\nOK, here we are assigning the SPARSE mode to sparse index entries that\nare inside a directory we want to move. Later iterations of the loop\nwill then process these entries, see this mode, and not error out if\nthe files are missing.\n\n> submodule_gitfile[argc + j] = NULL;\n> }\n> argc += last - first;\n> @@ -293,7 +297,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n> printf(_(\"Renaming %s to %s\\n\"), src, dst);\n> if (show_only)\n> continue;\n> - if (mode != INDEX && rename(src, dst) < 0) {\n> + if (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n\nAnd finally, we don't want to rename the SPARSE working tree file (if\nit exists) because its parent directory was already moved.\n\n> if (ignore_errors)\n> continue;\n> die_errno(_(\"renaming '%s' failed\"), src);\n> diff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\n> index 5397c6d07bd..517fd587fa8 100755\n> --- a/t/t7002-mv-sparse-checkout.sh\n> +++ b/t/t7002-mv-sparse-checkout.sh\n> @@ -31,7 +31,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse' '\n> echo b >>expect &&\n> echo e >>expect &&\n> cat sparse_hint >>expect &&\n> - test_cmp expect stderr\n> + test_cmp expect stderr &&\n> + git mv --sparse b e 2>stderr &&\n> + test_must_be_empty stderr\n> '\n>\n> test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n> @@ -44,7 +46,9 @@ test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n> echo b >>expect &&\n> echo e >>expect &&\n> cat sparse_hint >>expect &&\n> - test_cmp expect stderr\n> + test_cmp expect stderr &&\n> + git mv --sparse -k b e 2>stderr &&\n> + test_must_be_empty stderr\n\nNit: Isn't this case a bit redundant considering the test before this\none? That is, with `--sparse` there should be no error for `-k` to\nignore, and in the test above it we already checked that this command\nindeed succeeds with `--sparse`.\n\n>\n> test_expect_success 'recursive mv refuses to move (possible) sparse' '\n> @@ -80,7 +88,14 @@ test_expect_success 'recursive mv refuses to move (possible) sparse' '\n> echo sub >>expect &&\n> echo sub2 >>expect &&\n> cat sparse_hint >>expect &&\n> - test_cmp expect stderr\n> + test_cmp expect stderr &&\n> + git mv --sparse sub sub2 2>stderr &&\n> + test_must_be_empty stderr &&\n> + git commit -m \"moved sub to sub2\" &&\n> + git rev-parse HEAD~1:sub >expect &&\n> + git rev-parse HEAD:sub2 >actual &&\n> + test_cmp expect actual &&\n> + git reset --hard HEAD~1\n\nPerhaps this could be a `test_when_finished` (maybe right after the\n`git commit` invocation), so that we can restore the original state\nfor the next tests even if this one fails?\n"},{"id":"435128","messageId":"2826d2dd-5beb-e595-a5bc-9ffb1ef653ea@gmail.com","threadId":"56370","inReplyTo":"CAHd-oW7D1jikE5ByS36AjACfSJoZeekLCKzX2MRRfayKwKv=qQ@mail.gmail.com","subject":"Re: [PATCH 05/13] add: fail when adding an untracked sparse file","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T17:54:39Z","receivedAt":"2021-09-08T17:54:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:06 PM, Matheus Tavares Bernardino wrote:\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n> wrote:\n\nThanks for adding your review. I'm sorry I'm so late getting back to it.\n\n>> From: Derrick Stolee \n>>\n>> The add_files() method in builtin/add.c takes a set of untracked files\n>> that are being added by the input pathspec and inserts them into the\n>> index. If these files are outside of the sparse-checkout cone, then they\n>> gain the SKIP_WORKTREE bit at some point. However, this was not checked\n>> before inserting into the index, so these files are added even though we\n>> want to avoid modifying the index outside of the sparse-checkout cone.\n>>\n>> Add a check within add_files() for these files and write the advice\n>> about files outside of the sprase-checkout cone.\n> \n> s/sprase/sparse/\n\nThanks.\n\n>> diff --git a/builtin/add.c b/builtin/add.c\n>> index 88a6c0c69fb..3a109276b74 100644\n>> --- a/builtin/add.c\n>> +++ b/builtin/add.c\n>> @@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n>> static int add_files(struct dir_struct *dir, int flags)\n>> {\n>> int i, exit_status = 0;\n>> + struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n> \n> I see this reuses the logic from cmd_add() and refresh(). But since we\n> are operating on untracked files here, perhaps we could replace\n> \"skip_worktree\" by \"sparse_paths\" or something similar?\n\nHow about \"matched_sparse_paths\" as a whole name swap? The earlier uses\ncared only if every match was sparse, but here we are actually looking\nat cases that are untracked, and the pathspec could also match other\nnon-sparse cases.\n\n>> +\n>> + if (only_match_skip_worktree.nr) {\n>> + advise_on_updating_sparse_paths(&only_match_skip_worktree);\n> \n> \n> Hmm, advise_on_updating_sparse_paths() takes a list of pathspecs that\n> only matched sparse paths, but here we are passing a list of actual\n> pathnames... Well, these are technically pathspecs too, but the advice\n> message may be confusing.\n> \n> For example, if we ran `git add *.c` on a repo with the untracked\n> files `d1/file.c` and `d2/file.c`, we will get:\n> \n> The following pathspecs didn't match any eligible path, but they do match index\n> entries outside the current sparse checkout:\n> d1/file.c\n> d2/file.c\n> \n> However, `d1/file.c` and `d2/file.c` are neither index entries nor the\n> pathspecs that the user has given to `git add`. So perhaps we need to\n> change the error/advice message?\n\nI think the advice should be modified to refer to paths and/or pathspecs,\nand also it is not completely correct anymore.\n\nInstead of\n\n The following pathspecs didn't match any eligible path, but they do match index\n entries outside the current sparse checkout:\n\nperhaps\n\n The following paths and/or pathspecs matched paths that exist outside of your\n sparse-checkout definition, so will not be updated in the index:\n\nI'm going to save this one for a new patch at the end to make sure it handles\nall of the cases involved in this series.\n\nThanks,\n-Stolee\n"},{"id":"435129","messageId":"cb7881d7-754e-05fe-16c3-bf2afac4fb3f@gmail.com","threadId":"56370","inReplyTo":"CAHd-oW5JD56UerW6N18Ap9Dmt9eVyd7dZKUhJR6-hSjzzM0s5A@mail.gmail.com","subject":"Re: [PATCH 09/13] rm: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T18:04:02Z","receivedAt":"2021-09-08T18:04:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:17 PM, Matheus Tavares Bernardino wrote:\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> Subject: [PATCH 09/12] rm: add --sparse option\n> \n> Maybe mention in the commit message that, for now, rm's --sparse only\n> affects entries with the skip_worktree bit set? (Which will be changed\n> in the following patch.)\n\nI will expand with these details.\n \n>> test_expect_success 'recursive rm does not remove sparse entries' '\n>> git reset --hard &&\n>> - git sparse-checkout set sub/dir &&\n>> + git sparse-checkout set sub/dir/ &&\n> \n> Is this change necessary?\n\nNo, it is not. Thanks.\n\n-Stolee\n"},{"id":"435136","messageId":"939972a1-ce80-f72c-d30c-b0ead351d2bb@gmail.com","threadId":"56370","inReplyTo":"CAHd-oW5-f2Kb5DR-UTfu1qB1fm63oHf62WYsbGd5ajZueOWHtA@mail.gmail.com","subject":"Re: [PATCH 11/13] mv: refuse to move sparse paths","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T18:41:25Z","receivedAt":"2021-09-08T18:41:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:20 PM, Matheus Tavares Bernardino wrote:\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> diff --git a/builtin/mv.c b/builtin/mv.c\n>> index c2f96c8e895..b58fd4ce5ba 100644\n>> --- a/builtin/mv.c\n>> +++ b/builtin/mv.c\n>> @@ -176,10 +177,22 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n>> const char *src = source[i], *dst = destination[i];\n>> int length, src_is_dir;\n>> const char *bad = NULL;\n>> + int skip_sparse = 0;\n>>\n>> if (show_only)\n>> printf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n>>\n>> + if (!path_in_sparse_checkout(src, &the_index)) {\n> \n> `git mv` can only move/rename tracked paths, but since we check\n> whether `src` is sparse before checking if it is in the index, the\n> user will get the sparse error message instead. This is OK, but the\n> advice might be misleading, as it says they can use `--sparse` if they\n> really want to move the file, but repeating the command with\n> `--sparse` will now fail for another reason. I wonder if we should\n> check whether `src` is tracked before checking if it is sparse, or if\n> that is not really an issue we should bother with.\n\nI will move the logic to the last possible place before \"accepting\"\nthe move, then add a comment detailing why it should be there.\n\n>> + string_list_append(&only_match_skip_worktree, src);\n>> + skip_sparse = 1;\n>> + }\n>> + if (!path_in_sparse_checkout(dst, &the_index)) {\n>> + string_list_append(&only_match_skip_worktree, dst);\n>> + skip_sparse = 1;\n>> + }\n>> + if (skip_sparse)\n>> + continue;\n>> +\n...\n>> +test_expect_success 'mv refuses to move sparse-to-sparse' '\n>> + rm -f e &&\n> \n> At first glance, it confused me a bit that we are removing `e` when\n> the setup didn't create it. But then I realized the test itself might\n> create `e` if `git mv` succeeds in moving the `b` file. Could perhaps\n> this and the others `rm -f e` be a `test_when_finished`, to make it\n> clearer that it is a cleanup?\n\ntest_when_finished is cleaner.\n\n> \n>> + git reset --hard &&\n>> + git sparse-checkout set a &&\n>> + touch b &&\n>> + test_must_fail git mv b e 2>stderr &&\n> \n> Here we try to move a \"tracked sparse path\" to an \"untracked sparse\n> path\". Do we also want to test with a tracked to tracked operation?\n> (Although the code path will be the same, of course.)\n\nI can expand these tests to include tracked and untracked targets.\n\n>> + cat sparse_error_header >expect &&\n>> + echo b >>expect &&\n>> + echo e >>expect &&\n>> + cat sparse_hint >>expect &&\n>> + test_cmp expect stderr\n>> +'\n>> +\n>> +test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n>> + rm -f e &&\n>> + git reset --hard &&\n>> + git sparse-checkout set a &&\n>> + touch b &&\n>> + git mv -k b e 2>stderr &&\n> \n> Maybe also check that `b` is still there, and `e` is missing?\n\nGood idea.\n\nIn fact, there is a problem that the '-k' gets around the\nprotections because it doesn't return 1 early. I'll fix this\nby jumping to the end of the loop which removes the entries\nfrom the arrays.\n\nThanks,\n-Stolee\n"},{"id":"435173","messageId":"28762f8f-d9b4-caf8-18b5-3cdbf6b59a7d@gmail.com","threadId":"56370","inReplyTo":"20210827211310.108999-1-matheus.bernardino@usp.br","subject":"Re: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T19:46:31Z","receivedAt":"2021-09-08T19:46:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:13 PM, Matheus Tavares wrote:\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget wrote:\n>>\n>> Subject: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone\n> \n> Perhaps this could be \"skip _tracked_ paths that ...\" (to help\n> differentiate the end goal of this patch from the previous one).\n> \n>> diff --git a/pathspec.c b/pathspec.c\n>> index 44306fdaca2..0e6e60fdc5a 100644\n>> --- a/pathspec.c\n>> +++ b/pathspec.c\n>> @@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n>> return;\n>> for (i = 0; i < istate->cache_nr; i++) {\n>> const struct cache_entry *ce = istate->cache[i];\n>> - if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n>> + if (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n>> + (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n> \n> Hmm, even though we skip the sparse paths here, cmd_add() will call\n> add_files_to_cache() at the end and still update these paths in the\n> index. I think there are two ways to fix this. We could either change\n> run_diff_files() to skip these paths (but I don't know how other callers\n> of this functions want to handle this, so maybe this needs to hide\n> behind an option flag):\n\nYou are absolutely right to point this out. I had missed this interaction.\nBut, this is also already broken. The patch below adds a check to show that\n'git add' does not add the sparse_entry, but it does (even when applied\nbefore any patch in this series). That is: all the modified tests fail\nafter this change. I'll work to fix this issue before the next version of\nthis series.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom 21dab466d221e8632d98553f5f1fa900a2d47c7f Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Wed, 8 Sep 2021 15:40:32 -0400\nSubject: [PATCH] t3705: test that 'sparse_entry' is unstaged\n\nThe tests in t3705-add-sparse-checkout.sh check to see how 'git add'\nbehaves with paths outside the sparse-checkout definition. These\ncurrently check to see if a given warning is present but not that the\nindex is not updated with the sparse entries. Add a new\n'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\ncorrectly.\n\nSigned-off-by: Derrick Stolee \n---\n t/t3705-add-sparse-checkout.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 2b1fd0d0eef..d31a0d4f550 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -36,6 +36,11 @@ setup_gitignore () {\n \tEOF\n }\n \n+test_sparse_entry_unstaged () {\n+\tgit status --porcelain >actual &&\n+\t! grep \"A sparse_entry\" actual\n+}\n+\n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n \tThe following pathspecs didn't match any eligible path, but they do match index\n@@ -55,6 +60,7 @@ test_expect_success 'git add does not remove sparse entries' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -73,6 +79,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n \trm sparse_entry &&\n \tsetup_gitignore &&\n \ttest_must_fail git add . 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \n \tcat sparse_error_header >expect &&\n \techo . >>expect &&\n@@ -88,6 +95,7 @@ do\n \t\tsetup_sparse_entry &&\n \t\techo modified >sparse_entry &&\n \t\ttest_must_fail git add $opt sparse_entry 2>stderr &&\n+\t\ttest_sparse_entry_unstaged &&\n \t\ttest_cmp error_and_hint stderr &&\n \t\ttest_sparse_entry_unchanged\n \t'\n@@ -98,6 +106,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n \tgit ls-files --debug sparse_entry | grep mtime >before &&\n \ttest-tool chmtime -60 sparse_entry &&\n \ttest_must_fail git add --refresh sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \tgit ls-files --debug sparse_entry | grep mtime >after &&\n \ttest_cmp before after\n@@ -106,6 +115,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n test_expect_success 'git add --chmod does not update sparse entries' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged &&\n \t! test -x sparse_entry\n@@ -116,6 +126,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\n \ttest_must_fail git add --renormalize sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -124,6 +135,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -148,6 +160,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp sparse_entry_error stderr\n \n '\n-- \n2.33.0.vfs.signtest.55.g904c8365e91\n"},{"id":"435174","messageId":"41852f4f-5b37-2527-c907-1de2a6d2018d@gmail.com","threadId":"56370","inReplyTo":"28762f8f-d9b4-caf8-18b5-3cdbf6b59a7d@gmail.com","subject":"Re: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T20:02:24Z","receivedAt":"2021-09-08T20:02:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2021 3:46 PM, Derrick Stolee wrote:\n> On 8/27/2021 5:13 PM, Matheus Tavares wrote:\n>> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget wrote:\n>>>\n>>> Subject: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone\n>>\n>> Perhaps this could be \"skip _tracked_ paths that ...\" (to help\n>> differentiate the end goal of this patch from the previous one).\n>>\n>>> diff --git a/pathspec.c b/pathspec.c\n>>> index 44306fdaca2..0e6e60fdc5a 100644\n>>> --- a/pathspec.c\n>>> +++ b/pathspec.c\n>>> @@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n>>> return;\n>>> for (i = 0; i < istate->cache_nr; i++) {\n>>> const struct cache_entry *ce = istate->cache[i];\n>>> - if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n>>> + if (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n>>> + (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n>>\n>> Hmm, even though we skip the sparse paths here, cmd_add() will call\n>> add_files_to_cache() at the end and still update these paths in the\n>> index. I think there are two ways to fix this. We could either change\n>> run_diff_files() to skip these paths (but I don't know how other callers\n>> of this functions want to handle this, so maybe this needs to hide\n>> behind an option flag):\n> \n> You are absolutely right to point this out. I had missed this interaction.\n> But, this is also already broken. The patch below adds a check to show that\n> 'git add' does not add the sparse_entry, but it does (even when applied\n> before any patch in this series). That is: all the modified tests fail\n> after this change. I'll work to fix this issue before the next version of\n> this series.\n\nOf course, the reason for the failures is because the 'sparse_entry' is\nstaged as part of setup_sparse_entry. Not a bug.\n\nThis makes things more difficult to test, so I'll look around for an\nalternative way to test that 'git add' is behaving correctly.\n\nThanks,\n-Stolee\n"},{"id":"435178","messageId":"011f2b86-c0d0-c473-36f2-a2b8fc566050@gmail.com","threadId":"56370","inReplyTo":"20210827211310.108999-1-matheus.bernardino@usp.br","subject":"Re: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-08T21:06:24Z","receivedAt":"2021-09-08T21:06:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/27/2021 5:13 PM, Matheus Tavares wrote:\n> On Tue, Aug 24, 2021 at 6:54 PM Derrick Stolee via GitGitGadget wrote:\n>>\n>> Subject: [PATCH 06/13] add: skip paths that are outside sparse-checkout cone\n> \n> Perhaps this could be \"skip _tracked_ paths that ...\" (to help\n> differentiate the end goal of this patch from the previous one).\n> \n>> diff --git a/pathspec.c b/pathspec.c\n>> index 44306fdaca2..0e6e60fdc5a 100644\n>> --- a/pathspec.c\n>> +++ b/pathspec.c\n>> @@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n>> return;\n>> for (i = 0; i < istate->cache_nr; i++) {\n>> const struct cache_entry *ce = istate->cache[i];\n>> - if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n>> + if (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n>> + (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n> \n> Hmm, even though we skip the sparse paths here, cmd_add() will call\n> add_files_to_cache() at the end and still update these paths in the\n> index. I think there are two ways to fix this. We could either change\n> run_diff_files() to skip these paths (but I don't know how other callers\n> of this functions want to handle this, so maybe this needs to hide\n> behind an option flag):\n\nOk, this sent me off on a tangent (see other replies) trying to show\nthat 'git add ' is still modifying index entries. I finally\nadded enough checks that this fails in my merge/cherry-pick/rebase tests\nfor conflicts outside of the sparse-checkout cone. Here is a test that I\ncan add to t3705 to get a failure:\n\ntest_expect_success 'git add fails outside of sparse-checkout definition' '\n\ttest_when_finished git sparse-checkout disable &&\n\ttest_commit a &&\n\tgit sparse-checkout init &&\n\tgit sparse-checkout set a &&\n\n\tgit update-index --no-skip-worktree sparse_entry &&\n\ttest_must_fail git add sparse_entry &&\n\ttest_sparse_entry_unstaged\n'\n \n> diff --git a/diff-lib.c b/diff-lib.c\n> index f9eadc4fc1..4245d7ead5 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -198,7 +198,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t\tcontinue;\n> \t\t}\n> \n> -\t\tif (ce_uptodate(ce) || ce_skip_worktree(ce))\n> +\t\tif (ce_uptodate(ce) || ce_skip_worktree(ce) ||\n> +\t\t !path_in_sparse_checkout(ce->name, istate))\n> \t\t\tcontinue;\n> \n> \t\t/*\n> \n> Or we could change the callback in add itself:\n> \n> diff --git a/builtin/add.c b/builtin/add.c\n> index f675bdeae4..3d7762aac2 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,\n> \tfor (i = 0; i < q->nr; i++) {\n> \t\tstruct diff_filepair *p = q->queue[i];\n> \t\tconst char *path = p->one->path;\n> +\n> +\t\tif (!path_in_sparse_checkout(path, &the_index))\n> +\t\t\tcontinue;\n> +\n> \t\tswitch (fix_unmerged_status(p, data)) {\n> \t\tdefault:\n> \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\n\nIf I use this second callback, then I have access to 'include_sparse'\nin the later change that adds the --sparse option.\n\n> I believe we also need to update a few other places to use the\n> `(ce_skip_worktree(ce) || !path_in_sparse_checkout())` logic in order to\n> avoid updating tracked sparse paths: chmod_pathspec() for add's --chmod\n> option, renormalize_tracked_files() for --renormalize, and\n> read-cache.c:refresh_index() for --refresh.\n\nOK, I'll make sure to include and test these cases.\n \n>> continue;\n>> ce_path_match(istate, ce, pathspec, seen);\n>> }\n> \n> Hmm, don't we also want to update\n> find_pathspecs_matching_skip_worktree() in this patch to use\n> path_in_sparse_checkout()? I see you did that in patch 8, but I think\n> this should be together with this current patch as, without it, we stop\n> adding tracked sparse paths but we print no error/advice message about\n> it.\n\nI'll merge the patches to avoid confusion.\n\nThanks,\n-Stolee\n"},{"id":"435630","messageId":"8aefce6254c0bcbbbca909a62d033c74c90f980b.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:17Z","receivedAt":"2021-09-12T13:23:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe tests in t3705-add-sparse-checkout.sh check to see how 'git add'\nbehaves with paths outside the sparse-checkout definition. These\ncurrently check to see if a given warning is present but not that the\nindex is not updated with the sparse entries. Add a new\n'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\ncorrectly.\n\nWe need to modify setup_sparse_entry to actually commit the sparse_entry\nfile so it exists at HEAD but is not already staged in the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 2b1fd0d0eef..af81b4b6846 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n \n@@ -36,6 +37,11 @@ setup_gitignore () {\n \tEOF\n }\n \n+test_sparse_entry_unstaged () {\n+\tgit status --porcelain >actual &&\n+\t! grep \"^M sparse_entry\\$\" actual\n+}\n+\n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n \tThe following pathspecs didn't match any eligible path, but they do match index\n@@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n \trm sparse_entry &&\n \tsetup_gitignore &&\n \ttest_must_fail git add . 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \n \tcat sparse_error_header >expect &&\n \techo . >>expect &&\n@@ -88,6 +96,7 @@ do\n \t\tsetup_sparse_entry &&\n \t\techo modified >sparse_entry &&\n \t\ttest_must_fail git add $opt sparse_entry 2>stderr &&\n+\t\ttest_sparse_entry_unstaged &&\n \t\ttest_cmp error_and_hint stderr &&\n \t\ttest_sparse_entry_unchanged\n \t'\n@@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n \tgit ls-files --debug sparse_entry | grep mtime >before &&\n \ttest-tool chmtime -60 sparse_entry &&\n \ttest_must_fail git add --refresh sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \tgit ls-files --debug sparse_entry | grep mtime >after &&\n \ttest_cmp before after\n@@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n test_expect_success 'git add --chmod does not update sparse entries' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged &&\n \t! test -x sparse_entry\n@@ -116,6 +127,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\n \ttest_must_fail git add --renormalize sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -124,6 +136,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -148,6 +161,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp sparse_entry_error stderr\n \n '\n-- \ngitgitgadget\n\n"},{"id":"435631","messageId":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.git.1629842085.gitgitgadget@gmail.com","subject":"[PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:16Z","receivedAt":"2021-09-12T13:23:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/mergies-with-sparse-index.\n\nAs requested, this series looks to update the behavior of git add, git rm,\nand git mv when they attempt to modify paths outside of the sparse-checkout\ncone. In particular, this care is expanded to not just cache entries with\nthe SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\ndefinition.\n\nThis means that commands that worked before this series can now fail. In\nparticular, if 'git merge' results in a conflict outside of the\nsparse-checkout cone, then 'git add ' will now fail.\n\nIn order to allow users to circumvent these protections, a new '--sparse'\noption is added that ignores the sparse-checkout patterns and the\nSKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\nassist with discovery of this option.\n\nThere is a subtle issue with git mv in that it does not check the index\nuntil it discovers a directory and then uses the index to find the contained\nentries. This means that in non-cone-mode patterns, a pattern such as\n\"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n\nIn order to allow for checking arbitrary paths against the sparse-checkout\npatterns, some changes to the underlying pattern matching code is required.\nIt turns out that there are some bugs in the methods as advertised, but\nthese bugs were never discovered because of the way methods like\nunpack_trees() will check a directory for a pattern match before checking\nits contained paths. Our new \"check patterns on-demand\" approach pokes holes\nin that approach, specifically with patterns that match entire directories.\n\n\nUpdates in v2\n=============\n\n * I got no complaints about these restrictions, so this is now a full\n series, not RFC.\n\n * Thanks to Matheus, several holes are filled with extra testing and\n bugfixes.\n\n * New patches add --chmod and --renormalize improvements. These are added\n after the --sparse option to make them be one change each.\n\nThanks, -Stolee\n\nDerrick Stolee (14):\n t3705: test that 'sparse_entry' is unstaged\n t1092: behavior for adding sparse files\n dir: extract directory-matching logic\n dir: select directories correctly\n dir: fix pattern matching on dirs\n add: fail when adding an untracked sparse file\n add: skip tracked paths outside sparse-checkout cone\n add: implement the --sparse option\n add: update --chmod to skip sparse paths\n add: update --renormalize to skip sparse paths\n rm: add --sparse option\n rm: skip sparse paths with missing SKIP_WORKTREE\n mv: refuse to move sparse paths\n advice: update message to suggest '--sparse'\n\n Documentation/git-add.txt | 9 +-\n Documentation/git-rm.txt | 6 +\n advice.c | 10 +-\n builtin/add.c | 32 +++-\n builtin/mv.c | 52 +++++--\n builtin/rm.c | 10 +-\n dir.c | 54 ++++++-\n pathspec.c | 5 +-\n t/t1091-sparse-checkout-builtin.sh | 4 +-\n t/t1092-sparse-checkout-compatibility.sh | 75 +++++++--\n t/t3602-rm-sparse-checkout.sh | 37 ++++-\n t/t3705-add-sparse-checkout.sh | 65 +++++++-\n t/t7002-mv-sparse-checkout.sh | 186 +++++++++++++++++++++++\n 13 files changed, 494 insertions(+), 51 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\n\nbase-commit: 516680ba7704c473bb21628aa19cabbd787df4db\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1018\n\nRange-diff vs v1:\n\n -: ----------- > 1: 8aefce6254c t3705: test that 'sparse_entry' is unstaged\n 1: 69ca54877b4 ! 2: 61c23dc59a6 t1092: behavior for adding sparse files\n @@ Commit message\n \n Add some tests to demonstrate the current behavior around adding files\n outside of the sparse-checkout cone. Currently, untracked files are\n - handled differently from tracked files. A futuer change will make these\n + handled differently from tracked files. A future change will make these\n cases be handled the same way.\n \n + Further expand checking that a failed 'git add' does not stage changes\n + to the index.\n +\n Signed-off-by: Derrick Stolee \n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n + \ttest_cmp sparse-checkout-err sparse-index-err\n + }\n + \n ++test_sparse_unstaged () {\n ++\tfile=$1 &&\n ++\tfor repo in sparse-checkout sparse-index\n ++\tdo\n ++\t\tgit -C $repo status --porcelain >$repo-out &&\n ++\t\t! grep \"^A $file\\$\" $repo-out &&\n ++\t\t! grep \"^M $file\\$\" $repo-out || return 1\n ++\tdone\n ++}\n ++\n + test_expect_success 'sparse-index contents' '\n + \tinit_repos &&\n + \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n +\trun_on_sparse ../edit-contents folder1/a &&\n +\trun_on_sparse ../edit-contents folder1/newfile &&\n +\ttest_sparse_match test_must_fail git add folder1/a &&\n ++\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\ttest_sparse_unstaged folder1/a &&\n +\ttest_sparse_match git add folder1/newfile\n +'\n +\n test_expect_success 'commit including unstaged changes' '\n \tinit_repos &&\n \n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outside sparse cone' '\n + \n + \t# Adding the path outside of the sparse-checkout cone should fail.\n + \ttest_sparse_match test_must_fail git add folder1/a &&\n ++\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\ttest_sparse_unstaged folder1/a &&\n + \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n ++\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\ttest_sparse_unstaged folder1/a &&\n + \n + \t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n + \ttest_sparse_match git add folder1/new &&\n 2: 8cca5bcf405 = 3: 747b52e7bde dir: extract directory-matching logic\n 3: d47c7a1cf2a = 4: 723de4e4258 dir: select directories correctly\n 4: 58e7df44982 = 5: ed2472d33f1 dir: fix pattern matching on dirs\n 5: e80fcfa932c ! 6: bac6cedfc8d add: fail when adding an untracked sparse file\n @@ Commit message\n want to avoid modifying the index outside of the sparse-checkout cone.\n \n Add a check within add_files() for these files and write the advice\n - about files outside of the sprase-checkout cone.\n + about files outside of the sparse-checkout cone.\n \n This behavior change modifies some existing tests within t1092. These\n tests intended to document how a user could interact with the existing\n @@ builtin/add.c: static void check_embedded_repo(const char *path)\n static int add_files(struct dir_struct *dir, int flags)\n {\n \tint i, exit_status = 0;\n -+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n ++\tstruct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;\n \n \tif (dir->ignored_nr) {\n \t\tfprintf(stderr, _(ignore_error));\n @@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)\n \n \tfor (i = 0; i < dir->nr; i++) {\n +\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n -+\t\t\tstring_list_append(&only_match_skip_worktree,\n ++\t\t\tstring_list_append(&matched_sparse_paths,\n +\t\t\t\t\t dir->entries[i]->name);\n +\t\t\tcontinue;\n +\t\t}\n @@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)\n \t\t}\n \t}\n +\n -+\tif (only_match_skip_worktree.nr) {\n -+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n ++\tif (matched_sparse_paths.nr) {\n ++\t\tadvise_on_updating_sparse_paths(&matched_sparse_paths);\n +\t\texit_status = 1;\n +\t}\n +\n -+\tstring_list_clear(&only_match_skip_worktree, 0);\n ++\tstring_list_clear(&matched_sparse_paths, 0);\n +\n \treturn exit_status;\n }\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n \tinit_repos &&\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add outside sparse cone' '\n - \trun_on_sparse ../edit-contents folder1/a &&\n - \trun_on_sparse ../edit-contents folder1/newfile &&\n \ttest_sparse_match test_must_fail git add folder1/a &&\n + \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \ttest_sparse_unstaged folder1/a &&\n -\ttest_sparse_match git add folder1/newfile\n -+\ttest_sparse_match test_must_fail git add folder1/newfile\n ++\ttest_sparse_match test_must_fail git add folder1/newfile &&\n ++\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\ttest_sparse_unstaged folder1/newfile\n '\n \n test_expect_success 'commit including unstaged changes' '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including\n \n \t# folder1 is at HEAD, but outside the sparse cone\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outside sparse cone' '\n - \t# Adding the path outside of the sparse-checkout cone should fail.\n - \ttest_sparse_match test_must_fail git add folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n + \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \ttest_sparse_unstaged folder1/a &&\n +\ttest_sparse_match test_must_fail git add folder1/new &&\n ++\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\ttest_sparse_unstaged folder1/new &&\n \n -\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n -\ttest_sparse_match git add folder1/new &&\n 6: eeba97ad492 < -: ----------- add: skip paths that are outside sparse-checkout cone\n 8: f1764f9ed18 ! 7: d4706e17cae add: prevent adding sparse conflict files\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - add: prevent adding sparse conflict files\n + add: skip tracked paths outside sparse-checkout cone\n \n - When a merge results in a conflict outside of the sparse-checkout cone,\n - the conflicted file is written to the working tree and the index entry\n - loses the SKIP_WORKTREE bit. This allows users to add the file to the\n - index without realizing that the file might leave the working tree in a\n - later Git command.\n + When 'git add' adds a tracked file that is outside of the\n + sparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file\n + exists outside of the sparse-checkout cone. This is usually correct,\n + except in the case of a merge conflict outside of the cone.\n \n - Block this behavior, but keep in mind that the user can override the\n - failure using the '--sparse' option.\n + Modify add_pathspec_matched_against_index() to be more careful about\n + paths by checking the sparse-checkout patterns in addition to the\n + SKIP_WORKTREE bit. This causes 'git add' to no longer allow files\n + outside of the cone that removed the SKIP_WORKTREE bit due to a merge\n + conflict.\n +\n + With only this change, users will only be able to add the file after\n + adding the file to the sparse-checkout cone. A later change will allow\n + users to force adding even though the file is outside of the\n + sparse-checkout cone.\n \n Signed-off-by: Derrick Stolee \n \n + ## builtin/add.c ##\n +@@ builtin/add.c: static void update_callback(struct diff_queue_struct *q,\n + \tfor (i = 0; i < q->nr; i++) {\n + \t\tstruct diff_filepair *p = q->queue[i];\n + \t\tconst char *path = p->one->path;\n ++\n ++\t\tif (!path_in_sparse_checkout(path, &the_index))\n ++\t\t\tcontinue;\n ++\n + \t\tswitch (fix_unmerged_status(p, data)) {\n + \t\tdefault:\n + \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\n +\n ## pathspec.c ##\n +@@ pathspec.c: void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n + \t\treturn;\n + \tfor (i = 0; i < istate->cache_nr; i++) {\n + \t\tconst struct cache_entry *ce = istate->cache[i];\n +-\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n ++\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n ++\t\t (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n + \t\t\tcontinue;\n + \t\tce_path_match(istate, ce, pathspec, seen);\n + \t}\n @@ pathspec.c: char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n @@ t/t1091-sparse-checkout-builtin.sh: test_expect_success 'sparse-checkout reapply\n \ttest_must_be_empty err &&\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge with conflict outside cone' '\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n -\t# NEEDSWORK: Even though the merge conflict removed the\n -\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n --\t# warn that this is a problematic add when --sparse is not set.\n +-\t# warn that this is a problematic add.\n +-\ttest_all_match git add folder1/a &&\n +\ttest_sparse_match test_must_fail git add folder1/a &&\n +\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n - \ttest_all_match git add --sparse folder1/a &&\n ++\ttest_sparse_unstaged folder1/a &&\n ++\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n -@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge with conflict outside cone' '\n - \t# accept conflict markers as resolved content.\n - \trun_on_all mv folder2/a folder2/z &&\n - \ttest_sparse_match test_must_fail git add folder2 &&\n + \t# 3. Rename the file to another sparse filename and\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n + \t# NEEDSWORK: This mode now fails, because folder2/z is\n + \t# outside of the sparse-checkout cone and does not match an\n + \t# existing index entry with the SKIP_WORKTREE bit cleared.\n +-\ttest_all_match git add folder2 &&\n ++\ttest_sparse_match test_must_fail git add folder2 &&\n +\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n - \ttest_all_match git add --sparse folder2 &&\n ++\ttest_sparse_unstaged folder2/z &&\n ++\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n + \ttest_all_match git merge --continue &&\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n + \t\t# NEEDSWORK: Even though the merge conflict removed the\n + \t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n + \t\t# warn that this is a problematic add.\n +-\t\ttest_all_match git add folder1/a &&\n ++\t\ttest_sparse_match test_must_fail git add folder1/a &&\n ++\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\t\ttest_sparse_unstaged folder1/a &&\n + \t\ttest_all_match git status --porcelain=v2 &&\n + \n + \t\t# 3. Rename the file to another sparse filename and\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n + \t\t# outside of the sparse-checkout cone and does not match an\n + \t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n + \t\trun_on_all mv folder2/a folder2/z &&\n +-\t\ttest_all_match git add folder2 &&\n ++\t\ttest_sparse_match test_must_fail git add folder2 &&\n ++\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\t\ttest_sparse_unstaged folder2/z &&\n + \t\ttest_all_match git status --porcelain=v2 &&\n + \n + \t\ttest_all_match git $OPERATION --continue &&\n +\n + ## t/t3705-add-sparse-checkout.sh ##\n +@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'do not warn when pathspec matches dense entries' '\n + \tgit ls-files --error-unmatch dense_entry\n + '\n + \n ++test_expect_success 'git add fails outside of sparse-checkout definition' '\n ++\ttest_when_finished git sparse-checkout disable &&\n ++\ttest_commit a &&\n ++\tgit sparse-checkout init &&\n ++\tgit sparse-checkout set a &&\n ++\techo >>sparse_entry &&\n ++\n ++\tgit update-index --no-skip-worktree sparse_entry &&\n ++\ttest_must_fail git add sparse_entry &&\n ++\ttest_sparse_entry_unstaged\n ++'\n ++\n + test_expect_success 'add obeys advice.updateSparsePath' '\n + \tsetup_sparse_entry &&\n + \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n 7: 6edadbc2000 ! 8: 944ae2cffa8 add: implement the --sparse option\n @@ builtin/add.c: static int chmod_pathspec(struct pathspec *pathspec, char flip, i\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n +@@ builtin/add.c: static void update_callback(struct diff_queue_struct *q,\n + \t\tstruct diff_filepair *p = q->queue[i];\n + \t\tconst char *path = p->one->path;\n + \n +-\t\tif (!path_in_sparse_checkout(path, &the_index))\n ++\t\tif (!include_sparse && !path_in_sparse_checkout(path, &the_index))\n + \t\t\tcontinue;\n + \n + \t\tswitch (fix_unmerged_status(p, data)) {\n @@ builtin/add.c: static struct option builtin_add_options[] = {\n \tOPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n \tOPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n @@ builtin/add.c: static int add_files(struct dir_struct *dir, int flags)\n -\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n +\t\tif (!include_sparse &&\n +\t\t !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n - \t\t\tstring_list_append(&only_match_skip_worktree,\n + \t\t\tstring_list_append(&matched_sparse_paths,\n \t\t\t\t\t dir->entries[i]->name);\n \t\t\tcontinue;\n @@ builtin/add.c: int cmd_add(int argc, const char **argv, const char *prefix)\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including\n \n \t# folder1 is at HEAD, but outside the sparse cone\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'status/add: outside sparse cone' '\n - \ttest_sparse_match test_must_fail git add folder1/a &&\n - \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n \ttest_sparse_match test_must_fail git add folder1/new &&\n + \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \ttest_sparse_unstaged folder1/new &&\n +\ttest_sparse_match git add --sparse folder1/a &&\n +\ttest_sparse_match git add --sparse folder1/new &&\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge, cherry-pic\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n - \t# 2. Add the file with conflict markers\n - \t# NEEDSWORK: Even though the merge conflict removed the\n - \t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n --\t# warn that this is a problematic add.\n --\ttest_sparse_match test_must_fail git add folder1/a &&\n -+\t# warn that this is a problematic add when --sparse is not set.\n -+\ttest_all_match git add --sparse folder1/a &&\n - \ttest_all_match git status --porcelain=v2 &&\n - \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli\n -\t# outside of the sparse-checkout cone and does not match an\n -\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n -+\ttest_all_match git add --sparse folder2 &&\n - \ttest_all_match git status --porcelain=v2 &&\n + \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \ttest_sparse_unstaged folder2/z &&\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n + \ttest_all_match git rev-parse HEAD^{tree}\n + '\n + \n +-test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n ++test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n + \tinit_repos &&\n \n - \ttest_all_match git merge --continue &&\n + \tfor OPERATION in cherry-pick rebase\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n + \t\ttest_sparse_match test_must_fail git add folder1/a &&\n + \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \t\ttest_sparse_unstaged folder1/a &&\n ++\t\ttest_all_match git add --sparse folder1/a &&\n + \t\ttest_all_match git status --porcelain=v2 &&\n + \n + \t\t# 3. Rename the file to another sparse filename and\n +@@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n + \t\ttest_sparse_match test_must_fail git add folder2 &&\n + \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \t\ttest_sparse_unstaged folder2/z &&\n ++\t\ttest_all_match git add --sparse folder2 &&\n + \t\ttest_all_match git status --porcelain=v2 &&\n + \n + \t\ttest_all_match git $OPERATION --continue &&\n \n ## t/t3705-add-sparse-checkout.sh ##\n +@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'git add fails outside of sparse-checkout definition' '\n + \n + \tgit update-index --no-skip-worktree sparse_entry &&\n + \ttest_must_fail git add sparse_entry &&\n +-\ttest_sparse_entry_unstaged\n ++\ttest_sparse_entry_unstaged &&\n ++\n ++\t# Avoid munging CRLFs to avoid an error message\n ++\tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\ttest-tool read-cache --table >actual &&\n ++\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n + '\n + \n + test_expect_success 'add obeys advice.updateSparsePath' '\n @@ t/t3705-add-sparse-checkout.sh: test_expect_success 'add obeys advice.updateSparsePath' '\n \n '\n @@ t/t3705-add-sparse-checkout.sh: test_expect_success 'add obeys advice.updateSpar\n +\tgit sparse-checkout set a &&\n +\techo modified >sparse_entry &&\n +\ttest_must_fail git add sparse_entry &&\n ++\ttest_sparse_entry_unchanged &&\n +\tgit add --sparse sparse_entry 2>stderr &&\n +\ttest_must_be_empty stderr\n +'\n -: ----------- > 9: 7ace030c709 add: update --chmod to skip sparse paths\n -: ----------- > 10: f34be166097 add: update --renormalize to skip sparse paths\n 9: 6c9c986ff43 ! 11: fb3f86efa9b rm: add --sparse option\n @@ Metadata\n ## Commit message ##\n rm: add --sparse option\n \n + As we did previously in 'git add', add a '--sparse' option to 'git rm'\n + that allows modifying paths outside of the sparse-checkout definition.\n + The existing checks in 'git rm' are restricted to tracked files that\n + have the SKIP_WORKTREE bit in the current index. Future changes will\n + cause 'git rm' to reject removing paths outside of the sparse-checkout\n + definition, even if they are untracked or do not have the SKIP_WORKTREE\n + bit.\n +\n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-rm.txt ##\n @@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\t\tdie(_(\"pathspec '%s' did not match any files\"), original);\n \n ## t/t3602-rm-sparse-checkout.sh ##\n -@@ t/t3602-rm-sparse-checkout.sh: done\n - \n - test_expect_success 'recursive rm does not remove sparse entries' '\n - \tgit reset --hard &&\n --\tgit sparse-checkout set sub/dir &&\n -+\tgit sparse-checkout set sub/dir/ &&\n - \tgit rm -r sub &&\n - \tgit status --porcelain -uno >actual &&\n - \techo \"D sub/dir/e\" >expected &&\n +@@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'recursive rm does not remove sparse entries' '\n \ttest_cmp expected actual\n '\n \n 10: 5153accded4 ! 12: de0c9d09ef7 rm: skip sparse paths with missing SKIP_WORKTREE\n @@ Commit message\n if the '--sparse' option is specified, allowing users who accept the\n risks to continue with the removal.\n \n + This also removes a confusing behavior where a user asks for a directory\n + to be removed, but only the entries that are within the sparse-checkout\n + definition are removed. Now, 'git rm ' will fail without '--sparse'\n + and will succeed in removing all contained paths with '--sparse'.\n +\n Signed-off-by: Derrick Stolee \n \n ## builtin/rm.c ##\n @@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \n ## t/t3602-rm-sparse-checkout.sh ##\n +@@ t/t3602-rm-sparse-checkout.sh: done\n + test_expect_success 'recursive rm does not remove sparse entries' '\n + \tgit reset --hard &&\n + \tgit sparse-checkout set sub/dir &&\n +-\tgit rm -r sub &&\n ++\ttest_must_fail git rm -r sub &&\n ++\tgit rm --sparse -r sub &&\n + \tgit status --porcelain -uno >actual &&\n +-\techo \"D sub/dir/e\" >expected &&\n ++\tcat >expected <<-\\EOF &&\n ++\tD sub/d\n ++\tD sub/dir/e\n ++\tEOF\n + \ttest_cmp expected actual\n + '\n + \n @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n \tgit ls-files --error-unmatch b\n '\n 11: d31c6411806 ! 13: e17a529332d mv: refuse to move sparse paths\n @@ Commit message\n mechanism for seeing if a path is sparse or not. Be sure to skip\n returning a failure if '-k' is specified.\n \n + To ensure that the advice around sparse paths is the only reason a move\n + failed, be sure to check this as the very last thing before inserting\n + into the src_for_dst list.\n +\n + The tests cover a variety of cases such as whether the target is tracked\n + or untracked, and whether the source or destination are in or outside of\n + the sparse-checkout definition.\n +\n + Helped-by: Matheus Tavares Bernardino \n Signed-off-by: Derrick Stolee \n \n ## builtin/mv.c ##\n -@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n +@@ builtin/mv.c: static int index_range_of_same_dir(const char *src, int length,\n + int cmd_mv(int argc, const char **argv, const char *prefix)\n + {\n + \tint i, flags, gitmodules_modified = 0;\n +-\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n ++\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n + \tstruct option builtin_mv_options[] = {\n + \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n + \t\tOPT__DRY_RUN(&show_only, N_(\"dry run\")),\n + \t\tOPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n + \t\t\t PARSE_OPT_NOCOMPLETE),\n + \t\tOPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n ++\t\tOPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n + \t\tOPT_END(),\n + \t};\n + \tconst char **source, **destination, **dest_path, **submodule_gitfile;\n +-\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n ++\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n + \tstruct stat st;\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct cache_entry *ce;\n @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n -+\t\tif (!path_in_sparse_checkout(src, &the_index)) {\n -+\t\t\tstring_list_append(&only_match_skip_worktree, src);\n -+\t\t\tskip_sparse = 1;\n -+\t\t}\n -+\t\tif (!path_in_sparse_checkout(dst, &the_index)) {\n -+\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n -+\t\t\tskip_sparse = 1;\n -+\t\t}\n -+\t\tif (skip_sparse)\n -+\t\t\tcontinue;\n -+\n \t\tlength = strlen(src);\n - \t\tif (lstat(src, &st) < 0)\n - \t\t\tbad = _(\"bad source\");\n +-\t\tif (lstat(src, &st) < 0)\n +-\t\t\tbad = _(\"bad source\");\n +-\t\telse if (!strncmp(src, dst, length) &&\n ++\t\tif (lstat(src, &st) < 0) {\n ++\t\t\t/* only error if existence is expected. */\n ++\t\t\tif (modes[i] != SPARSE)\n ++\t\t\t\tbad = _(\"bad source\");\n ++\t\t} else if (!strncmp(src, dst, length) &&\n + \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n + \t\t\tbad = _(\"can not move directory into itself\");\n + \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n + \t\t\t\tdst_len = strlen(dst);\n + \n + \t\t\t\tfor (j = 0; j < last - first; j++) {\n +-\t\t\t\t\tconst char *path = active_cache[first + j]->name;\n ++\t\t\t\t\tconst struct cache_entry *ce = active_cache[first + j];\n ++\t\t\t\t\tconst char *path = ce->name;\n + \t\t\t\t\tsource[argc + j] = path;\n + \t\t\t\t\tdestination[argc + j] =\n + \t\t\t\t\t\tprefix_path(dst, dst_len, path + length + 1);\n +-\t\t\t\t\tmodes[argc + j] = INDEX;\n ++\t\t\t\t\tmodes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n + \t\t\t\t\tsubmodule_gitfile[argc + j] = NULL;\n + \t\t\t\t}\n + \t\t\t\targc += last - first;\n +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n + \t\t\tbad = _(\"multiple sources for the same target\");\n + \t\telse if (is_dir_sep(dst[strlen(dst) - 1]))\n + \t\t\tbad = _(\"destination directory does not exist\");\n +-\t\telse\n ++\t\telse {\n ++\t\t\t/*\n ++\t\t\t * We check if the paths are in the sparse-checkout\n ++\t\t\t * definition as a very final check, since that\n ++\t\t\t * allows us to point the user to the --sparse\n ++\t\t\t * option as a way to have a successful run.\n ++\t\t\t */\n ++\t\t\tif (!ignore_sparse &&\n ++\t\t\t !path_in_sparse_checkout(src, &the_index)) {\n ++\t\t\t\tstring_list_append(&only_match_skip_worktree, src);\n ++\t\t\t\tskip_sparse = 1;\n ++\t\t\t}\n ++\t\t\tif (!ignore_sparse &&\n ++\t\t\t !path_in_sparse_checkout(dst, &the_index)) {\n ++\t\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n ++\t\t\t\tskip_sparse = 1;\n ++\t\t\t}\n ++\n ++\t\t\tif (skip_sparse)\n ++\t\t\t\tgoto remove_entry;\n ++\n + \t\t\tstring_list_insert(&src_for_dst, dst);\n ++\t\t}\n + \n + \t\tif (!bad)\n + \t\t\tcontinue;\n + \t\tif (!ignore_errors)\n + \t\t\tdie(_(\"%s, source=%s, destination=%s\"),\n + \t\t\t bad, src, dst);\n ++remove_entry:\n + \t\tif (--argc > 0) {\n + \t\t\tint n = argc - i;\n + \t\t\tmemmove(source + i, source + i + 1,\n @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n @@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tenum update_mode mode = modes[i];\n +@@ builtin/mv.c: int cmd_mv(int argc, const char **argv, const char *prefix)\n + \t\t\tprintf(_(\"Renaming %s to %s\\n\"), src, dst);\n + \t\tif (show_only)\n + \t\t\tcontinue;\n +-\t\tif (mode != INDEX && rename(src, dst) < 0) {\n ++\t\tif (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n + \t\t\tif (ignore_errors)\n + \t\t\t\tcontinue;\n + \t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\n \n ## t/t7002-mv-sparse-checkout.sh (new) ##\n @@\n @@ t/t7002-mv-sparse-checkout.sh (new)\n +\"\n +\n +test_expect_success 'mv refuses to move sparse-to-sparse' '\n -+\trm -f e &&\n ++\ttest_when_finished rm -f e &&\n +\tgit reset --hard &&\n +\tgit sparse-checkout set a &&\n +\ttouch b &&\n @@ t/t7002-mv-sparse-checkout.sh (new)\n +\techo b >>expect &&\n +\techo e >>expect &&\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse b e 2>stderr &&\n ++\ttest_must_be_empty stderr\n +'\n +\n +test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n -+\trm -f e &&\n ++\ttest_when_finished rm -f b c e &&\n +\tgit reset --hard &&\n +\tgit sparse-checkout set a &&\n ++\n ++\t# tracked-to-untracked\n +\ttouch b &&\n +\tgit mv -k b e 2>stderr &&\n ++\ttest_path_exists b &&\n ++\ttest_path_is_missing e &&\n +\tcat sparse_error_header >expect &&\n +\techo b >>expect &&\n +\techo e >>expect &&\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\n ++\tgit mv --sparse b e 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\ttest_path_is_missing b &&\n ++\ttest_path_exists e &&\n ++\n ++\t# tracked-to-tracked\n ++\tgit reset --hard &&\n ++\ttouch b &&\n ++\tgit mv -k b c 2>stderr &&\n ++\ttest_path_exists b &&\n ++\ttest_path_is_missing c &&\n ++\tcat sparse_error_header >expect &&\n ++\techo b >>expect &&\n ++\techo c >>expect &&\n ++\tcat sparse_hint >>expect &&\n ++\ttest_cmp expect stderr &&\n ++\n ++\tgit mv --sparse b c 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\ttest_path_is_missing b &&\n ++\ttest_path_exists c\n +'\n +\n +test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n -+\trm -f e &&\n ++\ttest_when_finished rm -f b c e &&\n +\tgit reset --hard &&\n +\tgit sparse-checkout set a &&\n ++\n ++\t# tracked-to-untracked\n +\ttest_must_fail git mv a e 2>stderr &&\n ++\ttest_path_exists a &&\n ++\ttest_path_is_missing e &&\n +\tcat sparse_error_header >expect &&\n +\techo e >>expect &&\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse a e 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\ttest_path_is_missing a &&\n ++\ttest_path_exists e &&\n ++\n ++\t# tracked-to-tracked\n ++\trm e &&\n ++\tgit reset --hard &&\n ++\ttest_must_fail git mv a c 2>stderr &&\n ++\ttest_path_exists a &&\n ++\ttest_path_is_missing c &&\n ++\tcat sparse_error_header >expect &&\n ++\techo c >>expect &&\n ++\tcat sparse_hint >>expect &&\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse a c 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\ttest_path_is_missing a &&\n ++\ttest_path_exists c\n +'\n +\n +test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n -+\trm -f e &&\n ++\ttest_when_finished rm -f b c e &&\n +\tgit reset --hard &&\n +\tgit sparse-checkout set a e &&\n ++\n ++\t# tracked-to-untracked\n +\ttouch b &&\n +\ttest_must_fail git mv b e 2>stderr &&\n +\tcat sparse_error_header >expect &&\n +\techo b >>expect &&\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse b e 2>stderr &&\n ++\ttest_must_be_empty stderr\n +'\n +\n +test_expect_success 'recursive mv refuses to move (possible) sparse' '\n -+\trm -f e &&\n ++\ttest_when_finished rm -rf b c e sub2 &&\n +\tgit reset --hard &&\n +\t# Without cone mode, \"sub\" and \"sub2\" do not match\n +\tgit sparse-checkout set sub/dir sub2/dir &&\n ++\n ++\t# Add contained contents to ensure we avoid non-existence errors\n ++\tmkdir sub/dir2 &&\n ++\ttouch sub/d sub/dir2/e &&\n ++\n +\ttest_must_fail git mv sub sub2 2>stderr &&\n +\tcat sparse_error_header >expect &&\n -+\techo sub >>expect &&\n -+\techo sub2 >>expect &&\n ++\tcat >>expect <<-\\EOF &&\n ++\tsub/d\n ++\tsub2/d\n ++\tsub/dir/e\n ++\tsub2/dir/e\n ++\tsub/dir2/e\n ++\tsub2/dir2/e\n ++\tEOF\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse sub sub2 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\tgit commit -m \"moved sub to sub2\" &&\n ++\tgit rev-parse HEAD~1:sub >expect &&\n ++\tgit rev-parse HEAD:sub2 >actual &&\n ++\ttest_cmp expect actual &&\n ++\tgit reset --hard HEAD~1\n +'\n +\n +test_expect_success 'recursive mv refuses to move sparse' '\n @@ t/t7002-mv-sparse-checkout.sh (new)\n +\t# Use cone mode so \"sub/\" matches the sparse-checkout patterns\n +\tgit sparse-checkout init --cone &&\n +\tgit sparse-checkout set sub/dir sub2/dir &&\n ++\n ++\t# Add contained contents to ensure we avoid non-existence errors\n ++\tmkdir sub/dir2 &&\n ++\ttouch sub/dir2/e &&\n ++\n +\ttest_must_fail git mv sub sub2 2>stderr &&\n +\tcat sparse_error_header >expect &&\n -+\techo sub/dir2/e >>expect &&\n -+\techo sub2/dir2/e >>expect &&\n ++\tcat >>expect <<-\\EOF &&\n ++\tsub/dir2/e\n ++\tsub2/dir2/e\n ++\tEOF\n +\tcat sparse_hint >>expect &&\n -+\ttest_cmp expect stderr\n ++\ttest_cmp expect stderr &&\n ++\tgit mv --sparse sub sub2 2>stderr &&\n ++\ttest_must_be_empty stderr &&\n ++\tgit commit -m \"moved sub to sub2\" &&\n ++\tgit rev-parse HEAD~1:sub >expect &&\n ++\tgit rev-parse HEAD:sub2 >actual &&\n ++\ttest_cmp expect actual &&\n ++\tgit reset --hard HEAD~1\n +'\n +\n +test_done\n 12: f6c0d4e3a06 < -: ----------- mv: add '--sparse' option to ignore sparse-checkout\n 13: 7749a69a38f ! 14: f2abc387822 advice: update message to suggest '--sparse'\n @@ Commit message\n To improve the discoverability of this new functionality, add a message\n to advice.updateSparsePath that mentions the existence of the option.\n \n + The previous set of changes also modified the purpose of this message to\n + include possibly a list of paths instead of only a list of pathspecs.\n + Make the warning message more clear about this new behavior.\n +\n Signed-off-by: Derrick Stolee \n \n ## advice.c ##\n @@ advice.c: void advise_on_updating_sparse_paths(struct string_list *pathspec_list)\n + \tif (!pathspec_list->nr)\n + \t\treturn;\n + \n +-\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n +-\t\t\t \" eligible path, but they do match index\\n\"\n +-\t\t\t \"entries outside the current sparse checkout:\\n\"));\n ++\tfprintf(stderr, _(\"The following paths and/or pathspecs matched \"\n ++\t\t\t \"paths that exist outside of your\\n\"\n ++\t\t\t \"sparse-checkout definition, so will not be \"\n ++\t\t\t \"updated in the index:\\n\"));\n + \tfor_each_string_list_item(item, pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n @@ advice.c: void advise_on_updating_sparse_paths(struct string_list *pathspec_list\n \n ## t/t3602-rm-sparse-checkout.sh ##\n @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' \"\n + \tgit commit -m files &&\n + \n + \tcat >sparse_error_header <<-EOF &&\n +-\tThe following pathspecs didn't match any eligible path, but they do match index\n +-\tentries outside the current sparse checkout:\n ++\tThe following paths and/or pathspecs matched paths that exist outside of your\n ++\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' \"\n \n \n ## t/t3705-add-sparse-checkout.sh ##\n -@@ t/t3705-add-sparse-checkout.sh: test_expect_success 'setup' \"\n +@@ t/t3705-add-sparse-checkout.sh: test_sparse_entry_unstaged () {\n + \n + test_expect_success 'setup' \"\n + \tcat >sparse_error_header <<-EOF &&\n +-\tThe following pathspecs didn't match any eligible path, but they do match index\n +-\tentries outside the current sparse checkout:\n ++\tThe following paths and/or pathspecs matched paths that exist outside of your\n ++\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n @@ t/t3705-add-sparse-checkout.sh: test_expect_success 'setup' \"\n \n ## t/t7002-mv-sparse-checkout.sh ##\n @@ t/t7002-mv-sparse-checkout.sh: test_expect_success 'setup' \"\n + \tgit commit -m files &&\n + \n + \tcat >sparse_error_header <<-EOF &&\n +-\tThe following pathspecs didn't match any eligible path, but they do match index\n +-\tentries outside the current sparse checkout:\n ++\tThe following paths and/or pathspecs matched paths that exist outside of your\n ++\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n\n-- \ngitgitgadget\n"},{"id":"435632","messageId":"61c23dc59a6e062e073ba3bea135c370a78415aa.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 02/14] t1092: behavior for adding sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:18Z","receivedAt":"2021-09-12T13:23:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests to demonstrate the current behavior around adding files\noutside of the sparse-checkout cone. Currently, untracked files are\nhandled differently from tracked files. A future change will make these\ncases be handled the same way.\n\nFurther expand checking that a failed 'git add' does not stage changes\nto the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..5b3f55e355e 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -187,6 +187,16 @@ test_sparse_match () {\n \ttest_cmp sparse-checkout-err sparse-index-err\n }\n \n+test_sparse_unstaged () {\n+\tfile=$1 &&\n+\tfor repo in sparse-checkout sparse-index\n+\tdo\n+\t\tgit -C $repo status --porcelain >$repo-out &&\n+\t\t! grep \"^A $file\\$\" $repo-out &&\n+\t\t! grep \"^M $file\\$\" $repo-out || return 1\n+\tdone\n+}\n+\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n@@ -291,6 +301,20 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+# NEEDSWORK: This documents current behavior, but is not a desirable\n+# behavior (untracked files are handled differently than tracked).\n+test_expect_success 'add outside sparse cone' '\n+\tinit_repos &&\n+\n+\trun_on_sparse mkdir folder1 &&\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_sparse ../edit-contents folder1/newfile &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match git add folder1/newfile\n+'\n+\n test_expect_success 'commit including unstaged changes' '\n \tinit_repos &&\n \n@@ -339,7 +363,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \n \t# Adding the path outside of the sparse-checkout cone should fail.\n \ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \n \t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n \ttest_sparse_match git add folder1/new &&\n-- \ngitgitgadget\n\n"},{"id":"435633","messageId":"747b52e7bde1a33953849de233a47a063337548a.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 03/14] dir: extract directory-matching logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:19Z","receivedAt":"2021-09-12T13:23:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe last_matching_pattern_from_list() logic performs some checks on the\nfiletype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag\nis set. This works great when setting SKIP_WORKTREE bits within\nunpack_trees(), but doesn't work well when passing an arbitrary path\nsuch as a file within a matching directory.\n\nThis change only rearranges the logic but does not change its\nfunctionality.\n\nWe will expand the path_matches_dir_pattern() method in a following\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 22 +++++++++++++++++-----\n 1 file changed, 17 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 86afa2eae00..652135df896 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1303,6 +1303,19 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t\t\t WM_PATHNAME) == 0;\n }\n \n+static int path_matches_dir_pattern(const char *pathname,\n+\t\t\t\t int pathlen,\n+\t\t\t\t int *dtype,\n+\t\t\t\t struct path_pattern *pattern,\n+\t\t\t\t struct index_state *istate)\n+{\n+\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n+\tif (*dtype != DT_DIR)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n /*\n * Scan the given exclude list in reverse to see whether pathname\n * should be ignored. The first match (i.e. the last on the list), if\n@@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n-\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n-\t\t\tif (*dtype != DT_DIR)\n-\t\t\t\tcontinue;\n-\t\t}\n+\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\t\t\t\t dtype, pattern, istate))\n+\t\t\tcontinue;\n \n \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n \t\t\tif (match_basename(basename,\n-- \ngitgitgadget\n\n"},{"id":"435634","messageId":"723de4e42582afbe841ed96470fc02db44b24b5e.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 04/14] dir: select directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:20Z","receivedAt":"2021-09-12T13:23:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen matching a path against a list of patterns, the ones that require a\ndirectory match previously did not work when a filename is specified.\nThis was fine when all pattern-matching was done within methods such as\nunpack_trees() that check a directory before recursing into the\ncontained files. However, other commands will start matching individual\nfiles against pattern lists without that recursive approach.\n\nWe modify path_matches_dir_pattern() to take a strbuf 'path_parent' that\nis used to store the parent directory of 'pathname' between multiple\npattern matching tests. This is loaded lazily, only on the first pattern\nit finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n\nIf we find that a path has a parent directory, we start by checking to\nsee if that parent directory matches the pattern. If so, then we do not\nneed to query the index for the type (which can be expensive). If we\nfind that the parent does not match, then we still must check the type\nfrom the index for the given pathname.\n\nNote that this does not affect cone mode pattern matching, but instead\nthe more general -- and slower -- full pattern set. Thus, this does not\naffect the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 34 ++++++++++++++++++++++++++++++++--\n 1 file changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 652135df896..fe5ee87bb5f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,\n \n static int path_matches_dir_pattern(const char *pathname,\n \t\t\t\t int pathlen,\n+\t\t\t\t struct strbuf *path_parent,\n \t\t\t\t int *dtype,\n \t\t\t\t struct path_pattern *pattern,\n \t\t\t\t struct index_state *istate)\n {\n+\t/*\n+\t * Use 'alloc' as an indicator that the string has not been\n+\t * initialized, in case the parent is the root directory.\n+\t */\n+\tif (!path_parent->alloc) {\n+\t\tchar *slash;\n+\t\tstrbuf_addstr(path_parent, pathname);\n+\t\tslash = find_last_dir_sep(path_parent->buf);\n+\n+\t\tif (slash)\n+\t\t\t*slash = '\\0';\n+\t\telse\n+\t\t\tstrbuf_setlen(path_parent, 0);\n+\t}\n+\n+\t/*\n+\t * If the parent directory matches the pattern, then we do not\n+\t * need to check for dtype.\n+\t */\n+\tif (path_parent->len &&\n+\t match_pathname(path_parent->buf, path_parent->len,\n+\t\t\t pattern->base,\n+\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n+\t\t\t pattern->pattern, pattern->nowildcardlen,\n+\t\t\t pattern->patternlen, pattern->flags))\n+\t\treturn 1;\n+\n \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n \tif (*dtype != DT_DIR)\n \t\treturn 0;\n@@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n \tint i;\n+\tstruct strbuf path_parent = STRBUF_INIT;\n \n \tif (!pl->nr)\n \t\treturn NULL;\t/* undefined */\n@@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n-\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n \t\t\t\t\t dtype, pattern, istate))\n \t\t\tcontinue;\n \n@@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\t\tbreak;\n \t\t}\n \t}\n+\tstrbuf_release(&path_parent);\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"435635","messageId":"ed2472d33f12956dcafb0b600db9d7a6aada8cb8.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 05/14] dir: fix pattern matching on dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:21Z","receivedAt":"2021-09-12T13:23:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWithin match_pathname(), one successful matching category happens when\nthe pattern is equal to its non-wildcard prefix. At this point, we have\nchecked that the input 'pathname' matches the pattern up to the prefix\nlength, and then we subtraced that length from both 'patternlen' and\n'namelen'.\n\nIn the case of a directory match, this prefix match should be\nsufficient. However, the success condition only cared about _exact_\nequality here. Instead, we should allow any path that agrees on this\nprefix in the case of PATTERN_FLAG_MUSTBEDIR.\n\nThis case was not tested before because of the way unpack_trees() would\nmatch a parent directory before visiting the contained paths. This\napproach is changing, so we must change this comparison.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex fe5ee87bb5f..80fd0ad2fc0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t * then our prefix match is all we need; we\n \t\t * do not need to call fnmatch at all.\n \t\t */\n-\t\tif (!patternlen && !namelen)\n+\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n \t\t\treturn 1;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"435636","messageId":"bac6cedfc8d50511975f146c41f1577f41976c6f.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 06/14] add: fail when adding an untracked sparse file","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:22Z","receivedAt":"2021-09-12T13:23:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe add_files() method in builtin/add.c takes a set of untracked files\nthat are being added by the input pathspec and inserts them into the\nindex. If these files are outside of the sparse-checkout cone, then they\ngain the SKIP_WORKTREE bit at some point. However, this was not checked\nbefore inserting into the index, so these files are added even though we\nwant to avoid modifying the index outside of the sparse-checkout cone.\n\nAdd a check within add_files() for these files and write the advice\nabout files outside of the sparse-checkout cone.\n\nThis behavior change modifies some existing tests within t1092. These\ntests intended to document how a user could interact with the existing\nbehavior in place. Many of these tests need to be marked as expecting\nfailure. A future change will allow these tests to pass by adding a flag\nto 'git add' that allows users to modify index entries outside of the\nsparse-checkout cone.\n\nThe 'submodule handling' test is intended to document what happens to\ndirectories that contain a submodule when the sparse index is enabled.\nIt is not trying to say that users should be able to add submodules\noutside of the sparse-checkout cone, so that test can be modified to\navoid that operation.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 14 +++++++++\n t/t1092-sparse-checkout-compatibility.sh | 37 ++++++++++++++++++------\n 2 files changed, 42 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 88a6c0c69fb..8ea9cae0e7a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n static int add_files(struct dir_struct *dir, int flags)\n {\n \tint i, exit_status = 0;\n+\tstruct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;\n \n \tif (dir->ignored_nr) {\n \t\tfprintf(stderr, _(ignore_error));\n@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n+\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\t\tstring_list_append(&matched_sparse_paths,\n+\t\t\t\t\t dir->entries[i]->name);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {\n \t\t\tif (!ignore_add_errors)\n \t\t\t\tdie(_(\"adding files failed\"));\n@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)\n \t\t\tcheck_embedded_repo(dir->entries[i]->name);\n \t\t}\n \t}\n+\n+\tif (matched_sparse_paths.nr) {\n+\t\tadvise_on_updating_sparse_paths(&matched_sparse_paths);\n+\t\texit_status = 1;\n+\t}\n+\n+\tstring_list_clear(&matched_sparse_paths, 0);\n+\n \treturn exit_status;\n }\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 5b3f55e355e..997442b13e3 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -301,8 +301,6 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-# NEEDSWORK: This documents current behavior, but is not a desirable\n-# behavior (untracked files are handled differently than tracked).\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n@@ -312,7 +310,9 @@ test_expect_success 'add outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match git add folder1/newfile\n+\ttest_sparse_match test_must_fail git add folder1/newfile &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/newfile\n '\n \n test_expect_success 'commit including unstaged changes' '\n@@ -343,7 +343,11 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-test_expect_success 'status/add: outside sparse cone' '\n+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n+# attempted here fail for the sparse-checkout and sparse-index repos.\n+# We must enable a way for adding files outside the sparse-checkout\n+# done, even if it is by an optional flag.\n+test_expect_failure 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -368,10 +372,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/new &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/new &&\n \n-\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n-\ttest_sparse_match git add folder1/new &&\n-\n+\t# NEEDSWORK: behavior begins to deviate here.\n \ttest_all_match git add . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n@@ -527,7 +532,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n # Right now, users might be using this flow to work through conflicts,\n # so any solution should present advice to users who try this sequence\n # of commands to follow whatever new method we create.\n-test_expect_success 'merge with conflict outside cone' '\n+test_expect_failure 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -541,12 +546,18 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n+\t# NEEDSWORK: Even though the merge conflict removed the\n+\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t# warn that this is a problematic add.\n \ttest_all_match git add folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n+\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t# outside of the sparse-checkout cone and does not match an\n+\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_all_match git add folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n@@ -555,7 +566,7 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -572,11 +583,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 2. Add the file with conflict markers\n+\t\t# NEEDSWORK: Even though the merge conflict removed the\n+\t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t\t# warn that this is a problematic add.\n \t\ttest_all_match git add folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n \t\t# accept conflict markers as resolved content.\n+\t\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t\t# outside of the sparse-checkout cone and does not match an\n+\t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n \t\ttest_all_match git add folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n@@ -654,6 +671,7 @@ test_expect_success 'clean' '\n test_expect_success 'submodule handling' '\n \tinit_repos &&\n \n+\ttest_sparse_match git sparse-checkout add modules &&\n \ttest_all_match mkdir modules &&\n \ttest_all_match touch modules/a &&\n \ttest_all_match git add modules &&\n@@ -663,6 +681,7 @@ test_expect_success 'submodule handling' '\n \ttest_all_match git commit -m \"add submodule\" &&\n \n \t# having a submodule prevents \"modules\" from collapse\n+\ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n \ttest-tool -C sparse-index read-cache --table >cache &&\n \tgrep \"100644 blob .*\tmodules/a\" cache &&\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n-- \ngitgitgadget\n\n"},{"id":"435637","messageId":"d4706e17cae03d68d5c9108b3570f7fd9d2500d7.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 07/14] add: skip tracked paths outside sparse-checkout cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:23Z","receivedAt":"2021-09-12T13:23:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen 'git add' adds a tracked file that is outside of the\nsparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file\nexists outside of the sparse-checkout cone. This is usually correct,\nexcept in the case of a merge conflict outside of the cone.\n\nModify add_pathspec_matched_against_index() to be more careful about\npaths by checking the sparse-checkout patterns in addition to the\nSKIP_WORKTREE bit. This causes 'git add' to no longer allow files\noutside of the cone that removed the SKIP_WORKTREE bit due to a merge\nconflict.\n\nWith only this change, users will only be able to add the file after\nadding the file to the sparse-checkout cone. A later change will allow\nusers to force adding even though the file is outside of the\nsparse-checkout cone.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 ++++\n pathspec.c | 5 +++--\n t/t1091-sparse-checkout-builtin.sh | 4 +++-\n t/t1092-sparse-checkout-compatibility.sh | 21 ++++++++++++++-------\n t/t3705-add-sparse-checkout.sh | 12 ++++++++++++\n 5 files changed, 36 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8ea9cae0e7a..09c3fad6321 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n+\n+\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\t\tcontinue;\n+\n \t\tswitch (fix_unmerged_status(p, data)) {\n \t\tdefault:\n \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\ndiff --git a/pathspec.c b/pathspec.c\nindex 44306fdaca2..ddeeba79114 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\treturn;\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n-\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n+\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n+\t\t (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n \t\t\tcontinue;\n \t\tce_path_match(istate, ce, pathspec, seen);\n \t}\n@@ -70,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))\n \t\t ce_path_match(istate, ce, pathspec, seen);\n \t}\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 71236981e64..af99ae81b1d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat\n \tgit -C unmerged sparse-checkout disable\n '\n \n-test_expect_success 'sparse-checkout reapply' '\n+test_expect_failure 'sparse-checkout reapply' '\n \tgit clone repo tweak &&\n \n \techo dirty >tweak/deep/deeper2/a &&\n@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '\n \ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n \ttest_path_is_file tweak/folder1/a &&\n \n+\t# NEEDSWORK: We are asking to update a file outside of the\n+\t# sparse-checkout cone, but this is no longer allowed.\n \tgit -C tweak add folder1/a &&\n \tgit -C tweak sparse-checkout reapply 2>err &&\n \ttest_must_be_empty err &&\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 997442b13e3..0fdc5c7098c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -546,10 +546,10 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n-\t# NEEDSWORK: Even though the merge conflict removed the\n-\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n-\t# warn that this is a problematic add.\n-\ttest_all_match git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n+\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n@@ -558,7 +558,10 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# NEEDSWORK: This mode now fails, because folder2/z is\n \t# outside of the sparse-checkout cone and does not match an\n \t# existing index entry with the SKIP_WORKTREE bit cleared.\n-\ttest_all_match git add folder2 &&\n+\ttest_sparse_match test_must_fail git add folder2 &&\n+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder2/z &&\n+\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -586,7 +589,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# NEEDSWORK: Even though the merge conflict removed the\n \t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n \t\t# warn that this is a problematic add.\n-\t\ttest_all_match git add folder1/a &&\n+\t\ttest_sparse_match test_must_fail git add folder1/a &&\n+\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -595,7 +600,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# outside of the sparse-checkout cone and does not match an\n \t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n-\t\ttest_all_match git add folder2 &&\n+\t\ttest_sparse_match test_must_fail git add folder2 &&\n+\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder2/z &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex af81b4b6846..678be1acbf9 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -158,6 +158,18 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n \tgit ls-files --error-unmatch dense_entry\n '\n \n+test_expect_success 'git add fails outside of sparse-checkout definition' '\n+\ttest_when_finished git sparse-checkout disable &&\n+\ttest_commit a &&\n+\tgit sparse-checkout init &&\n+\tgit sparse-checkout set a &&\n+\techo >>sparse_entry &&\n+\n+\tgit update-index --no-skip-worktree sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unstaged\n+'\n+\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n-- \ngitgitgadget\n\n"},{"id":"435638","messageId":"7ace030c7093f6a0822fe45389d88df39df4d66b.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 09/14] add: update --chmod to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:25Z","receivedAt":"2021-09-12T13:23:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent staging a modification, but we skipped the\n--chmod mode. Update chmod_pathspec() to ignore cache entries whose path\nis outside of the sparse-checkout cone (unless --sparse is provided).\nAdd a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 12 ++++++++++--\n 2 files changed, 13 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f8e3930608d..f87b8134b67 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -47,7 +47,9 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 0f7e03b5326..6fc49a1845f 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -39,7 +39,7 @@ setup_gitignore () {\n \n test_sparse_entry_unstaged () {\n \tgit status --porcelain >actual &&\n-\t! grep \"^M sparse_entry\\$\" actual\n+\t! grep \"^M[ M] sparse_entry\\$\" actual\n }\n \n test_expect_success 'setup' \"\n@@ -169,11 +169,19 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --chmod=+x sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"435639","messageId":"944ae2cffa8ff175cd1cee0b3a25060ec5599973.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 08/14] add: implement the --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:24Z","receivedAt":"2021-09-12T13:23:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe previously modified 'git add' to refuse updating index entries\noutside of the sparse-checkout cone. This is justified to prevent users\nfrom accidentally getting into a confusing state when Git removes those\nfiles from the working tree at some later point.\n\nUnfortunately, this caused some workflows that were previously possible\nto become impossible, especially around merge conflicts outside of the\nsparse-checkout cone. These were documented in tests within t1092.\n\nWe now re-enable these workflows using a new '--sparse' option to 'git\nadd'. This allows users to signal \"Yes, I do know what I'm doing with\nthese files,\" and accept the consequences of the files leaving the\nworktree later.\n\nWe delay updating the advice message until implementing a similar option\nin 'git rm' and 'git mv'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-add.txt | 9 +++++++-\n builtin/add.c | 12 +++++++----\n t/t1092-sparse-checkout-compatibility.sh | 27 ++++++++----------------\n t/t3705-add-sparse-checkout.sh | 17 ++++++++++++++-\n 4 files changed, 41 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex be5e3ac54b8..bb79016d2ca 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -9,7 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n-\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n+\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]\n \t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--pathspec-from-file= [--pathspec-file-nul]]\n \t [--] [...]\n@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].\n --force::\n \tAllow adding otherwise ignored files.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git add` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone, since those files might\n+\tbe removed from the working tree without warning. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -i::\n --interactive::\n \tAdd modified contents in the working tree interactively to\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 09c3fad6321..f8e3930608d 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n static int add_renormalize;\n static int pathspec_file_nul;\n+static int include_sparse;\n static const char *pathspec_from_file;\n static int legacy_stash_p; /* support for the scripted `git stash` */\n \n@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n@@ -95,7 +96,7 @@ static void update_callback(struct diff_queue_struct *q,\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n \n-\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\tif (!include_sparse && !path_in_sparse_checkout(path, &the_index))\n \t\t\tcontinue;\n \n \t\tswitch (fix_unmerged_status(p, data)) {\n@@ -383,6 +384,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n \tOPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n \tOPT_BOOL( 0 , \"ignore-missing\", &ignore_missing, N_(\"check if - even missing - files are ignored in dry run\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_STRING(0, \"chmod\", &chmod_arg, \"(+|-)x\",\n \t\t N_(\"override the executable bit of the listed files\")),\n \tOPT_HIDDEN_BOOL(0, \"warn-embedded-repo\", &warn_on_embedded_repo,\n@@ -461,7 +463,8 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n-\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\tif (!include_sparse &&\n+\t\t !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n \t\t\tstring_list_append(&matched_sparse_paths,\n \t\t\t\t\t dir->entries[i]->name);\n \t\t\tcontinue;\n@@ -646,7 +649,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\tif (seen[i])\n \t\t\t\tcontinue;\n \n-\t\t\tif (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n+\t\t\tif (!include_sparse &&\n+\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec.items[i].original);\n \t\t\t\tcontinue;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0fdc5c7098c..7d64d9deb22 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -343,11 +343,7 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n-# attempted here fail for the sparse-checkout and sparse-index repos.\n-# We must enable a way for adding files outside the sparse-checkout\n-# done, even if it is by an optional flag.\n-test_expect_failure 'status/add: outside sparse cone' '\n+test_expect_success 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -375,15 +371,16 @@ test_expect_failure 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/new &&\n+\ttest_sparse_match git add --sparse folder1/a &&\n+\ttest_sparse_match git add --sparse folder1/new &&\n \n-\t# NEEDSWORK: behavior begins to deviate here.\n-\ttest_all_match git add . &&\n+\ttest_all_match git add --sparse . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n \ttest_all_match git rev-parse HEAD^{tree} &&\n \n \trun_on_all ../edit-contents folder1/newer &&\n-\ttest_all_match git add folder1/ &&\n+\ttest_all_match git add --sparse folder1/ &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/newer &&\n \ttest_all_match git rev-parse HEAD^{tree}\n@@ -527,12 +524,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n \tdone\n '\n \n-# NEEDSWORK: This test is documenting current behavior, but that\n-# behavior can be confusing to users so there is desire to change it.\n-# Right now, users might be using this flow to work through conflicts,\n-# so any solution should present advice to users who try this sequence\n-# of commands to follow whatever new method we create.\n-test_expect_failure 'merge with conflict outside cone' '\n+test_expect_success 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -555,9 +547,6 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n-\t# NEEDSWORK: This mode now fails, because folder2/z is\n-\t# outside of the sparse-checkout cone and does not match an\n-\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder2/z &&\n@@ -569,7 +558,7 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -592,6 +581,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder1/a &&\n \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder1/a &&\n+\t\ttest_all_match git add --sparse folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -603,6 +593,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder2 &&\n \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder2/z &&\n+\t\ttest_all_match git add --sparse folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 678be1acbf9..0f7e03b5326 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -167,7 +167,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \n \tgit update-index --no-skip-worktree sparse_entry &&\n \ttest_must_fail git add sparse_entry &&\n-\ttest_sparse_entry_unstaged\n+\ttest_sparse_entry_unstaged &&\n+\n+\t# Avoid munging CRLFs to avoid an error message\n+\tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n@@ -178,4 +184,13 @@ test_expect_success 'add obeys advice.updateSparsePath' '\n \n '\n \n+test_expect_success 'add allows sparse entries with --sparse' '\n+\tgit sparse-checkout set a &&\n+\techo modified >sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unchanged &&\n+\tgit add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"435640","messageId":"f34be1660978ead503db13b81f55cac744230ec8.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 10/14] add: update --renormalize to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:26Z","receivedAt":"2021-09-12T13:23:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent stagins a modification, but we skipped the\n--renormalize mode. Update renormalize_tracked_files() to ignore cache\nentries whose path is outside of the sparse-checkout cone (unless\n--sparse is provided). Add a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 12 +++++++++++-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f87b8134b67..f8f0dfa4046 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -154,7 +154,9 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (ce_stage(ce))\n \t\t\tcontinue; /* do not touch unmerged paths */\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 6fc49a1845f..cf2ccb87cf2 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -172,6 +172,9 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add --chmod=+x sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --renormalize sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n@@ -181,7 +184,14 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit reset &&\n+\n+\t# This will print a message over stderr on Windows.\n+\tgit add --sparse --renormalize sparse_entry &&\n+\tgit status --porcelain >actual &&\n+\tgrep \"^M sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"435641","messageId":"fb3f86efa9b2308883bad0b39057ef1e3eefd898.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 11/14] rm: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:27Z","receivedAt":"2021-09-12T13:23:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we did previously in 'git add', add a '--sparse' option to 'git rm'\nthat allows modifying paths outside of the sparse-checkout definition.\nThe existing checks in 'git rm' are restricted to tracked files that\nhave the SKIP_WORKTREE bit in the current index. Future changes will\ncause 'git rm' to reject removing paths outside of the sparse-checkout\ndefinition, even if they are untracked or do not have the SKIP_WORKTREE\nbit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-rm.txt | 6 ++++++\n builtin/rm.c | 8 ++++++--\n t/t3602-rm-sparse-checkout.sh | 12 ++++++++++++\n 3 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-rm.txt b/Documentation/git-rm.txt\nindex 26e9b284704..81bc23f3cdb 100644\n--- a/Documentation/git-rm.txt\n+++ b/Documentation/git-rm.txt\n@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].\n --ignore-unmatch::\n \tExit with a zero status even if no files matched.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git rm` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -q::\n --quiet::\n \t`git rm` normally outputs one line (in the form of an `rm` command)\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 8a24c715e02..4208f3f9a5f 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)\n \n static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;\n static int ignore_unmatch = 0, pathspec_file_nul;\n+static int include_sparse;\n static char *pathspec_from_file;\n \n static struct option builtin_rm_options[] = {\n@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {\n \tOPT_BOOL('r', NULL, &recursive, N_(\"allow recursive removal\")),\n \tOPT_BOOL( 0 , \"ignore-unmatch\", &ignore_unmatch,\n \t\t\t\tN_(\"exit with a zero status even if nothing matched\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_PATHSPEC_FROM_FILE(&pathspec_from_file),\n \tOPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),\n \tOPT_END(),\n@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\n@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\t\tseen_any = 1;\n \t\t\telse if (ignore_unmatch)\n \t\t\t\tcontinue;\n-\t\t\telse if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n+\t\t\telse if (!include_sparse &&\n+\t\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n \t\t\t\tstring_list_append(&only_match_skip_worktree, original);\n \t\t\telse\n \t\t\t\tdie(_(\"pathspec '%s' did not match any files\"), original);\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex e9e9a15c74c..493c8f636b8 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -43,6 +43,18 @@ test_expect_success 'recursive rm does not remove sparse entries' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'recursive rm --sparse removes sparse entries' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set \"sub/dir\" &&\n+\tgit rm --sparse -r sub &&\n+\tgit status --porcelain -uno >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'rm obeys advice.updateSparsePath' '\n \tgit reset --hard &&\n \tgit sparse-checkout set a &&\n-- \ngitgitgadget\n\n"},{"id":"435642","messageId":"f2abc387822378e02d0b221baf9a09ac91d44d7d.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 14/14] advice: update message to suggest '--sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:30Z","receivedAt":"2021-09-12T13:23:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous changes modified the behavior of 'git add', 'git rm', and\n'git mv' to not adjust paths outside the sparse-checkout cone, even if\nthey exist in the working tree and their cache entries lack the\nSKIP_WORKTREE bit. The intention is to warn users that they are doing\nsomething potentially dangerous. The '--sparse' option was added to each\ncommand to allow careful users the same ability they had before.\n\nTo improve the discoverability of this new functionality, add a message\nto advice.updateSparsePath that mentions the existence of the option.\n\nThe previous set of changes also modified the purpose of this message to\ninclude possibly a list of paths instead of only a list of pathspecs.\nMake the warning message more clear about this new behavior.\n\nSigned-off-by: Derrick Stolee \n---\n advice.c | 10 ++++++----\n t/t3602-rm-sparse-checkout.sh | 6 +++---\n t/t3705-add-sparse-checkout.sh | 6 +++---\n t/t7002-mv-sparse-checkout.sh | 6 +++---\n 4 files changed, 15 insertions(+), 13 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex 0b9c89c48ab..0deaf111795 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -293,14 +293,16 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)\n \tif (!pathspec_list->nr)\n \t\treturn;\n \n-\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n-\t\t\t \" eligible path, but they do match index\\n\"\n-\t\t\t \"entries outside the current sparse checkout:\\n\"));\n+\tfprintf(stderr, _(\"The following paths and/or pathspecs matched \"\n+\t\t\t \"paths that exist outside of your\\n\"\n+\t\t\t \"sparse-checkout definition, so will not be \"\n+\t\t\t \"updated in the index:\\n\"));\n \tfor_each_string_list_item(item, pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n-\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n+\t\t\t _(\"Disable or modify the sparsity rules or\"\n+\t\t\t \" use the --sparse option if you intend\"\n \t\t\t \" to update such entries.\"));\n }\n \ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 5f92b60a56a..712ddae2b15 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -11,12 +11,12 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist outside of your\n+\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex cf2ccb87cf2..d51c96f8d72 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -44,12 +44,12 @@ test_sparse_entry_unstaged () {\n \n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist outside of your\n+\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 07dbfeb6d17..3db7ddaba9a 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -11,12 +11,12 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist outside of your\n+\tsparse-checkout definition, so will not be updated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \"\n-- \ngitgitgadget\n"},{"id":"435643","messageId":"de0c9d09ef774b25e9d587610f482c08032683aa.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 12/14] rm: skip sparse paths with missing SKIP_WORKTREE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:28Z","receivedAt":"2021-09-12T13:23:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a path does not match the sparse-checkout cone but is somehow missing\nthe SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the\nfile. One reason a user might be in this situation is a merge conflict\noutside of the sparse-checkout cone. Removing such a file might be\nproblematic for users who are not sure what they are doing.\n\nAdd a check to path_in_sparse_checkout() when 'git rm' is checking if a\npath should be considered for deletion. Of course, this check is ignored\nif the '--sparse' option is specified, allowing users who accept the\nrisks to continue with the removal.\n\nThis also removes a confusing behavior where a user asks for a directory\nto be removed, but only the entries that are within the sparse-checkout\ndefinition are removed. Now, 'git rm ' will fail without '--sparse'\nand will succeed in removing all contained paths with '--sparse'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 4 +++-\n t/t3602-rm-sparse-checkout.sh | 19 +++++++++++++++++--\n 2 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4208f3f9a5f..a6da03da2be 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 493c8f636b8..5f92b60a56a 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -37,9 +37,13 @@ done\n test_expect_success 'recursive rm does not remove sparse entries' '\n \tgit reset --hard &&\n \tgit sparse-checkout set sub/dir &&\n-\tgit rm -r sub &&\n+\ttest_must_fail git rm -r sub &&\n+\tgit rm --sparse -r sub &&\n \tgit status --porcelain -uno >actual &&\n-\techo \"D sub/dir/e\" >expected &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n \ttest_cmp expected actual\n '\n \n@@ -87,4 +91,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n \tgit ls-files --error-unmatch b\n '\n \n+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\tgit update-index --no-skip-worktree b &&\n+\ttest_must_fail git rm b 2>stderr &&\n+\ttest_cmp b_error_and_hint stderr &&\n+\tgit rm --sparse b 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"435644","messageId":"e17a529332d220f0943a8496d7dc33de5ea4e873.1631453010.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v2 13/14] mv: refuse to move sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-12T13:23:29Z","receivedAt":"2021-09-12T13:23:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince cmd_mv() does not operate on cache entries and instead directly\nchecks the filesystem, we can only use path_in_sparse_checkout() as a\nmechanism for seeing if a path is sparse or not. Be sure to skip\nreturning a failure if '-k' is specified.\n\nTo ensure that the advice around sparse paths is the only reason a move\nfailed, be sure to check this as the very last thing before inserting\ninto the src_for_dst list.\n\nThe tests cover a variety of cases such as whether the target is tracked\nor untracked, and whether the source or destination are in or outside of\nthe sparse-checkout definition.\n\nHelped-by: Matheus Tavares Bernardino \nSigned-off-by: Derrick Stolee \n---\n builtin/mv.c | 52 ++++++++--\n t/t7002-mv-sparse-checkout.sh | 186 ++++++++++++++++++++++++++++++++++\n 2 files changed, 229 insertions(+), 9 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex c2f96c8e895..83a465ba831 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -118,21 +118,23 @@ static int index_range_of_same_dir(const char *src, int length,\n int cmd_mv(int argc, const char **argv, const char *prefix)\n {\n \tint i, flags, gitmodules_modified = 0;\n-\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n+\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n \tstruct option builtin_mv_options[] = {\n \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n \t\tOPT__DRY_RUN(&show_only, N_(\"dry run\")),\n \t\tOPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n+\t\tOPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \t\tOPT_END(),\n \t};\n \tconst char **source, **destination, **dest_path, **submodule_gitfile;\n-\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n+\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n \tstruct stat st;\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct cache_entry *ce;\n+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \n \tgit_config(git_default_config, NULL);\n \n@@ -176,14 +178,17 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tint length, src_is_dir;\n \t\tconst char *bad = NULL;\n+\t\tint skip_sparse = 0;\n \n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n \t\tlength = strlen(src);\n-\t\tif (lstat(src, &st) < 0)\n-\t\t\tbad = _(\"bad source\");\n-\t\telse if (!strncmp(src, dst, length) &&\n+\t\tif (lstat(src, &st) < 0) {\n+\t\t\t/* only error if existence is expected. */\n+\t\t\tif (modes[i] != SPARSE)\n+\t\t\t\tbad = _(\"bad source\");\n+\t\t} else if (!strncmp(src, dst, length) &&\n \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n \t\t\tbad = _(\"can not move directory into itself\");\n \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n@@ -212,11 +217,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\t\tdst_len = strlen(dst);\n \n \t\t\t\tfor (j = 0; j < last - first; j++) {\n-\t\t\t\t\tconst char *path = active_cache[first + j]->name;\n+\t\t\t\t\tconst struct cache_entry *ce = active_cache[first + j];\n+\t\t\t\t\tconst char *path = ce->name;\n \t\t\t\t\tsource[argc + j] = path;\n \t\t\t\t\tdestination[argc + j] =\n \t\t\t\t\t\tprefix_path(dst, dst_len, path + length + 1);\n-\t\t\t\t\tmodes[argc + j] = INDEX;\n+\t\t\t\t\tmodes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n \t\t\t\t\tsubmodule_gitfile[argc + j] = NULL;\n \t\t\t\t}\n \t\t\t\targc += last - first;\n@@ -244,14 +250,36 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tbad = _(\"multiple sources for the same target\");\n \t\telse if (is_dir_sep(dst[strlen(dst) - 1]))\n \t\t\tbad = _(\"destination directory does not exist\");\n-\t\telse\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * We check if the paths are in the sparse-checkout\n+\t\t\t * definition as a very final check, since that\n+\t\t\t * allows us to point the user to the --sparse\n+\t\t\t * option as a way to have a successful run.\n+\t\t\t */\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(src, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, src);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(dst, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\n+\t\t\tif (skip_sparse)\n+\t\t\t\tgoto remove_entry;\n+\n \t\t\tstring_list_insert(&src_for_dst, dst);\n+\t\t}\n \n \t\tif (!bad)\n \t\t\tcontinue;\n \t\tif (!ignore_errors)\n \t\t\tdie(_(\"%s, source=%s, destination=%s\"),\n \t\t\t bad, src, dst);\n+remove_entry:\n \t\tif (--argc > 0) {\n \t\t\tint n = argc - i;\n \t\t\tmemmove(source + i, source + i + 1,\n@@ -266,6 +294,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (only_match_skip_worktree.nr) {\n+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n+\t\tif (!ignore_errors)\n+\t\t\treturn 1;\n+\t}\n+\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tenum update_mode mode = modes[i];\n@@ -274,7 +308,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(_(\"Renaming %s to %s\\n\"), src, dst);\n \t\tif (show_only)\n \t\t\tcontinue;\n-\t\tif (mode != INDEX && rename(src, dst) < 0) {\n+\t\tif (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n \t\t\tif (ignore_errors)\n \t\t\t\tcontinue;\n \t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..07dbfeb6d17\n--- /dev/null\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -0,0 +1,186 @@\n+#!/bin/sh\n+\n+test_description='git mv in sparse working trees'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' \"\n+\tmkdir -p sub/dir sub/dir2 &&\n+\ttouch a b c sub/d sub/dir/e sub/dir2/e &&\n+\tgit add -A &&\n+\tgit commit -m files &&\n+\n+\tcat >sparse_error_header <<-EOF &&\n+\tThe following pathspecs didn't match any eligible path, but they do match index\n+\tentries outside the current sparse checkout:\n+\tEOF\n+\n+\tcat >sparse_hint <<-EOF\n+\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n+\tEOF\n+\"\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse' '\n+\ttest_when_finished rm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\tgit mv -k b e 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\tgit reset --hard &&\n+\ttouch b &&\n+\tgit mv -k b c 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttest_must_fail git mv a e 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\trm e &&\n+\tgit reset --hard &&\n+\ttest_must_fail git mv a c 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a e &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'recursive mv refuses to move (possible) sparse' '\n+\ttest_when_finished rm -rf b c e sub2 &&\n+\tgit reset --hard &&\n+\t# Without cone mode, \"sub\" and \"sub2\" do not match\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/d sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/d\n+\tsub2/d\n+\tsub/dir/e\n+\tsub2/dir/e\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_expect_success 'recursive mv refuses to move sparse' '\n+\tgit reset --hard &&\n+\t# Use cone mode so \"sub/\" matches the sparse-checkout patterns\n+\tgit sparse-checkout init --cone &&\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"435660","messageId":"87pmtd5tkj.fsf@evledraar.gmail.com","threadId":"56370","inReplyTo":"f2abc387822378e02d0b221baf9a09ac91d44d7d.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 14/14] advice: update message to suggest '--sparse'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-12T21:58:35Z","receivedAt":"2021-09-12T22:15:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Sep 12 2021, Derrick Stolee via GitGitGadget wrote:\n\n> -\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n> -\t\t\t \" eligible path, but they do match index\\n\"\n> -\t\t\t \"entries outside the current sparse checkout:\\n\"));\n> +\tfprintf(stderr, _(\"The following paths and/or pathspecs matched \"\n> +\t\t\t \"paths that exist outside of your\\n\"\n> +\t\t\t \"sparse-checkout definition, so will not be \"\n> +\t\t\t \"updated in the index:\\n\"));\n> \tfor_each_string_list_item(item, pathspec_list)\n> \t\tfprintf(stderr, \"%s\\n\", item->string);\n\nThis before and after looks about as well line-wrapped...\n\n> \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n> -\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n> +\t\t\t _(\"Disable or modify the sparsity rules or\"\n> +\t\t\t \" use the --sparse option if you intend\"\n> \t\t\t \" to update such entries.\"));\n> }\n\n...but here..\n\n> \tcat >sparse_error_header <<-EOF &&\n> -\tThe following pathspecs didn't match any eligible path, but they do match index\n> -\tentries outside the current sparse checkout:\n> +\tThe following paths and/or pathspecs matched paths that exist outside of your\n> +\tsparse-checkout definition, so will not be updated in the index:\n> \tEOF\n> \n> \tcat >sparse_hint <<-EOF &&\n> -\thint: Disable or modify the sparsity rules if you intend to update such entries.\n> +\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n> \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n> \tEOF\n\n...this used to line-wrap at 80 characters, but is now a bit beyond\nthat.\n\nMaybe instead make these two into bullet-points?\n\nAlso the third \"Disable\" looks a bit jarring at first, it seems like a\ncontinuation of the first message, but it's just the standard \"disable\nthis message\" we tend to print out.\n\nThis commentary pre-dates this commit, but just in general:\n\nI think the advice system is best used where there's an initial\nnon-optional message, and then the advice elaborates on what happened,\nhow to fix it. A good example is the \"short object ID %s is ambiguous\"\nin object-name.c.\n\nBut in this case both messages are rather long. I'd think better would\nbe something like (and I didn't look very deeply at the involved code):\n\n error(\"pathspec '%s' matched only outside sparse checkout\")\n\nI.e. in e.g. cmd_rm() we loop through the pathspecs, and we error on the\nfirst one, so to a first approximation why do we need to for sparse emit\nALL the pathspecs we didn't match? if we're going to error out anyway\nshouldn'w we just error out on the first one?\n\nBut going on, I'd think this would be better overall (pseudocode):\n\n error(\"pathspec '%s' matched only outside sparse checkout\")\n if (advice_enabled(ADVICE_UPDATE_SPARSE_PATH)) {\n char *list_str;\n list_of_bad_pathspecs = make_that_list(my_pathspec_string_list, &list_str);\n \n if (list_of_bad_pathspecs.nr > 1)\n /* Emit a message that details what's wrong, but also has a\n * list of all the other pathspecs we'd also die on if the user */\n else\n /* Ditto, but no list *?\n\nMaybe I'm missing something with the sparse implemention, but I'd think\ngoing above & beyond and listing all failures is a bit much in either\ncase, i.e. for non-sparse we have:\n\n $ git rm 'file-i-do-not-have' 'directory-i-do-not-have/'\n fatal: pathspec 'file-i-do-not-have' did not match any files\n $\n\nI'd think in general a user who's screwed up and typo'd both isn't going\nto be much harmed by us noting the first, maybe they'll get another\nerror then.\n\nBut usually it's obvious (e.g. you just ran the command in the wrong\ndirectory), so if you have a large list of pathspecs getting a firehose\nof all the things that didn't match can be less helpful due to being\noverrly verbose.\n\n\n"},{"id":"435661","messageId":"87lf415tge.fsf@evledraar.gmail.com","threadId":"56370","inReplyTo":"61c23dc59a6e062e073ba3bea135c370a78415aa.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/14] t1092: behavior for adding sparse files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-12T22:17:12Z","receivedAt":"2021-09-12T22:18:16Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Sep 12 2021, Derrick Stolee via GitGitGadget wrote:\n\n> +# NEEDSWORK: This documents current behavior, but is not a desirable\n> +# behavior (untracked files are handled differently than tracked).\n\nI wonder if a test_expect_failure test would be better for the thing\nthat is the desired behavior, but maybe we don't know what the CLI UI\nfor that would look like yet.\n\n\n> +test_expect_success 'add outside sparse cone' '\n> +\tinit_repos &&\n> +\n> +\trun_on_sparse mkdir folder1 &&\n> +\trun_on_sparse ../edit-contents folder1/a &&\n> +\trun_on_sparse ../edit-contents folder1/newfile &&\n> +\ttest_sparse_match test_must_fail git add folder1/a &&\n> +\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n\nJust \"grep\" is preferred over \"test_i18ngrep\" now, the GETTEXT_POISON\nwent away.\n"},{"id":"435664","messageId":"87h7ep5t5t.fsf@evledraar.gmail.com","threadId":"56370","inReplyTo":"723de4e42582afbe841ed96470fc02db44b24b5e.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/14] dir: select directories correctly","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-09-12T22:21:42Z","receivedAt":"2021-09-12T22:24:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Sep 12 2021, Derrick Stolee via GitGitGadget wrote:\n\n> +\t/*\n> +\t * Use 'alloc' as an indicator that the string has not been\n> +\t * initialized, in case the parent is the root directory.\n> +\t */\n> +\tif (!path_parent->alloc) {\n\nThis isn't wrong, but seems to be way too cozy with the internal\nimplementation details of strbuf. For what it's worth I renamed it to\n\"alloc2\" and found that this would be only the 3rd bit of code out of\nstrbuf.[ch] that cares about that member.\n\n> +\t\tchar *slash;\n> +\t\tstrbuf_addstr(path_parent, pathname);\n\nSo is \"pathname\" ever the empty string? If not we could check the\nlength?\n\nOr probably better: ...\n\n> @@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> {\n> \tstruct path_pattern *res = NULL; /* undecided */\n> \tint i;\n> +\tstruct strbuf path_parent = STRBUF_INIT;\n\nJust malloc + strbuf_init() this in the above function and have a\n\"struct strbuf *\" initialized to NULL here? Then we can use a much more\nidiomatic \"is it NULL?\" to check if it's initialized.\n"},{"id":"435730","messageId":"0c691195-cb06-aa91-639f-cc1added81b6@gmail.com","threadId":"56370","inReplyTo":"87lf415tge.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 02/14] t1092: behavior for adding sparse files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-13T15:02:39Z","receivedAt":"2021-09-13T15:41:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/12/2021 6:17 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Sun, Sep 12 2021, Derrick Stolee via GitGitGadget wrote:\n> \n>> +# NEEDSWORK: This documents current behavior, but is not a desirable\n>> +# behavior (untracked files are handled differently than tracked).\n> \n> I wonder if a test_expect_failure test would be better for the thing\n> that is the desired behavior, but maybe we don't know what the CLI UI\n> for that would look like yet.\n\nThe problem with test_expect_failure is that you don't know which\nline of the test is the problem. That's probably all fine and good\nwhen we completely understand the situation we want to solve but\ndon't have a good approach to fixing it, but here I want to document\na change in behavior.\n\nUsing test_expect_success allows me to demonstrate \"it works this\nway now\" and then \"this is how behavior changes\".\n\n>> +test_expect_success 'add outside sparse cone' '\n>> +\tinit_repos &&\n>> +\n>> +\trun_on_sparse mkdir folder1 &&\n>> +\trun_on_sparse ../edit-contents folder1/a &&\n>> +\trun_on_sparse ../edit-contents folder1/newfile &&\n>> +\ttest_sparse_match test_must_fail git add folder1/a &&\n>> +\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> \n> Just \"grep\" is preferred over \"test_i18ngrep\" now, the GETTEXT_POISON\n> went away.\n\nRight. Force of habit.\n\nThanks,\n-Stolee\n"},{"id":"435964","messageId":"CABPp-BF9rhAK47KQUJtZ9yDTkJ-qoeZ0Q7T4ycLcLSgLoxkzTw@mail.gmail.com","threadId":"56370","inReplyTo":"8aefce6254c0bcbbbca909a62d033c74c90f980b.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-15T05:22:56Z","receivedAt":"2021-09-15T05:23:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The tests in t3705-add-sparse-checkout.sh check to see how 'git add'\n> behaves with paths outside the sparse-checkout definition. These\n> currently check to see if a given warning is present but not that the\n> index is not updated with the sparse entries. Add a new\n> 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\n> correctly.\n>\n> We need to modify setup_sparse_entry to actually commit the sparse_entry\n> file so it exists at HEAD but is not already staged in the index.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n> 1 file changed, 14 insertions(+)\n>\n> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n> index 2b1fd0d0eef..af81b4b6846 100755\n> --- a/t/t3705-add-sparse-checkout.sh\n> +++ b/t/t3705-add-sparse-checkout.sh\n> @@ -19,6 +19,7 @@ setup_sparse_entry () {\n> fi &&\n> git add sparse_entry &&\n> git update-index --skip-worktree sparse_entry &&\n> + git commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n> SPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n> }\n>\n> @@ -36,6 +37,11 @@ setup_gitignore () {\n> EOF\n> }\n>\n> +test_sparse_entry_unstaged () {\n> + git status --porcelain >actual &&\n> + ! grep \"^M sparse_entry\\$\" actual\n\nIs there a reason this is ^M rather than ^D? Granted, both would be\nbugs so I wouldn't expect either to appear, but the point of the check\nis looking for likely errors. Wouldn't the more likely error case for\na file not present in the working tree be that we stage the deletion\nof the file?\n\n> +}\n> +\n> test_expect_success 'setup' \"\n> cat >sparse_error_header <<-EOF &&\n> The following pathspecs didn't match any eligible path, but they do match index\n> @@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '\n> setup_sparse_entry &&\n> rm sparse_entry &&\n> test_must_fail git add sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> test_sparse_entry_unchanged\n> '\n> @@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n> rm sparse_entry &&\n> setup_gitignore &&\n> test_must_fail git add . 2>stderr &&\n> + test_sparse_entry_unstaged &&\n>\n> cat sparse_error_header >expect &&\n> echo . >>expect &&\n> @@ -88,6 +96,7 @@ do\n> setup_sparse_entry &&\n> echo modified >sparse_entry &&\n> test_must_fail git add $opt sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> test_sparse_entry_unchanged\n> '\n> @@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n> git ls-files --debug sparse_entry | grep mtime >before &&\n> test-tool chmtime -60 sparse_entry &&\n> test_must_fail git add --refresh sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> git ls-files --debug sparse_entry | grep mtime >after &&\n> test_cmp before after\n> @@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n> test_expect_success 'git add --chmod does not update sparse entries' '\n> setup_sparse_entry &&\n> test_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> test_sparse_entry_unchanged &&\n> ! test -x sparse_entry\n> @@ -116,6 +127,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '\n> setup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n> echo \"sparse_entry text=auto\" >.gitattributes &&\n> test_must_fail git add --renormalize sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> test_sparse_entry_unchanged\n> '\n> @@ -124,6 +136,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '\n> setup_sparse_entry &&\n> rm sparse_entry &&\n> test_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp error_and_hint stderr &&\n> test_sparse_entry_unchanged\n> '\n> @@ -148,6 +161,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n> test_expect_success 'add obeys advice.updateSparsePath' '\n> setup_sparse_entry &&\n> test_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n> + test_sparse_entry_unstaged &&\n> test_cmp sparse_entry_error stderr\n>\n> '\n> --\n> gitgitgadget\n\nLooks fine otherwise.\n"},{"id":"435981","messageId":"3262acae-3ae1-7309-a1dd-b2e1472391a2@gmail.com","threadId":"56370","inReplyTo":"87h7ep5t5t.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 04/14] dir: select directories correctly","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-15T14:41:42Z","receivedAt":"2021-09-15T14:41:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/12/2021 6:21 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Sun, Sep 12 2021, Derrick Stolee via GitGitGadget wrote:\n> \n>> +\t/*\n>> +\t * Use 'alloc' as an indicator that the string has not been\n>> +\t * initialized, in case the parent is the root directory.\n>> +\t */\n>> +\tif (!path_parent->alloc) {\n> \n> This isn't wrong, but seems to be way too cozy with the internal\n> implementation details of strbuf. For what it's worth I renamed it to\n> \"alloc2\" and found that this would be only the 3rd bit of code out of\n> strbuf.[ch] that cares about that member.\n\nI can understand not wanting to poke into the internals.\n\n>> +\t\tchar *slash;\n>> +\t\tstrbuf_addstr(path_parent, pathname);\n> \n> So is \"pathname\" ever the empty string? If not we could check the\n> length?\n\nWe are given 'pathlen' as a parameter, so this should just use\nstrbuf_add() instead.\n\n> Or probably better: ...\n> \n>> @@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n>> {\n>> \tstruct path_pattern *res = NULL; /* undecided */\n>> \tint i;\n>> +\tstruct strbuf path_parent = STRBUF_INIT;\n> \n> Just malloc + strbuf_init() this in the above function and have a\n> \"struct strbuf *\" initialized to NULL here? Then we can use a much more\n> idiomatic \"is it NULL?\" to check if it's initialized.\n \nThat makes sense. Can do.\n\nThanks,\n-Stolee\n"},{"id":"435982","messageId":"CABPp-BFitDu8UFkH5Syd3w7_e4sT5nJCJ_6N5sn3a+Acv6B4aQ@mail.gmail.com","threadId":"56370","inReplyTo":"723de4e42582afbe841ed96470fc02db44b24b5e.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/14] dir: select directories correctly","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-15T14:54:16Z","receivedAt":"2021-09-15T14:54:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When matching a path against a list of patterns, the ones that require a\n> directory match previously did not work when a filename is specified.\n> This was fine when all pattern-matching was done within methods such as\n> unpack_trees() that check a directory before recursing into the\n> contained files. However, other commands will start matching individual\n> files against pattern lists without that recursive approach.\n>\n> We modify path_matches_dir_pattern() to take a strbuf 'path_parent' that\n> is used to store the parent directory of 'pathname' between multiple\n> pattern matching tests. This is loaded lazily, only on the first pattern\n> it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n>\n> If we find that a path has a parent directory, we start by checking to\n> see if that parent directory matches the pattern. If so, then we do not\n> need to query the index for the type (which can be expensive). If we\n> find that the parent does not match, then we still must check the type\n> from the index for the given pathname.\n>\n> Note that this does not affect cone mode pattern matching, but instead\n> the more general -- and slower -- full pattern set. Thus, this does not\n> affect the sparse index.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 34 ++++++++++++++++++++++++++++++++--\n> 1 file changed, 32 insertions(+), 2 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index 652135df896..fe5ee87bb5f 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,\n>\n> static int path_matches_dir_pattern(const char *pathname,\n> int pathlen,\n> + struct strbuf *path_parent,\n> int *dtype,\n> struct path_pattern *pattern,\n> struct index_state *istate)\n> {\n> + /*\n> + * Use 'alloc' as an indicator that the string has not been\n> + * initialized, in case the parent is the root directory.\n> + */\n> + if (!path_parent->alloc) {\n> + char *slash;\n> + strbuf_addstr(path_parent, pathname);\n> + slash = find_last_dir_sep(path_parent->buf);\n> +\n> + if (slash)\n> + *slash = '\\0';\n\nAre you breaking strbuf invariants here? path_parent->len will not be\ncorrected by this string manipulation. Perhaps replace this if-else\nblock with\n\n strbuf_setlen(path_parent, slash ? slash - path_parent->buf : 0)\n\n> + else\n> + strbuf_setlen(path_parent, 0);\n> + }\n> +\n> + /*\n> + * If the parent directory matches the pattern, then we do not\n> + * need to check for dtype.\n> + */\n> + if (path_parent->len &&\n> + match_pathname(path_parent->buf, path_parent->len,\n> + pattern->base,\n> + pattern->baselen ? pattern->baselen - 1 : 0,\n> + pattern->pattern, pattern->nowildcardlen,\n> + pattern->patternlen, pattern->flags))\n> + return 1;\n> +\n> *dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n> if (*dtype != DT_DIR)\n> return 0;\n> @@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> {\n> struct path_pattern *res = NULL; /* undecided */\n> int i;\n> + struct strbuf path_parent = STRBUF_INIT;\n>\n> if (!pl->nr)\n> return NULL; /* undefined */\n> @@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> const char *exclude = pattern->pattern;\n> int prefix = pattern->nowildcardlen;\n>\n> - if ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n> - !path_matches_dir_pattern(pathname, pathlen,\n> + if (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n> + !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n> dtype, pattern, istate))\n> continue;\n>\n> @@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> break;\n> }\n> }\n> + strbuf_release(&path_parent);\n> return res;\n> }\n>\n> --\n> gitgitgadget\n"},{"id":"435984","messageId":"ba372f28-3b90-db30-72fd-0242c1c1c3b3@gmail.com","threadId":"56370","inReplyTo":"CABPp-BF9rhAK47KQUJtZ9yDTkJ-qoeZ0Q7T4ycLcLSgLoxkzTw@mail.gmail.com","subject":"Re: [PATCH v2 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-15T16:17:31Z","receivedAt":"2021-09-15T16:17:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/15/2021 1:22 AM, Elijah Newren wrote:\n> On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The tests in t3705-add-sparse-checkout.sh check to see how 'git add'\n>> behaves with paths outside the sparse-checkout definition. These\n>> currently check to see if a given warning is present but not that the\n>> index is not updated with the sparse entries. Add a new\n>> 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\n>> correctly.\n>>\n>> We need to modify setup_sparse_entry to actually commit the sparse_entry\n>> file so it exists at HEAD but is not already staged in the index.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n>> 1 file changed, 14 insertions(+)\n>>\n>> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n>> index 2b1fd0d0eef..af81b4b6846 100755\n>> --- a/t/t3705-add-sparse-checkout.sh\n>> +++ b/t/t3705-add-sparse-checkout.sh\n>> @@ -19,6 +19,7 @@ setup_sparse_entry () {\n>> fi &&\n>> git add sparse_entry &&\n>> git update-index --skip-worktree sparse_entry &&\n>> + git commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n>> SPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n>> }\n>>\n>> @@ -36,6 +37,11 @@ setup_gitignore () {\n>> EOF\n>> }\n>>\n>> +test_sparse_entry_unstaged () {\n>> + git status --porcelain >actual &&\n>> + ! grep \"^M sparse_entry\\$\" actual\n> \n> Is there a reason this is ^M rather than ^D? Granted, both would be\n> bugs so I wouldn't expect either to appear, but the point of the check\n> is looking for likely errors. Wouldn't the more likely error case for\n> a file not present in the working tree be that we stage the deletion\n> of the file?\n\nYou are right that we should be checking for deletions or adds\n_as well_ as modifications. The test_sparse_entry_unstaged helper\nis used in a variety of situations that typically would trigger\na modification, but at least one instance in this test is a\npossible deletion.\n\n> \n>> +}\n>> +\n>> test_expect_success 'setup' \"\n>> cat >sparse_error_header <<-EOF &&\n>> The following pathspecs didn't match any eligible path, but they do match index\n>> @@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '\n>> setup_sparse_entry &&\n>> rm sparse_entry &&\n>> test_must_fail git add sparse_entry 2>stderr &&\n>> + test_sparse_entry_unstaged &&\n\nHere, sparse_entry could be staged as a deletion.\n\n>> test_cmp error_and_hint stderr &&\n>> test_sparse_entry_unchanged\n>> '\n>> @@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n>> rm sparse_entry &&\n>> setup_gitignore &&\n>> test_must_fail git add . 2>stderr &&\n>> + test_sparse_entry_unstaged &&\n\nDeletion here.\n\n>> cat sparse_error_header >expect &&\n>> echo . >>expect &&\n>> @@ -88,6 +96,7 @@ do\n>> setup_sparse_entry &&\n>> echo modified >sparse_entry &&\n>> test_must_fail git add $opt sparse_entry 2>stderr &&\n>> + test_sparse_entry_unstaged &&\n\nBut here would be modified.\n\n>> test_cmp error_and_hint stderr &&\n>> test_sparse_entry_unchanged\n>> '\n>> @@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n>> git ls-files --debug sparse_entry | grep mtime >before &&\n>> test-tool chmtime -60 sparse_entry &&\n>> test_must_fail git add --refresh sparse_entry 2>stderr &&\n>> + test_sparse_entry_unstaged &&\n\nSame here.\n\n>> test_cmp error_and_hint stderr &&\n>> git ls-files --debug sparse_entry | grep mtime >after &&\n>> test_cmp before after\n>> @@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n>> test_expect_success 'git add --chmod does not update sparse entries' '\n>> setup_sparse_entry &&\n>> test_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n>> + test_sparse_entry_unstaged &&\n\nHere it would be modified with a mode change.\n\nUsing the pattern \"^[MDARCU][M ] sparse_entry\\$\" seems to work while also\ncovering these other cases.\n\nThanks,\n-Stolee\n"},{"id":"435989","messageId":"CAHd-oW59G6MF04aUA-XJewa=0VnqbHenb2khkeO7U9egW_6S4A@mail.gmail.com","threadId":"56370","inReplyTo":"8aefce6254c0bcbbbca909a62d033c74c90f980b.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-09-15T16:32:57Z","receivedAt":"2021-09-15T16:33:11Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Sun, Sep 12, 2021 at 10:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The tests in t3705-add-sparse-checkout.sh check to see how 'git add'\n> behaves with paths outside the sparse-checkout definition. These\n> currently check to see if a given warning is present but not that the\n> index is not updated with the sparse entries.\n\nHmm, I probably missed something, but don't we already check that with\nthe `test_sparse_entry_unchanged` helper? The only test case that we\ndon't call it is 'git add --refresh does not update sparse entries',\nbut we explicitly compare the cached 'mtime' from before and after\n`git add` there.\n"},{"id":"435995","messageId":"96e63435-f8d6-73c7-15f9-f713fd742931@gmail.com","threadId":"56370","inReplyTo":"CAHd-oW59G6MF04aUA-XJewa=0VnqbHenb2khkeO7U9egW_6S4A@mail.gmail.com","subject":"Re: [PATCH v2 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-15T16:42:10Z","receivedAt":"2021-09-15T16:42:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/15/2021 12:32 PM, Matheus Tavares wrote:\n> On Sun, Sep 12, 2021 at 10:23 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The tests in t3705-add-sparse-checkout.sh check to see how 'git add'\n>> behaves with paths outside the sparse-checkout definition. These\n>> currently check to see if a given warning is present but not that the\n>> index is not updated with the sparse entries.\n> \n> Hmm, I probably missed something, but don't we already check that with\n> the `test_sparse_entry_unchanged` helper? The only test case that we\n> don't call it is 'git add --refresh does not update sparse entries',\n> but we explicitly compare the cached 'mtime' from before and after\n> `git add` there.\n\ntest_sparse_entry_unchanged does a bit more by actually\nrequiring that we fully know the mode and OID of the object\nin the index. \n\nSince some of the tests modify sparse_entry and then update\nthe index using the --sparse option (including a mode change\nwith --chmod=x), it seems more robust to avoid an exact\nmatch from ls-files.\n\nThanks,\n-Stolee\n"},{"id":"435996","messageId":"3252cc19-46ec-bbbe-6a02-fe02cb767dff@gmail.com","threadId":"56370","inReplyTo":"CABPp-BFitDu8UFkH5Syd3w7_e4sT5nJCJ_6N5sn3a+Acv6B4aQ@mail.gmail.com","subject":"Re: [PATCH v2 04/14] dir: select directories correctly","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-15T16:43:08Z","receivedAt":"2021-09-15T16:43:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/15/2021 10:54 AM, Elijah Newren wrote:\n>> + /*\n>> + * Use 'alloc' as an indicator that the string has not been\n>> + * initialized, in case the parent is the root directory.\n>> + */\n>> + if (!path_parent->alloc) {\n>> + char *slash;\n>> + strbuf_addstr(path_parent, pathname);\n>> + slash = find_last_dir_sep(path_parent->buf);\n>> +\n>> + if (slash)\n>> + *slash = '\\0';\n> \n> Are you breaking strbuf invariants here? path_parent->len will not be\n> corrected by this string manipulation. Perhaps replace this if-else\n> block with\n> \n> strbuf_setlen(path_parent, slash ? slash - path_parent->buf : 0)\n\nYes, I am. I noticed and fixed this when I was rewriting this\npatch for Ævar's feedback. Thanks for pointing it out.\n\nThanks,\n-Stolee\n"},{"id":"435997","messageId":"07454056-0123-876e-1a03-2b6b01497c1c@gmail.com","threadId":"56370","inReplyTo":"87pmtd5tkj.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 14/14] advice: update message to suggest '--sparse'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-15T16:54:11Z","receivedAt":"2021-09-15T16:54:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/12/2021 5:58 PM, Ævar Arnfjörð Bjarmason wrote:\n...\n>> \tcat >sparse_error_header <<-EOF &&\n>> -\tThe following pathspecs didn't match any eligible path, but they do match index\n>> -\tentries outside the current sparse checkout:\n>> +\tThe following paths and/or pathspecs matched paths that exist outside of your\n>> +\tsparse-checkout definition, so will not be updated in the index:\n>> \tEOF\n>> \n>> \tcat >sparse_hint <<-EOF &&\n>> -\thint: Disable or modify the sparsity rules if you intend to update such entries.\n>> +\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n>> \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n>> \tEOF\n> \n> ...this used to line-wrap at 80 characters, but is now a bit beyond\n> that.\n> \n> Maybe instead make these two into bullet-points?\n\nDo you mean something like this?\n\n\thint: If you intend to update such entries, try one of the following:\n\thint: * Use the --sparse option.\n\thint: * Disable or modify the sparsity rules.\n\n> Also the third \"Disable\" looks a bit jarring at first, it seems like a\n> continuation of the first message, but it's just the standard \"disable\n> this message\" we tend to print out.\n\nWith the bullet points, this is no longer a concern.\n\n> This commentary pre-dates this commit, but just in general:\n> \n> I think the advice system is best used where there's an initial\n> non-optional message, and then the advice elaborates on what happened,\n> how to fix it. A good example is the \"short object ID %s is ambiguous\"\n> in object-name.c.\n>\n> But in this case both messages are rather long. I'd think better would\n> be something like (and I didn't look very deeply at the involved code):\n> \n> error(\"pathspec '%s' matched only outside sparse checkout\")\n> \n> I.e. in e.g. cmd_rm() we loop through the pathspecs, and we error on the\n> first one, so to a first approximation why do we need to for sparse emit\n> ALL the pathspecs we didn't match? if we're going to error out anyway\n> shouldn'w we just error out on the first one?\n\nIf we don't list the entire set, then users will need to use trial\nand error to discover how to get out of a bad state.\n \n> But going on, I'd think this would be better overall (pseudocode):\n> \n> error(\"pathspec '%s' matched only outside sparse checkout\")\n> if (advice_enabled(ADVICE_UPDATE_SPARSE_PATH)) {\n> char *list_str;\n> list_of_bad_pathspecs = make_that_list(my_pathspec_string_list, &list_str);\n> \n> if (list_of_bad_pathspecs.nr > 1)\n> /* Emit a message that details what's wrong, but also has a\n> * list of all the other pathspecs we'd also die on if the user */\n> else\n> /* Ditto, but no list *?\n\nI'm not a fan of this \"here's an error message for the first thing, but\nthe advice gives all the details\" approach.\n\n> Maybe I'm missing something with the sparse implemention, but I'd think\n> going above & beyond and listing all failures is a bit much in either\n> case, i.e. for non-sparse we have:\n> \n> $ git rm 'file-i-do-not-have' 'directory-i-do-not-have/'\n> fatal: pathspec 'file-i-do-not-have' did not match any files\n> $\n> \n> I'd think in general a user who's screwed up and typo'd both isn't going\n> to be much harmed by us noting the first, maybe they'll get another\n> error then.\n> \n> But usually it's obvious (e.g. you just ran the command in the wrong\n> directory), so if you have a large list of pathspecs getting a firehose\n> of all the things that didn't match can be less helpful due to being\n> overrly verbose.\n\nThe difference here is a die() versus an error(). It would probably\nbe better to convert the die() into an error() and report all failures\nrather than have the sparse-checkout changes start short-circuiting\nand providing less data.\n\nThanks,\n-Stolee\n"},{"id":"436001","messageId":"CABPp-BFvCpfq66oJoxgABJy3LP=VNDwtd3YkTwV-nEQjXm2_Bg@mail.gmail.com","threadId":"56370","inReplyTo":"944ae2cffa8ff175cd1cee0b3a25060ec5599973.1631453010.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/14] add: implement the --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-15T16:59:16Z","receivedAt":"2021-09-15T16:59:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> We previously modified 'git add' to refuse updating index entries\n> outside of the sparse-checkout cone. This is justified to prevent users\n> from accidentally getting into a confusing state when Git removes those\n> files from the working tree at some later point.\n>\n> Unfortunately, this caused some workflows that were previously possible\n> to become impossible, especially around merge conflicts outside of the\n> sparse-checkout cone. These were documented in tests within t1092.\n>\n> We now re-enable these workflows using a new '--sparse' option to 'git\n> add'. This allows users to signal \"Yes, I do know what I'm doing with\n> these files,\" and accept the consequences of the files leaving the\n> worktree later.\n>\n> We delay updating the advice message until implementing a similar option\n> in 'git rm' and 'git mv'.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-add.txt | 9 +++++++-\n> builtin/add.c | 12 +++++++----\n> t/t1092-sparse-checkout-compatibility.sh | 27 ++++++++----------------\n> t/t3705-add-sparse-checkout.sh | 17 ++++++++++++++-\n> 4 files changed, 41 insertions(+), 24 deletions(-)\n>\n> diff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\n> index be5e3ac54b8..bb79016d2ca 100644\n> --- a/Documentation/git-add.txt\n> +++ b/Documentation/git-add.txt\n> @@ -9,7 +9,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n> - [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n> + [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]\n> [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n> [--chmod=(+|-)x] [--pathspec-from-file= [--pathspec-file-nul]]\n> [--] [...]\n> @@ -79,6 +79,13 @@ in linkgit:gitglossary[7].\n> --force::\n> Allow adding otherwise ignored files.\n>\n> +--sparse::\n> + Allow updating index entries outside of the sparse-checkout cone.\n> + Normally, `git add` refuses to update index entries whose paths do\n> + not fit within the sparse-checkout cone, since those files might\n> + be removed from the working tree without warning. See\n> + linkgit:git-sparse-checkout[1] for more.\n\nfor more ...? details? I find the last sentence incomplete.\n\nFollowing that track for a moment and thinking out loud, I wonder if\nwe want more details somewhere in the sparse-checkout docs about this\nissue and if so, if we should point to that specific part of that\npage. The 'reapply' section of the sparse-checkout page kind of\ntouches on the topic of the worktree not exactly matching sparsity\npatterns (due to other commands), but focuses on unsparsifying files\ndue to conflicts and kind of ignores the re-sparsification that\nhappens in other commands after the working copy matches the index.\n(Such a documentation improvement could come after your series, as I\nsaid, I'm just thinking out loud.)\n\n> +\n> -i::\n> --interactive::\n> Add modified contents in the working tree interactively to\n> diff --git a/builtin/add.c b/builtin/add.c\n> index 09c3fad6321..f8e3930608d 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;\n> static int take_worktree_changes;\n> static int add_renormalize;\n> static int pathspec_file_nul;\n> +static int include_sparse;\n> static const char *pathspec_from_file;\n> static int legacy_stash_p; /* support for the scripted `git stash` */\n>\n> @@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n> struct cache_entry *ce = active_cache[i];\n> int err;\n>\n> - if (ce_skip_worktree(ce))\n> + if (!include_sparse && ce_skip_worktree(ce))\n> continue;\n>\n> if (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n> @@ -95,7 +96,7 @@ static void update_callback(struct diff_queue_struct *q,\n> struct diff_filepair *p = q->queue[i];\n> const char *path = p->one->path;\n>\n> - if (!path_in_sparse_checkout(path, &the_index))\n> + if (!include_sparse && !path_in_sparse_checkout(path, &the_index))\n> continue;\n>\n> switch (fix_unmerged_status(p, data)) {\n> @@ -383,6 +384,7 @@ static struct option builtin_add_options[] = {\n> OPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n> OPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n> OPT_BOOL( 0 , \"ignore-missing\", &ignore_missing, N_(\"check if - even missing - files are ignored in dry run\")),\n> + OPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n> OPT_STRING(0, \"chmod\", &chmod_arg, \"(+|-)x\",\n> N_(\"override the executable bit of the listed files\")),\n> OPT_HIDDEN_BOOL(0, \"warn-embedded-repo\", &warn_on_embedded_repo,\n> @@ -461,7 +463,8 @@ static int add_files(struct dir_struct *dir, int flags)\n> }\n>\n> for (i = 0; i < dir->nr; i++) {\n> - if (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n> + if (!include_sparse &&\n> + !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n> string_list_append(&matched_sparse_paths,\n> dir->entries[i]->name);\n> continue;\n> @@ -646,7 +649,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n> if (seen[i])\n> continue;\n>\n> - if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n> + if (!include_sparse &&\n> + matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n> string_list_append(&only_match_skip_worktree,\n> pathspec.items[i].original);\n> continue;\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 0fdc5c7098c..7d64d9deb22 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -343,11 +343,7 @@ test_expect_success 'commit including unstaged changes' '\n> test_all_match git status --porcelain=v2\n> '\n>\n> -# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n> -# attempted here fail for the sparse-checkout and sparse-index repos.\n> -# We must enable a way for adding files outside the sparse-checkout\n> -# done, even if it is by an optional flag.\n> -test_expect_failure 'status/add: outside sparse cone' '\n> +test_expect_success 'status/add: outside sparse cone' '\n> init_repos &&\n>\n> # folder1 is at HEAD, but outside the sparse cone\n> @@ -375,15 +371,16 @@ test_expect_failure 'status/add: outside sparse cone' '\n> test_sparse_match test_must_fail git add folder1/new &&\n> test_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> test_sparse_unstaged folder1/new &&\n> + test_sparse_match git add --sparse folder1/a &&\n> + test_sparse_match git add --sparse folder1/new &&\n>\n> - # NEEDSWORK: behavior begins to deviate here.\n> - test_all_match git add . &&\n> + test_all_match git add --sparse . &&\n> test_all_match git status --porcelain=v2 &&\n> test_all_match git commit -m folder1/new &&\n> test_all_match git rev-parse HEAD^{tree} &&\n>\n> run_on_all ../edit-contents folder1/newer &&\n> - test_all_match git add folder1/ &&\n> + test_all_match git add --sparse folder1/ &&\n> test_all_match git status --porcelain=v2 &&\n> test_all_match git commit -m folder1/newer &&\n> test_all_match git rev-parse HEAD^{tree}\n> @@ -527,12 +524,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n> done\n> '\n>\n> -# NEEDSWORK: This test is documenting current behavior, but that\n> -# behavior can be confusing to users so there is desire to change it.\n> -# Right now, users might be using this flow to work through conflicts,\n> -# so any solution should present advice to users who try this sequence\n> -# of commands to follow whatever new method we create.\n> -test_expect_failure 'merge with conflict outside cone' '\n> +test_expect_success 'merge with conflict outside cone' '\n\nBased on the comments on the next hunk, I also wonder if this hunk\ndoesn't belong in the previous commit...\n\n> init_repos &&\n>\n> test_all_match git checkout -b merge-tip merge-left &&\n> @@ -555,9 +547,6 @@ test_expect_failure 'merge with conflict outside cone' '\n> # 3. Rename the file to another sparse filename and\n> # accept conflict markers as resolved content.\n> run_on_all mv folder2/a folder2/z &&\n> - # NEEDSWORK: This mode now fails, because folder2/z is\n> - # outside of the sparse-checkout cone and does not match an\n> - # existing index entry with the SKIP_WORKTREE bit cleared.\n> test_sparse_match test_must_fail git add folder2 &&\n> test_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> test_sparse_unstaged folder2/z &&\n\nIs this hunk in the wrong commit? You added a --sparse flag to the\ngit add a few lines below in the previous commit, so it seems the\nNEEDSWORK comment should have been removed at the same time.\n\n> @@ -569,7 +558,7 @@ test_expect_failure 'merge with conflict outside cone' '\n> test_all_match git rev-parse HEAD^{tree}\n> '\n>\n> -test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n> +test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n> init_repos &&\n>\n> for OPERATION in cherry-pick rebase\n> @@ -592,6 +581,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n> test_sparse_match test_must_fail git add folder1/a &&\n> test_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> test_sparse_unstaged folder1/a &&\n> + test_all_match git add --sparse folder1/a &&\n> test_all_match git status --porcelain=v2 &&\n>\n> # 3. Rename the file to another sparse filename and\n> @@ -603,6 +593,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n> test_sparse_match test_must_fail git add folder2 &&\n> test_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n> test_sparse_unstaged folder2/z &&\n> + test_all_match git add --sparse folder2 &&\n> test_all_match git status --porcelain=v2 &&\n>\n> test_all_match git $OPERATION --continue &&\n> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n> index 678be1acbf9..0f7e03b5326 100755\n> --- a/t/t3705-add-sparse-checkout.sh\n> +++ b/t/t3705-add-sparse-checkout.sh\n> @@ -167,7 +167,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n>\n> git update-index --no-skip-worktree sparse_entry &&\n> test_must_fail git add sparse_entry &&\n> - test_sparse_entry_unstaged\n> + test_sparse_entry_unstaged &&\n> +\n> + # Avoid munging CRLFs to avoid an error message\n> + git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n> + test_must_be_empty stderr &&\n> + test-tool read-cache --table >actual &&\n> + grep \"^100644 blob.*sparse_entry\\$\" actual\n\nDoes this CRLF anti-munging belong in a separate commit somewhere? I\nwas surprised to see it, and it's not clear to me how it relates to\nthe other changes. Am I missing something?\n\n> '\n>\n> test_expect_success 'add obeys advice.updateSparsePath' '\n> @@ -178,4 +184,13 @@ test_expect_success 'add obeys advice.updateSparsePath' '\n>\n> '\n>\n> +test_expect_success 'add allows sparse entries with --sparse' '\n> + git sparse-checkout set a &&\n> + echo modified >sparse_entry &&\n> + test_must_fail git add sparse_entry &&\n> + test_sparse_entry_unchanged &&\n> + git add --sparse sparse_entry 2>stderr &&\n> + test_must_be_empty stderr\n> +'\n> +\n> test_done\n> --\n> gitgitgadget\n"},{"id":"436041","messageId":"CABPp-BHCovLmjpHG0mhYJ5CSR0p2kMqmLaKq4GihyokJ5YnJoQ@mail.gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-15T20:18:42Z","receivedAt":"2021-09-15T20:19:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This series is based on ds/mergies-with-sparse-index.\n>\n> As requested, this series looks to update the behavior of git add, git rm,\n> and git mv when they attempt to modify paths outside of the sparse-checkout\n> cone. In particular, this care is expanded to not just cache entries with\n> the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\n> definition.\n>\n> This means that commands that worked before this series can now fail. In\n> particular, if 'git merge' results in a conflict outside of the\n> sparse-checkout cone, then 'git add ' will now fail.\n>\n> In order to allow users to circumvent these protections, a new '--sparse'\n> option is added that ignores the sparse-checkout patterns and the\n> SKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\n> assist with discovery of this option.\n>\n> There is a subtle issue with git mv in that it does not check the index\n> until it discovers a directory and then uses the index to find the contained\n> entries. This means that in non-cone-mode patterns, a pattern such as\n> \"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n>\n> In order to allow for checking arbitrary paths against the sparse-checkout\n> patterns, some changes to the underlying pattern matching code is required.\n> It turns out that there are some bugs in the methods as advertised, but\n> these bugs were never discovered because of the way methods like\n> unpack_trees() will check a directory for a pattern match before checking\n> its contained paths. Our new \"check patterns on-demand\" approach pokes holes\n> in that approach, specifically with patterns that match entire directories.\n>\n>\n> Updates in v2\n> =============\n>\n> * I got no complaints about these restrictions, so this is now a full\n> series, not RFC.\n>\n> * Thanks to Matheus, several holes are filled with extra testing and\n> bugfixes.\n>\n> * New patches add --chmod and --renormalize improvements. These are added\n> after the --sparse option to make them be one change each.\n\nSorry for taking so long, but I finally read through the series. Only\nhad a few small comments here and there; the high level direction\nlooks good.\n"},{"id":"436432","messageId":"3c4332f0-5c88-033f-b294-4fe50d010b1c@gmail.com","threadId":"56370","inReplyTo":"CABPp-BFvCpfq66oJoxgABJy3LP=VNDwtd3YkTwV-nEQjXm2_Bg@mail.gmail.com","subject":"Re: [PATCH v2 08/14] add: implement the --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-20T15:45:05Z","receivedAt":"2021-09-20T15:45:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/15/2021 12:59 PM, Elijah Newren wrote:\n> On Sun, Sep 12, 2021 at 6:23 AM Derrick Stolee via GitGitGadget\n...\n>> +--sparse::\n>> + Allow updating index entries outside of the sparse-checkout cone.\n>> + Normally, `git add` refuses to update index entries whose paths do\n>> + not fit within the sparse-checkout cone, since those files might\n>> + be removed from the working tree without warning. See\n>> + linkgit:git-sparse-checkout[1] for more.\n> \n> for more ...? details? I find the last sentence incomplete.\n\nI'll add \"details\"\n\n> Following that track for a moment and thinking out loud, I wonder if\n> we want more details somewhere in the sparse-checkout docs about this\n> issue and if so, if we should point to that specific part of that\n> page. The 'reapply' section of the sparse-checkout page kind of\n> touches on the topic of the worktree not exactly matching sparsity\n> patterns (due to other commands), but focuses on unsparsifying files\n> due to conflicts and kind of ignores the re-sparsification that\n> happens in other commands after the working copy matches the index.\n> (Such a documentation improvement could come after your series, as I\n> said, I'm just thinking out loud.)\n\nThis is a good idea to include in a new \"Troubleshooting\" section.\n\n>> -# NEEDSWORK: This test is documenting current behavior, but that\n>> -# behavior can be confusing to users so there is desire to change it.\n>> -# Right now, users might be using this flow to work through conflicts,\n>> -# so any solution should present advice to users who try this sequence\n>> -# of commands to follow whatever new method we create.\n>> -test_expect_failure 'merge with conflict outside cone' '\n>> +test_expect_success 'merge with conflict outside cone' '\n> \n> Based on the comments on the next hunk, I also wonder if this hunk\n> doesn't belong in the previous commit...\n\nYou are absolutely right, I squashed the wrong commits. Thanks.\n\n-Stolee\n"},{"id":"436462","messageId":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"[PATCH v3 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:23Z","receivedAt":"2021-09-20T23:37:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/mergies-with-sparse-index.\n\nAs requested, this series looks to update the behavior of git add, git rm,\nand git mv when they attempt to modify paths outside of the sparse-checkout\ncone. In particular, this care is expanded to not just cache entries with\nthe SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\ndefinition.\n\nThis means that commands that worked before this series can now fail. In\nparticular, if 'git merge' results in a conflict outside of the\nsparse-checkout cone, then 'git add ' will now fail.\n\nIn order to allow users to circumvent these protections, a new '--sparse'\noption is added that ignores the sparse-checkout patterns and the\nSKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\nassist with discovery of this option.\n\nThere is a subtle issue with git mv in that it does not check the index\nuntil it discovers a directory and then uses the index to find the contained\nentries. This means that in non-cone-mode patterns, a pattern such as\n\"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n\nIn order to allow for checking arbitrary paths against the sparse-checkout\npatterns, some changes to the underlying pattern matching code is required.\nIt turns out that there are some bugs in the methods as advertised, but\nthese bugs were never discovered because of the way methods like\nunpack_trees() will check a directory for a pattern match before checking\nits contained paths. Our new \"check patterns on-demand\" approach pokes holes\nin that approach, specifically with patterns that match entire directories.\n\n\nUpdates in v3\n=============\n\n * Fixed an incorrectly-squashed commit. Spread out some changes in a better\n way. For example, I don't add --sparse to tests before introducing the\n option.\n\n * Use a NULL struct strbuf pointer to indicate an uninitialized value\n instead of relying on an internal member.\n\n * Use grep over test_i18ngrep.\n\n * Fixed line wrapping for error messages.\n\n * Use strbuf_setlen() over modifying the len member manually.\n\n\nUpdates in v2\n=============\n\n * I got no complaints about these restrictions, so this is now a full\n series, not RFC.\n\n * Thanks to Matheus, several holes are filled with extra testing and\n bugfixes.\n\n * New patches add --chmod and --renormalize improvements. These are added\n after the --sparse option to make them be one change each.\n\nThanks, -Stolee\n\nDerrick Stolee (14):\n t3705: test that 'sparse_entry' is unstaged\n t1092: behavior for adding sparse files\n dir: extract directory-matching logic\n dir: select directories correctly\n dir: fix pattern matching on dirs\n add: fail when adding an untracked sparse file\n add: skip tracked paths outside sparse-checkout cone\n add: implement the --sparse option\n add: update --chmod to skip sparse paths\n add: update --renormalize to skip sparse paths\n rm: add --sparse option\n rm: skip sparse paths with missing SKIP_WORKTREE\n mv: refuse to move sparse paths\n advice: update message to suggest '--sparse'\n\n Documentation/git-add.txt | 9 +-\n Documentation/git-rm.txt | 6 +\n advice.c | 11 +-\n builtin/add.c | 32 +++-\n builtin/mv.c | 52 +++++--\n builtin/rm.c | 10 +-\n dir.c | 56 ++++++-\n pathspec.c | 5 +-\n t/t1091-sparse-checkout-builtin.sh | 4 +-\n t/t1092-sparse-checkout-compatibility.sh | 75 +++++++--\n t/t3602-rm-sparse-checkout.sh | 40 ++++-\n t/t3705-add-sparse-checkout.sh | 68 +++++++-\n t/t7002-mv-sparse-checkout.sh | 189 +++++++++++++++++++++++\n 13 files changed, 505 insertions(+), 52 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\n\nbase-commit: 516680ba7704c473bb21628aa19cabbd787df4db\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1018\n\nRange-diff vs v2:\n\n 1: 8aefce6254c ! 1: ea940f10a7c t3705: test that 'sparse_entry' is unstaged\n @@ Commit message\n correctly.\n \n We need to modify setup_sparse_entry to actually commit the sparse_entry\n - file so it exists at HEAD but is not already staged in the index.\n + file so it exists at HEAD and as an entry in the index, but its exact\n + contents are not staged in the index.\n \n Signed-off-by: Derrick Stolee \n \n @@ t/t3705-add-sparse-checkout.sh: setup_gitignore () {\n \n +test_sparse_entry_unstaged () {\n +\tgit status --porcelain >actual &&\n -+\t! grep \"^M sparse_entry\\$\" actual\n ++\t! grep \"^[MDARCU][M ] sparse_entry\\$\" actual\n +}\n +\n test_expect_success 'setup' \"\n 2: 61c23dc59a6 ! 2: c7dedb41291 t1092: behavior for adding sparse files\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n +\trun_on_sparse ../edit-contents folder1/a &&\n +\trun_on_sparse ../edit-contents folder1/newfile &&\n +\ttest_sparse_match test_must_fail git add folder1/a &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/a &&\n +\ttest_sparse_match git add folder1/newfile\n +'\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outsi\n \n \t# Adding the path outside of the sparse-checkout cone should fail.\n \ttest_sparse_match test_must_fail git add folder1/a &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/a &&\n \n \t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n 3: 747b52e7bde = 3: b1f6468f9cd dir: extract directory-matching logic\n 4: 723de4e4258 ! 4: 0252c7ee15c dir: select directories correctly\n @@ Commit message\n contained files. However, other commands will start matching individual\n files against pattern lists without that recursive approach.\n \n - We modify path_matches_dir_pattern() to take a strbuf 'path_parent' that\n - is used to store the parent directory of 'pathname' between multiple\n - pattern matching tests. This is loaded lazily, only on the first pattern\n - it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n + We modify path_matches_dir_pattern() to take a strbuf pointer\n + 'path_parent' that is used to store the parent directory of 'pathname'\n + between multiple pattern matching tests. This is loaded lazily, only on\n + the first pattern it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n \n If we find that a path has a parent directory, we start by checking to\n see if that parent directory matches the pattern. If so, then we do not\n @@ Commit message\n the more general -- and slower -- full pattern set. Thus, this does not\n affect the sparse index.\n \n + Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n \n ## dir.c ##\n @@ dir.c: int match_pathname(const char *pathname, int pathlen,\n \n static int path_matches_dir_pattern(const char *pathname,\n \t\t\t\t int pathlen,\n -+\t\t\t\t struct strbuf *path_parent,\n ++\t\t\t\t struct strbuf **path_parent,\n \t\t\t\t int *dtype,\n \t\t\t\t struct path_pattern *pattern,\n \t\t\t\t struct index_state *istate)\n {\n -+\t/*\n -+\t * Use 'alloc' as an indicator that the string has not been\n -+\t * initialized, in case the parent is the root directory.\n -+\t */\n -+\tif (!path_parent->alloc) {\n ++\tif (!*path_parent) {\n +\t\tchar *slash;\n -+\t\tstrbuf_addstr(path_parent, pathname);\n -+\t\tslash = find_last_dir_sep(path_parent->buf);\n ++\t\tCALLOC_ARRAY(*path_parent, 1);\n ++\t\tstrbuf_add(*path_parent, pathname, pathlen);\n ++\t\tslash = find_last_dir_sep((*path_parent)->buf);\n +\n +\t\tif (slash)\n -+\t\t\t*slash = '\\0';\n ++\t\t\tstrbuf_setlen(*path_parent, slash - (*path_parent)->buf);\n +\t\telse\n -+\t\t\tstrbuf_setlen(path_parent, 0);\n ++\t\t\tstrbuf_setlen(*path_parent, 0);\n +\t}\n +\n +\t/*\n +\t * If the parent directory matches the pattern, then we do not\n +\t * need to check for dtype.\n +\t */\n -+\tif (path_parent->len &&\n -+\t match_pathname(path_parent->buf, path_parent->len,\n ++\tif ((*path_parent)->len &&\n ++\t match_pathname((*path_parent)->buf, (*path_parent)->len,\n +\t\t\t pattern->base,\n +\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n +\t\t\t pattern->pattern, pattern->nowildcardlen,\n @@ dir.c: static struct path_pattern *last_matching_pattern_from_list(const char *p\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n \tint i;\n -+\tstruct strbuf path_parent = STRBUF_INIT;\n ++\tstruct strbuf *path_parent = NULL;\n \n \tif (!pl->nr)\n \t\treturn NULL;\t/* undefined */\n @@ dir.c: static struct path_pattern *last_matching_pattern_from_list(const char *p\n \t\t\tbreak;\n \t\t}\n \t}\n -+\tstrbuf_release(&path_parent);\n ++\n ++\tif (path_parent) {\n ++\t\tstrbuf_release(path_parent);\n ++\t\tfree(path_parent);\n ++\t}\n ++\n \treturn res;\n }\n \n 5: ed2472d33f1 = 5: c6d17df5e5d dir: fix pattern matching on dirs\n 6: bac6cedfc8d ! 6: 3dd1d6c228c add: fail when adding an untracked sparse file\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n - \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n -\ttest_sparse_match git add folder1/newfile\n +\ttest_sparse_match test_must_fail git add folder1/newfile &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/newfile\n '\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including\n \t# folder1 is at HEAD, but outside the sparse cone\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n - \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n +\ttest_sparse_match test_must_fail git add folder1/new &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/new &&\n \n -\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n 7: d4706e17cae ! 7: 15039e031e5 add: skip tracked paths outside sparse-checkout cone\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli\n -\t# warn that this is a problematic add.\n -\ttest_all_match git add folder1/a &&\n +\ttest_sparse_match test_must_fail git add folder1/a &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder1/a &&\n -+\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli\n \t# existing index entry with the SKIP_WORKTREE bit cleared.\n -\ttest_all_match git add folder2 &&\n +\ttest_sparse_match test_must_fail git add folder2 &&\n -+\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\ttest_sparse_unstaged folder2/z &&\n -+\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebas\n \t\t# warn that this is a problematic add.\n -\t\ttest_all_match git add folder1/a &&\n +\t\ttest_sparse_match test_must_fail git add folder1/a &&\n -+\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\t\ttest_sparse_unstaged folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebas\n \t\trun_on_all mv folder2/a folder2/z &&\n -\t\ttest_all_match git add folder2 &&\n +\t\ttest_sparse_match test_must_fail git add folder2 &&\n -+\t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n ++\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n +\t\ttest_sparse_unstaged folder2/z &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n 8: 944ae2cffa8 ! 8: 6014ac8ab9e add: implement the --sparse option\n @@ Documentation/git-add.txt: in linkgit:gitglossary[7].\n +\tNormally, `git add` refuses to update index entries whose paths do\n +\tnot fit within the sparse-checkout cone, since those files might\n +\tbe removed from the working tree without warning. See\n -+\tlinkgit:git-sparse-checkout[1] for more.\n ++\tlinkgit:git-sparse-checkout[1] for more details.\n +\n -i::\n --interactive::\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'commit including\n \t# folder1 is at HEAD, but outside the sparse cone\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/new &&\n - \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/new &&\n +\ttest_sparse_match git add --sparse folder1/a &&\n +\ttest_sparse_match git add --sparse folder1/new &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'merge, cherry-pic\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n + \ttest_sparse_match test_must_fail git add folder1/a &&\n + \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \ttest_sparse_unstaged folder1/a &&\n ++\ttest_all_match git add --sparse folder1/a &&\n + \ttest_all_match git status --porcelain=v2 &&\n + \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli\n -\t# outside of the sparse-checkout cone and does not match an\n -\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n - \ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder2/z &&\n ++\ttest_all_match git add --sparse folder2 &&\n + \ttest_all_match git status --porcelain=v2 &&\n + \n + \ttest_all_match git merge --continue &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'merge with confli\n \tfor OPERATION in cherry-pick rebase\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder1/a &&\n - \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder1/a &&\n +\t\ttest_all_match git add --sparse folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebas\n \t\t# 3. Rename the file to another sparse filename and\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder2 &&\n - \t\ttest_i18ngrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n + \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder2/z &&\n +\t\ttest_all_match git add --sparse folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n 9: 7ace030c709 ! 9: 2bd3448be5f add: update --chmod to skip sparse paths\n @@ builtin/add.c: static int chmod_pathspec(struct pathspec *pathspec, char flip, i\n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n \n ## t/t3705-add-sparse-checkout.sh ##\n -@@ t/t3705-add-sparse-checkout.sh: setup_gitignore () {\n - \n - test_sparse_entry_unstaged () {\n - \tgit status --porcelain >actual &&\n --\t! grep \"^M sparse_entry\\$\" actual\n -+\t! grep \"^M[ M] sparse_entry\\$\" actual\n - }\n - \n - test_expect_success 'setup' \"\n @@ t/t3705-add-sparse-checkout.sh: test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n 10: f34be166097 = 10: 131beda1bc3 add: update --renormalize to skip sparse paths\n 11: fb3f86efa9b = 11: 837a9314893 rm: add --sparse option\n 12: de0c9d09ef7 = 12: cc25ce17162 rm: skip sparse paths with missing SKIP_WORKTREE\n 13: e17a529332d = 13: 63a9cd80ade mv: refuse to move sparse paths\n 14: f2abc387822 ! 14: 79a3518dc15 advice: update message to suggest '--sparse'\n @@ advice.c: void advise_on_updating_sparse_paths(struct string_list *pathspec_list\n -\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n -\t\t\t \" eligible path, but they do match index\\n\"\n -\t\t\t \"entries outside the current sparse checkout:\\n\"));\n -+\tfprintf(stderr, _(\"The following paths and/or pathspecs matched \"\n -+\t\t\t \"paths that exist outside of your\\n\"\n -+\t\t\t \"sparse-checkout definition, so will not be \"\n ++\tfprintf(stderr, _(\"The following paths and/or pathspecs matched paths that exist\\n\"\n ++\t\t\t \"outside of your sparse-checkout definition, so will not be\\n\"\n +\t\t\t \"updated in the index:\\n\"));\n \tfor_each_string_list_item(item, pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n -\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n -+\t\t\t _(\"Disable or modify the sparsity rules or\"\n -+\t\t\t \" use the --sparse option if you intend\"\n - \t\t\t \" to update such entries.\"));\n +-\t\t\t \" to update such entries.\"));\n ++\t\t\t _(\"If you intend to update such entries, try one of the following:\\n\"\n ++\t\t\t \"* Use the --sparse option.\\n\"\n ++\t\t\t \"* Disable or modify the sparsity rules.\"));\n }\n \n + void detach_advice(const char *new_name)\n \n ## t/t3602-rm-sparse-checkout.sh ##\n @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' \"\n @@ t/t3602-rm-sparse-checkout.sh: test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n -\tThe following pathspecs didn't match any eligible path, but they do match index\n -\tentries outside the current sparse checkout:\n -+\tThe following paths and/or pathspecs matched paths that exist outside of your\n -+\tsparse-checkout definition, so will not be updated in the index:\n ++\tThe following paths and/or pathspecs matched paths that exist\n ++\toutside of your sparse-checkout definition, so will not be\n ++\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n -\thint: Disable or modify the sparsity rules if you intend to update such entries.\n -+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n ++\thint: If you intend to update such entries, try one of the following:\n ++\thint: * Use the --sparse option.\n ++\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \n @@ t/t3705-add-sparse-checkout.sh: test_sparse_entry_unstaged () {\n \tcat >sparse_error_header <<-EOF &&\n -\tThe following pathspecs didn't match any eligible path, but they do match index\n -\tentries outside the current sparse checkout:\n -+\tThe following paths and/or pathspecs matched paths that exist outside of your\n -+\tsparse-checkout definition, so will not be updated in the index:\n ++\tThe following paths and/or pathspecs matched paths that exist\n ++\toutside of your sparse-checkout definition, so will not be\n ++\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n -\thint: Disable or modify the sparsity rules if you intend to update such entries.\n -+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n ++\thint: If you intend to update such entries, try one of the following:\n ++\thint: * Use the --sparse option.\n ++\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \n @@ t/t7002-mv-sparse-checkout.sh: test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n -\tThe following pathspecs didn't match any eligible path, but they do match index\n -\tentries outside the current sparse checkout:\n -+\tThe following paths and/or pathspecs matched paths that exist outside of your\n -+\tsparse-checkout definition, so will not be updated in the index:\n ++\tThe following paths and/or pathspecs matched paths that exist\n ++\toutside of your sparse-checkout definition, so will not be\n ++\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n -\thint: Disable or modify the sparsity rules if you intend to update such entries.\n -+\thint: Disable or modify the sparsity rules or use the --sparse option if you intend to update such entries.\n ++\thint: If you intend to update such entries, try one of the following:\n ++\thint: * Use the --sparse option.\n ++\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \"\n\n-- \ngitgitgadget\n"},{"id":"436463","messageId":"c7dedb41291ed6ff48dd9b9b9814e3040f4fa92c.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 02/14] t1092: behavior for adding sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:25Z","receivedAt":"2021-09-20T23:37:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests to demonstrate the current behavior around adding files\noutside of the sparse-checkout cone. Currently, untracked files are\nhandled differently from tracked files. A future change will make these\ncases be handled the same way.\n\nFurther expand checking that a failed 'git add' does not stage changes\nto the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..3fb764f5eb9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -187,6 +187,16 @@ test_sparse_match () {\n \ttest_cmp sparse-checkout-err sparse-index-err\n }\n \n+test_sparse_unstaged () {\n+\tfile=$1 &&\n+\tfor repo in sparse-checkout sparse-index\n+\tdo\n+\t\tgit -C $repo status --porcelain >$repo-out &&\n+\t\t! grep \"^A $file\\$\" $repo-out &&\n+\t\t! grep \"^M $file\\$\" $repo-out || return 1\n+\tdone\n+}\n+\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n@@ -291,6 +301,20 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+# NEEDSWORK: This documents current behavior, but is not a desirable\n+# behavior (untracked files are handled differently than tracked).\n+test_expect_success 'add outside sparse cone' '\n+\tinit_repos &&\n+\n+\trun_on_sparse mkdir folder1 &&\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_sparse ../edit-contents folder1/newfile &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match git add folder1/newfile\n+'\n+\n test_expect_success 'commit including unstaged changes' '\n \tinit_repos &&\n \n@@ -339,7 +363,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \n \t# Adding the path outside of the sparse-checkout cone should fail.\n \ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \n \t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n \ttest_sparse_match git add folder1/new &&\n-- \ngitgitgadget\n\n"},{"id":"436464","messageId":"ea940f10a7cd26c6be3693b5a800afb7d6f752f2.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:24Z","receivedAt":"2021-09-20T23:37:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe tests in t3705-add-sparse-checkout.sh check to see how 'git add'\nbehaves with paths outside the sparse-checkout definition. These\ncurrently check to see if a given warning is present but not that the\nindex is not updated with the sparse entries. Add a new\n'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\ncorrectly.\n\nWe need to modify setup_sparse_entry to actually commit the sparse_entry\nfile so it exists at HEAD and as an entry in the index, but its exact\ncontents are not staged in the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 2b1fd0d0eef..e202a2ff74a 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n \n@@ -36,6 +37,11 @@ setup_gitignore () {\n \tEOF\n }\n \n+test_sparse_entry_unstaged () {\n+\tgit status --porcelain >actual &&\n+\t! grep \"^[MDARCU][M ] sparse_entry\\$\" actual\n+}\n+\n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n \tThe following pathspecs didn't match any eligible path, but they do match index\n@@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n \trm sparse_entry &&\n \tsetup_gitignore &&\n \ttest_must_fail git add . 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \n \tcat sparse_error_header >expect &&\n \techo . >>expect &&\n@@ -88,6 +96,7 @@ do\n \t\tsetup_sparse_entry &&\n \t\techo modified >sparse_entry &&\n \t\ttest_must_fail git add $opt sparse_entry 2>stderr &&\n+\t\ttest_sparse_entry_unstaged &&\n \t\ttest_cmp error_and_hint stderr &&\n \t\ttest_sparse_entry_unchanged\n \t'\n@@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n \tgit ls-files --debug sparse_entry | grep mtime >before &&\n \ttest-tool chmtime -60 sparse_entry &&\n \ttest_must_fail git add --refresh sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \tgit ls-files --debug sparse_entry | grep mtime >after &&\n \ttest_cmp before after\n@@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n test_expect_success 'git add --chmod does not update sparse entries' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged &&\n \t! test -x sparse_entry\n@@ -116,6 +127,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\n \ttest_must_fail git add --renormalize sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -124,6 +136,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -148,6 +161,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp sparse_entry_error stderr\n \n '\n-- \ngitgitgadget\n\n"},{"id":"436465","messageId":"b1f6468f9cdb7d16f6317c71b21f4459af158e87.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:26Z","receivedAt":"2021-09-20T23:37:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe last_matching_pattern_from_list() logic performs some checks on the\nfiletype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag\nis set. This works great when setting SKIP_WORKTREE bits within\nunpack_trees(), but doesn't work well when passing an arbitrary path\nsuch as a file within a matching directory.\n\nThis change only rearranges the logic but does not change its\nfunctionality.\n\nWe will expand the path_matches_dir_pattern() method in a following\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 22 +++++++++++++++++-----\n 1 file changed, 17 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 86afa2eae00..652135df896 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1303,6 +1303,19 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t\t\t WM_PATHNAME) == 0;\n }\n \n+static int path_matches_dir_pattern(const char *pathname,\n+\t\t\t\t int pathlen,\n+\t\t\t\t int *dtype,\n+\t\t\t\t struct path_pattern *pattern,\n+\t\t\t\t struct index_state *istate)\n+{\n+\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n+\tif (*dtype != DT_DIR)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n /*\n * Scan the given exclude list in reverse to see whether pathname\n * should be ignored. The first match (i.e. the last on the list), if\n@@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n-\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n-\t\t\tif (*dtype != DT_DIR)\n-\t\t\t\tcontinue;\n-\t\t}\n+\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\t\t\t\t dtype, pattern, istate))\n+\t\t\tcontinue;\n \n \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n \t\t\tif (match_basename(basename,\n-- \ngitgitgadget\n\n"},{"id":"436466","messageId":"0252c7ee15c3cb364a7610b63d74c193cbda0728.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 04/14] dir: select directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:27Z","receivedAt":"2021-09-20T23:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen matching a path against a list of patterns, the ones that require a\ndirectory match previously did not work when a filename is specified.\nThis was fine when all pattern-matching was done within methods such as\nunpack_trees() that check a directory before recursing into the\ncontained files. However, other commands will start matching individual\nfiles against pattern lists without that recursive approach.\n\nWe modify path_matches_dir_pattern() to take a strbuf pointer\n'path_parent' that is used to store the parent directory of 'pathname'\nbetween multiple pattern matching tests. This is loaded lazily, only on\nthe first pattern it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n\nIf we find that a path has a parent directory, we start by checking to\nsee if that parent directory matches the pattern. If so, then we do not\nneed to query the index for the type (which can be expensive). If we\nfind that the parent does not match, then we still must check the type\nfrom the index for the given pathname.\n\nNote that this does not affect cone mode pattern matching, but instead\nthe more general -- and slower -- full pattern set. Thus, this does not\naffect the sparse index.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n dir.c | 36 ++++++++++++++++++++++++++++++++++--\n 1 file changed, 34 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 652135df896..9ea6cfe61cb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1305,10 +1305,35 @@ int match_pathname(const char *pathname, int pathlen,\n \n static int path_matches_dir_pattern(const char *pathname,\n \t\t\t\t int pathlen,\n+\t\t\t\t struct strbuf **path_parent,\n \t\t\t\t int *dtype,\n \t\t\t\t struct path_pattern *pattern,\n \t\t\t\t struct index_state *istate)\n {\n+\tif (!*path_parent) {\n+\t\tchar *slash;\n+\t\tCALLOC_ARRAY(*path_parent, 1);\n+\t\tstrbuf_add(*path_parent, pathname, pathlen);\n+\t\tslash = find_last_dir_sep((*path_parent)->buf);\n+\n+\t\tif (slash)\n+\t\t\tstrbuf_setlen(*path_parent, slash - (*path_parent)->buf);\n+\t\telse\n+\t\t\tstrbuf_setlen(*path_parent, 0);\n+\t}\n+\n+\t/*\n+\t * If the parent directory matches the pattern, then we do not\n+\t * need to check for dtype.\n+\t */\n+\tif ((*path_parent)->len &&\n+\t match_pathname((*path_parent)->buf, (*path_parent)->len,\n+\t\t\t pattern->base,\n+\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n+\t\t\t pattern->pattern, pattern->nowildcardlen,\n+\t\t\t pattern->patternlen, pattern->flags))\n+\t\treturn 1;\n+\n \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n \tif (*dtype != DT_DIR)\n \t\treturn 0;\n@@ -1331,6 +1356,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n \tint i;\n+\tstruct strbuf *path_parent = NULL;\n \n \tif (!pl->nr)\n \t\treturn NULL;\t/* undefined */\n@@ -1340,8 +1366,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n-\t\t !path_matches_dir_pattern(pathname, pathlen,\n+\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n \t\t\t\t\t dtype, pattern, istate))\n \t\t\tcontinue;\n \n@@ -1367,6 +1393,12 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\t\tbreak;\n \t\t}\n \t}\n+\n+\tif (path_parent) {\n+\t\tstrbuf_release(path_parent);\n+\t\tfree(path_parent);\n+\t}\n+\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"436467","messageId":"c6d17df5e5d94ab6b53663dac419c20b5e7ae824.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 05/14] dir: fix pattern matching on dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:28Z","receivedAt":"2021-09-20T23:37:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWithin match_pathname(), one successful matching category happens when\nthe pattern is equal to its non-wildcard prefix. At this point, we have\nchecked that the input 'pathname' matches the pattern up to the prefix\nlength, and then we subtraced that length from both 'patternlen' and\n'namelen'.\n\nIn the case of a directory match, this prefix match should be\nsufficient. However, the success condition only cared about _exact_\nequality here. Instead, we should allow any path that agrees on this\nprefix in the case of PATTERN_FLAG_MUSTBEDIR.\n\nThis case was not tested before because of the way unpack_trees() would\nmatch a parent directory before visiting the contained paths. This\napproach is changing, so we must change this comparison.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9ea6cfe61cb..174d336c30e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t * then our prefix match is all we need; we\n \t\t * do not need to call fnmatch at all.\n \t\t */\n-\t\tif (!patternlen && !namelen)\n+\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n \t\t\treturn 1;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"436468","messageId":"3dd1d6c228cc8feef95a016fadbda0d620a7caff.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 06/14] add: fail when adding an untracked sparse file","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:29Z","receivedAt":"2021-09-20T23:37:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe add_files() method in builtin/add.c takes a set of untracked files\nthat are being added by the input pathspec and inserts them into the\nindex. If these files are outside of the sparse-checkout cone, then they\ngain the SKIP_WORKTREE bit at some point. However, this was not checked\nbefore inserting into the index, so these files are added even though we\nwant to avoid modifying the index outside of the sparse-checkout cone.\n\nAdd a check within add_files() for these files and write the advice\nabout files outside of the sparse-checkout cone.\n\nThis behavior change modifies some existing tests within t1092. These\ntests intended to document how a user could interact with the existing\nbehavior in place. Many of these tests need to be marked as expecting\nfailure. A future change will allow these tests to pass by adding a flag\nto 'git add' that allows users to modify index entries outside of the\nsparse-checkout cone.\n\nThe 'submodule handling' test is intended to document what happens to\ndirectories that contain a submodule when the sparse index is enabled.\nIt is not trying to say that users should be able to add submodules\noutside of the sparse-checkout cone, so that test can be modified to\navoid that operation.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 14 +++++++++\n t/t1092-sparse-checkout-compatibility.sh | 37 ++++++++++++++++++------\n 2 files changed, 42 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 88a6c0c69fb..8ea9cae0e7a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n static int add_files(struct dir_struct *dir, int flags)\n {\n \tint i, exit_status = 0;\n+\tstruct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;\n \n \tif (dir->ignored_nr) {\n \t\tfprintf(stderr, _(ignore_error));\n@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n+\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\t\tstring_list_append(&matched_sparse_paths,\n+\t\t\t\t\t dir->entries[i]->name);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {\n \t\t\tif (!ignore_add_errors)\n \t\t\t\tdie(_(\"adding files failed\"));\n@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)\n \t\t\tcheck_embedded_repo(dir->entries[i]->name);\n \t\t}\n \t}\n+\n+\tif (matched_sparse_paths.nr) {\n+\t\tadvise_on_updating_sparse_paths(&matched_sparse_paths);\n+\t\texit_status = 1;\n+\t}\n+\n+\tstring_list_clear(&matched_sparse_paths, 0);\n+\n \treturn exit_status;\n }\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 3fb764f5eb9..be8f6f1399e 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -301,8 +301,6 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-# NEEDSWORK: This documents current behavior, but is not a desirable\n-# behavior (untracked files are handled differently than tracked).\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n@@ -312,7 +310,9 @@ test_expect_success 'add outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match git add folder1/newfile\n+\ttest_sparse_match test_must_fail git add folder1/newfile &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/newfile\n '\n \n test_expect_success 'commit including unstaged changes' '\n@@ -343,7 +343,11 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-test_expect_success 'status/add: outside sparse cone' '\n+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n+# attempted here fail for the sparse-checkout and sparse-index repos.\n+# We must enable a way for adding files outside the sparse-checkout\n+# done, even if it is by an optional flag.\n+test_expect_failure 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -368,10 +372,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/new &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/new &&\n \n-\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n-\ttest_sparse_match git add folder1/new &&\n-\n+\t# NEEDSWORK: behavior begins to deviate here.\n \ttest_all_match git add . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n@@ -527,7 +532,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n # Right now, users might be using this flow to work through conflicts,\n # so any solution should present advice to users who try this sequence\n # of commands to follow whatever new method we create.\n-test_expect_success 'merge with conflict outside cone' '\n+test_expect_failure 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -541,12 +546,18 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n+\t# NEEDSWORK: Even though the merge conflict removed the\n+\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t# warn that this is a problematic add.\n \ttest_all_match git add folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n+\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t# outside of the sparse-checkout cone and does not match an\n+\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_all_match git add folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n@@ -555,7 +566,7 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -572,11 +583,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 2. Add the file with conflict markers\n+\t\t# NEEDSWORK: Even though the merge conflict removed the\n+\t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t\t# warn that this is a problematic add.\n \t\ttest_all_match git add folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n \t\t# accept conflict markers as resolved content.\n+\t\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t\t# outside of the sparse-checkout cone and does not match an\n+\t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n \t\ttest_all_match git add folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n@@ -654,6 +671,7 @@ test_expect_success 'clean' '\n test_expect_success 'submodule handling' '\n \tinit_repos &&\n \n+\ttest_sparse_match git sparse-checkout add modules &&\n \ttest_all_match mkdir modules &&\n \ttest_all_match touch modules/a &&\n \ttest_all_match git add modules &&\n@@ -663,6 +681,7 @@ test_expect_success 'submodule handling' '\n \ttest_all_match git commit -m \"add submodule\" &&\n \n \t# having a submodule prevents \"modules\" from collapse\n+\ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n \ttest-tool -C sparse-index read-cache --table >cache &&\n \tgrep \"100644 blob .*\tmodules/a\" cache &&\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n-- \ngitgitgadget\n\n"},{"id":"436469","messageId":"15039e031e5252179eb16fb04c02ad91fbbe4952.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 07/14] add: skip tracked paths outside sparse-checkout cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:30Z","receivedAt":"2021-09-20T23:37:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen 'git add' adds a tracked file that is outside of the\nsparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file\nexists outside of the sparse-checkout cone. This is usually correct,\nexcept in the case of a merge conflict outside of the cone.\n\nModify add_pathspec_matched_against_index() to be more careful about\npaths by checking the sparse-checkout patterns in addition to the\nSKIP_WORKTREE bit. This causes 'git add' to no longer allow files\noutside of the cone that removed the SKIP_WORKTREE bit due to a merge\nconflict.\n\nWith only this change, users will only be able to add the file after\nadding the file to the sparse-checkout cone. A later change will allow\nusers to force adding even though the file is outside of the\nsparse-checkout cone.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 ++++\n pathspec.c | 5 +++--\n t/t1091-sparse-checkout-builtin.sh | 4 +++-\n t/t1092-sparse-checkout-compatibility.sh | 19 ++++++++++++-------\n t/t3705-add-sparse-checkout.sh | 12 ++++++++++++\n 5 files changed, 34 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8ea9cae0e7a..09c3fad6321 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n+\n+\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\t\tcontinue;\n+\n \t\tswitch (fix_unmerged_status(p, data)) {\n \t\tdefault:\n \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\ndiff --git a/pathspec.c b/pathspec.c\nindex 44306fdaca2..ddeeba79114 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\treturn;\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n-\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n+\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n+\t\t (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n \t\t\tcontinue;\n \t\tce_path_match(istate, ce, pathspec, seen);\n \t}\n@@ -70,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))\n \t\t ce_path_match(istate, ce, pathspec, seen);\n \t}\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 71236981e64..af99ae81b1d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat\n \tgit -C unmerged sparse-checkout disable\n '\n \n-test_expect_success 'sparse-checkout reapply' '\n+test_expect_failure 'sparse-checkout reapply' '\n \tgit clone repo tweak &&\n \n \techo dirty >tweak/deep/deeper2/a &&\n@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '\n \ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n \ttest_path_is_file tweak/folder1/a &&\n \n+\t# NEEDSWORK: We are asking to update a file outside of the\n+\t# sparse-checkout cone, but this is no longer allowed.\n \tgit -C tweak add folder1/a &&\n \tgit -C tweak sparse-checkout reapply 2>err &&\n \ttest_must_be_empty err &&\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex be8f6f1399e..ba2483732ff 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -546,10 +546,9 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n-\t# NEEDSWORK: Even though the merge conflict removed the\n-\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n-\t# warn that this is a problematic add.\n-\ttest_all_match git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n@@ -558,7 +557,9 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# NEEDSWORK: This mode now fails, because folder2/z is\n \t# outside of the sparse-checkout cone and does not match an\n \t# existing index entry with the SKIP_WORKTREE bit cleared.\n-\ttest_all_match git add folder2 &&\n+\ttest_sparse_match test_must_fail git add folder2 &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder2/z &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -586,7 +587,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# NEEDSWORK: Even though the merge conflict removed the\n \t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n \t\t# warn that this is a problematic add.\n-\t\ttest_all_match git add folder1/a &&\n+\t\ttest_sparse_match test_must_fail git add folder1/a &&\n+\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -595,7 +598,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# outside of the sparse-checkout cone and does not match an\n \t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n-\t\ttest_all_match git add folder2 &&\n+\t\ttest_sparse_match test_must_fail git add folder2 &&\n+\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder2/z &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex e202a2ff74a..5899f270d02 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -158,6 +158,18 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n \tgit ls-files --error-unmatch dense_entry\n '\n \n+test_expect_success 'git add fails outside of sparse-checkout definition' '\n+\ttest_when_finished git sparse-checkout disable &&\n+\ttest_commit a &&\n+\tgit sparse-checkout init &&\n+\tgit sparse-checkout set a &&\n+\techo >>sparse_entry &&\n+\n+\tgit update-index --no-skip-worktree sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unstaged\n+'\n+\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n-- \ngitgitgadget\n\n"},{"id":"436470","messageId":"6014ac8ab9ed699065dca21d7dfba059c98b8a52.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 08/14] add: implement the --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:31Z","receivedAt":"2021-09-20T23:37:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe previously modified 'git add' to refuse updating index entries\noutside of the sparse-checkout cone. This is justified to prevent users\nfrom accidentally getting into a confusing state when Git removes those\nfiles from the working tree at some later point.\n\nUnfortunately, this caused some workflows that were previously possible\nto become impossible, especially around merge conflicts outside of the\nsparse-checkout cone. These were documented in tests within t1092.\n\nWe now re-enable these workflows using a new '--sparse' option to 'git\nadd'. This allows users to signal \"Yes, I do know what I'm doing with\nthese files,\" and accept the consequences of the files leaving the\nworktree later.\n\nWe delay updating the advice message until implementing a similar option\nin 'git rm' and 'git mv'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-add.txt | 9 +++++++-\n builtin/add.c | 12 ++++++----\n t/t1092-sparse-checkout-compatibility.sh | 29 +++++++++---------------\n t/t3705-add-sparse-checkout.sh | 17 +++++++++++++-\n 4 files changed, 43 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex be5e3ac54b8..11eb70f16c7 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -9,7 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n-\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n+\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]\n \t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--pathspec-from-file= [--pathspec-file-nul]]\n \t [--] [...]\n@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].\n --force::\n \tAllow adding otherwise ignored files.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git add` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone, since those files might\n+\tbe removed from the working tree without warning. See\n+\tlinkgit:git-sparse-checkout[1] for more details.\n+\n -i::\n --interactive::\n \tAdd modified contents in the working tree interactively to\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 09c3fad6321..f8e3930608d 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n static int add_renormalize;\n static int pathspec_file_nul;\n+static int include_sparse;\n static const char *pathspec_from_file;\n static int legacy_stash_p; /* support for the scripted `git stash` */\n \n@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n@@ -95,7 +96,7 @@ static void update_callback(struct diff_queue_struct *q,\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n \n-\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\tif (!include_sparse && !path_in_sparse_checkout(path, &the_index))\n \t\t\tcontinue;\n \n \t\tswitch (fix_unmerged_status(p, data)) {\n@@ -383,6 +384,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n \tOPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n \tOPT_BOOL( 0 , \"ignore-missing\", &ignore_missing, N_(\"check if - even missing - files are ignored in dry run\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_STRING(0, \"chmod\", &chmod_arg, \"(+|-)x\",\n \t\t N_(\"override the executable bit of the listed files\")),\n \tOPT_HIDDEN_BOOL(0, \"warn-embedded-repo\", &warn_on_embedded_repo,\n@@ -461,7 +463,8 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n-\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\tif (!include_sparse &&\n+\t\t !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n \t\t\tstring_list_append(&matched_sparse_paths,\n \t\t\t\t\t dir->entries[i]->name);\n \t\t\tcontinue;\n@@ -646,7 +649,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\tif (seen[i])\n \t\t\t\tcontinue;\n \n-\t\t\tif (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n+\t\t\tif (!include_sparse &&\n+\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec.items[i].original);\n \t\t\t\tcontinue;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ba2483732ff..b11fc540415 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -343,11 +343,7 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n-# attempted here fail for the sparse-checkout and sparse-index repos.\n-# We must enable a way for adding files outside the sparse-checkout\n-# done, even if it is by an optional flag.\n-test_expect_failure 'status/add: outside sparse cone' '\n+test_expect_success 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -375,15 +371,16 @@ test_expect_failure 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/new &&\n+\ttest_sparse_match git add --sparse folder1/a &&\n+\ttest_sparse_match git add --sparse folder1/new &&\n \n-\t# NEEDSWORK: behavior begins to deviate here.\n-\ttest_all_match git add . &&\n+\ttest_all_match git add --sparse . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n \ttest_all_match git rev-parse HEAD^{tree} &&\n \n \trun_on_all ../edit-contents folder1/newer &&\n-\ttest_all_match git add folder1/ &&\n+\ttest_all_match git add --sparse folder1/ &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/newer &&\n \ttest_all_match git rev-parse HEAD^{tree}\n@@ -527,12 +524,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n \tdone\n '\n \n-# NEEDSWORK: This test is documenting current behavior, but that\n-# behavior can be confusing to users so there is desire to change it.\n-# Right now, users might be using this flow to work through conflicts,\n-# so any solution should present advice to users who try this sequence\n-# of commands to follow whatever new method we create.\n-test_expect_failure 'merge with conflict outside cone' '\n+test_expect_success 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -549,17 +541,16 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n+\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n-\t# NEEDSWORK: This mode now fails, because folder2/z is\n-\t# outside of the sparse-checkout cone and does not match an\n-\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder2/z &&\n+\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -567,7 +558,7 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -590,6 +581,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder1/a &&\n \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder1/a &&\n+\t\ttest_all_match git add --sparse folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -601,6 +593,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder2 &&\n \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder2/z &&\n+\t\ttest_all_match git add --sparse folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 5899f270d02..0434cdfd9c6 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -167,7 +167,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \n \tgit update-index --no-skip-worktree sparse_entry &&\n \ttest_must_fail git add sparse_entry &&\n-\ttest_sparse_entry_unstaged\n+\ttest_sparse_entry_unstaged &&\n+\n+\t# Avoid munging CRLFs to avoid an error message\n+\tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n@@ -178,4 +184,13 @@ test_expect_success 'add obeys advice.updateSparsePath' '\n \n '\n \n+test_expect_success 'add allows sparse entries with --sparse' '\n+\tgit sparse-checkout set a &&\n+\techo modified >sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unchanged &&\n+\tgit add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"436471","messageId":"2bd3448be5f688adecc7ccc2947532b2af26c5e4.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 09/14] add: update --chmod to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:32Z","receivedAt":"2021-09-20T23:37:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent staging a modification, but we skipped the\n--chmod mode. Update chmod_pathspec() to ignore cache entries whose path\nis outside of the sparse-checkout cone (unless --sparse is provided).\nAdd a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 10 +++++++++-\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f8e3930608d..f87b8134b67 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -47,7 +47,9 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 0434cdfd9c6..9347705ad1c 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -169,11 +169,19 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --chmod=+x sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"436472","messageId":"131beda1bc32d61b1063a51210048feb26755bb2.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 10/14] add: update --renormalize to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:33Z","receivedAt":"2021-09-20T23:37:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent stagins a modification, but we skipped the\n--renormalize mode. Update renormalize_tracked_files() to ignore cache\nentries whose path is outside of the sparse-checkout cone (unless\n--sparse is provided). Add a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 12 +++++++++++-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f87b8134b67..f8f0dfa4046 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -154,7 +154,9 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (ce_stage(ce))\n \t\t\tcontinue; /* do not touch unmerged paths */\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 9347705ad1c..10969efc23e 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -172,6 +172,9 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add --chmod=+x sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --renormalize sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n@@ -181,7 +184,14 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit reset &&\n+\n+\t# This will print a message over stderr on Windows.\n+\tgit add --sparse --renormalize sparse_entry &&\n+\tgit status --porcelain >actual &&\n+\tgrep \"^M sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"436473","messageId":"837a93148936d3138be47bff78069373342ce3d8.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 11/14] rm: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:34Z","receivedAt":"2021-09-20T23:37:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we did previously in 'git add', add a '--sparse' option to 'git rm'\nthat allows modifying paths outside of the sparse-checkout definition.\nThe existing checks in 'git rm' are restricted to tracked files that\nhave the SKIP_WORKTREE bit in the current index. Future changes will\ncause 'git rm' to reject removing paths outside of the sparse-checkout\ndefinition, even if they are untracked or do not have the SKIP_WORKTREE\nbit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-rm.txt | 6 ++++++\n builtin/rm.c | 8 ++++++--\n t/t3602-rm-sparse-checkout.sh | 12 ++++++++++++\n 3 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-rm.txt b/Documentation/git-rm.txt\nindex 26e9b284704..81bc23f3cdb 100644\n--- a/Documentation/git-rm.txt\n+++ b/Documentation/git-rm.txt\n@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].\n --ignore-unmatch::\n \tExit with a zero status even if no files matched.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git rm` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -q::\n --quiet::\n \t`git rm` normally outputs one line (in the form of an `rm` command)\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 8a24c715e02..4208f3f9a5f 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)\n \n static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;\n static int ignore_unmatch = 0, pathspec_file_nul;\n+static int include_sparse;\n static char *pathspec_from_file;\n \n static struct option builtin_rm_options[] = {\n@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {\n \tOPT_BOOL('r', NULL, &recursive, N_(\"allow recursive removal\")),\n \tOPT_BOOL( 0 , \"ignore-unmatch\", &ignore_unmatch,\n \t\t\t\tN_(\"exit with a zero status even if nothing matched\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_PATHSPEC_FROM_FILE(&pathspec_from_file),\n \tOPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),\n \tOPT_END(),\n@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\n@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\t\tseen_any = 1;\n \t\t\telse if (ignore_unmatch)\n \t\t\t\tcontinue;\n-\t\t\telse if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n+\t\t\telse if (!include_sparse &&\n+\t\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n \t\t\t\tstring_list_append(&only_match_skip_worktree, original);\n \t\t\telse\n \t\t\t\tdie(_(\"pathspec '%s' did not match any files\"), original);\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex e9e9a15c74c..493c8f636b8 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -43,6 +43,18 @@ test_expect_success 'recursive rm does not remove sparse entries' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'recursive rm --sparse removes sparse entries' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set \"sub/dir\" &&\n+\tgit rm --sparse -r sub &&\n+\tgit status --porcelain -uno >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'rm obeys advice.updateSparsePath' '\n \tgit reset --hard &&\n \tgit sparse-checkout set a &&\n-- \ngitgitgadget\n\n"},{"id":"436474","messageId":"cc25ce171627f60896d3f70800f207fa630aec22.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 12/14] rm: skip sparse paths with missing SKIP_WORKTREE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:35Z","receivedAt":"2021-09-20T23:39:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a path does not match the sparse-checkout cone but is somehow missing\nthe SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the\nfile. One reason a user might be in this situation is a merge conflict\noutside of the sparse-checkout cone. Removing such a file might be\nproblematic for users who are not sure what they are doing.\n\nAdd a check to path_in_sparse_checkout() when 'git rm' is checking if a\npath should be considered for deletion. Of course, this check is ignored\nif the '--sparse' option is specified, allowing users who accept the\nrisks to continue with the removal.\n\nThis also removes a confusing behavior where a user asks for a directory\nto be removed, but only the entries that are within the sparse-checkout\ndefinition are removed. Now, 'git rm ' will fail without '--sparse'\nand will succeed in removing all contained paths with '--sparse'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 4 +++-\n t/t3602-rm-sparse-checkout.sh | 19 +++++++++++++++++--\n 2 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4208f3f9a5f..a6da03da2be 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 493c8f636b8..5f92b60a56a 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -37,9 +37,13 @@ done\n test_expect_success 'recursive rm does not remove sparse entries' '\n \tgit reset --hard &&\n \tgit sparse-checkout set sub/dir &&\n-\tgit rm -r sub &&\n+\ttest_must_fail git rm -r sub &&\n+\tgit rm --sparse -r sub &&\n \tgit status --porcelain -uno >actual &&\n-\techo \"D sub/dir/e\" >expected &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n \ttest_cmp expected actual\n '\n \n@@ -87,4 +91,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n \tgit ls-files --error-unmatch b\n '\n \n+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\tgit update-index --no-skip-worktree b &&\n+\ttest_must_fail git rm b 2>stderr &&\n+\ttest_cmp b_error_and_hint stderr &&\n+\tgit rm --sparse b 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"436475","messageId":"63a9cd80ade83d0307374f46516782ee6a89dd30.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 13/14] mv: refuse to move sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:36Z","receivedAt":"2021-09-20T23:39:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince cmd_mv() does not operate on cache entries and instead directly\nchecks the filesystem, we can only use path_in_sparse_checkout() as a\nmechanism for seeing if a path is sparse or not. Be sure to skip\nreturning a failure if '-k' is specified.\n\nTo ensure that the advice around sparse paths is the only reason a move\nfailed, be sure to check this as the very last thing before inserting\ninto the src_for_dst list.\n\nThe tests cover a variety of cases such as whether the target is tracked\nor untracked, and whether the source or destination are in or outside of\nthe sparse-checkout definition.\n\nHelped-by: Matheus Tavares Bernardino \nSigned-off-by: Derrick Stolee \n---\n builtin/mv.c | 52 ++++++++--\n t/t7002-mv-sparse-checkout.sh | 186 ++++++++++++++++++++++++++++++++++\n 2 files changed, 229 insertions(+), 9 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex c2f96c8e895..83a465ba831 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -118,21 +118,23 @@ static int index_range_of_same_dir(const char *src, int length,\n int cmd_mv(int argc, const char **argv, const char *prefix)\n {\n \tint i, flags, gitmodules_modified = 0;\n-\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n+\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n \tstruct option builtin_mv_options[] = {\n \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n \t\tOPT__DRY_RUN(&show_only, N_(\"dry run\")),\n \t\tOPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n+\t\tOPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \t\tOPT_END(),\n \t};\n \tconst char **source, **destination, **dest_path, **submodule_gitfile;\n-\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n+\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n \tstruct stat st;\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct cache_entry *ce;\n+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \n \tgit_config(git_default_config, NULL);\n \n@@ -176,14 +178,17 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tint length, src_is_dir;\n \t\tconst char *bad = NULL;\n+\t\tint skip_sparse = 0;\n \n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n \t\tlength = strlen(src);\n-\t\tif (lstat(src, &st) < 0)\n-\t\t\tbad = _(\"bad source\");\n-\t\telse if (!strncmp(src, dst, length) &&\n+\t\tif (lstat(src, &st) < 0) {\n+\t\t\t/* only error if existence is expected. */\n+\t\t\tif (modes[i] != SPARSE)\n+\t\t\t\tbad = _(\"bad source\");\n+\t\t} else if (!strncmp(src, dst, length) &&\n \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n \t\t\tbad = _(\"can not move directory into itself\");\n \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n@@ -212,11 +217,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\t\tdst_len = strlen(dst);\n \n \t\t\t\tfor (j = 0; j < last - first; j++) {\n-\t\t\t\t\tconst char *path = active_cache[first + j]->name;\n+\t\t\t\t\tconst struct cache_entry *ce = active_cache[first + j];\n+\t\t\t\t\tconst char *path = ce->name;\n \t\t\t\t\tsource[argc + j] = path;\n \t\t\t\t\tdestination[argc + j] =\n \t\t\t\t\t\tprefix_path(dst, dst_len, path + length + 1);\n-\t\t\t\t\tmodes[argc + j] = INDEX;\n+\t\t\t\t\tmodes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n \t\t\t\t\tsubmodule_gitfile[argc + j] = NULL;\n \t\t\t\t}\n \t\t\t\targc += last - first;\n@@ -244,14 +250,36 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tbad = _(\"multiple sources for the same target\");\n \t\telse if (is_dir_sep(dst[strlen(dst) - 1]))\n \t\t\tbad = _(\"destination directory does not exist\");\n-\t\telse\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * We check if the paths are in the sparse-checkout\n+\t\t\t * definition as a very final check, since that\n+\t\t\t * allows us to point the user to the --sparse\n+\t\t\t * option as a way to have a successful run.\n+\t\t\t */\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(src, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, src);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(dst, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\n+\t\t\tif (skip_sparse)\n+\t\t\t\tgoto remove_entry;\n+\n \t\t\tstring_list_insert(&src_for_dst, dst);\n+\t\t}\n \n \t\tif (!bad)\n \t\t\tcontinue;\n \t\tif (!ignore_errors)\n \t\t\tdie(_(\"%s, source=%s, destination=%s\"),\n \t\t\t bad, src, dst);\n+remove_entry:\n \t\tif (--argc > 0) {\n \t\t\tint n = argc - i;\n \t\t\tmemmove(source + i, source + i + 1,\n@@ -266,6 +294,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (only_match_skip_worktree.nr) {\n+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n+\t\tif (!ignore_errors)\n+\t\t\treturn 1;\n+\t}\n+\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tenum update_mode mode = modes[i];\n@@ -274,7 +308,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(_(\"Renaming %s to %s\\n\"), src, dst);\n \t\tif (show_only)\n \t\t\tcontinue;\n-\t\tif (mode != INDEX && rename(src, dst) < 0) {\n+\t\tif (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n \t\t\tif (ignore_errors)\n \t\t\t\tcontinue;\n \t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..07dbfeb6d17\n--- /dev/null\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -0,0 +1,186 @@\n+#!/bin/sh\n+\n+test_description='git mv in sparse working trees'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' \"\n+\tmkdir -p sub/dir sub/dir2 &&\n+\ttouch a b c sub/d sub/dir/e sub/dir2/e &&\n+\tgit add -A &&\n+\tgit commit -m files &&\n+\n+\tcat >sparse_error_header <<-EOF &&\n+\tThe following pathspecs didn't match any eligible path, but they do match index\n+\tentries outside the current sparse checkout:\n+\tEOF\n+\n+\tcat >sparse_hint <<-EOF\n+\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n+\tEOF\n+\"\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse' '\n+\ttest_when_finished rm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\tgit mv -k b e 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\tgit reset --hard &&\n+\ttouch b &&\n+\tgit mv -k b c 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttest_must_fail git mv a e 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\trm e &&\n+\tgit reset --hard &&\n+\ttest_must_fail git mv a c 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a e &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'recursive mv refuses to move (possible) sparse' '\n+\ttest_when_finished rm -rf b c e sub2 &&\n+\tgit reset --hard &&\n+\t# Without cone mode, \"sub\" and \"sub2\" do not match\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/d sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/d\n+\tsub2/d\n+\tsub/dir/e\n+\tsub2/dir/e\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_expect_success 'recursive mv refuses to move sparse' '\n+\tgit reset --hard &&\n+\t# Use cone mode so \"sub/\" matches the sparse-checkout patterns\n+\tgit sparse-checkout init --cone &&\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"436476","messageId":"79a3518dc15cbf07991535a779b141e3ccefc6f4.1632159937.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v3 14/14] advice: update message to suggest '--sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-20T17:45:37Z","receivedAt":"2021-09-20T23:39:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous changes modified the behavior of 'git add', 'git rm', and\n'git mv' to not adjust paths outside the sparse-checkout cone, even if\nthey exist in the working tree and their cache entries lack the\nSKIP_WORKTREE bit. The intention is to warn users that they are doing\nsomething potentially dangerous. The '--sparse' option was added to each\ncommand to allow careful users the same ability they had before.\n\nTo improve the discoverability of this new functionality, add a message\nto advice.updateSparsePath that mentions the existence of the option.\n\nThe previous set of changes also modified the purpose of this message to\ninclude possibly a list of paths instead of only a list of pathspecs.\nMake the warning message more clear about this new behavior.\n\nSigned-off-by: Derrick Stolee \n---\n advice.c | 11 ++++++-----\n t/t3602-rm-sparse-checkout.sh | 9 ++++++---\n t/t3705-add-sparse-checkout.sh | 9 ++++++---\n t/t7002-mv-sparse-checkout.sh | 9 ++++++---\n 4 files changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex 0b9c89c48ab..713fff49ee3 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -293,15 +293,16 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)\n \tif (!pathspec_list->nr)\n \t\treturn;\n \n-\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n-\t\t\t \" eligible path, but they do match index\\n\"\n-\t\t\t \"entries outside the current sparse checkout:\\n\"));\n+\tfprintf(stderr, _(\"The following paths and/or pathspecs matched paths that exist\\n\"\n+\t\t\t \"outside of your sparse-checkout definition, so will not be\\n\"\n+\t\t\t \"updated in the index:\\n\"));\n \tfor_each_string_list_item(item, pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n-\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n-\t\t\t \" to update such entries.\"));\n+\t\t\t _(\"If you intend to update such entries, try one of the following:\\n\"\n+\t\t\t \"* Use the --sparse option.\\n\"\n+\t\t\t \"* Disable or modify the sparsity rules.\"));\n }\n \n void detach_advice(const char *new_name)\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 5f92b60a56a..ecce497a9ca 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -11,12 +11,15 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 10969efc23e..754b06a9663 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -44,12 +44,15 @@ test_sparse_entry_unstaged () {\n \n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 07dbfeb6d17..545748949aa 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -11,12 +11,15 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \"\n-- \ngitgitgadget\n"},{"id":"436794","messageId":"xmqq7df85ilg.fsf@gitster.g","threadId":"56370","inReplyTo":"ea940f10a7cd26c6be3693b5a800afb7d6f752f2.1632159937.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/14] t3705: test that 'sparse_entry' is unstaged","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-22T22:52:11Z","receivedAt":"2021-09-22T22:52:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The tests in t3705-add-sparse-checkout.sh check to see how 'git add'\n> behaves with paths outside the sparse-checkout definition. These\n> currently check to see if a given warning is present but not that the\n> index is not updated with the sparse entries. Add a new\n> 'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\n> correctly.\n>\n> We need to modify setup_sparse_entry to actually commit the sparse_entry\n> file so it exists at HEAD and as an entry in the index, but its exact\n> contents are not staged in the index.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n> 1 file changed, 14 insertions(+)\n>\n> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n> index 2b1fd0d0eef..e202a2ff74a 100755\n> --- a/t/t3705-add-sparse-checkout.sh\n> +++ b/t/t3705-add-sparse-checkout.sh\n> @@ -19,6 +19,7 @@ setup_sparse_entry () {\n> \tfi &&\n> \tgit add sparse_entry &&\n> \tgit update-index --skip-worktree sparse_entry &&\n> +\tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n> \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n> }\n> \n> @@ -36,6 +37,11 @@ setup_gitignore () {\n> \tEOF\n> }\n> \n> +test_sparse_entry_unstaged () {\n> +\tgit status --porcelain >actual &&\n> +\t! grep \"^[MDARCU][M ] sparse_entry\\$\" actual\n\nDoes this say \"we do not want any difference from the index, be it\nmodification, deletion, addtion, etc.\"?\n\nJust wondering if there were a reason why the pattern is more\ncomplex than \"^[^ ][M ]\" (i.e. anything but \"unmodified since the\nindex\"), not necessarily suggesting to spell the test differently.\n\nThanks.\n"},{"id":"436797","messageId":"xmqq35pw5hx5.fsf@gitster.g","threadId":"56370","inReplyTo":"c7dedb41291ed6ff48dd9b9b9814e3040f4fa92c.1632159937.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/14] t1092: behavior for adding sparse files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-22T23:06:46Z","receivedAt":"2021-09-22T23:06:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Add some tests to demonstrate the current behavior around adding files\n> outside of the sparse-checkout cone. Currently, untracked files are\n> handled differently from tracked files. A future change will make these\n> cases be handled the same way.\n>\n> Further expand checking that a failed 'git add' does not stage changes\n> to the index.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++\n> 1 file changed, 28 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 886e78715fe..3fb764f5eb9 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -187,6 +187,16 @@ test_sparse_match () {\n> \ttest_cmp sparse-checkout-err sparse-index-err\n> }\n> \n> +test_sparse_unstaged () {\n> +\tfile=$1 &&\n> +\tfor repo in sparse-checkout sparse-index\n> +\tdo\n> +\t\tgit -C $repo status --porcelain >$repo-out &&\n> +\t\t! grep \"^A $file\\$\" $repo-out &&\n> +\t\t! grep \"^M $file\\$\" $repo-out || return 1\n\nIs addition and modification very special, as opposed to other kinds\nof changes? Is the reason why we say \"we do not want to see\naddition nor modification\" here because there is no concrete X that\nwe can say \"we do want to see X\" in various callers of this helper?\n\nI am also wondering, if this is asserting that $file is not added to\nthe index, why we are using \"git status\" and not \"ls-files\", for\nexample. Wouldn't\n\n test_must_fail git ls-files --error-unmatch \"$file\"\n\nbe a more direct way to do so?\n"},{"id":"436800","messageId":"xmqqwnn8430s.fsf@gitster.g","threadId":"56370","inReplyTo":"b1f6468f9cdb7d16f6317c71b21f4459af158e87.1632159937.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-22T23:13:55Z","receivedAt":"2021-09-22T23:13:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static int path_matches_dir_pattern(const char *pathname,\n> +\t\t\t\t int pathlen,\n> +\t\t\t\t int *dtype,\n> +\t\t\t\t struct path_pattern *pattern,\n> +\t\t\t\t struct index_state *istate)\n> +{\n> +\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n> +\tif (*dtype != DT_DIR)\n> +\t\treturn 0;\n> +\n> +\treturn 1;\n> +}\n\nThe function name and parameter list have \"pattern\" but as far as I\ncan see any \"matches\" or \"pattern\" comes into the picture. The code\nin the caller after calling this function may be doing pattern\nmatching, but not this one.\n\nWhat this helper is doing is \"signal if the pathname in the working\ntree is supposed to be a directory with the return value, while\nfilling *dtype with what kind of thing it is.\"\n\npath_must_be_dir_in_working_tree() or something, perhaps?\n\n> @@ -1327,11 +1340,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> \t\tconst char *exclude = pattern->pattern;\n> \t\tint prefix = pattern->nowildcardlen;\n> \n> -\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n> -\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n> -\t\t\tif (*dtype != DT_DIR)\n> -\t\t\t\tcontinue;\n> -\t\t}\n> +\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n> +\t\t !path_matches_dir_pattern(pathname, pathlen,\n> +\t\t\t\t\t dtype, pattern, istate))\n> +\t\t\tcontinue;\n> \n> \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n> \t\t\tif (match_basename(basename,\n"},{"id":"436850","messageId":"c1efc50d-23c1-29f1-27f4-8638ecbf38dc@gmail.com","threadId":"56370","inReplyTo":"xmqq35pw5hx5.fsf@gitster.g","subject":"Re: [PATCH v3 02/14] t1092: behavior for adding sparse files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-23T13:37:00Z","receivedAt":"2021-09-23T13:37:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/22/2021 7:06 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> Add some tests to demonstrate the current behavior around adding files\n>> outside of the sparse-checkout cone. Currently, untracked files are\n>> handled differently from tracked files. A future change will make these\n>> cases be handled the same way.\n>>\n>> Further expand checking that a failed 'git add' does not stage changes\n>> to the index.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++\n>> 1 file changed, 28 insertions(+)\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 886e78715fe..3fb764f5eb9 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -187,6 +187,16 @@ test_sparse_match () {\n>> \ttest_cmp sparse-checkout-err sparse-index-err\n>> }\n>> \n>> +test_sparse_unstaged () {\n>> +\tfile=$1 &&\n>> +\tfor repo in sparse-checkout sparse-index\n>> +\tdo\n>> +\t\tgit -C $repo status --porcelain >$repo-out &&\n>> +\t\t! grep \"^A $file\\$\" $repo-out &&\n>> +\t\t! grep \"^M $file\\$\" $repo-out || return 1\n> \n> Is addition and modification very special, as opposed to other kinds\n> of changes? Is the reason why we say \"we do not want to see\n> addition nor modification\" here because there is no concrete X that\n> we can say \"we do want to see X\" in various callers of this helper?\n> \n> I am also wondering, if this is asserting that $file is not added to\n> the index, why we are using \"git status\" and not \"ls-files\", for\n> example. Wouldn't\n> \n> test_must_fail git ls-files --error-unmatch \"$file\"\n> \n> be a more direct way to do so?\n\nThe intention is to avoid a staged change, even if that is a modification\nor addition. The --error-unmatch approach only works if we are trying to\navoid an add, and we need to know that the path does not exist in the\nindex already if we want to take that approach.\n\nUpon reflection, perhaps what we really want is this:\n\n\tgit diff --staged -- $path >diff &&\n\ttest_must_be_empty diff\n\nThis applies to the previous patch, too.\n\nThanks,\n-Stolee\n"},{"id":"436851","messageId":"36475af0-1021-f59d-cb33-b3b402c13767@gmail.com","threadId":"56370","inReplyTo":"xmqqwnn8430s.fsf@gitster.g","subject":"Re: [PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-23T13:39:06Z","receivedAt":"2021-09-23T13:39:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/22/2021 7:13 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +static int path_matches_dir_pattern(const char *pathname,\n>> +\t\t\t\t int pathlen,\n>> +\t\t\t\t int *dtype,\n>> +\t\t\t\t struct path_pattern *pattern,\n>> +\t\t\t\t struct index_state *istate)\n>> +{\n>> +\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n>> +\tif (*dtype != DT_DIR)\n>> +\t\treturn 0;\n>> +\n>> +\treturn 1;\n>> +}\n> \n> The function name and parameter list have \"pattern\" but as far as I\n> can see any \"matches\" or \"pattern\" comes into the picture. The code\n> in the caller after calling this function may be doing pattern\n> matching, but not this one.\n> \n> What this helper is doing is \"signal if the pathname in the working\n> tree is supposed to be a directory with the return value, while\n> filling *dtype with what kind of thing it is.\"\n> \n> path_must_be_dir_in_working_tree() or something, perhaps?\n\nYes, a rename would be prudent here. Thanks.\n\n-Stolee\n"},{"id":"436852","messageId":"a99a58ee-c3fe-501b-3116-8b54eaed0e53@gmail.com","threadId":"56370","inReplyTo":"36475af0-1021-f59d-cb33-b3b402c13767@gmail.com","subject":"Re: [PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-23T13:42:56Z","receivedAt":"2021-09-23T13:43:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/23/2021 9:39 AM, Derrick Stolee wrote:\n> On 9/22/2021 7:13 PM, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> +static int path_matches_dir_pattern(const char *pathname,\n>>> +\t\t\t\t int pathlen,\n>>> +\t\t\t\t int *dtype,\n>>> +\t\t\t\t struct path_pattern *pattern,\n>>> +\t\t\t\t struct index_state *istate)\n>>> +{\n>>> +\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n>>> +\tif (*dtype != DT_DIR)\n>>> +\t\treturn 0;\n>>> +\n>>> +\treturn 1;\n>>> +}\n>>\n>> The function name and parameter list have \"pattern\" but as far as I\n>> can see any \"matches\" or \"pattern\" comes into the picture. The code\n>> in the caller after calling this function may be doing pattern\n>> matching, but not this one.\n>>\n>> What this helper is doing is \"signal if the pathname in the working\n>> tree is supposed to be a directory with the return value, while\n>> filling *dtype with what kind of thing it is.\"\n>>\n>> path_must_be_dir_in_working_tree() or something, perhaps?\n> \n> Yes, a rename would be prudent here. Thanks.\n\nOf course, when I go to amend the commit, the commit message says\n\n\tWe will expand the path_matches_dir_pattern() method in a following\n\tchange.\n\nwhich means that more will follow that will actually care about the\npattern and matching as a directory.\n\nAfter looking at the extension in the next patch, do you still think a\nrename is necessary? Specifically, this diff hunk:\n\ndiff --git a/dir.c b/dir.c\nindex 652135df896..9ea6cfe61cb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1305,10 +1305,35 @@ int match_pathname(const char *pathname, int pathlen,\n \n static int path_matches_dir_pattern(const char *pathname,\n \t\t\t\t int pathlen,\n+\t\t\t\t struct strbuf **path_parent,\n \t\t\t\t int *dtype,\n \t\t\t\t struct path_pattern *pattern,\n \t\t\t\t struct index_state *istate)\n {\n+\tif (!*path_parent) {\n+\t\tchar *slash;\n+\t\tCALLOC_ARRAY(*path_parent, 1);\n+\t\tstrbuf_add(*path_parent, pathname, pathlen);\n+\t\tslash = find_last_dir_sep((*path_parent)->buf);\n+\n+\t\tif (slash)\n+\t\t\tstrbuf_setlen(*path_parent, slash - (*path_parent)->buf);\n+\t\telse\n+\t\t\tstrbuf_setlen(*path_parent, 0);\n+\t}\n+\n+\t/*\n+\t * If the parent directory matches the pattern, then we do not\n+\t * need to check for dtype.\n+\t */\n+\tif ((*path_parent)->len &&\n+\t match_pathname((*path_parent)->buf, (*path_parent)->len,\n+\t\t\t pattern->base,\n+\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n+\t\t\t pattern->pattern, pattern->nowildcardlen,\n+\t\t\t pattern->patternlen, pattern->flags))\n+\t\treturn 1;\n+\n \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n \tif (*dtype != DT_DIR)\n \t\treturn 0;\n\nThanks,\n-Stolee\n"},{"id":"436887","messageId":"xmqqlf3nywuw.fsf@gitster.g","threadId":"56370","inReplyTo":"a99a58ee-c3fe-501b-3116-8b54eaed0e53@gmail.com","subject":"Re: [PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-23T18:23:35Z","receivedAt":"2021-09-23T18:23:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 9/23/2021 9:39 AM, Derrick Stolee wrote:\n>> On 9/22/2021 7:13 PM, Junio C Hamano wrote:\n>>> \"Derrick Stolee via GitGitGadget\" writes:\n>>>\n>>>> +static int path_matches_dir_pattern(const char *pathname,\n>>>> +\t\t\t\t int pathlen,\n>>>> +\t\t\t\t int *dtype,\n>>>> +\t\t\t\t struct path_pattern *pattern,\n>>>> +\t\t\t\t struct index_state *istate)\n>>>> +{\n>>>> +\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n>>>> +\tif (*dtype != DT_DIR)\n>>>> +\t\treturn 0;\n>>>> +\n>>>> +\treturn 1;\n>>>> +}\n>>>\n>>> The function name and parameter list have \"pattern\" but as far as I\n>>> can see any \"matches\" or \"pattern\" comes into the picture. The code\n>>> in the caller after calling this function may be doing pattern\n>>> matching, but not this one.\n>>>\n>>> What this helper is doing is \"signal if the pathname in the working\n>>> tree is supposed to be a directory with the return value, while\n>>> filling *dtype with what kind of thing it is.\"\n>>>\n>>> path_must_be_dir_in_working_tree() or something, perhaps?\n>> \n>> Yes, a rename would be prudent here. Thanks.\n>\n> Of course, when I go to amend the commit, the commit message says\n>\n> \tWe will expand the path_matches_dir_pattern() method in a following\n> \tchange.\n>\n> which means that more will follow that will actually care about the\n> pattern and matching as a directory.\n>\n> After looking at the extension in the next patch, do you still think a\n> rename is necessary?\n\nWhen the focus and purpose of the function changes, it may warrant a\nrename to include \"matching\" or \"pattern\", but not before.\n\nOr we might be seeing a premature refactoring with these two steps.\nAre we gaining multiple callers of this function before it gets\nextended to care about pattern and matching? If not, perhaps\nteaching the inlined codepath about the pattern and matching in\nplace first before extracting the code to a helper function for\nreadability and reusability may help make the resulting series\neasier to follow, and we do not have to see a function with a\nmisleading name.\n\n"},{"id":"436923","messageId":"CABPp-BGU3HgG5B7wXc-Wq1gwGYy_R9iveM7VRmUc80YhejLF1A@mail.gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-24T06:08:25Z","receivedAt":"2021-09-24T06:08:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Sep 20, 2021 at 10:45 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This series is based on ds/mergies-with-sparse-index.\n>\n> As requested, this series looks to update the behavior of git add, git rm,\n> and git mv when they attempt to modify paths outside of the sparse-checkout\n> cone. In particular, this care is expanded to not just cache entries with\n> the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\n> definition.\n>\n> This means that commands that worked before this series can now fail. In\n> particular, if 'git merge' results in a conflict outside of the\n> sparse-checkout cone, then 'git add ' will now fail.\n>\n> In order to allow users to circumvent these protections, a new '--sparse'\n> option is added that ignores the sparse-checkout patterns and the\n> SKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\n> assist with discovery of this option.\n>\n> There is a subtle issue with git mv in that it does not check the index\n> until it discovers a directory and then uses the index to find the contained\n> entries. This means that in non-cone-mode patterns, a pattern such as\n> \"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n>\n> In order to allow for checking arbitrary paths against the sparse-checkout\n> patterns, some changes to the underlying pattern matching code is required.\n> It turns out that there are some bugs in the methods as advertised, but\n> these bugs were never discovered because of the way methods like\n> unpack_trees() will check a directory for a pattern match before checking\n> its contained paths. Our new \"check patterns on-demand\" approach pokes holes\n> in that approach, specifically with patterns that match entire directories.\n>\n>\n> Updates in v3\n> =============\n>\n> * Fixed an incorrectly-squashed commit. Spread out some changes in a better\n> way. For example, I don't add --sparse to tests before introducing the\n> option.\n>\n> * Use a NULL struct strbuf pointer to indicate an uninitialized value\n> instead of relying on an internal member.\n>\n> * Use grep over test_i18ngrep.\n>\n> * Fixed line wrapping for error messages.\n>\n> * Use strbuf_setlen() over modifying the len member manually.\n\nI see that you and Junio had some interesting comments on the first 3\npatches, so I look forward to seeing how those play out, but you\naddressed all my feedback from the previous rounds here.\n"},{"id":"436934","messageId":"e51adbad-c633-b030-ff01-3cfc260a0d65@web.de","threadId":"56370","inReplyTo":"d47c7a1cf2a3fa8cfdcfc6be1ac800af123e7efc.1629842085.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 03/13] dir: select directories correctly","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2021-09-24T07:44:29Z","receivedAt":"2021-09-24T07:44:41Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 24.08.21 um 23:54 schrieb Derrick Stolee via GitGitGadget:\n> From: Derrick Stolee \n>\n> When matching a path against a list of patterns, the ones that require a\n> directory match previously did not work when a filename is specified.\n> This was fine when all pattern-matching was done within methods such as\n> unpack_trees() that check a directory before recursing into the\n> contained files. However, other commands will start matching individual\n> files against pattern lists without that recursive approach.\n>\n> We modify path_matches_dir_pattern() to take a strbuf 'path_parent' that\n> is used to store the parent directory of 'pathname' between multiple\n> pattern matching tests. This is loaded lazily, only on the first pattern\n> it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n>\n> If we find that a path has a parent directory, we start by checking to\n> see if that parent directory matches the pattern. If so, then we do not\n> need to query the index for the type (which can be expensive). If we\n> find that the parent does not match, then we still must check the type\n> from the index for the given pathname.\n>\n> Note that this does not affect cone mode pattern matching, but instead\n> the more general -- and slower -- full pattern set. Thus, this does not\n> affect the sparse index.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 34 ++++++++++++++++++++++++++++++++--\n> 1 file changed, 32 insertions(+), 2 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index 652135df896..fe5ee87bb5f 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1305,10 +1305,38 @@ int match_pathname(const char *pathname, int pathlen,\n>\n> static int path_matches_dir_pattern(const char *pathname,\n> \t\t\t\t int pathlen,\n> +\t\t\t\t struct strbuf *path_parent,\n> \t\t\t\t int *dtype,\n> \t\t\t\t struct path_pattern *pattern,\n> \t\t\t\t struct index_state *istate)\n> {\n> +\t/*\n> +\t * Use 'alloc' as an indicator that the string has not been\n> +\t * initialized, in case the parent is the root directory.\n> +\t */\n\nThis means the caller needs to take care to release the strbuf between\ncalls for files from different directories. Seems a bit fragile. The\ncurrent caller is only ever passing in the same pathname before throwing\naway the strbuf, so it's doing the right thing.\n\n> +\tif (!path_parent->alloc) {\n> +\t\tchar *slash;\n> +\t\tstrbuf_addstr(path_parent, pathname);\n> +\t\tslash = find_last_dir_sep(path_parent->buf);\n\nThe caller has pathname, pathlen and basename. If basename is\nguaranteed to be a substring of pathname then the parent directory name\nlength could be calculated without requiring a string copy or scan.\n\nIIUC if pathname and basename can be pointers to different objects then\njust checking if basename is between pathname and pathname + pathlen\nwould already be undefined behavior.\n\nUsing pathname, pathlen and dirlen instead would be safer for such\ncalculations, as it enforces basename to be a substring. Seems like\nthis would require a lot of function signature changes, though, as the\ncall tree is quite high. :|\n\n> +\n> +\t\tif (slash)\n> +\t\t\t*slash = '\\0';\n\nThis doesn't update path_parent->len...\n\n> +\t\telse\n> +\t\t\tstrbuf_setlen(path_parent, 0);\n> +\t}\n> +\n> +\t/*\n> +\t * If the parent directory matches the pattern, then we do not\n> +\t * need to check for dtype.\n> +\t */\n> +\tif (path_parent->len &&\n> +\t match_pathname(path_parent->buf, path_parent->len,\n\n... so this checks if \"\\0\" matches. Intended?\n\n> +\t\t\t pattern->base,\n> +\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n> +\t\t\t pattern->pattern, pattern->nowildcardlen,\n> +\t\t\t pattern->patternlen, pattern->flags))\n> +\t\treturn 1;\n> +\n> \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n> \tif (*dtype != DT_DIR)\n> \t\treturn 0;\n> @@ -1331,6 +1359,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> {\n> \tstruct path_pattern *res = NULL; /* undecided */\n> \tint i;\n> +\tstruct strbuf path_parent = STRBUF_INIT;\n>\n> \tif (!pl->nr)\n> \t\treturn NULL;\t/* undefined */\n> @@ -1340,8 +1369,8 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> \t\tconst char *exclude = pattern->pattern;\n> \t\tint prefix = pattern->nowildcardlen;\n>\n> -\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n> -\t\t !path_matches_dir_pattern(pathname, pathlen,\n> +\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n> +\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n\n\"a & b && c\" is equivalent to \"(a & b) && c\", but removing the\nparentheses here serves no apparent purpose and distracts a bit from\nthe actual change, i.e. adding a parameter.\n\n> \t\t\t\t\t dtype, pattern, istate))\n> \t\t\tcontinue;\n>\n> @@ -1367,6 +1396,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n> \t\t\tbreak;\n> \t\t}\n> \t}\n> +\tstrbuf_release(&path_parent);\n> \treturn res;\n> }\n>\n>\n\n"},{"id":"436948","messageId":"a848a476-3993-6fa4-9fe0-3c96c46041a4@gmail.com","threadId":"56370","inReplyTo":"xmqqlf3nywuw.fsf@gitster.g","subject":"Re: [PATCH v3 03/14] dir: extract directory-matching logic","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-09-24T13:29:14Z","receivedAt":"2021-09-24T13:33:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/23/2021 2:23 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 9/23/2021 9:39 AM, Derrick Stolee wrote:\n>>> On 9/22/2021 7:13 PM, Junio C Hamano wrote:\n>>>> \"Derrick Stolee via GitGitGadget\" writes:\n>>>>\n>>>>> +static int path_matches_dir_pattern(const char *pathname,\n>>>>> +\t\t\t\t int pathlen,\n>>>>> +\t\t\t\t int *dtype,\n>>>>> +\t\t\t\t struct path_pattern *pattern,\n>>>>> +\t\t\t\t struct index_state *istate)\n>>>>> +{\n>>>>> +\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n>>>>> +\tif (*dtype != DT_DIR)\n>>>>> +\t\treturn 0;\n>>>>> +\n>>>>> +\treturn 1;\n>>>>> +}\n>>>>\n>>>> The function name and parameter list have \"pattern\" but as far as I\n>>>> can see any \"matches\" or \"pattern\" comes into the picture. The code\n>>>> in the caller after calling this function may be doing pattern\n>>>> matching, but not this one.\n>>>>\n>>>> What this helper is doing is \"signal if the pathname in the working\n>>>> tree is supposed to be a directory with the return value, while\n>>>> filling *dtype with what kind of thing it is.\"\n>>>>\n>>>> path_must_be_dir_in_working_tree() or something, perhaps?\n>>>\n>>> Yes, a rename would be prudent here. Thanks.\n>>\n>> Of course, when I go to amend the commit, the commit message says\n>>\n>> \tWe will expand the path_matches_dir_pattern() method in a following\n>> \tchange.\n>>\n>> which means that more will follow that will actually care about the\n>> pattern and matching as a directory.\n>>\n>> After looking at the extension in the next patch, do you still think a\n>> rename is necessary?\n> \n> When the focus and purpose of the function changes, it may warrant a\n> rename to include \"matching\" or \"pattern\", but not before.\n> \n> Or we might be seeing a premature refactoring with these two steps.\n> Are we gaining multiple callers of this function before it gets\n> extended to care about pattern and matching? If not, perhaps\n> teaching the inlined codepath about the pattern and matching in\n> place first before extracting the code to a helper function for\n> readability and reusability may help make the resulting series\n> easier to follow, and we do not have to see a function with a\n> misleading name.\n \nSquashing these two patches together has the same effect, but\ntakes a little bit extra work to see that the re-used code is\nthe same. It's small enough that I don't see that as a huge\nhurdle.\n\nThanks,\n-Stolee\n"},{"id":"436954","messageId":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v3.git.1632159937.gitgitgadget@gmail.com","subject":"[PATCH v4 00/13] Sparse-checkout: modify 'git add', 'git rm', and 'git mv' behavior","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:01Z","receivedAt":"2021-09-24T15:39:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/mergies-with-sparse-index.\n\nAs requested, this series looks to update the behavior of git add, git rm,\nand git mv when they attempt to modify paths outside of the sparse-checkout\ncone. In particular, this care is expanded to not just cache entries with\nthe SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\ndefinition.\n\nThis means that commands that worked before this series can now fail. In\nparticular, if 'git merge' results in a conflict outside of the\nsparse-checkout cone, then 'git add ' will now fail.\n\nIn order to allow users to circumvent these protections, a new '--sparse'\noption is added that ignores the sparse-checkout patterns and the\nSKIP_WORKTREE bit. The message for advice.updateSparsePath is adjusted to\nassist with discovery of this option.\n\nThere is a subtle issue with git mv in that it does not check the index\nuntil it discovers a directory and then uses the index to find the contained\nentries. This means that in non-cone-mode patterns, a pattern such as\n\"sub/dir\" will not match the path \"sub\" and this can cause an issue.\n\nIn order to allow for checking arbitrary paths against the sparse-checkout\npatterns, some changes to the underlying pattern matching code is required.\nIt turns out that there are some bugs in the methods as advertised, but\nthese bugs were never discovered because of the way methods like\nunpack_trees() will check a directory for a pattern match before checking\nits contained paths. Our new \"check patterns on-demand\" approach pokes holes\nin that approach, specifically with patterns that match entire directories.\n\n\nUpdates in v4\n=============\n\n * Instead of using 'git status' and 'grep' to detect staged changes, we use\n 'git diff --staged'. t1092 uses an additional --diff-filter because it\n tests with merge conflicts, so it needs this extra flag.\n\n * Patches 3 and 4 are merged into the new patch 3 to avoid temporarily\n having a poorly named method.\n\n\nUpdates in v3\n=============\n\n * Fixed an incorrectly-squashed commit. Spread out some changes in a better\n way. For example, I don't add --sparse to tests before introducing the\n option.\n\n * Use a NULL struct strbuf pointer to indicate an uninitialized value\n instead of relying on an internal member.\n\n * Use grep over test_i18ngrep.\n\n * Fixed line wrapping for error messages.\n\n * Use strbuf_setlen() over modifying the len member manually.\n\n\nUpdates in v2\n=============\n\n * I got no complaints about these restrictions, so this is now a full\n series, not RFC.\n\n * Thanks to Matheus, several holes are filled with extra testing and\n bugfixes.\n\n * New patches add --chmod and --renormalize improvements. These are added\n after the --sparse option to make them be one change each.\n\nThanks, -Stolee\n\nDerrick Stolee (13):\n t3705: test that 'sparse_entry' is unstaged\n t1092: behavior for adding sparse files\n dir: select directories correctly\n dir: fix pattern matching on dirs\n add: fail when adding an untracked sparse file\n add: skip tracked paths outside sparse-checkout cone\n add: implement the --sparse option\n add: update --chmod to skip sparse paths\n add: update --renormalize to skip sparse paths\n rm: add --sparse option\n rm: skip sparse paths with missing SKIP_WORKTREE\n mv: refuse to move sparse paths\n advice: update message to suggest '--sparse'\n\n Documentation/git-add.txt | 9 +-\n Documentation/git-rm.txt | 6 +\n advice.c | 11 +-\n builtin/add.c | 32 +++-\n builtin/mv.c | 52 +++++--\n builtin/rm.c | 10 +-\n dir.c | 56 ++++++-\n pathspec.c | 5 +-\n t/t1091-sparse-checkout-builtin.sh | 4 +-\n t/t1092-sparse-checkout-compatibility.sh | 75 +++++++--\n t/t3602-rm-sparse-checkout.sh | 40 ++++-\n t/t3705-add-sparse-checkout.sh | 68 +++++++-\n t/t7002-mv-sparse-checkout.sh | 189 +++++++++++++++++++++++\n 13 files changed, 505 insertions(+), 52 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\n\nbase-commit: 516680ba7704c473bb21628aa19cabbd787df4db\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1018%2Fderrickstolee%2Fsparse-index%2Fadd-rm-mv-behavior-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1018/derrickstolee/sparse-index/add-rm-mv-behavior-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1018\n\nRange-diff vs v3:\n\n 1: ea940f10a7c ! 1: 642b05fc020 t3705: test that 'sparse_entry' is unstaged\n @@ t/t3705-add-sparse-checkout.sh: setup_gitignore () {\n }\n \n +test_sparse_entry_unstaged () {\n -+\tgit status --porcelain >actual &&\n -+\t! grep \"^[MDARCU][M ] sparse_entry\\$\" actual\n ++\tgit diff --staged -- sparse_entry >diff &&\n ++\ttest_must_be_empty diff\n +}\n +\n test_expect_success 'setup' \"\n 2: c7dedb41291 ! 2: 58389edc76c t1092: behavior for adding sparse files\n @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n +\tfile=$1 &&\n +\tfor repo in sparse-checkout sparse-index\n +\tdo\n -+\t\tgit -C $repo status --porcelain >$repo-out &&\n -+\t\t! grep \"^A $file\\$\" $repo-out &&\n -+\t\t! grep \"^M $file\\$\" $repo-out || return 1\n ++\t\t# Skip \"unmerged\" paths\n ++\t\tgit -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n ++\t\ttest_must_be_empty diff || return 1\n +\tdone\n +}\n +\n 3: b1f6468f9cd < -: ----------- dir: extract directory-matching logic\n 4: 0252c7ee15c ! 3: 2ebaf8e68c2 dir: select directories correctly\n @@ Commit message\n contained files. However, other commands will start matching individual\n files against pattern lists without that recursive approach.\n \n - We modify path_matches_dir_pattern() to take a strbuf pointer\n - 'path_parent' that is used to store the parent directory of 'pathname'\n - between multiple pattern matching tests. This is loaded lazily, only on\n - the first pattern it finds that has the PATTERN_FLAG_MUSTBEDIR flag.\n + The last_matching_pattern_from_list() logic performs some checks on the\n + filetype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag\n + is set. This works great when setting SKIP_WORKTREE bits within\n + unpack_trees(), but doesn't work well when passing an arbitrary path\n + such as a file within a matching directory.\n +\n + We extract the logic around determining the file type, but attempt to\n + avoid checking the filesystem if the parent directory already matches\n + the sparse-checkout patterns. The new path_matches_dir_pattern() method\n + includes a 'path_parent' parameter that is used to store the parent\n + directory of 'pathname' between multiple pattern matching tests. This is\n + loaded lazily, only on the first pattern it finds that has the\n + PATTERN_FLAG_MUSTBEDIR flag.\n \n If we find that a path has a parent directory, we start by checking to\n see if that parent directory matches the pattern. If so, then we do not\n @@ Commit message\n \n ## dir.c ##\n @@ dir.c: int match_pathname(const char *pathname, int pathlen,\n + \t\t\t\t WM_PATHNAME) == 0;\n + }\n \n - static int path_matches_dir_pattern(const char *pathname,\n - \t\t\t\t int pathlen,\n ++static int path_matches_dir_pattern(const char *pathname,\n ++\t\t\t\t int pathlen,\n +\t\t\t\t struct strbuf **path_parent,\n - \t\t\t\t int *dtype,\n - \t\t\t\t struct path_pattern *pattern,\n - \t\t\t\t struct index_state *istate)\n - {\n ++\t\t\t\t int *dtype,\n ++\t\t\t\t struct path_pattern *pattern,\n ++\t\t\t\t struct index_state *istate)\n ++{\n +\tif (!*path_parent) {\n +\t\tchar *slash;\n +\t\tCALLOC_ARRAY(*path_parent, 1);\n @@ dir.c: int match_pathname(const char *pathname, int pathlen,\n +\t\t\t pattern->patternlen, pattern->flags))\n +\t\treturn 1;\n +\n - \t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n - \tif (*dtype != DT_DIR)\n - \t\treturn 0;\n ++\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n ++\tif (*dtype != DT_DIR)\n ++\t\treturn 0;\n ++\n ++\treturn 1;\n ++}\n ++\n + /*\n + * Scan the given exclude list in reverse to see whether pathname\n + * should be ignored. The first match (i.e. the last on the list), if\n @@ dir.c: static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n @@ dir.c: static struct path_pattern *last_matching_pattern_from_list(const char *p\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n --\t\tif ((pattern->flags & PATTERN_FLAG_MUSTBEDIR) &&\n --\t\t !path_matches_dir_pattern(pathname, pathlen,\n +-\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n +-\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n +-\t\t\tif (*dtype != DT_DIR)\n +-\t\t\t\tcontinue;\n +-\t\t}\n +\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n +\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n - \t\t\t\t\t dtype, pattern, istate))\n - \t\t\tcontinue;\n ++\t\t\t\t\t dtype, pattern, istate))\n ++\t\t\tcontinue;\n \n + \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n + \t\t\tif (match_basename(basename,\n @@ dir.c: static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\t\tbreak;\n \t\t}\n 5: c6d17df5e5d = 4: 24bffdab139 dir: fix pattern matching on dirs\n 6: 3dd1d6c228c = 5: e3a749e3182 add: fail when adding an untracked sparse file\n 7: 15039e031e5 = 6: 2c5c834bc9f add: skip tracked paths outside sparse-checkout cone\n 8: 6014ac8ab9e = 7: 430ab44e4f1 add: implement the --sparse option\n 9: 2bd3448be5f = 8: 4f7b5cdfa36 add: update --chmod to skip sparse paths\n 10: 131beda1bc3 = 9: 30ec6096939 add: update --renormalize to skip sparse paths\n 11: 837a9314893 = 10: 99d50921ef4 rm: add --sparse option\n 12: cc25ce17162 = 11: 47a1444115b rm: skip sparse paths with missing SKIP_WORKTREE\n 13: 63a9cd80ade = 12: 28e703d80d3 mv: refuse to move sparse paths\n 14: 79a3518dc15 = 13: 9fbc88ee0da advice: update message to suggest '--sparse'\n\n-- \ngitgitgadget\n"},{"id":"436953","messageId":"642b05fc0204f2f9f655afd13695f22d30c2fc83.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 01/13] t3705: test that 'sparse_entry' is unstaged","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:02Z","receivedAt":"2021-09-24T15:39:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe tests in t3705-add-sparse-checkout.sh check to see how 'git add'\nbehaves with paths outside the sparse-checkout definition. These\ncurrently check to see if a given warning is present but not that the\nindex is not updated with the sparse entries. Add a new\n'test_sparse_entry_unstaged' helper to be sure 'git add' is behaving\ncorrectly.\n\nWe need to modify setup_sparse_entry to actually commit the sparse_entry\nfile so it exists at HEAD and as an entry in the index, but its exact\ncontents are not staged in the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t3705-add-sparse-checkout.sh | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 2b1fd0d0eef..b2d798662ee 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -19,6 +19,7 @@ setup_sparse_entry () {\n \tfi &&\n \tgit add sparse_entry &&\n \tgit update-index --skip-worktree sparse_entry &&\n+\tgit commit --allow-empty -m \"ensure sparse_entry exists at HEAD\" &&\n \tSPARSE_ENTRY_BLOB=$(git rev-parse :sparse_entry)\n }\n \n@@ -36,6 +37,11 @@ setup_gitignore () {\n \tEOF\n }\n \n+test_sparse_entry_unstaged () {\n+\tgit diff --staged -- sparse_entry >diff &&\n+\ttest_must_be_empty diff\n+}\n+\n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n \tThe following pathspecs didn't match any eligible path, but they do match index\n@@ -55,6 +61,7 @@ test_expect_success 'git add does not remove sparse entries' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -73,6 +80,7 @@ test_expect_success 'git add . does not remove sparse entries' '\n \trm sparse_entry &&\n \tsetup_gitignore &&\n \ttest_must_fail git add . 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \n \tcat sparse_error_header >expect &&\n \techo . >>expect &&\n@@ -88,6 +96,7 @@ do\n \t\tsetup_sparse_entry &&\n \t\techo modified >sparse_entry &&\n \t\ttest_must_fail git add $opt sparse_entry 2>stderr &&\n+\t\ttest_sparse_entry_unstaged &&\n \t\ttest_cmp error_and_hint stderr &&\n \t\ttest_sparse_entry_unchanged\n \t'\n@@ -98,6 +107,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n \tgit ls-files --debug sparse_entry | grep mtime >before &&\n \ttest-tool chmtime -60 sparse_entry &&\n \ttest_must_fail git add --refresh sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \tgit ls-files --debug sparse_entry | grep mtime >after &&\n \ttest_cmp before after\n@@ -106,6 +116,7 @@ test_expect_success 'git add --refresh does not update sparse entries' '\n test_expect_success 'git add --chmod does not update sparse entries' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git add --chmod=+x sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged &&\n \t! test -x sparse_entry\n@@ -116,6 +127,7 @@ test_expect_success 'git add --renormalize does not update sparse entries' '\n \tsetup_sparse_entry \"LINEONE\\r\\nLINETWO\\r\\n\" &&\n \techo \"sparse_entry text=auto\" >.gitattributes &&\n \ttest_must_fail git add --renormalize sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -124,6 +136,7 @@ test_expect_success 'git add --dry-run --ignore-missing warn on sparse path' '\n \tsetup_sparse_entry &&\n \trm sparse_entry &&\n \ttest_must_fail git add --dry-run --ignore-missing sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp error_and_hint stderr &&\n \ttest_sparse_entry_unchanged\n '\n@@ -148,6 +161,7 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n+\ttest_sparse_entry_unstaged &&\n \ttest_cmp sparse_entry_error stderr\n \n '\n-- \ngitgitgadget\n\n"},{"id":"436955","messageId":"58389edc76ca8f14951ee9387a0067c1ead08f41.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 02/13] t1092: behavior for adding sparse files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:03Z","receivedAt":"2021-09-24T15:39:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd some tests to demonstrate the current behavior around adding files\noutside of the sparse-checkout cone. Currently, untracked files are\nhandled differently from tracked files. A future change will make these\ncases be handled the same way.\n\nFurther expand checking that a failed 'git add' does not stage changes\nto the index.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 28 ++++++++++++++++++++++++\n 1 file changed, 28 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 886e78715fe..7edc2060221 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -187,6 +187,16 @@ test_sparse_match () {\n \ttest_cmp sparse-checkout-err sparse-index-err\n }\n \n+test_sparse_unstaged () {\n+\tfile=$1 &&\n+\tfor repo in sparse-checkout sparse-index\n+\tdo\n+\t\t# Skip \"unmerged\" paths\n+\t\tgit -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n+\t\ttest_must_be_empty diff || return 1\n+\tdone\n+}\n+\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n@@ -291,6 +301,20 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+# NEEDSWORK: This documents current behavior, but is not a desirable\n+# behavior (untracked files are handled differently than tracked).\n+test_expect_success 'add outside sparse cone' '\n+\tinit_repos &&\n+\n+\trun_on_sparse mkdir folder1 &&\n+\trun_on_sparse ../edit-contents folder1/a &&\n+\trun_on_sparse ../edit-contents folder1/newfile &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match git add folder1/newfile\n+'\n+\n test_expect_success 'commit including unstaged changes' '\n \tinit_repos &&\n \n@@ -339,7 +363,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \n \t# Adding the path outside of the sparse-checkout cone should fail.\n \ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \n \t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n \ttest_sparse_match git add folder1/new &&\n-- \ngitgitgadget\n\n"},{"id":"436956","messageId":"2ebaf8e68c2c46f700f7523d96e7494ebe3fb8cb.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 03/13] dir: select directories correctly","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:04Z","receivedAt":"2021-09-24T15:39:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen matching a path against a list of patterns, the ones that require a\ndirectory match previously did not work when a filename is specified.\nThis was fine when all pattern-matching was done within methods such as\nunpack_trees() that check a directory before recursing into the\ncontained files. However, other commands will start matching individual\nfiles against pattern lists without that recursive approach.\n\nThe last_matching_pattern_from_list() logic performs some checks on the\nfiletype of a path within the index when the PATTERN_FLAG_MUSTBEDIR flag\nis set. This works great when setting SKIP_WORKTREE bits within\nunpack_trees(), but doesn't work well when passing an arbitrary path\nsuch as a file within a matching directory.\n\nWe extract the logic around determining the file type, but attempt to\navoid checking the filesystem if the parent directory already matches\nthe sparse-checkout patterns. The new path_matches_dir_pattern() method\nincludes a 'path_parent' parameter that is used to store the parent\ndirectory of 'pathname' between multiple pattern matching tests. This is\nloaded lazily, only on the first pattern it finds that has the\nPATTERN_FLAG_MUSTBEDIR flag.\n\nIf we find that a path has a parent directory, we start by checking to\nsee if that parent directory matches the pattern. If so, then we do not\nneed to query the index for the type (which can be expensive). If we\nfind that the parent does not match, then we still must check the type\nfrom the index for the given pathname.\n\nNote that this does not affect cone mode pattern matching, but instead\nthe more general -- and slower -- full pattern set. Thus, this does not\naffect the sparse index.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n dir.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++-----\n 1 file changed, 49 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 86afa2eae00..9ea6cfe61cb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1303,6 +1303,44 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t\t\t WM_PATHNAME) == 0;\n }\n \n+static int path_matches_dir_pattern(const char *pathname,\n+\t\t\t\t int pathlen,\n+\t\t\t\t struct strbuf **path_parent,\n+\t\t\t\t int *dtype,\n+\t\t\t\t struct path_pattern *pattern,\n+\t\t\t\t struct index_state *istate)\n+{\n+\tif (!*path_parent) {\n+\t\tchar *slash;\n+\t\tCALLOC_ARRAY(*path_parent, 1);\n+\t\tstrbuf_add(*path_parent, pathname, pathlen);\n+\t\tslash = find_last_dir_sep((*path_parent)->buf);\n+\n+\t\tif (slash)\n+\t\t\tstrbuf_setlen(*path_parent, slash - (*path_parent)->buf);\n+\t\telse\n+\t\t\tstrbuf_setlen(*path_parent, 0);\n+\t}\n+\n+\t/*\n+\t * If the parent directory matches the pattern, then we do not\n+\t * need to check for dtype.\n+\t */\n+\tif ((*path_parent)->len &&\n+\t match_pathname((*path_parent)->buf, (*path_parent)->len,\n+\t\t\t pattern->base,\n+\t\t\t pattern->baselen ? pattern->baselen - 1 : 0,\n+\t\t\t pattern->pattern, pattern->nowildcardlen,\n+\t\t\t pattern->patternlen, pattern->flags))\n+\t\treturn 1;\n+\n+\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n+\tif (*dtype != DT_DIR)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n /*\n * Scan the given exclude list in reverse to see whether pathname\n * should be ignored. The first match (i.e. the last on the list), if\n@@ -1318,6 +1356,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n {\n \tstruct path_pattern *res = NULL; /* undecided */\n \tint i;\n+\tstruct strbuf *path_parent = NULL;\n \n \tif (!pl->nr)\n \t\treturn NULL;\t/* undefined */\n@@ -1327,11 +1366,10 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tconst char *exclude = pattern->pattern;\n \t\tint prefix = pattern->nowildcardlen;\n \n-\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n-\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n-\t\t\tif (*dtype != DT_DIR)\n-\t\t\t\tcontinue;\n-\t\t}\n+\t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t\t !path_matches_dir_pattern(pathname, pathlen, &path_parent,\n+\t\t\t\t\t dtype, pattern, istate))\n+\t\t\tcontinue;\n \n \t\tif (pattern->flags & PATTERN_FLAG_NODIR) {\n \t\t\tif (match_basename(basename,\n@@ -1355,6 +1393,12 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\t\tbreak;\n \t\t}\n \t}\n+\n+\tif (path_parent) {\n+\t\tstrbuf_release(path_parent);\n+\t\tfree(path_parent);\n+\t}\n+\n \treturn res;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"436957","messageId":"24bffdab139435173712101aaf72f7277298c99d.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:05Z","receivedAt":"2021-09-24T15:39:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWithin match_pathname(), one successful matching category happens when\nthe pattern is equal to its non-wildcard prefix. At this point, we have\nchecked that the input 'pathname' matches the pattern up to the prefix\nlength, and then we subtraced that length from both 'patternlen' and\n'namelen'.\n\nIn the case of a directory match, this prefix match should be\nsufficient. However, the success condition only cared about _exact_\nequality here. Instead, we should allow any path that agrees on this\nprefix in the case of PATTERN_FLAG_MUSTBEDIR.\n\nThis case was not tested before because of the way unpack_trees() would\nmatch a parent directory before visiting the contained paths. This\napproach is changing, so we must change this comparison.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9ea6cfe61cb..174d336c30e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t * then our prefix match is all we need; we\n \t\t * do not need to call fnmatch at all.\n \t\t */\n-\t\tif (!patternlen && !namelen)\n+\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n \t\t\treturn 1;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"436958","messageId":"e3a749e3182b07f2b219ff40f16e38868d0ba283.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 05/13] add: fail when adding an untracked sparse file","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:06Z","receivedAt":"2021-09-24T15:39:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe add_files() method in builtin/add.c takes a set of untracked files\nthat are being added by the input pathspec and inserts them into the\nindex. If these files are outside of the sparse-checkout cone, then they\ngain the SKIP_WORKTREE bit at some point. However, this was not checked\nbefore inserting into the index, so these files are added even though we\nwant to avoid modifying the index outside of the sparse-checkout cone.\n\nAdd a check within add_files() for these files and write the advice\nabout files outside of the sparse-checkout cone.\n\nThis behavior change modifies some existing tests within t1092. These\ntests intended to document how a user could interact with the existing\nbehavior in place. Many of these tests need to be marked as expecting\nfailure. A future change will allow these tests to pass by adding a flag\nto 'git add' that allows users to modify index entries outside of the\nsparse-checkout cone.\n\nThe 'submodule handling' test is intended to document what happens to\ndirectories that contain a submodule when the sparse index is enabled.\nIt is not trying to say that users should be able to add submodules\noutside of the sparse-checkout cone, so that test can be modified to\navoid that operation.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 14 +++++++++\n t/t1092-sparse-checkout-compatibility.sh | 37 ++++++++++++++++++------\n 2 files changed, 42 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 88a6c0c69fb..8ea9cae0e7a 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -443,6 +443,7 @@ static void check_embedded_repo(const char *path)\n static int add_files(struct dir_struct *dir, int flags)\n {\n \tint i, exit_status = 0;\n+\tstruct string_list matched_sparse_paths = STRING_LIST_INIT_NODUP;\n \n \tif (dir->ignored_nr) {\n \t\tfprintf(stderr, _(ignore_error));\n@@ -456,6 +457,11 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n+\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\t\tstring_list_append(&matched_sparse_paths,\n+\t\t\t\t\t dir->entries[i]->name);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (add_file_to_index(&the_index, dir->entries[i]->name, flags)) {\n \t\t\tif (!ignore_add_errors)\n \t\t\t\tdie(_(\"adding files failed\"));\n@@ -464,6 +470,14 @@ static int add_files(struct dir_struct *dir, int flags)\n \t\t\tcheck_embedded_repo(dir->entries[i]->name);\n \t\t}\n \t}\n+\n+\tif (matched_sparse_paths.nr) {\n+\t\tadvise_on_updating_sparse_paths(&matched_sparse_paths);\n+\t\texit_status = 1;\n+\t}\n+\n+\tstring_list_clear(&matched_sparse_paths, 0);\n+\n \treturn exit_status;\n }\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 7edc2060221..17525ff13fc 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -301,8 +301,6 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-# NEEDSWORK: This documents current behavior, but is not a desirable\n-# behavior (untracked files are handled differently than tracked).\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n@@ -312,7 +310,9 @@ test_expect_success 'add outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n-\ttest_sparse_match git add folder1/newfile\n+\ttest_sparse_match test_must_fail git add folder1/newfile &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/newfile\n '\n \n test_expect_success 'commit including unstaged changes' '\n@@ -343,7 +343,11 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-test_expect_success 'status/add: outside sparse cone' '\n+# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n+# attempted here fail for the sparse-checkout and sparse-index repos.\n+# We must enable a way for adding files outside the sparse-checkout\n+# done, even if it is by an optional flag.\n+test_expect_failure 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -368,10 +372,11 @@ test_expect_success 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add --refresh folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/new &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/new &&\n \n-\t# NEEDSWORK: Adding a newly-tracked file outside the cone succeeds\n-\ttest_sparse_match git add folder1/new &&\n-\n+\t# NEEDSWORK: behavior begins to deviate here.\n \ttest_all_match git add . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n@@ -527,7 +532,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n # Right now, users might be using this flow to work through conflicts,\n # so any solution should present advice to users who try this sequence\n # of commands to follow whatever new method we create.\n-test_expect_success 'merge with conflict outside cone' '\n+test_expect_failure 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -541,12 +546,18 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n+\t# NEEDSWORK: Even though the merge conflict removed the\n+\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t# warn that this is a problematic add.\n \ttest_all_match git add folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n+\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t# outside of the sparse-checkout cone and does not match an\n+\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_all_match git add folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n@@ -555,7 +566,7 @@ test_expect_success 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -572,11 +583,17 @@ test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 2. Add the file with conflict markers\n+\t\t# NEEDSWORK: Even though the merge conflict removed the\n+\t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n+\t\t# warn that this is a problematic add.\n \t\ttest_all_match git add folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n \t\t# accept conflict markers as resolved content.\n+\t\t# NEEDSWORK: This mode now fails, because folder2/z is\n+\t\t# outside of the sparse-checkout cone and does not match an\n+\t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n \t\ttest_all_match git add folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n@@ -654,6 +671,7 @@ test_expect_success 'clean' '\n test_expect_success 'submodule handling' '\n \tinit_repos &&\n \n+\ttest_sparse_match git sparse-checkout add modules &&\n \ttest_all_match mkdir modules &&\n \ttest_all_match touch modules/a &&\n \ttest_all_match git add modules &&\n@@ -663,6 +681,7 @@ test_expect_success 'submodule handling' '\n \ttest_all_match git commit -m \"add submodule\" &&\n \n \t# having a submodule prevents \"modules\" from collapse\n+\ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n \ttest-tool -C sparse-index read-cache --table >cache &&\n \tgrep \"100644 blob .*\tmodules/a\" cache &&\n \tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n-- \ngitgitgadget\n\n"},{"id":"436959","messageId":"2c5c834bc9fb42aeaff7befbba477aec727184c0.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 06/13] add: skip tracked paths outside sparse-checkout cone","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:07Z","receivedAt":"2021-09-24T15:39:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen 'git add' adds a tracked file that is outside of the\nsparse-checkout cone, it checks the SKIP_WORKTREE bit to see if the file\nexists outside of the sparse-checkout cone. This is usually correct,\nexcept in the case of a merge conflict outside of the cone.\n\nModify add_pathspec_matched_against_index() to be more careful about\npaths by checking the sparse-checkout patterns in addition to the\nSKIP_WORKTREE bit. This causes 'git add' to no longer allow files\noutside of the cone that removed the SKIP_WORKTREE bit due to a merge\nconflict.\n\nWith only this change, users will only be able to add the file after\nadding the file to the sparse-checkout cone. A later change will allow\nusers to force adding even though the file is outside of the\nsparse-checkout cone.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 ++++\n pathspec.c | 5 +++--\n t/t1091-sparse-checkout-builtin.sh | 4 +++-\n t/t1092-sparse-checkout-compatibility.sh | 19 ++++++++++++-------\n t/t3705-add-sparse-checkout.sh | 12 ++++++++++++\n 5 files changed, 34 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 8ea9cae0e7a..09c3fad6321 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -94,6 +94,10 @@ static void update_callback(struct diff_queue_struct *q,\n \tfor (i = 0; i < q->nr; i++) {\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n+\n+\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\t\tcontinue;\n+\n \t\tswitch (fix_unmerged_status(p, data)) {\n \t\tdefault:\n \t\t\tdie(_(\"unexpected diff status %c\"), p->status);\ndiff --git a/pathspec.c b/pathspec.c\nindex 44306fdaca2..ddeeba79114 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -39,7 +39,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\treturn;\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n-\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce))\n+\t\tif (sw_action == PS_IGNORE_SKIP_WORKTREE &&\n+\t\t (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate)))\n \t\t\tcontinue;\n \t\tce_path_match(istate, ce, pathspec, seen);\n \t}\n@@ -70,7 +71,7 @@ char *find_pathspecs_matching_skip_worktree(const struct pathspec *pathspec)\n \n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (ce_skip_worktree(ce) || !path_in_sparse_checkout(ce->name, istate))\n \t\t ce_path_match(istate, ce, pathspec, seen);\n \t}\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 71236981e64..af99ae81b1d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -406,7 +406,7 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat\n \tgit -C unmerged sparse-checkout disable\n '\n \n-test_expect_success 'sparse-checkout reapply' '\n+test_expect_failure 'sparse-checkout reapply' '\n \tgit clone repo tweak &&\n \n \techo dirty >tweak/deep/deeper2/a &&\n@@ -438,6 +438,8 @@ test_expect_success 'sparse-checkout reapply' '\n \ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n \ttest_path_is_file tweak/folder1/a &&\n \n+\t# NEEDSWORK: We are asking to update a file outside of the\n+\t# sparse-checkout cone, but this is no longer allowed.\n \tgit -C tweak add folder1/a &&\n \tgit -C tweak sparse-checkout reapply 2>err &&\n \ttest_must_be_empty err &&\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 17525ff13fc..062b42f2c10 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -546,10 +546,9 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 2. Add the file with conflict markers\n-\t# NEEDSWORK: Even though the merge conflict removed the\n-\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n-\t# warn that this is a problematic add.\n-\ttest_all_match git add folder1/a &&\n+\ttest_sparse_match test_must_fail git add folder1/a &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n@@ -558,7 +557,9 @@ test_expect_failure 'merge with conflict outside cone' '\n \t# NEEDSWORK: This mode now fails, because folder2/z is\n \t# outside of the sparse-checkout cone and does not match an\n \t# existing index entry with the SKIP_WORKTREE bit cleared.\n-\ttest_all_match git add folder2 &&\n+\ttest_sparse_match test_must_fail git add folder2 &&\n+\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\ttest_sparse_unstaged folder2/z &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -586,7 +587,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# NEEDSWORK: Even though the merge conflict removed the\n \t\t# SKIP_WORKTREE bit from the index entry for folder1/a, we should\n \t\t# warn that this is a problematic add.\n-\t\ttest_all_match git add folder1/a &&\n+\t\ttest_sparse_match test_must_fail git add folder1/a &&\n+\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -595,7 +598,9 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\t# outside of the sparse-checkout cone and does not match an\n \t\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \t\trun_on_all mv folder2/a folder2/z &&\n-\t\ttest_all_match git add folder2 &&\n+\t\ttest_sparse_match test_must_fail git add folder2 &&\n+\t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n+\t\ttest_sparse_unstaged folder2/z &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex b2d798662ee..be6809eed23 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -158,6 +158,18 @@ test_expect_success 'do not warn when pathspec matches dense entries' '\n \tgit ls-files --error-unmatch dense_entry\n '\n \n+test_expect_success 'git add fails outside of sparse-checkout definition' '\n+\ttest_when_finished git sparse-checkout disable &&\n+\ttest_commit a &&\n+\tgit sparse-checkout init &&\n+\tgit sparse-checkout set a &&\n+\techo >>sparse_entry &&\n+\n+\tgit update-index --no-skip-worktree sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unstaged\n+'\n+\n test_expect_success 'add obeys advice.updateSparsePath' '\n \tsetup_sparse_entry &&\n \ttest_must_fail git -c advice.updateSparsePath=false add sparse_entry 2>stderr &&\n-- \ngitgitgadget\n\n"},{"id":"436960","messageId":"430ab44e4f1c7998cee5f034b6a32b065f197900.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 07/13] add: implement the --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:08Z","receivedAt":"2021-09-24T15:39:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe previously modified 'git add' to refuse updating index entries\noutside of the sparse-checkout cone. This is justified to prevent users\nfrom accidentally getting into a confusing state when Git removes those\nfiles from the working tree at some later point.\n\nUnfortunately, this caused some workflows that were previously possible\nto become impossible, especially around merge conflicts outside of the\nsparse-checkout cone. These were documented in tests within t1092.\n\nWe now re-enable these workflows using a new '--sparse' option to 'git\nadd'. This allows users to signal \"Yes, I do know what I'm doing with\nthese files,\" and accept the consequences of the files leaving the\nworktree later.\n\nWe delay updating the advice message until implementing a similar option\nin 'git rm' and 'git mv'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-add.txt | 9 +++++++-\n builtin/add.c | 12 ++++++----\n t/t1092-sparse-checkout-compatibility.sh | 29 +++++++++---------------\n t/t3705-add-sparse-checkout.sh | 17 +++++++++++++-\n 4 files changed, 43 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex be5e3ac54b8..11eb70f16c7 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -9,7 +9,7 @@ SYNOPSIS\n --------\n [verse]\n 'git add' [--verbose | -v] [--dry-run | -n] [--force | -f] [--interactive | -i] [--patch | -p]\n-\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]]\n+\t [--edit | -e] [--[no-]all | --[no-]ignore-removal | [--update | -u]] [--sparse]\n \t [--intent-to-add | -N] [--refresh] [--ignore-errors] [--ignore-missing] [--renormalize]\n \t [--chmod=(+|-)x] [--pathspec-from-file= [--pathspec-file-nul]]\n \t [--] [...]\n@@ -79,6 +79,13 @@ in linkgit:gitglossary[7].\n --force::\n \tAllow adding otherwise ignored files.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git add` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone, since those files might\n+\tbe removed from the working tree without warning. See\n+\tlinkgit:git-sparse-checkout[1] for more details.\n+\n -i::\n --interactive::\n \tAdd modified contents in the working tree interactively to\ndiff --git a/builtin/add.c b/builtin/add.c\nindex 09c3fad6321..f8e3930608d 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -30,6 +30,7 @@ static int patch_interactive, add_interactive, edit_interactive;\n static int take_worktree_changes;\n static int add_renormalize;\n static int pathspec_file_nul;\n+static int include_sparse;\n static const char *pathspec_from_file;\n static int legacy_stash_p; /* support for the scripted `git stash` */\n \n@@ -46,7 +47,7 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\n@@ -95,7 +96,7 @@ static void update_callback(struct diff_queue_struct *q,\n \t\tstruct diff_filepair *p = q->queue[i];\n \t\tconst char *path = p->one->path;\n \n-\t\tif (!path_in_sparse_checkout(path, &the_index))\n+\t\tif (!include_sparse && !path_in_sparse_checkout(path, &the_index))\n \t\t\tcontinue;\n \n \t\tswitch (fix_unmerged_status(p, data)) {\n@@ -383,6 +384,7 @@ static struct option builtin_add_options[] = {\n \tOPT_BOOL( 0 , \"refresh\", &refresh_only, N_(\"don't add, only refresh the index\")),\n \tOPT_BOOL( 0 , \"ignore-errors\", &ignore_add_errors, N_(\"just skip files which cannot be added because of errors\")),\n \tOPT_BOOL( 0 , \"ignore-missing\", &ignore_missing, N_(\"check if - even missing - files are ignored in dry run\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_STRING(0, \"chmod\", &chmod_arg, \"(+|-)x\",\n \t\t N_(\"override the executable bit of the listed files\")),\n \tOPT_HIDDEN_BOOL(0, \"warn-embedded-repo\", &warn_on_embedded_repo,\n@@ -461,7 +463,8 @@ static int add_files(struct dir_struct *dir, int flags)\n \t}\n \n \tfor (i = 0; i < dir->nr; i++) {\n-\t\tif (!path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n+\t\tif (!include_sparse &&\n+\t\t !path_in_sparse_checkout(dir->entries[i]->name, &the_index)) {\n \t\t\tstring_list_append(&matched_sparse_paths,\n \t\t\t\t\t dir->entries[i]->name);\n \t\t\tcontinue;\n@@ -646,7 +649,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\t\tif (seen[i])\n \t\t\t\tcontinue;\n \n-\t\t\tif (matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n+\t\t\tif (!include_sparse &&\n+\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen)) {\n \t\t\t\tstring_list_append(&only_match_skip_worktree,\n \t\t\t\t\t\t pathspec.items[i].original);\n \t\t\t\tcontinue;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 062b42f2c10..a00e42fa233 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -343,11 +343,7 @@ test_expect_success 'commit including unstaged changes' '\n \ttest_all_match git status --porcelain=v2\n '\n \n-# NEEDSWORK: Now that 'git add folder1/new' fails, the changes being\n-# attempted here fail for the sparse-checkout and sparse-index repos.\n-# We must enable a way for adding files outside the sparse-checkout\n-# done, even if it is by an optional flag.\n-test_expect_failure 'status/add: outside sparse cone' '\n+test_expect_success 'status/add: outside sparse cone' '\n \tinit_repos &&\n \n \t# folder1 is at HEAD, but outside the sparse cone\n@@ -375,15 +371,16 @@ test_expect_failure 'status/add: outside sparse cone' '\n \ttest_sparse_match test_must_fail git add folder1/new &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/new &&\n+\ttest_sparse_match git add --sparse folder1/a &&\n+\ttest_sparse_match git add --sparse folder1/new &&\n \n-\t# NEEDSWORK: behavior begins to deviate here.\n-\ttest_all_match git add . &&\n+\ttest_all_match git add --sparse . &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/new &&\n \ttest_all_match git rev-parse HEAD^{tree} &&\n \n \trun_on_all ../edit-contents folder1/newer &&\n-\ttest_all_match git add folder1/ &&\n+\ttest_all_match git add --sparse folder1/ &&\n \ttest_all_match git status --porcelain=v2 &&\n \ttest_all_match git commit -m folder1/newer &&\n \ttest_all_match git rev-parse HEAD^{tree}\n@@ -527,12 +524,7 @@ test_expect_success 'merge, cherry-pick, and rebase' '\n \tdone\n '\n \n-# NEEDSWORK: This test is documenting current behavior, but that\n-# behavior can be confusing to users so there is desire to change it.\n-# Right now, users might be using this flow to work through conflicts,\n-# so any solution should present advice to users who try this sequence\n-# of commands to follow whatever new method we create.\n-test_expect_failure 'merge with conflict outside cone' '\n+test_expect_success 'merge with conflict outside cone' '\n \tinit_repos &&\n \n \ttest_all_match git checkout -b merge-tip merge-left &&\n@@ -549,17 +541,16 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_sparse_match test_must_fail git add folder1/a &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder1/a &&\n+\ttest_all_match git add --sparse folder1/a &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \t# 3. Rename the file to another sparse filename and\n \t# accept conflict markers as resolved content.\n \trun_on_all mv folder2/a folder2/z &&\n-\t# NEEDSWORK: This mode now fails, because folder2/z is\n-\t# outside of the sparse-checkout cone and does not match an\n-\t# existing index entry with the SKIP_WORKTREE bit cleared.\n \ttest_sparse_match test_must_fail git add folder2 &&\n \tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \ttest_sparse_unstaged folder2/z &&\n+\ttest_all_match git add --sparse folder2 &&\n \ttest_all_match git status --porcelain=v2 &&\n \n \ttest_all_match git merge --continue &&\n@@ -567,7 +558,7 @@ test_expect_failure 'merge with conflict outside cone' '\n \ttest_all_match git rev-parse HEAD^{tree}\n '\n \n-test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n+test_expect_success 'cherry-pick/rebase with conflict outside cone' '\n \tinit_repos &&\n \n \tfor OPERATION in cherry-pick rebase\n@@ -590,6 +581,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder1/a &&\n \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder1/a &&\n+\t\ttest_all_match git add --sparse folder1/a &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\t# 3. Rename the file to another sparse filename and\n@@ -601,6 +593,7 @@ test_expect_failure 'cherry-pick/rebase with conflict outside cone' '\n \t\ttest_sparse_match test_must_fail git add folder2 &&\n \t\tgrep \"Disable or modify the sparsity rules\" sparse-checkout-err &&\n \t\ttest_sparse_unstaged folder2/z &&\n+\t\ttest_all_match git add --sparse folder2 &&\n \t\ttest_all_match git status --porcelain=v2 &&\n \n \t\ttest_all_match git $OPERATION --continue &&\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex be6809eed23..3cab82092d4 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -167,7 +167,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \n \tgit update-index --no-skip-worktree sparse_entry &&\n \ttest_must_fail git add sparse_entry &&\n-\ttest_sparse_entry_unstaged\n+\ttest_sparse_entry_unstaged &&\n+\n+\t# Avoid munging CRLFs to avoid an error message\n+\tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n@@ -178,4 +184,13 @@ test_expect_success 'add obeys advice.updateSparsePath' '\n \n '\n \n+test_expect_success 'add allows sparse entries with --sparse' '\n+\tgit sparse-checkout set a &&\n+\techo modified >sparse_entry &&\n+\ttest_must_fail git add sparse_entry &&\n+\ttest_sparse_entry_unchanged &&\n+\tgit add --sparse sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"436961","messageId":"4f7b5cdfa36b2edef666aa12ed8aee87405fc884.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 08/13] add: update --chmod to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:09Z","receivedAt":"2021-09-24T15:39:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent staging a modification, but we skipped the\n--chmod mode. Update chmod_pathspec() to ignore cache entries whose path\nis outside of the sparse-checkout cone (unless --sparse is provided).\nAdd a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 10 +++++++++-\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f8e3930608d..f87b8134b67 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -47,7 +47,9 @@ static int chmod_pathspec(struct pathspec *pathspec, char flip, int show_only)\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tint err;\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \n \t\tif (pathspec && !ce_path_match(&the_index, ce, pathspec, NULL))\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 3cab82092d4..0ae674a17a9 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -169,11 +169,19 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --chmod=+x sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest-tool read-cache --table >actual &&\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"436962","messageId":"30ec60969399e2d876d34fdf2150e4152bb9438a.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 09/13] add: update --renormalize to skip sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:10Z","receivedAt":"2021-09-24T15:39:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe added checks for path_in_sparse_checkout() to portions of 'git add'\nthat add warnings and prevent stagins a modification, but we skipped the\n--renormalize mode. Update renormalize_tracked_files() to ignore cache\nentries whose path is outside of the sparse-checkout cone (unless\n--sparse is provided). Add a test in t3705.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 4 +++-\n t/t3705-add-sparse-checkout.sh | 12 +++++++++++-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex f87b8134b67..f8f0dfa4046 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -154,7 +154,9 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-\t\tif (ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (ce_stage(ce))\n \t\t\tcontinue; /* do not touch unmerged paths */\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 0ae674a17a9..339ec0ed2d6 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -172,6 +172,9 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \ttest_must_fail git add --chmod=+x sparse_entry &&\n \ttest_sparse_entry_unstaged &&\n \n+\ttest_must_fail git add --renormalize sparse_entry &&\n+\ttest_sparse_entry_unstaged &&\n+\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n@@ -181,7 +184,14 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n \ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual\n+\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\n+\tgit reset &&\n+\n+\t# This will print a message over stderr on Windows.\n+\tgit add --sparse --renormalize sparse_entry &&\n+\tgit status --porcelain >actual &&\n+\tgrep \"^M sparse_entry\\$\" actual\n '\n \n test_expect_success 'add obeys advice.updateSparsePath' '\n-- \ngitgitgadget\n\n"},{"id":"436963","messageId":"99d50921ef4b787a0cc67d05a1a92f64a91b25d3.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 10/13] rm: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:11Z","receivedAt":"2021-09-24T15:39:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we did previously in 'git add', add a '--sparse' option to 'git rm'\nthat allows modifying paths outside of the sparse-checkout definition.\nThe existing checks in 'git rm' are restricted to tracked files that\nhave the SKIP_WORKTREE bit in the current index. Future changes will\ncause 'git rm' to reject removing paths outside of the sparse-checkout\ndefinition, even if they are untracked or do not have the SKIP_WORKTREE\nbit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-rm.txt | 6 ++++++\n builtin/rm.c | 8 ++++++--\n t/t3602-rm-sparse-checkout.sh | 12 ++++++++++++\n 3 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-rm.txt b/Documentation/git-rm.txt\nindex 26e9b284704..81bc23f3cdb 100644\n--- a/Documentation/git-rm.txt\n+++ b/Documentation/git-rm.txt\n@@ -72,6 +72,12 @@ For more details, see the 'pathspec' entry in linkgit:gitglossary[7].\n --ignore-unmatch::\n \tExit with a zero status even if no files matched.\n \n+--sparse::\n+\tAllow updating index entries outside of the sparse-checkout cone.\n+\tNormally, `git rm` refuses to update index entries whose paths do\n+\tnot fit within the sparse-checkout cone. See\n+\tlinkgit:git-sparse-checkout[1] for more.\n+\n -q::\n --quiet::\n \t`git rm` normally outputs one line (in the form of an `rm` command)\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 8a24c715e02..4208f3f9a5f 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -237,6 +237,7 @@ static int check_local_mod(struct object_id *head, int index_only)\n \n static int show_only = 0, force = 0, index_only = 0, recursive = 0, quiet = 0;\n static int ignore_unmatch = 0, pathspec_file_nul;\n+static int include_sparse;\n static char *pathspec_from_file;\n \n static struct option builtin_rm_options[] = {\n@@ -247,6 +248,7 @@ static struct option builtin_rm_options[] = {\n \tOPT_BOOL('r', NULL, &recursive, N_(\"allow recursive removal\")),\n \tOPT_BOOL( 0 , \"ignore-unmatch\", &ignore_unmatch,\n \t\t\t\tN_(\"exit with a zero status even if nothing matched\")),\n+\tOPT_BOOL(0, \"sparse\", &include_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \tOPT_PATHSPEC_FROM_FILE(&pathspec_from_file),\n \tOPT_PATHSPEC_FILE_NUL(&pathspec_file_nul),\n \tOPT_END(),\n@@ -298,7 +300,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n-\t\tif (ce_skip_worktree(ce))\n+\n+\t\tif (!include_sparse && ce_skip_worktree(ce))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\n@@ -322,7 +325,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \t\t\t\tseen_any = 1;\n \t\t\telse if (ignore_unmatch)\n \t\t\t\tcontinue;\n-\t\t\telse if (matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n+\t\t\telse if (!include_sparse &&\n+\t\t\t\t matches_skip_worktree(&pathspec, i, &skip_worktree_seen))\n \t\t\t\tstring_list_append(&only_match_skip_worktree, original);\n \t\t\telse\n \t\t\t\tdie(_(\"pathspec '%s' did not match any files\"), original);\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex e9e9a15c74c..493c8f636b8 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -43,6 +43,18 @@ test_expect_success 'recursive rm does not remove sparse entries' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'recursive rm --sparse removes sparse entries' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set \"sub/dir\" &&\n+\tgit rm --sparse -r sub &&\n+\tgit status --porcelain -uno >actual &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n test_expect_success 'rm obeys advice.updateSparsePath' '\n \tgit reset --hard &&\n \tgit sparse-checkout set a &&\n-- \ngitgitgadget\n\n"},{"id":"436964","messageId":"47a1444115b517b2842f179ba692962a982860cb.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 11/13] rm: skip sparse paths with missing SKIP_WORKTREE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:12Z","receivedAt":"2021-09-24T15:39:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a path does not match the sparse-checkout cone but is somehow missing\nthe SKIP_WORKTREE bit, then 'git rm' currently succeeds in removing the\nfile. One reason a user might be in this situation is a merge conflict\noutside of the sparse-checkout cone. Removing such a file might be\nproblematic for users who are not sure what they are doing.\n\nAdd a check to path_in_sparse_checkout() when 'git rm' is checking if a\npath should be considered for deletion. Of course, this check is ignored\nif the '--sparse' option is specified, allowing users who accept the\nrisks to continue with the removal.\n\nThis also removes a confusing behavior where a user asks for a directory\nto be removed, but only the entries that are within the sparse-checkout\ndefinition are removed. Now, 'git rm ' will fail without '--sparse'\nand will succeed in removing all contained paths with '--sparse'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 4 +++-\n t/t3602-rm-sparse-checkout.sh | 19 +++++++++++++++++--\n 2 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4208f3f9a5f..a6da03da2be 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -301,7 +301,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \n-\t\tif (!include_sparse && ce_skip_worktree(ce))\n+\t\tif (!include_sparse &&\n+\t\t (ce_skip_worktree(ce) ||\n+\t\t !path_in_sparse_checkout(ce->name, &the_index)))\n \t\t\tcontinue;\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n \t\t\tcontinue;\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 493c8f636b8..5f92b60a56a 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -37,9 +37,13 @@ done\n test_expect_success 'recursive rm does not remove sparse entries' '\n \tgit reset --hard &&\n \tgit sparse-checkout set sub/dir &&\n-\tgit rm -r sub &&\n+\ttest_must_fail git rm -r sub &&\n+\tgit rm --sparse -r sub &&\n \tgit status --porcelain -uno >actual &&\n-\techo \"D sub/dir/e\" >expected &&\n+\tcat >expected <<-\\EOF &&\n+\tD sub/d\n+\tD sub/dir/e\n+\tEOF\n \ttest_cmp expected actual\n '\n \n@@ -87,4 +91,15 @@ test_expect_success 'do not warn about sparse entries with --ignore-unmatch' '\n \tgit ls-files --error-unmatch b\n '\n \n+test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\tgit update-index --no-skip-worktree b &&\n+\ttest_must_fail git rm b 2>stderr &&\n+\ttest_cmp b_error_and_hint stderr &&\n+\tgit rm --sparse b 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"436965","messageId":"28e703d80d38b5589dc117823d40ca99f8ad7cf3.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 12/13] mv: refuse to move sparse paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:13Z","receivedAt":"2021-09-24T15:39:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince cmd_mv() does not operate on cache entries and instead directly\nchecks the filesystem, we can only use path_in_sparse_checkout() as a\nmechanism for seeing if a path is sparse or not. Be sure to skip\nreturning a failure if '-k' is specified.\n\nTo ensure that the advice around sparse paths is the only reason a move\nfailed, be sure to check this as the very last thing before inserting\ninto the src_for_dst list.\n\nThe tests cover a variety of cases such as whether the target is tracked\nor untracked, and whether the source or destination are in or outside of\nthe sparse-checkout definition.\n\nHelped-by: Matheus Tavares Bernardino \nSigned-off-by: Derrick Stolee \n---\n builtin/mv.c | 52 ++++++++--\n t/t7002-mv-sparse-checkout.sh | 186 ++++++++++++++++++++++++++++++++++\n 2 files changed, 229 insertions(+), 9 deletions(-)\n create mode 100755 t/t7002-mv-sparse-checkout.sh\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex c2f96c8e895..83a465ba831 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -118,21 +118,23 @@ static int index_range_of_same_dir(const char *src, int length,\n int cmd_mv(int argc, const char **argv, const char *prefix)\n {\n \tint i, flags, gitmodules_modified = 0;\n-\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0;\n+\tint verbose = 0, show_only = 0, force = 0, ignore_errors = 0, ignore_sparse = 0;\n \tstruct option builtin_mv_options[] = {\n \t\tOPT__VERBOSE(&verbose, N_(\"be verbose\")),\n \t\tOPT__DRY_RUN(&show_only, N_(\"dry run\")),\n \t\tOPT__FORCE(&force, N_(\"force move/rename even if target exists\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n \t\tOPT_BOOL('k', NULL, &ignore_errors, N_(\"skip move/rename errors\")),\n+\t\tOPT_BOOL(0, \"sparse\", &ignore_sparse, N_(\"allow updating entries outside of the sparse-checkout cone\")),\n \t\tOPT_END(),\n \t};\n \tconst char **source, **destination, **dest_path, **submodule_gitfile;\n-\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX } *modes;\n+\tenum update_mode { BOTH = 0, WORKING_DIRECTORY, INDEX, SPARSE } *modes;\n \tstruct stat st;\n \tstruct string_list src_for_dst = STRING_LIST_INIT_NODUP;\n \tstruct lock_file lock_file = LOCK_INIT;\n \tstruct cache_entry *ce;\n+\tstruct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP;\n \n \tgit_config(git_default_config, NULL);\n \n@@ -176,14 +178,17 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tint length, src_is_dir;\n \t\tconst char *bad = NULL;\n+\t\tint skip_sparse = 0;\n \n \t\tif (show_only)\n \t\t\tprintf(_(\"Checking rename of '%s' to '%s'\\n\"), src, dst);\n \n \t\tlength = strlen(src);\n-\t\tif (lstat(src, &st) < 0)\n-\t\t\tbad = _(\"bad source\");\n-\t\telse if (!strncmp(src, dst, length) &&\n+\t\tif (lstat(src, &st) < 0) {\n+\t\t\t/* only error if existence is expected. */\n+\t\t\tif (modes[i] != SPARSE)\n+\t\t\t\tbad = _(\"bad source\");\n+\t\t} else if (!strncmp(src, dst, length) &&\n \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n \t\t\tbad = _(\"can not move directory into itself\");\n \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n@@ -212,11 +217,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\t\tdst_len = strlen(dst);\n \n \t\t\t\tfor (j = 0; j < last - first; j++) {\n-\t\t\t\t\tconst char *path = active_cache[first + j]->name;\n+\t\t\t\t\tconst struct cache_entry *ce = active_cache[first + j];\n+\t\t\t\t\tconst char *path = ce->name;\n \t\t\t\t\tsource[argc + j] = path;\n \t\t\t\t\tdestination[argc + j] =\n \t\t\t\t\t\tprefix_path(dst, dst_len, path + length + 1);\n-\t\t\t\t\tmodes[argc + j] = INDEX;\n+\t\t\t\t\tmodes[argc + j] = ce_skip_worktree(ce) ? SPARSE : INDEX;\n \t\t\t\t\tsubmodule_gitfile[argc + j] = NULL;\n \t\t\t\t}\n \t\t\t\targc += last - first;\n@@ -244,14 +250,36 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tbad = _(\"multiple sources for the same target\");\n \t\telse if (is_dir_sep(dst[strlen(dst) - 1]))\n \t\t\tbad = _(\"destination directory does not exist\");\n-\t\telse\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * We check if the paths are in the sparse-checkout\n+\t\t\t * definition as a very final check, since that\n+\t\t\t * allows us to point the user to the --sparse\n+\t\t\t * option as a way to have a successful run.\n+\t\t\t */\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(src, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, src);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\t\t\tif (!ignore_sparse &&\n+\t\t\t !path_in_sparse_checkout(dst, &the_index)) {\n+\t\t\t\tstring_list_append(&only_match_skip_worktree, dst);\n+\t\t\t\tskip_sparse = 1;\n+\t\t\t}\n+\n+\t\t\tif (skip_sparse)\n+\t\t\t\tgoto remove_entry;\n+\n \t\t\tstring_list_insert(&src_for_dst, dst);\n+\t\t}\n \n \t\tif (!bad)\n \t\t\tcontinue;\n \t\tif (!ignore_errors)\n \t\t\tdie(_(\"%s, source=%s, destination=%s\"),\n \t\t\t bad, src, dst);\n+remove_entry:\n \t\tif (--argc > 0) {\n \t\t\tint n = argc - i;\n \t\t\tmemmove(source + i, source + i + 1,\n@@ -266,6 +294,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n+\tif (only_match_skip_worktree.nr) {\n+\t\tadvise_on_updating_sparse_paths(&only_match_skip_worktree);\n+\t\tif (!ignore_errors)\n+\t\t\treturn 1;\n+\t}\n+\n \tfor (i = 0; i < argc; i++) {\n \t\tconst char *src = source[i], *dst = destination[i];\n \t\tenum update_mode mode = modes[i];\n@@ -274,7 +308,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\tprintf(_(\"Renaming %s to %s\\n\"), src, dst);\n \t\tif (show_only)\n \t\t\tcontinue;\n-\t\tif (mode != INDEX && rename(src, dst) < 0) {\n+\t\tif (mode != INDEX && mode != SPARSE && rename(src, dst) < 0) {\n \t\t\tif (ignore_errors)\n \t\t\t\tcontinue;\n \t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nnew file mode 100755\nindex 00000000000..07dbfeb6d17\n--- /dev/null\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -0,0 +1,186 @@\n+#!/bin/sh\n+\n+test_description='git mv in sparse working trees'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' \"\n+\tmkdir -p sub/dir sub/dir2 &&\n+\ttouch a b c sub/d sub/dir/e sub/dir2/e &&\n+\tgit add -A &&\n+\tgit commit -m files &&\n+\n+\tcat >sparse_error_header <<-EOF &&\n+\tThe following pathspecs didn't match any eligible path, but they do match index\n+\tentries outside the current sparse checkout:\n+\tEOF\n+\n+\tcat >sparse_hint <<-EOF\n+\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n+\tEOF\n+\"\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse' '\n+\ttest_when_finished rm -f e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-sparse, ignores failure' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\tgit mv -k b e 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\tgit reset --hard &&\n+\ttouch b &&\n+\tgit mv -k b c 2>stderr &&\n+\ttest_path_exists b &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\n+\tgit mv --sparse b c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing b &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move non-sparse-to-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a &&\n+\n+\t# tracked-to-untracked\n+\ttest_must_fail git mv a e 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing e &&\n+\tcat sparse_error_header >expect &&\n+\techo e >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a e 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists e &&\n+\n+\t# tracked-to-tracked\n+\trm e &&\n+\tgit reset --hard &&\n+\ttest_must_fail git mv a c 2>stderr &&\n+\ttest_path_exists a &&\n+\ttest_path_is_missing c &&\n+\tcat sparse_error_header >expect &&\n+\techo c >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse a c 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\ttest_path_is_missing a &&\n+\ttest_path_exists c\n+'\n+\n+test_expect_success 'mv refuses to move sparse-to-non-sparse' '\n+\ttest_when_finished rm -f b c e &&\n+\tgit reset --hard &&\n+\tgit sparse-checkout set a e &&\n+\n+\t# tracked-to-untracked\n+\ttouch b &&\n+\ttest_must_fail git mv b e 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\techo b >>expect &&\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse b e 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'recursive mv refuses to move (possible) sparse' '\n+\ttest_when_finished rm -rf b c e sub2 &&\n+\tgit reset --hard &&\n+\t# Without cone mode, \"sub\" and \"sub2\" do not match\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/d sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/d\n+\tsub2/d\n+\tsub/dir/e\n+\tsub2/dir/e\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_expect_success 'recursive mv refuses to move sparse' '\n+\tgit reset --hard &&\n+\t# Use cone mode so \"sub/\" matches the sparse-checkout patterns\n+\tgit sparse-checkout init --cone &&\n+\tgit sparse-checkout set sub/dir sub2/dir &&\n+\n+\t# Add contained contents to ensure we avoid non-existence errors\n+\tmkdir sub/dir2 &&\n+\ttouch sub/dir2/e &&\n+\n+\ttest_must_fail git mv sub sub2 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tsub/dir2/e\n+\tsub2/dir2/e\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr &&\n+\tgit mv --sparse sub sub2 2>stderr &&\n+\ttest_must_be_empty stderr &&\n+\tgit commit -m \"moved sub to sub2\" &&\n+\tgit rev-parse HEAD~1:sub >expect &&\n+\tgit rev-parse HEAD:sub2 >actual &&\n+\ttest_cmp expect actual &&\n+\tgit reset --hard HEAD~1\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"436966","messageId":"9fbc88ee0dab830bdc34575e9808b8f2a04a59e9.1632497954.git.gitgitgadget@gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"[PATCH v4 13/13] advice: update message to suggest '--sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-09-24T15:39:14Z","receivedAt":"2021-09-24T15:39:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous changes modified the behavior of 'git add', 'git rm', and\n'git mv' to not adjust paths outside the sparse-checkout cone, even if\nthey exist in the working tree and their cache entries lack the\nSKIP_WORKTREE bit. The intention is to warn users that they are doing\nsomething potentially dangerous. The '--sparse' option was added to each\ncommand to allow careful users the same ability they had before.\n\nTo improve the discoverability of this new functionality, add a message\nto advice.updateSparsePath that mentions the existence of the option.\n\nThe previous set of changes also modified the purpose of this message to\ninclude possibly a list of paths instead of only a list of pathspecs.\nMake the warning message more clear about this new behavior.\n\nSigned-off-by: Derrick Stolee \n---\n advice.c | 11 ++++++-----\n t/t3602-rm-sparse-checkout.sh | 9 ++++++---\n t/t3705-add-sparse-checkout.sh | 9 ++++++---\n t/t7002-mv-sparse-checkout.sh | 9 ++++++---\n 4 files changed, 24 insertions(+), 14 deletions(-)\n\ndiff --git a/advice.c b/advice.c\nindex 0b9c89c48ab..713fff49ee3 100644\n--- a/advice.c\n+++ b/advice.c\n@@ -293,15 +293,16 @@ void advise_on_updating_sparse_paths(struct string_list *pathspec_list)\n \tif (!pathspec_list->nr)\n \t\treturn;\n \n-\tfprintf(stderr, _(\"The following pathspecs didn't match any\"\n-\t\t\t \" eligible path, but they do match index\\n\"\n-\t\t\t \"entries outside the current sparse checkout:\\n\"));\n+\tfprintf(stderr, _(\"The following paths and/or pathspecs matched paths that exist\\n\"\n+\t\t\t \"outside of your sparse-checkout definition, so will not be\\n\"\n+\t\t\t \"updated in the index:\\n\"));\n \tfor_each_string_list_item(item, pathspec_list)\n \t\tfprintf(stderr, \"%s\\n\", item->string);\n \n \tadvise_if_enabled(ADVICE_UPDATE_SPARSE_PATH,\n-\t\t\t _(\"Disable or modify the sparsity rules if you intend\"\n-\t\t\t \" to update such entries.\"));\n+\t\t\t _(\"If you intend to update such entries, try one of the following:\\n\"\n+\t\t\t \"* Use the --sparse option.\\n\"\n+\t\t\t \"* Disable or modify the sparsity rules.\"));\n }\n \n void detach_advice(const char *new_name)\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex 5f92b60a56a..ecce497a9ca 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -11,12 +11,15 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 339ec0ed2d6..5b904988d49 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -44,12 +44,15 @@ test_sparse_entry_unstaged () {\n \n test_expect_success 'setup' \"\n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF &&\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 07dbfeb6d17..545748949aa 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -11,12 +11,15 @@ test_expect_success 'setup' \"\n \tgit commit -m files &&\n \n \tcat >sparse_error_header <<-EOF &&\n-\tThe following pathspecs didn't match any eligible path, but they do match index\n-\tentries outside the current sparse checkout:\n+\tThe following paths and/or pathspecs matched paths that exist\n+\toutside of your sparse-checkout definition, so will not be\n+\tupdated in the index:\n \tEOF\n \n \tcat >sparse_hint <<-EOF\n-\thint: Disable or modify the sparsity rules if you intend to update such entries.\n+\thint: If you intend to update such entries, try one of the following:\n+\thint: * Use the --sparse option.\n+\thint: * Disable or modify the sparsity rules.\n \thint: Disable this message with \\\"git config advice.updateSparsePath false\\\"\n \tEOF\n \"\n-- \ngitgitgadget\n"},{"id":"437149","messageId":"CABPp-BEks6ntKbVAK2_eJCYXejSArFKEgD5v28W2K5RAHCjp0A@mail.gmail.com","threadId":"56370","inReplyTo":"pull.1018.v4.git.1632497954.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/13] Sparse-checkout: modify 'git add', 'git rm', and 'git mv' behavior","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-27T15:51:22Z","receivedAt":"2021-09-27T15:51:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Sep 24, 2021 at 8:39 AM Derrick Stolee via GitGitGadget\n wrote:\n...\n> Updates in v4\n> =============\n>\n> * Instead of using 'git status' and 'grep' to detect staged changes, we use\n> 'git diff --staged'. t1092 uses an additional --diff-filter because it\n> tests with merge conflicts, so it needs this extra flag.\n>\n> * Patches 3 and 4 are merged into the new patch 3 to avoid temporarily\n> having a poorly named method.\n>\n>\n...\n>\n> Range-diff vs v3:\n>\n> 1: ea940f10a7c ! 1: 642b05fc020 t3705: test that 'sparse_entry' is unstaged\n> @@ t/t3705-add-sparse-checkout.sh: setup_gitignore () {\n> }\n>\n> +test_sparse_entry_unstaged () {\n> -+ git status --porcelain >actual &&\n> -+ ! grep \"^[MDARCU][M ] sparse_entry\\$\" actual\n> ++ git diff --staged -- sparse_entry >diff &&\n> ++ test_must_be_empty diff\n> +}\n> +\n> test_expect_success 'setup' \"\n> 2: c7dedb41291 ! 2: 58389edc76c t1092: behavior for adding sparse files\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n> + file=$1 &&\n> + for repo in sparse-checkout sparse-index\n> + do\n> -+ git -C $repo status --porcelain >$repo-out &&\n> -+ ! grep \"^A $file\\$\" $repo-out &&\n> -+ ! grep \"^M $file\\$\" $repo-out || return 1\n> ++ # Skip \"unmerged\" paths\n> ++ git -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n\nWouldn't this be more naturally spelled as --diff-filter=u ? (Note:\nlowercase 'u', not uppercase.) Then you could drop the comment too.\n\nOther than that nit, this round looks good to me. Feel free to add a\n\nReviewed-by: Elijah Newren \n"},{"id":"437189","messageId":"xmqq35pppwsm.fsf@gitster.g","threadId":"56370","inReplyTo":"CABPp-BEks6ntKbVAK2_eJCYXejSArFKEgD5v28W2K5RAHCjp0A@mail.gmail.com","subject":"Re: [PATCH v4 00/13] Sparse-checkout: modify 'git add', 'git rm', and 'git mv' behavior","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-27T20:51:05Z","receivedAt":"2021-09-27T20:51:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> 2: c7dedb41291 ! 2: 58389edc76c t1092: behavior for adding sparse files\n>> @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_match () {\n>> + file=$1 &&\n>> + for repo in sparse-checkout sparse-index\n>> + do\n>> -+ git -C $repo status --porcelain >$repo-out &&\n>> -+ ! grep \"^A $file\\$\" $repo-out &&\n>> -+ ! grep \"^M $file\\$\" $repo-out || return 1\n>> ++ # Skip \"unmerged\" paths\n>> ++ git -C $repo diff --staged --diff-filter=ACDMRTXB -- \"$file\" >diff &&\n>\n> Wouldn't this be more naturally spelled as --diff-filter=u ? (Note:\n> lowercase 'u', not uppercase.) Then you could drop the comment too.\n\nExcellent.\n\n>\n> Other than that nit, this round looks good to me. Feel free to add a\n>\n> Reviewed-by: Elijah Newren \n"},{"id":"439007","messageId":"YW3nAKAUj/HF15OR@google.com","threadId":"56370","inReplyTo":"pull.1018.v2.git.1631453010.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Sean Christopherson","fromEmail":"seanjc@google.com","sentAt":"2021-10-18T21:28:32Z","receivedAt":"2021-10-18T21:28:39Z","isPatch":true,"sender":{"key":"seanjc@google.com","avatar":null},"body":"On Sun, Sep 12, 2021, Derrick Stolee via GitGitGadget wrote:\n> This series is based on ds/mergies-with-sparse-index.\n> \n> As requested, this series looks to update the behavior of git add, git rm,\n> and git mv when they attempt to modify paths outside of the sparse-checkout\n> cone. In particular, this care is expanded to not just cache entries with\n> the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\n> definition.\n\nI suspect something in this series broke 'git add' and friends with \"odd\" sparse\ndefinitions (I haven't actually bisected). git 2.33.0 rejects attempts to add\nfiles with the below sparse-checkout and modified files. There appears to be a\ndiscrepancy in the query vs. checkout logic as the rejected files are checked out\nin the working tree, e.g. git sees that the local file was deleted, yet will not\nstage the deletion.\n\nThere's also arguably a flaw in the \"advise\" trigger. AFAICT, the help message\nis displayed if and only if the entire path is excluded from the working tree.\nIn my perfect world, git would complain and advise if there are unstaged changes\nfor tracked files covered by the specified path.\n\nNote, my sparse-checkout is very much the result of trial and error to get the\nexact files I care about. It's entirely possible I'm doing something weird, but\nat the same time git itself is obviously confused.\n\nThanks!\n\n$ cat .git/info/sparse-checkout\n!arch/*\n!tools/arch/*\n!virt/kvm/arm/*\n/*\narch/.gitignore\narch/Kconfig\narch/x86\ntools/arch/x86\ntools/include/uapi/linux/kvm.h\n!Documentation\n!drivers\n\n$ git read-tree -mu HEAD\n\n$ rm arch/x86/kvm/x86.c\n\n$ git commit -a\nOn branch x86/kvm_find_cpuid_entry_index\nYour branch is up to date with 'kvm/queue'.\n\nYou are in a sparse checkout with 40% of tracked files present.\n\nChanges not staged for commit:\n (use \"git add/rm ...\" to update what will be committed)\n (use \"git restore ...\" to discard changes in working directory)\n\tdeleted: arch/x86/kvm/x86.c\n\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\n$ git add arch\n\n$ git add .\n\n$ git add arch/x86\nThe following paths and/or pathspecs matched paths that exist\noutside of your sparse-checkout definition, so will not be\nupdated in the index:\narch/x86\nhint: If you intend to update such entries, try one of the following:\nhint: * Use the --sparse option.\nhint: * Disable or modify the sparsity rules.\nhint: Disable this message with \"git config advice.updateSparsePath false\"\n"},{"id":"439032","messageId":"00188c99-386f-8d4f-08f6-11a49d31184c@gmail.com","threadId":"56370","inReplyTo":"YW3nAKAUj/HF15OR@google.com","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-10-19T12:29:58Z","receivedAt":"2021-10-19T12:30:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/18/2021 5:28 PM, Sean Christopherson wrote:\n> On Sun, Sep 12, 2021, Derrick Stolee via GitGitGadget wrote:\n>> This series is based on ds/mergies-with-sparse-index.\n>>\n>> As requested, this series looks to update the behavior of git add, git rm,\n>> and git mv when they attempt to modify paths outside of the sparse-checkout\n>> cone. In particular, this care is expanded to not just cache entries with\n>> the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\n>> definition.\n> \n> I suspect something in this series broke 'git add' and friends with \"odd\" sparse\n> definitions (I haven't actually bisected). git 2.33.0 rejects attempts to add\n> files with the below sparse-checkout and modified files. There appears to be a\n> discrepancy in the query vs. checkout logic as the rejected files are checked out\n> in the working tree, e.g. git sees that the local file was deleted, yet will not\n> stage the deletion.\n\nAre you using v2.33.0? This change is not in that version.\n\nHowever, mt/add-rm-in-sparse-checkout [1] was introduced in v2.33.0 and\nintroduced these advice suggestions.\n\n[1] https://github.com/git/git/compare/a5828ae6b52137b913b978e16cd2334482eb4c1f...d5f4b8260f623d6fdef36d5eaa8a0c2350390472\n\nThe series you are commenting on goes even farther in restricting adds to\nbe within the sparse-checkout definitions, even for unstaged files or files\nthat removed the skip-worktree bit due to a merge conflict. It also creates\nan override '--sparse' option that allows you to ignore these protections.\n\n> There's also arguably a flaw in the \"advise\" trigger. AFAICT, the help message\n> is displayed if and only if the entire path is excluded from the working tree.\n> In my perfect world, git would complain and advise if there are unstaged changes\n> for tracked files covered by the specified path.\n>> Note, my sparse-checkout is very much the result of trial and error to get the\n> exact files I care about. It's entirely possible I'm doing something weird, but\n> at the same time git itself is obviously confused.\n> \n> Thanks!\n> \n> $ cat .git/info/sparse-checkout\n> !arch/*\n> !tools/arch/*\n> !virt/kvm/arm/*\n> /*\n> arch/.gitignore\n> arch/Kconfig\n> arch/x86\n> tools/arch/x86\n> tools/include/uapi/linux/kvm.h\n> !Documentation\n> !drivers\n\nHave you tried using 'arch/x86/' and 'tools/arch/x86/' to specify\nthat these are directories? Just a thought.\n\n> $ git read-tree -mu HEAD\n> \n> $ rm arch/x86/kvm/x86.c\n> \n> $ git commit -a\n...\n> \tdeleted: arch/x86/kvm/x86.c\n\nThis is certainly odd. Worth more investigation that I don't have\ntime for at this moment.\n\nThanks,\n-Stolee\n"},{"id":"439035","messageId":"YW73QD7a+NzHd8dq@google.com","threadId":"56370","inReplyTo":"00188c99-386f-8d4f-08f6-11a49d31184c@gmail.com","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Sean Christopherson","fromEmail":"seanjc@google.com","sentAt":"2021-10-19T16:50:08Z","receivedAt":"2021-10-19T16:50:16Z","isPatch":true,"sender":{"key":"seanjc@google.com","avatar":null},"body":"On Tue, Oct 19, 2021, Derrick Stolee wrote:\n> On 10/18/2021 5:28 PM, Sean Christopherson wrote:\n> > On Sun, Sep 12, 2021, Derrick Stolee via GitGitGadget wrote:\n> >> This series is based on ds/mergies-with-sparse-index.\n> >>\n> >> As requested, this series looks to update the behavior of git add, git rm,\n> >> and git mv when they attempt to modify paths outside of the sparse-checkout\n> >> cone. In particular, this care is expanded to not just cache entries with\n> >> the SKIP_WORKTREE bit, but also paths that do not match the sparse-checkout\n> >> definition.\n> > \n> > I suspect something in this series broke 'git add' and friends with \"odd\" sparse\n> > definitions (I haven't actually bisected). git 2.33.0 rejects attempts to add\n> > files with the below sparse-checkout and modified files. There appears to be a\n> > discrepancy in the query vs. checkout logic as the rejected files are checked out\n> > in the working tree, e.g. git sees that the local file was deleted, yet will not\n> > stage the deletion.\n> \n> Are you using v2.33.0? This change is not in that version.\n\nHrm, it's an internal build that says v2.33.0 is the bsae, but the --sparse option\nis available so who knows what's actually underneath the hood. I can try vanilla\nupstream builds if that would help narrow down the issue.\n\n> However, mt/add-rm-in-sparse-checkout [1] was introduced in v2.33.0 and\n> introduced these advice suggestions.\n> \n> [1] https://github.com/git/git/compare/a5828ae6b52137b913b978e16cd2334482eb4c1f...d5f4b8260f623d6fdef36d5eaa8a0c2350390472\n>\n> The series you are commenting on goes even farther in restricting adds to\n> be within the sparse-checkout definitions, even for unstaged files or files\n> that removed the skip-worktree bit due to a merge conflict. It also creates\n> an override '--sparse' option that allows you to ignore these protections.\n> \n> > There's also arguably a flaw in the \"advise\" trigger. AFAICT, the help message\n> > is displayed if and only if the entire path is excluded from the working tree.\n> > In my perfect world, git would complain and advise if there are unstaged changes\n> > for tracked files covered by the specified path.\n> >> Note, my sparse-checkout is very much the result of trial and error to get the\n> > exact files I care about. It's entirely possible I'm doing something weird, but\n> > at the same time git itself is obviously confused.\n> > \n> > Thanks!\n> > \n> > $ cat .git/info/sparse-checkout\n> > !arch/*\n> > !tools/arch/*\n> > !virt/kvm/arm/*\n> > /*\n> > arch/.gitignore\n> > arch/Kconfig\n> > arch/x86\n> > tools/arch/x86\n> > tools/include/uapi/linux/kvm.h\n> > !Documentation\n> > !drivers\n> \n> Have you tried using 'arch/x86/' and 'tools/arch/x86/' to specify\n> that these are directories? Just a thought.\n\nNice! That workaround resolves the issue. I vaguely recall intentionally omitting\nthe trailing slash, but adding it back doesn't seem to have any unwanted side effects\non the current git versions I'm using.\n\n> > $ git read-tree -mu HEAD\n> > \n> > $ rm arch/x86/kvm/x86.c\n> > \n> > $ git commit -a\n> ...\n> > \tdeleted: arch/x86/kvm/x86.c\n> \n> This is certainly odd. Worth more investigation that I don't have\n> time for at this moment.\n\nI've no objection to punting on this now that I have a workaround. The man pages\nare quite clear that sparse checkouts are still experimental and it's no trouble\nfor me to whine again if something breaks in the future :-)\n\nThanks again!\n"},{"id":"439102","messageId":"xmqqa6j3su57.fsf@gitster.g","threadId":"56370","inReplyTo":"YW73QD7a+NzHd8dq@google.com","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-20T13:28:04Z","receivedAt":"2021-10-20T13:28:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sean Christopherson writes:\n\n>> Are you using v2.33.0? This change is not in that version.\n>\n> Hrm, it's an internal build that says v2.33.0 is the bsae, but the --sparse option\n> is available so who knows what's actually underneath the hood. I can try vanilla\n> upstream builds if that would help narrow down the issue.\n\n$ git version\n\nGuessing from the e-mail address, perhaps you are using something\nderived from the next branch of the day, maintained by Jonathan\nNieder's group, for internal consumption at Google.\n\n"},{"id":"439107","messageId":"YXAnnHfH8sv2ZJqF@google.com","threadId":"56370","inReplyTo":"xmqqa6j3su57.fsf@gitster.g","subject":"Re: [PATCH v2 00/14] Sparse-checkout: modify 'git add', 'git rm', and 'git add' behavior","fromName":"Sean Christopherson","fromEmail":"seanjc@google.com","sentAt":"2021-10-20T14:28:44Z","receivedAt":"2021-10-20T14:29:19Z","isPatch":true,"sender":{"key":"seanjc@google.com","avatar":null},"body":"On Wed, Oct 20, 2021, Junio C Hamano wrote:\n> Sean Christopherson writes:\n> \n> >> Are you using v2.33.0? This change is not in that version.\n> >\n> > Hrm, it's an internal build that says v2.33.0 is the bsae, but the --sparse option\n> > is available so who knows what's actually underneath the hood. I can try vanilla\n> > upstream builds if that would help narrow down the issue.\n> \n> $ git version\n> \n> Guessing from the e-mail address, perhaps you are using something\n> derived from the next branch of the day, maintained by Jonathan\n> Nieder's group, for internal consumption at Google.\n\nThat's more than likely the case. 2.33.0.1079.g6e70778dc9-goog\n"},{"id":"439323","messageId":"80b5ba61861193daf7132aa64b65fc7dde90dacb.1634866698.git.matheus.bernardino@usp.br","threadId":"56370","inReplyTo":"YW3nAKAUj/HF15OR@google.com","subject":"[RFC PATCH] add|rm|mv: fix bug that prevent the update of non-sparse","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-10-22T02:28:08Z","receivedAt":"2021-10-22T02:28:20Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Mon, Oct 18, 2021 at 6:28 PM Sean Christopherson wrote:\n>\n> $ cat .git/info/sparse-checkout\n> !arch/*\n> !tools/arch/*\n> !virt/kvm/arm/*\n> /*\n> arch/.gitignore\n> arch/Kconfig\n> arch/x86\n> tools/arch/x86\n> tools/include/uapi/linux/kvm.h\n> !Documentation\n> !drivers\n>\n> $ git read-tree -mu HEAD\n>\n> $ rm arch/x86/kvm/x86.c\n[...]\n> $ git add arch/x86\n> The following paths and/or pathspecs matched paths that exist\n> outside of your sparse-checkout definition, so will not be\n> updated in the index:\n> arch/x86\n\nI think the problem may be that we are performing pattern matching\nslightly different in add, mv, and rm, in comparison to \"git\nsparse-checkout\". On \"git sparse-checkout init\" (or reapply), we call\nclear_ce_flags() which calls path_matches_pattern_list() for each\ncomponent of the working tree paths. If the full path gives a match\nresult of UNDECIDED, we recursively try to use the match result from\nthe parent dir (or NOT_MATCHED if we reach the top with UNDECIDED).\n\nIn Sean's example, we get UNDECIDED for \"arch/x86/kvm/x86.c\", but\n\"arch/x86\" gives MATCHED, so we end up using that for the full path.\n\nHowever, in add|mv|rm we only call path_matches_pattern_list() for the\nfull path and get UNDECIDED, which we consider the same as NOT_MATCHED,\nand end up disallowing the path update operation with a warning message.\n\nThe commands do work if we replace the sparsity pattern \"arch/x86\" with\n\"arch/x86/\" (with a trailing slash), but note that it only works\nbecause the pattern is relative to the root (see dir.c:1297). If we\nchange it to \"x86/\", it would no longer work.\n\nSo far, the only way I could think of to fix this would be to perform\npattern matching for the leading components of the paths too. That\ndoesn't seem very nice, though, as it can probably be quite expensive...\nBut here is a patch for discussion:\n\n-- >8 --\nSubject: [RFC PATCH] add|rm|mv: fix bug that prevent the update of non-sparse dirs\n\nThese three commands recently learned to avoid updating paths that do\nnot match the sparse-checkout patterns even if they are missing the\nSKIP_WORKTREE bit. This is done using path_in_sparse_checkout(), which\ntries to match the path with the current set of sparsity rules using\npath_matches_pattern_list(). This is similar to what clear_ce_flags()\ndoes when we run \"git sparse-checkout init\" or \"git sparse-checkout\nreapply\". But note that clear_ce_flags() has a recursive behavior,\ncalling path_matches_pattern_list() for each component in a path,\nwhereas path_in_sparse_checkout() only calls it for the full path. This\nmakes the function miss matches such as the one between path \"a/b/c\" and\nthe pattern \"b/\". So if the user has the sparsity rules \"!/a\" and \"b/\",\nfor example, add, rm, and mv will fail to update the path \"a/b/c\" and\nend up displaying a warning about \"a/b/c\" being outside the sparse\ncheckout even though it isn't. Note that this problem only occurs with\nnon-cone mode.\n\nFix this by making path_in_sparse_checkout() perform pattern matching\nfor every component in the given path when cone mode is disabled. (This\ncan be expensive, and we might want to do some form of caching for the\nmatch results of the leading components. However, this is not\nimplemented in this patch.) Also add two tests for each command (add,\nrm, and mv) to check that they behave correctly with the said pattern\nmatching. The first test would previously fail without this patch, while\nthe second already succeeded. It is added mostly to make sure that we\nare not breaking the existing pattern matching for directories that are\nreally sparse, and also as a protection against any future\nregressions.\n\nNote that two other existing tests had to be changed: one test in t3602\nchecks that \"git rm -r \" won't remove sparse entries, but it\ndidn't allow the non-sparse entries inside to be removed. The\nother one, in t7002, tested that \"git mv\" would correctly display a\nwarning message for sparse paths, but it accidentally expected the\nmessage to include two non-sparse paths as well.\n\nSigned-off-by: Matheus Tavares \n---\n dir.c | 33 ++++++++++++++++++++++++------\n t/t3602-rm-sparse-checkout.sh | 37 +++++++++++++++++++++++++++++++---\n t/t3705-add-sparse-checkout.sh | 18 +++++++++++++++++\n t/t7002-mv-sparse-checkout.sh | 28 +++++++++++++++++++++++--\n 4 files changed, 105 insertions(+), 11 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex a4306ab874..225487a59c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1504,8 +1504,9 @@ static int path_in_sparse_checkout_1(const char *path,\n \t\t\t\t struct index_state *istate,\n \t\t\t\t int require_cone_mode)\n {\n-\tconst char *base;\n \tint dtype = DT_REG;\n+\tenum pattern_match_result ret = NOT_MATCHED;\n+\tconst char *p, *last_slash = NULL;\n \n \t/*\n \t * We default to accepting a path if there are no patterns or\n@@ -1516,11 +1517,31 @@ static int path_in_sparse_checkout_1(const char *path,\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n \t\treturn 1;\n \n-\tbase = strrchr(path, '/');\n-\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n-\t\t\t\t\t &dtype,\n-\t\t\t\t\t istate->sparse_checkout_patterns,\n-\t\t\t\t\t istate) > 0;\n+\tif (istate->sparse_checkout_patterns->use_cone_patterns) {\n+\t\tconst char *base = strrchr(path, '/');\n+\t\treturn path_matches_pattern_list(path, strlen(path),\n+\t\t\t\tbase ? base + 1 : path, &dtype,\n+\t\t\t\tistate->sparse_checkout_patterns, istate) > 0;\n+\t}\n+\n+\tfor (p = path; ; p++) {\n+\t\tenum pattern_match_result match;\n+\n+\t\tif (*p && *p != '/')\n+\t\t\tcontinue;\n+\n+\t\tmatch = path_matches_pattern_list(path, p - path,\n+\t\t\t\tlast_slash ? last_slash + 1 : path, &dtype,\n+\t\t\t\tistate->sparse_checkout_patterns, istate);\n+\n+\t\tif (match != UNDECIDED)\n+\t\t\tret = match;\n+\t\tif (!*p)\n+\t\t\tbreak;\n+\t\tlast_slash = p;\n+\t}\n+\n+\treturn ret;\n }\n \n int path_in_sparse_checkout(const char *path,\ndiff --git a/t/t3602-rm-sparse-checkout.sh b/t/t3602-rm-sparse-checkout.sh\nindex ecce497a9c..6e127b966e 100755\n--- a/t/t3602-rm-sparse-checkout.sh\n+++ b/t/t3602-rm-sparse-checkout.sh\n@@ -40,14 +40,20 @@ done\n test_expect_success 'recursive rm does not remove sparse entries' '\n \tgit reset --hard &&\n \tgit sparse-checkout set sub/dir &&\n-\ttest_must_fail git rm -r sub &&\n-\tgit rm --sparse -r sub &&\n+\tgit rm -r sub &&\n \tgit status --porcelain -uno >actual &&\n \tcat >expected <<-\\EOF &&\n+\tD sub/dir/e\n+\tEOF\n+\ttest_cmp expected actual &&\n+\n+\tgit rm --sparse -r sub &&\n+\tgit status --porcelain -uno >actual2 &&\n+\tcat >expected2 <<-\\EOF &&\n \tD sub/d\n \tD sub/dir/e\n \tEOF\n-\ttest_cmp expected actual\n+\ttest_cmp expected2 actual2\n '\n \n test_expect_success 'recursive rm --sparse removes sparse entries' '\n@@ -105,4 +111,29 @@ test_expect_success 'refuse to rm a non-skip-worktree path outside sparse cone'\n \ttest_path_is_missing b\n '\n \n+test_expect_success 'can remove files from non-sparse dir' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout disable &&\n+\tmkdir -p w x/y &&\n+\ttest_commit w/f &&\n+\ttest_commit x/y/f &&\n+\n+\tgit sparse-checkout set w !/x y &&\n+\tgit rm w/f.t x/y/f.t 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'refuse to remove non-skip-worktree file from sparse dir' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout disable &&\n+\tmkdir -p x/y/z &&\n+\ttest_commit x/y/z/f &&\n+\tgit sparse-checkout set !/x y !x/y/z &&\n+\n+\tgit update-index --no-skip-worktree x/y/z/f.t &&\n+\ttest_must_fail git rm x/y/z/f.t 2>stderr &&\n+\techo x/y/z/f.t | cat sparse_error_header - sparse_hint >expect &&\n+\ttest_cmp expect stderr\n+'\n+\n test_done\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex 5b904988d4..63f888af12 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -214,4 +214,22 @@ test_expect_success 'add allows sparse entries with --sparse' '\n \ttest_must_be_empty stderr\n '\n \n+test_expect_success 'can add files from non-sparse dir' '\n+\tgit sparse-checkout set w !/x y &&\n+\tmkdir -p w x/y &&\n+\ttouch w/f x/y/f &&\n+\tgit add w/f x/y/f 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'refuse to add non-skip-worktree file from sparse dir' '\n+\tgit sparse-checkout set !/x y !x/y/z &&\n+\tmkdir -p x/y/z &&\n+\ttouch x/y/z/f &&\n+\ttest_must_fail git add x/y/z/f 2>stderr &&\n+\techo x/y/z/f | cat sparse_error_header - sparse_hint >expect &&\n+\ttest_cmp expect stderr\n+\n+'\n+\n test_done\ndiff --git a/t/t7002-mv-sparse-checkout.sh b/t/t7002-mv-sparse-checkout.sh\nindex 545748949a..91a857bf05 100755\n--- a/t/t7002-mv-sparse-checkout.sh\n+++ b/t/t7002-mv-sparse-checkout.sh\n@@ -143,8 +143,6 @@ test_expect_success 'recursive mv refuses to move (possible) sparse' '\n \tcat >>expect <<-\\EOF &&\n \tsub/d\n \tsub2/d\n-\tsub/dir/e\n-\tsub2/dir/e\n \tsub/dir2/e\n \tsub2/dir2/e\n \tEOF\n@@ -186,4 +184,30 @@ test_expect_success 'recursive mv refuses to move sparse' '\n \tgit reset --hard HEAD~1\n '\n \n+test_expect_success 'can move files to non-sparse dir' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout init --no-cone &&\n+\tgit sparse-checkout set a b c w !/x y &&\n+\tmkdir -p w x/y &&\n+\n+\tgit mv a w/new-a 2>stderr &&\n+\tgit mv b x/y/new-b 2>stderr &&\n+\ttest_must_be_empty stderr\n+'\n+\n+test_expect_success 'refuse to move file to non-skip-worktree sparse path' '\n+\tgit reset --hard &&\n+\tgit sparse-checkout init --no-cone &&\n+\tgit sparse-checkout set a !/x y !x/y/z &&\n+\tmkdir -p x/y/z &&\n+\n+\ttest_must_fail git mv a x/y/z/new-a 2>stderr &&\n+\tcat sparse_error_header >expect &&\n+\tcat >>expect <<-\\EOF &&\n+\tx/y/z/new-a\n+\tEOF\n+\tcat sparse_hint >>expect &&\n+\ttest_cmp expect stderr\n+'\n+\n test_done\n-- \n2.33.0\n\n"},{"id":"439335","messageId":"20211022040352.16773-1-matheus.bernardino@usp.br","threadId":"56370","inReplyTo":"80b5ba61861193daf7132aa64b65fc7dde90dacb.1634866698.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH] add|rm|mv: fix bug that prevent the update of non-sparse","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-10-22T04:03:52Z","receivedAt":"2021-10-22T04:04:07Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Thu, Oct 21, 2021 at 11:28 PM Matheus Tavares wrote:\n>\n> diff --git a/dir.c b/dir.c\n> index a4306ab874..225487a59c 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1516,11 +1517,31 @@ static int path_in_sparse_checkout_1(const char *path,\n> \t !istate->sparse_checkout_patterns->use_cone_patterns))\n> \t\treturn 1;\n> \n> -\tbase = strrchr(path, '/');\n> -\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n> -\t\t\t\t\t &dtype,\n> -\t\t\t\t\t istate->sparse_checkout_patterns,\n> -\t\t\t\t\t istate) > 0;\n> +\tif (istate->sparse_checkout_patterns->use_cone_patterns) {\n> +\t\tconst char *base = strrchr(path, '/');\n> +\t\treturn path_matches_pattern_list(path, strlen(path),\n> +\t\t\t\tbase ? base + 1 : path, &dtype,\n> +\t\t\t\tistate->sparse_checkout_patterns, istate) > 0;\n> +\t}\n> +\n> +\tfor (p = path; ; p++) {\n> +\t\tenum pattern_match_result match;\n> +\n> +\t\tif (*p && *p != '/')\n> +\t\t\tcontinue;\n> +\n> +\t\tmatch = path_matches_pattern_list(path, p - path,\n> +\t\t\t\tlast_slash ? last_slash + 1 : path, &dtype,\n> +\t\t\t\tistate->sparse_checkout_patterns, istate);\n> +\n> +\t\tif (match != UNDECIDED)\n> +\t\t\tret = match;\n> +\t\tif (!*p)\n> +\t\t\tbreak;\n> +\t\tlast_slash = p;\n> +\t}\n> +\n> +\treturn ret;\n> }\n\nOf course, after hitting send I realized it would make a lot more sense\nto start the pattern matching from the full path and only go backwards\nthrough the parent dirs until we find the first non-UNDECIDED result.\nI.e. something like this:\n\nstatic int path_in_sparse_checkout_1(const char *path,\n\t\t\t\t struct index_state *istate,\n\t\t\t\t int require_cone_mode)\n{\n\tint dtype = DT_REG;\n\tenum pattern_match_result ret;\n\tconst char *p, *base;\n\n\t/*\n\t * We default to accepting a path if there are no patterns or\n\t * they are of the wrong type.\n\t */\n\tif (init_sparse_checkout_patterns(istate) ||\n\t (require_cone_mode &&\n\t !istate->sparse_checkout_patterns->use_cone_patterns))\n\t\treturn 1;\n\n\tif (istate->sparse_checkout_patterns->use_cone_patterns) {\n\t\tbase = strrchr(path, '/');\n\t\treturn path_matches_pattern_list(path, strlen(path),\n\t\t\t\tbase ? base + 1 : path, &dtype,\n\t\t\t\tistate->sparse_checkout_patterns, istate) > 0;\n\t}\n\n\t/*\n\t * If the match for the path is UNDECIDED, try to match the parent dir\n\t * recursively.\n\t */\n\tfor (p = path + strlen(path); p && p > path; p = base) {\n\t\tbase = memrchr(path, '/', p - path);\n\t\tret = path_matches_pattern_list(path, p - path,\n\t\t\t\tbase ? base + 1 : path, &dtype,\n\t\t\t\tistate->sparse_checkout_patterns, istate);\n\n\t\tif (ret != UNDECIDED)\n\t\t\tbreak;\n\t}\n\n\treturn ret == UNDECIDED ? NOT_MATCHED : ret;\n}\n\nBut I will let others comment on the overall idea and/or other\nalternatives before sending a possible v2.\n"},{"id":"439559","messageId":"a11d4fb0-184a-b811-64a0-e29e9cfd91a3@gmail.com","threadId":"56370","inReplyTo":"80b5ba61861193daf7132aa64b65fc7dde90dacb.1634866698.git.matheus.bernardino@usp.br","subject":"Re: [RFC PATCH] add|rm|mv: fix bug that prevent the update of non-sparse","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-10-25T16:40:17Z","receivedAt":"2021-10-25T16:40:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/21/2021 10:28 PM, Matheus Tavares wrote:\n> On Mon, Oct 18, 2021 at 6:28 PM Sean Christopherson wrote:\n>>\n>> $ cat .git/info/sparse-checkout\n>> !arch/*\n>> !tools/arch/*\n>> !virt/kvm/arm/*\n>> /*\n>> arch/.gitignore\n>> arch/Kconfig\n>> arch/x86\n>> tools/arch/x86\n>> tools/include/uapi/linux/kvm.h\n>> !Documentation\n>> !drivers\n>>\n>> $ git read-tree -mu HEAD\n>>\n>> $ rm arch/x86/kvm/x86.c\n> [...]\n>> $ git add arch/x86\n>> The following paths and/or pathspecs matched paths that exist\n>> outside of your sparse-checkout definition, so will not be\n>> updated in the index:\n>> arch/x86\n> \n> I think the problem may be that we are performing pattern matching\n> slightly different in add, mv, and rm, in comparison to \"git\n> sparse-checkout\". On \"git sparse-checkout init\" (or reapply), we call\n> clear_ce_flags() which calls path_matches_pattern_list() for each\n> component of the working tree paths. If the full path gives a match\n> result of UNDECIDED, we recursively try to use the match result from\n> the parent dir (or NOT_MATCHED if we reach the top with UNDECIDED).\n\nYes! I think this is absolutely the problem. Thanks for pointing\nthis out!\n \n> In Sean's example, we get UNDECIDED for \"arch/x86/kvm/x86.c\", but\n> \"arch/x86\" gives MATCHED, so we end up using that for the full path.\n> \n> However, in add|mv|rm we only call path_matches_pattern_list() for the\n> full path and get UNDECIDED, which we consider the same as NOT_MATCHED,\n> and end up disallowing the path update operation with a warning message.\n> \n> The commands do work if we replace the sparsity pattern \"arch/x86\" with\n> \"arch/x86/\" (with a trailing slash), but note that it only works\n> because the pattern is relative to the root (see dir.c:1297). If we\n> change it to \"x86/\", it would no longer work.\n> \n> So far, the only way I could think of to fix this would be to perform\n> pattern matching for the leading components of the paths too. That\n> doesn't seem very nice, though, as it can probably be quite expensive...\n> But here is a patch for discussion:\n\nI agree that it is expensive, but that's already the case for the\nnon-cone sparse-checkout patterns. Hopefully it is sufficient that\nthese cases are restricted to modified files (in the case of `git add .`)\nor specific pathspecs (in the case of `git mv` and `git rm`).\n\n> -- >8 --\n> Subject: [RFC PATCH] add|rm|mv: fix bug that prevent the update of non-sparse dirs\n> \n> These three commands recently learned to avoid updating paths that do\n> not match the sparse-checkout patterns even if they are missing the\n> SKIP_WORKTREE bit. This is done using path_in_sparse_checkout(), which\n> tries to match the path with the current set of sparsity rules using\n> path_matches_pattern_list(). This is similar to what clear_ce_flags()\n> does when we run \"git sparse-checkout init\" or \"git sparse-checkout\n> reapply\". But note that clear_ce_flags() has a recursive behavior,\n> calling path_matches_pattern_list() for each component in a path,\n> whereas path_in_sparse_checkout() only calls it for the full path. This\n> makes the function miss matches such as the one between path \"a/b/c\" and\n> the pattern \"b/\". So if the user has the sparsity rules \"!/a\" and \"b/\",\n> for example, add, rm, and mv will fail to update the path \"a/b/c\" and\n> end up displaying a warning about \"a/b/c\" being outside the sparse\n> checkout even though it isn't. Note that this problem only occurs with\n> non-cone mode.\n> \n> Fix this by making path_in_sparse_checkout() perform pattern matching\n> for every component in the given path when cone mode is disabled. (This\n> can be expensive, and we might want to do some form of caching for the\n> match results of the leading components. However, this is not\n> implemented in this patch.) Also add two tests for each command (add,\n> rm, and mv) to check that they behave correctly with the said pattern\n> matching. The first test would previously fail without this patch, while\n> the second already succeeded. It is added mostly to make sure that we\n> are not breaking the existing pattern matching for directories that are\n> really sparse, and also as a protection against any future\n> regressions.\n> \n> Note that two other existing tests had to be changed: one test in t3602\n> checks that \"git rm -r \" won't remove sparse entries, but it\n> didn't allow the non-sparse entries inside to be removed. The\n> other one, in t7002, tested that \"git mv\" would correctly display a\n> warning message for sparse paths, but it accidentally expected the\n> message to include two non-sparse paths as well.\n\n\n> @@ -1504,8 +1504,9 @@ static int path_in_sparse_checkout_1(const char *path,\n> \t\t\t\t struct index_state *istate,\n> \t\t\t\t int require_cone_mode)\n> {\n> -\tconst char *base;\n> \tint dtype = DT_REG;\n> +\tenum pattern_match_result ret = NOT_MATCHED;\n> +\tconst char *p, *last_slash = NULL;\n> \n> \t/*\n> \t * We default to accepting a path if there are no patterns or\n> @@ -1516,11 +1517,31 @@ static int path_in_sparse_checkout_1(const char *path,\n> \t !istate->sparse_checkout_patterns->use_cone_patterns))\n> \t\treturn 1;\n> \n> -\tbase = strrchr(path, '/');\n> -\treturn path_matches_pattern_list(path, strlen(path), base ? base + 1 : path,\n> -\t\t\t\t\t &dtype,\n> -\t\t\t\t\t istate->sparse_checkout_patterns,\n> -\t\t\t\t\t istate) > 0;\n> +\tif (istate->sparse_checkout_patterns->use_cone_patterns) {\n> +\t\tconst char *base = strrchr(path, '/');\n> +\t\treturn path_matches_pattern_list(path, strlen(path),\n> +\t\t\t\tbase ? base + 1 : path, &dtype,\n> +\t\t\t\tistate->sparse_checkout_patterns, istate) > 0;\n> +\t}\n> +\n> +\tfor (p = path; ; p++) {\n> +\t\tenum pattern_match_result match;\n> +\n> +\t\tif (*p && *p != '/')\n> +\t\t\tcontinue;\n> +\n> +\t\tmatch = path_matches_pattern_list(path, p - path,\n> +\t\t\t\tlast_slash ? last_slash + 1 : path, &dtype,\n> +\t\t\t\tistate->sparse_checkout_patterns, istate);\n> +\n> +\t\tif (match != UNDECIDED)\n> +\t\t\tret = match;\n> +\t\tif (!*p)\n> +\t\t\tbreak;\n> +\t\tlast_slash = p;\n> +\t}\n> +\n> +\treturn ret;\n\nThis implementation makes sense to me.\n\n> test_expect_success 'recursive rm does not remove sparse entries' '\n> \tgit reset --hard &&\n> \tgit sparse-checkout set sub/dir &&\n> -\ttest_must_fail git rm -r sub &&\n> -\tgit rm --sparse -r sub &&\n> +\tgit rm -r sub &&\n\nInteresting that the new pattern-matching already presents a change of\nbehavior in this test case.\n\n> \tgit status --porcelain -uno >actual &&\n> \tcat >expected <<-\\EOF &&\n> +\tD sub/dir/e\n> +\tEOF\n> +\ttest_cmp expected actual &&\n\nAnd here is why. Excellent. I suppose that setting the pattern to be\n\"sub/dir/\" would have shown this behavior before.\n\n> +\n> +\tgit rm --sparse -r sub &&\n> +\tgit status --porcelain -uno >actual2 &&\n> +\tcat >expected2 <<-\\EOF &&\n> \tD sub/d\n> \tD sub/dir/e\n> \tEOF\n> -\ttest_cmp expected actual\n> +\ttest_cmp expected2 actual2\n> '\n\nThe rest of the test cases add new checks that are very valuable.\n\nI love this idea and I agree that it would be better to change the\nloop direction to match the full path first (as you mention in your\nresponse).\n\nThanks,\n-Stolee\n"},{"id":"440281","messageId":"kl6lzgqnv2dn.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"56370","inReplyTo":"24bffdab139435173712101aaf72f7277298c99d.1632497954.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2021-11-02T00:15:48Z","receivedAt":"2021-11-02T00:15:55Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nThis patch changes the behavior of .gitignore such that directories are\nnow matched by prefix instead of matching exactly.\n\nThe failure that we observed is something like the following:\n\nIn \"a/.gitignore\", we have the pattern \"git/\". We should expect that\n\"a/git/foo\" to be ignored because \"git/\" should be matched exactly.\nHowever, \"a/git-foo/bar\" is also ignored because \"git-foo\" matches the\nprefix.\n\nI'll prepare a test case for this as soon as I figure out how to write\nit..\n"},{"id":"440282","messageId":"xmqq7ddr9z0i.fsf@gitster.g","threadId":"56370","inReplyTo":"kl6lzgqnv2dn.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-02T00:34:05Z","receivedAt":"2021-11-02T00:34:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> This patch changes the behavior of .gitignore such that directories are\n> now matched by prefix instead of matching exactly.\n>\n> The failure that we observed is something like the following:\n>\n> In \"a/.gitignore\", we have the pattern \"git/\". We should expect that\n> \"a/git/foo\" to be ignored because \"git/\" should be matched exactly.\n> However, \"a/git-foo/bar\" is also ignored because \"git-foo\" matches the\n> prefix.\n>\n> I'll prepare a test case for this as soon as I figure out how to write\n> it..\n\nFWIW, reverting this commit (and nothing else) from 'master' does not\nseem to break any test. That does not necessarily mean much (it may\nbe indicating a gap in the test coverage).\n"},{"id":"440306","messageId":"2f1d2e09-faf5-0caf-6be8-7bcede9fe8d1@gmail.com","threadId":"56370","inReplyTo":"xmqq7ddr9z0i.fsf@gitster.g","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-02T13:42:52Z","receivedAt":"2021-11-02T13:42:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/2021 8:34 PM, Junio C Hamano wrote:\n> Glen Choo writes:\n> \n>> This patch changes the behavior of .gitignore such that directories are\n>> now matched by prefix instead of matching exactly.\n\nThank you for pointing out an unintended consequence.\n\n>> The failure that we observed is something like the following:\n>>\n>> In \"a/.gitignore\", we have the pattern \"git/\". We should expect that\n>> \"a/git/foo\" to be ignored because \"git/\" should be matched exactly.\n>> However, \"a/git-foo/bar\" is also ignored because \"git-foo\" matches the\n>> prefix.\n>>\n>> I'll prepare a test case for this as soon as I figure out how to write\n>> it..\n> \n> FWIW, reverting this commit (and nothing else) from 'master' does not\n> seem to break any test. That does not necessarily mean much (it may\n> be indicating a gap in the test coverage).\n\nHm. I definitely seemed confident in my commit message that this change\nwas important for a test I was introducing at some point. Let me revert\nit in microsoft/git and see if our Scalar functional tests find an\nissue.\n\nIn the meantime, I'll try to create a Git test that demonstrates a\nproblem one way or another.\n\nThanks,\n-Stolee\n"},{"id":"440316","messageId":"7a62be4e-aa69-8a79-4608-971b96ee4d7c@gmail.com","threadId":"56370","inReplyTo":"2f1d2e09-faf5-0caf-6be8-7bcede9fe8d1@gmail.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-02T14:50:16Z","receivedAt":"2021-11-02T14:50:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2021 9:42 AM, Derrick Stolee wrote:\n> On 11/1/2021 8:34 PM, Junio C Hamano wrote:\n>> Glen Choo writes:\n>>\n>>> This patch changes the behavior of .gitignore such that directories are\n>>> now matched by prefix instead of matching exactly.\n> \n> Thank you for pointing out an unintended consequence.\n> \n>>> The failure that we observed is something like the following:\n>>>\n>>> In \"a/.gitignore\", we have the pattern \"git/\". We should expect that\n>>> \"a/git/foo\" to be ignored because \"git/\" should be matched exactly.\n>>> However, \"a/git-foo/bar\" is also ignored because \"git-foo\" matches the\n>>> prefix.\n>>>\n>>> I'll prepare a test case for this as soon as I figure out how to write\n>>> it..\n...\n> In the meantime, I'll try to create a Git test that demonstrates a\n> problem one way or another.\n\nI created a test, but had some trouble reproducing it due to some\nsubtleties higher in the call stack. Here is a patch that reverts\nthe change and adds some tests.\n\nThe Scalar functional tests passed with the revert, so the original\npatch was worthless to begin with. I don't recall what motivated the\nchange, but clearly it was a mistake. Sorry.\n\n---- >8 ----\n\nFrom d1cfc8efeab015273bfebd6cd93465e6f38dc40f Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 2 Nov 2021 10:40:06 -0400\nSubject: [PATCH] dir: fix directory-matching bug\n\nThis reverts the change from ed49584 (dir: fix pattern matching on dirs,\n2021-09-24), which claimed to fix a directory-matching problem without a\ntest case. It turns out to _create_ a bug, but it is a bit subtle.\n\nThe bug would have been revealed by the first of two tests being added to\nt0008-ignores.sh. The first uses a pattern \"/git/\" inside the a/.gitignores\nfile, which matches against 'a/git/foo' but not 'a/git-foo/bar'. This test\nwould fail before the revert.\n\nThe second test shows what happens if the test instead uses a pattern \"git/\"\nand this test passes both before and after the revert.\n\nThe difference in these two cases are due to how\nlast_matching_pattern_from_list() checks patterns both if they have the\nPATTERN_FLAG_MUSTBEDIR and PATTERN_FLAG_NODIR flags. In the case of \"git/\",\nthe PATTERN_FLAG_NODIR is also provided, making the change in behavior in\nmatch_pathname() not affect the end result of\nlast_matching_pattern_from_list().\n\nReported-by: Glen Choo \nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 +-\n t/t0008-ignores.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex c6d7a8647b9..94489298f4c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n \t\t * then our prefix match is all we need; we\n \t\t * do not need to call fnmatch at all.\n \t\t */\n-\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n+\t\tif (!patternlen && !namelen)\n \t\t\treturn 1;\n \t}\n \ndiff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh\nindex 532637de882..1889cfc60e0 100755\n--- a/t/t0008-ignores.sh\n+++ b/t/t0008-ignores.sh\n@@ -803,6 +803,32 @@ test_expect_success 'existing directory and file' '\n \tgrep top-level-dir actual\n '\n \n+test_expect_success 'exact prefix matching (with root)' '\n+\ttest_when_finished rm -r a &&\n+\tmkdir -p a/git a/git-foo &&\n+\ttouch a/git/foo a/git-foo/bar &&\n+\techo /git/ >a/.gitignore &&\n+\tgit check-ignore a/git a/git/foo a/git-foo a/git-foo/bar >actual &&\n+\tcat >expect <<-\\EOF &&\n+\ta/git\n+\ta/git/foo\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'exact prefix matching (without root)' '\n+\ttest_when_finished rm -r a &&\n+\tmkdir -p a/git a/git-foo &&\n+\ttouch a/git/foo a/git-foo/bar &&\n+\techo git/ >a/.gitignore &&\n+\tgit check-ignore a/git a/git/foo a/git-foo a/git-foo/bar >actual &&\n+\tcat >expect <<-\\EOF &&\n+\ta/git\n+\ta/git/foo\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n ############################################################################\n #\n # test whitespace handling\n-- \n2.34.0.vfs.0.0.rc0.dirty\n\n"},{"id":"440333","messageId":"211102.86pmri1rv7.gmgdl@evledraar.gmail.com","threadId":"56370","inReplyTo":"7a62be4e-aa69-8a79-4608-971b96ee4d7c@gmail.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-02T15:33:04Z","receivedAt":"2021-11-02T15:48:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 02 2021, Derrick Stolee wrote:\n\n> On 11/2/2021 9:42 AM, Derrick Stolee wrote:\n>> On 11/1/2021 8:34 PM, Junio C Hamano wrote:\n>>> Glen Choo writes:\n>>>\n>>>> This patch changes the behavior of .gitignore such that directories are\n>>>> now matched by prefix instead of matching exactly.\n>> \n>> Thank you for pointing out an unintended consequence.\n>> \n>>>> The failure that we observed is something like the following:\n>>>>\n>>>> In \"a/.gitignore\", we have the pattern \"git/\". We should expect that\n>>>> \"a/git/foo\" to be ignored because \"git/\" should be matched exactly.\n>>>> However, \"a/git-foo/bar\" is also ignored because \"git-foo\" matches the\n>>>> prefix.\n>>>>\n>>>> I'll prepare a test case for this as soon as I figure out how to write\n>>>> it..\n> ...\n>> In the meantime, I'll try to create a Git test that demonstrates a\n>> problem one way or another.\n>\n> I created a test, but had some trouble reproducing it due to some\n> subtleties higher in the call stack. Here is a patch that reverts\n> the change and adds some tests.\n>\n> The Scalar functional tests passed with the revert, so the original\n> patch was worthless to begin with. I don't recall what motivated the\n> change, but clearly it was a mistake. Sorry.\n>\n> ---- >8 ----\n>\n> From d1cfc8efeab015273bfebd6cd93465e6f38dc40f Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Tue, 2 Nov 2021 10:40:06 -0400\n> Subject: [PATCH] dir: fix directory-matching bug\n>\n> This reverts the change from ed49584 (dir: fix pattern matching on dirs,\n> 2021-09-24), which claimed to fix a directory-matching problem without a\n> test case. It turns out to _create_ a bug, but it is a bit subtle.\n>\n> The bug would have been revealed by the first of two tests being added to\n> t0008-ignores.sh. The first uses a pattern \"/git/\" inside the a/.gitignores\n> file, which matches against 'a/git/foo' but not 'a/git-foo/bar'. This test\n> would fail before the revert.\n>\n> The second test shows what happens if the test instead uses a pattern \"git/\"\n> and this test passes both before and after the revert.\n>\n> The difference in these two cases are due to how\n> last_matching_pattern_from_list() checks patterns both if they have the\n> PATTERN_FLAG_MUSTBEDIR and PATTERN_FLAG_NODIR flags. In the case of \"git/\",\n> the PATTERN_FLAG_NODIR is also provided, making the change in behavior in\n> match_pathname() not affect the end result of\n> last_matching_pattern_from_list().\n>\n> Reported-by: Glen Choo \n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 2 +-\n> t/t0008-ignores.sh | 26 ++++++++++++++++++++++++++\n> 2 files changed, 27 insertions(+), 1 deletion(-)\n>\n> diff --git a/dir.c b/dir.c\n> index c6d7a8647b9..94489298f4c 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1294,7 +1294,7 @@ int match_pathname(const char *pathname, int pathlen,\n> \t\t * then our prefix match is all we need; we\n> \t\t * do not need to call fnmatch at all.\n> \t\t */\n> -\t\tif (!patternlen && (!namelen || (flags & PATTERN_FLAG_MUSTBEDIR)))\n> +\t\tif (!patternlen && !namelen)\n> \t\t\treturn 1;\n> \t}\n> \n> diff --git a/t/t0008-ignores.sh b/t/t0008-ignores.sh\n> index 532637de882..1889cfc60e0 100755\n> --- a/t/t0008-ignores.sh\n> +++ b/t/t0008-ignores.sh\n> @@ -803,6 +803,32 @@ test_expect_success 'existing directory and file' '\n> \tgrep top-level-dir actual\n> '\n> \n> +test_expect_success 'exact prefix matching (with root)' '\n> +\ttest_when_finished rm -r a &&\n> +\tmkdir -p a/git a/git-foo &&\n> +\ttouch a/git/foo a/git-foo/bar &&\n> +\techo /git/ >a/.gitignore &&\n> +\tgit check-ignore a/git a/git/foo a/git-foo a/git-foo/bar >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\ta/git\n> +\ta/git/foo\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'exact prefix matching (without root)' '\n> +\ttest_when_finished rm -r a &&\n> +\tmkdir -p a/git a/git-foo &&\n> +\ttouch a/git/foo a/git-foo/bar &&\n> +\techo git/ >a/.gitignore &&\n> +\tgit check-ignore a/git a/git/foo a/git-foo a/git-foo/bar >actual &&\n> +\tcat >expect <<-\\EOF &&\n> +\ta/git\n> +\ta/git/foo\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> ############################################################################\n> #\n> # test whitespace handling\n\nWe have t3070-wildmatch.sh testing various combinations of these, and\nindeed this code resolves back to wildmatch().\n\nBut I think in this case this is due to dir.c's particular behavior of\nsplitting paths before feeding them to wildmatch, as it needs to match\nthings relative to the subdirectory.\n\nStill, we've got a matrix of these in t3070-wildmatch.sh, which already\ntests some (but apparently not all) cases where we need to create an\nactual file on disk. These sorts of test blindspots are why I added that\nin de8bada2bf6 (wildmatch test: create & test files on disk in addition\nto in-memory, 2018-01-30).\n\nWouldn't it be better & more exhaustive here to change its test lines\nlike:\n\n\n match 0 0 1 1 foo/bar/baz 'bar'\n\nTo say:\n\n match 0 0 1 1 ? ? foo/bar/baz 'bar'\n\nAnd just add to its match() function so that if we have a subject with a\nslash, we mkdir up to that first slash (here: \"mkdir foo\"), and create a\n.gitignore file therein with the \"foo\" directory with the \"bar\" content,\nperhaps adding \"/bar\", \"bar/\" and \"/bar/\" variants implicitly.\n\nThen create a \"bar.txt\" in the directory as well, and a\nbar-otherdir/somefile.txt or whatever.\n\nAnd fill in the \"? ?\" depending on whether those variants matched or\nnot...\n\nAnyway, just an idea, but if you pursue that you should be able to get\nmuch more exhaustive testing in this area that we've apparently had a\nblindspot in.\n"},{"id":"440375","messageId":"1b9e4d3b-faa6-c9fc-c734-4720f9f832df@gmail.com","threadId":"56370","inReplyTo":"211102.86pmri1rv7.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-03T14:40:51Z","receivedAt":"2021-11-03T14:40:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/2021 11:33 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> We have t3070-wildmatch.sh testing various combinations of these, and\n> indeed this code resolves back to wildmatch().\n> \n> But I think in this case this is due to dir.c's particular behavior of\n> splitting paths before feeding them to wildmatch, as it needs to match\n> things relative to the subdirectory.\n> \n> Still, we've got a matrix of these in t3070-wildmatch.sh, which already\n> tests some (but apparently not all) cases where we need to create an\n> actual file on disk. These sorts of test blindspots are why I added that\n> in de8bada2bf6 (wildmatch test: create & test files on disk in addition\n> to in-memory, 2018-01-30).\n> \n> Wouldn't it be better & more exhaustive here to change its test lines\n> like:\n> \n> \n> match 0 0 1 1 foo/bar/baz 'bar'\n> \n> To say:\n> \n> match 0 0 1 1 ? ? foo/bar/baz 'bar'\n> \n> And just add to its match() function so that if we have a subject with a\n> slash, we mkdir up to that first slash (here: \"mkdir foo\"), and create a\n> .gitignore file therein with the \"foo\" directory with the \"bar\" content,\n> perhaps adding \"/bar\", \"bar/\" and \"/bar/\" variants implicitly.\n> \n> Then create a \"bar.txt\" in the directory as well, and a\n> bar-otherdir/somefile.txt or whatever.\n> \n> And fill in the \"? ?\" depending on whether those variants matched or\n> not...\n> \n> Anyway, just an idea, but if you pursue that you should be able to get\n> much more exhaustive testing in this area that we've apparently had a\n> blindspot in.\n\nThose tests are quite exhaustive, but the test script is pretty\ninscrutable and the refactor you suggest is pretty major. I'd prefer\nto keep to the focused test for the sake of fixing this in time for\nthe release.\n\nThanks,\n-Stolee\n"},{"id":"440385","messageId":"xmqq8ry588mf.fsf@gitster.g","threadId":"56370","inReplyTo":"1b9e4d3b-faa6-c9fc-c734-4720f9f832df@gmail.com","subject":"Re: [PATCH v4 04/13] dir: fix pattern matching on dirs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-03T17:14:00Z","receivedAt":"2021-11-03T17:14:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Those tests are quite exhaustive, but the test script is pretty\n> inscrutable and the refactor you suggest is pretty major. I'd prefer\n> to keep to the focused test for the sake of fixing this in time for\n> the release.\n\nThanks, both.\n"}]}