{"thread":{"id":"57029","subject":"[PATCH 0/2] Sparse Index: fix a checkout bug with deep sparse-checkout patterns","startedAt":"2021-12-04T02:55:39Z","lastAt":"2021-12-06T14:10:49Z","messageCount":9,"participants":["Derrick Stolee via GitGitGadget","Elijah Newren","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"443056","messageId":"pull.1092.git.1638586534.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":null,"subject":"[PATCH 0/2] Sparse Index: fix a checkout bug with deep sparse-checkout patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-04T02:55:32Z","receivedAt":"2021-12-04T02:55:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This week, we rolled out the sparse index to a large internal monorepo. We\ngot two very similar bug reports that dealt with a strange error that\ninvolved the same set of paths. One was during git pull (pull was a red\nherring) and the other was git checkout. The git checkout case gave enough\nof a reproduction to debug deep into unpack-trees.c and find the problem.\n\nThis bug dates back to 523506d (unpack-trees: unpack sparse directory\nentries, 2021-07-14). The reason we didn't hit this before is because it\nrequires the following:\n\n 1. The sparse-checkout definition needs to have recursive inclusion of deep\n    folders (depth 3 or more).\n 2. Adjacent to those deep folders, we need a deep sparse directory entry\n    that receives changes.\n 3. In this particular repo, deep directories are only added to the\n    sparse-checkout in rare occasions and those adjacent folders are rarely\n    updated. They happened to update this week and hit our sparse index\n    dogfooders in surprising ways.\n\nThe first patch adds a test that fails without the fix. It requires\nmodifying our test data to make adjacent, deep sparse directory entries\npossible. It's a rather simple test after we have that data change.\n\nThe second patch includes the actual fix. It's really just an error of not\nunderstanding the difference between the name and traverse_path members of\nthe struct traverse_info structure. name only stores a single tree entry\nwhile traverse_path actually includes the full name from root. The method we\nare editing also has an additional struct name_entry that fills in the tree\nentry on top of the traverse_path, which explains how this worked to depth\ntwo, but not depth three.\n\nThanks, -Stolee\n\nDerrick Stolee (2):\n  t1092: add deeper changes during a checkout\n  unpack-trees: use traverse_path instead of name\n\n t/t1092-sparse-checkout-compatibility.sh | 16 +++++++++++++++-\n unpack-trees.c                           | 10 +++++-----\n 2 files changed, 20 insertions(+), 6 deletions(-)\n\n\nbase-commit: cd3e606211bb1cf8bc57f7d76bab98cc17a150bc\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1092%2Fderrickstolee%2Fsparse-index%2Fcheckout-bug-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1092/derrickstolee/sparse-index/checkout-bug-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1092\n-- \ngitgitgadget\n"},{"id":"443057","messageId":"c91421996562ecedab18f8af4ed8a02896584540.1638586534.git.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":"pull.1092.git.1638586534.gitgitgadget@gmail.com","subject":"[PATCH 2/2] unpack-trees: use traverse_path instead of name","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-04T02:55:34Z","receivedAt":"2021-12-04T02:55:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe sparse_dir_matches_path() method compares a cache entry that is a\nsparse directory entry against a 'struct traverse_info *info' and a\n'struct name_entry *p' to see if the cache entry has exactly the right\nname for those other inputs.\n\nThis method was introduced in 523506d (unpack-trees: unpack sparse\ndirectory entries, 2021-07-14), but included a significant mistake. The\npath comparisons used 'info->name' instead of 'info->traverse_path'.\nSince 'info->name' only stores a single tree entry name while\n'info->traverse_path' stores the full path from root, this method does\nnot work when 'info' is in a subdirectory of a directory. Replacing the\nright strings and their corresponding lengths make the method work\nproperly.\n\nThe previous change included a failing test that exposes this issue.\nThat test now passes. The critical detail is that as we go deep into\nunpack_trees(), the logic for merging a sparse directory entry with a\ntree entry during 'git checkout' relies on this\nsparse_dir_matches_path() in order to avoid calling\ntraverse_trees_recursive() during unpack_callback() in this hunk:\n\n\tif (!is_sparse_directory_entry(src[0], names, info) &&\n\t    traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n\t\t\t\t\t    names, info) < 0) {\n\t\treturn -1;\n\t}\n\nFor deep paths, the short-circuit never occurred and\ntraverse_trees_recursive() was being called incorrectly and that was\ncausing other strange issues. Specifically, the error message from the\nnow-passing test previously included this:\n\n      error: Your local changes to the following files would be overwritten by checkout:\n              deep/deeper1/deepest2/a\n              deep/deeper1/deepest3/a\n      Please commit your changes or stash them before you switch branches.\n      Aborting\n\nThese messages occurred because the 'current' cache entry in\ntwoway_merge() was showing as NULL because the index did not contain\nentries for the paths contained within the sparse directory entries. We\ninstead had 'oldtree' given as the entry at HEAD and 'newtree' as the\nentry in the target tree. This led to reject_merge() listing these\npaths.\n\nNow that sparse_dir_matches_path() works the same for deep paths as it\ndoes for shallow depths, the rest of the logic kicks in to properly\nhandle modifying the sparse directory entries as designed.\n\nReported-by: Gustave Granroth <gus.gran@gmail.com>\nReported-by: Mike Marcelais <michmarc@exchange.microsoft.com>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t1092-sparse-checkout-compatibility.sh |  2 +-\n unpack-trees.c                           | 10 +++++-----\n 2 files changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e6aef40e9b3..f04a02c6b20 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -307,7 +307,7 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-test_expect_failure 'deep changes during checkout' '\n+test_expect_success 'deep changes during checkout' '\n \tinit_repos &&\n \n \ttest_sparse_match git sparse-checkout set deep/deeper1/deepest &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 89ca95ce90b..7381c275768 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1243,11 +1243,11 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n \tassert(S_ISSPARSEDIR(ce->ce_mode));\n \tassert(ce->name[ce->ce_namelen - 1] == '/');\n \n-\tif (info->namelen)\n-\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n-\t\t       ce->name[info->namelen] == '/' &&\n-\t\t       !strncmp(ce->name, info->name, info->namelen) &&\n-\t\t       !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\tif (info->pathlen)\n+\t\treturn ce->ce_namelen == info->pathlen + p->pathlen + 1 &&\n+\t\t       ce->name[info->pathlen - 1] == '/' &&\n+\t\t       !strncmp(ce->name, info->traverse_path, info->pathlen) &&\n+\t\t       !strncmp(ce->name + info->pathlen, p->path, p->pathlen);\n \treturn ce->ce_namelen == p->pathlen + 1 &&\n \t       !strncmp(ce->name, p->path, p->pathlen);\n }\n-- \ngitgitgadget\n"},{"id":"443058","messageId":"ba05d7d4149bc4e9e931aa225cd1fc013796d7b3.1638586534.git.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":"pull.1092.git.1638586534.gitgitgadget@gmail.com","subject":"[PATCH 1/2] t1092: add deeper changes during a checkout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-04T02:55:33Z","receivedAt":"2021-12-04T02:55:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nExtend the repository data in the setup of t1092 to include more\ndirectories within two parent directories. This reproduces a bug found\nby users of the sparse index feature with suitably-complicated\nsparse-checkout definitions.\n\nAdd a failing test that fails in its first 'git checkout deepest' run in\nthe sparse index case with this error:\n\n  error: Your local changes to the following files would be overwritten by checkout:\n          deep/deeper1/deepest2/a\n          deep/deeper1/deepest3/a\n  Please commit your changes or stash them before you switch branches.\n  Aborting\n\nThe next change will fix this error, and that fix will make it clear why\nthe extra depth is necessary for revealing this bug. The assignment of\nthe sparse-checkout definition to include deep/deeper1/deepest as a\nsibling directory is important to ensure that deep/deeper1 is not a\nsparse directory entry, but deep/deeper1/deepest2 is.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t1092-sparse-checkout-compatibility.sh | 16 +++++++++++++++-\n 1 file changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 16fbd2c6db9..e6aef40e9b3 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -19,6 +19,8 @@ test_expect_success 'setup' '\n \t\tmkdir folder1 folder2 deep x &&\n \t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n+\t\tmkdir deep/deeper1/deepest2 &&\n+\t\tmkdir deep/deeper1/deepest3 &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n \t\tcp a folder1 &&\n@@ -30,7 +32,9 @@ test_expect_success 'setup' '\n \t\tcp a deep/deeper2 &&\n \t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n-\t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest2 &&\n+\t\tcp a deep/deeper1/deepest3 &&\n+\t\tcp -r deep/deeper1/ deep/deeper2 &&\n \t\tmkdir deep/deeper1/0 &&\n \t\tmkdir deep/deeper1/0/0 &&\n \t\ttouch deep/deeper1/0/1 &&\n@@ -126,6 +130,8 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b deepest base &&\n \t\techo \"updated deepest\" >deep/deeper1/deepest/a &&\n+\t\techo \"updated deepest2\" >deep/deeper1/deepest2/a &&\n+\t\techo \"updated deepest3\" >deep/deeper1/deepest3/a &&\n \t\tgit commit -a -m \"update deepest\" &&\n \n \t\tgit checkout -f base &&\n@@ -301,6 +307,14 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_failure 'deep changes during checkout' '\n+\tinit_repos &&\n+\n+\ttest_sparse_match git sparse-checkout set deep/deeper1/deepest &&\n+\ttest_all_match git checkout deepest &&\n+\ttest_all_match git checkout base\n+'\n+\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"443065","messageId":"CABPp-BEs8_F7SKsxDMd8p03oMn9M3H47=hrPP=AzxsacUkGg_A@mail.gmail.com","threadId":"57029","inReplyTo":"c91421996562ecedab18f8af4ed8a02896584540.1638586534.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] unpack-trees: use traverse_path instead of name","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-04T05:42:50Z","receivedAt":"2021-12-04T05:43:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Dec 3, 2021 at 6:55 PM Derrick Stolee via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> The sparse_dir_matches_path() method compares a cache entry that is a\n> sparse directory entry against a 'struct traverse_info *info' and a\n> 'struct name_entry *p' to see if the cache entry has exactly the right\n> name for those other inputs.\n>\n> This method was introduced in 523506d (unpack-trees: unpack sparse\n> directory entries, 2021-07-14), but included a significant mistake. The\n> path comparisons used 'info->name' instead of 'info->traverse_path'.\n> Since 'info->name' only stores a single tree entry name while\n> 'info->traverse_path' stores the full path from root, this method does\n> not work when 'info' is in a subdirectory of a directory. Replacing the\n> right strings and their corresponding lengths make the method work\n> properly.\n>\n> The previous change included a failing test that exposes this issue.\n> That test now passes. The critical detail is that as we go deep into\n> unpack_trees(), the logic for merging a sparse directory entry with a\n> tree entry during 'git checkout' relies on this\n> sparse_dir_matches_path() in order to avoid calling\n> traverse_trees_recursive() during unpack_callback() in this hunk:\n>\n>         if (!is_sparse_directory_entry(src[0], names, info) &&\n>             traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n>                                             names, info) < 0) {\n>                 return -1;\n>         }\n>\n> For deep paths, the short-circuit never occurred and\n> traverse_trees_recursive() was being called incorrectly and that was\n> causing other strange issues. Specifically, the error message from the\n> now-passing test previously included this:\n>\n>       error: Your local changes to the following files would be overwritten by checkout:\n>               deep/deeper1/deepest2/a\n>               deep/deeper1/deepest3/a\n>       Please commit your changes or stash them before you switch branches.\n>       Aborting\n>\n> These messages occurred because the 'current' cache entry in\n> twoway_merge() was showing as NULL because the index did not contain\n> entries for the paths contained within the sparse directory entries. We\n> instead had 'oldtree' given as the entry at HEAD and 'newtree' as the\n> entry in the target tree. This led to reject_merge() listing these\n> paths.\n>\n> Now that sparse_dir_matches_path() works the same for deep paths as it\n> does for shallow depths, the rest of the logic kicks in to properly\n> handle modifying the sparse directory entries as designed.\n\nEek, sorry for not catching this in my earlier review.  Thanks for the\ndetailed explanation; well analyzed.\n\n>\n> Reported-by: Gustave Granroth <gus.gran@gmail.com>\n> Reported-by: Mike Marcelais <michmarc@exchange.microsoft.com>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  t/t1092-sparse-checkout-compatibility.sh |  2 +-\n>  unpack-trees.c                           | 10 +++++-----\n>  2 files changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index e6aef40e9b3..f04a02c6b20 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -307,7 +307,7 @@ test_expect_success 'add, commit, checkout' '\n>         test_all_match git checkout -\n>  '\n>\n> -test_expect_failure 'deep changes during checkout' '\n> +test_expect_success 'deep changes during checkout' '\n>         init_repos &&\n>\n>         test_sparse_match git sparse-checkout set deep/deeper1/deepest &&\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 89ca95ce90b..7381c275768 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1243,11 +1243,11 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n>         assert(S_ISSPARSEDIR(ce->ce_mode));\n>         assert(ce->name[ce->ce_namelen - 1] == '/');\n>\n> -       if (info->namelen)\n> -               return ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n> -                      ce->name[info->namelen] == '/' &&\n> -                      !strncmp(ce->name, info->name, info->namelen) &&\n> -                      !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n> +       if (info->pathlen)\n> +               return ce->ce_namelen == info->pathlen + p->pathlen + 1 &&\n> +                      ce->name[info->pathlen - 1] == '/' &&\n> +                      !strncmp(ce->name, info->traverse_path, info->pathlen) &&\n> +                      !strncmp(ce->name + info->pathlen, p->path, p->pathlen);\n>         return ce->ce_namelen == p->pathlen + 1 &&\n>                !strncmp(ce->name, p->path, p->pathlen);\n>  }\n> --\n\nThe comment at the beginning of this function (not shown in this\npatch) is now stale and misleading; it should be corrected too.\n"},{"id":"443066","messageId":"CABPp-BHHqLvnMbR5Q+UmfTy7GPy4vz73zZdNc99PogCyxTFJHw@mail.gmail.com","threadId":"57029","inReplyTo":"pull.1092.git.1638586534.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] Sparse Index: fix a checkout bug with deep sparse-checkout patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-04T05:45:30Z","receivedAt":"2021-12-04T05:45:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Dec 3, 2021 at 6:55 PM Derrick Stolee via GitGitGadget\n<gitgitgadget@gmail.com> wrote:\n>\n> This week, we rolled out the sparse index to a large internal monorepo. We\n> got two very similar bug reports that dealt with a strange error that\n> involved the same set of paths. One was during git pull (pull was a red\n> herring) and the other was git checkout. The git checkout case gave enough\n> of a reproduction to debug deep into unpack-trees.c and find the problem.\n>\n> This bug dates back to 523506d (unpack-trees: unpack sparse directory\n> entries, 2021-07-14). The reason we didn't hit this before is because it\n> requires the following:\n>\n>  1. The sparse-checkout definition needs to have recursive inclusion of deep\n>     folders (depth 3 or more).\n>  2. Adjacent to those deep folders, we need a deep sparse directory entry\n>     that receives changes.\n>  3. In this particular repo, deep directories are only added to the\n>     sparse-checkout in rare occasions and those adjacent folders are rarely\n>     updated. They happened to update this week and hit our sparse index\n>     dogfooders in surprising ways.\n>\n> The first patch adds a test that fails without the fix. It requires\n> modifying our test data to make adjacent, deep sparse directory entries\n> possible. It's a rather simple test after we have that data change.\n>\n> The second patch includes the actual fix. It's really just an error of not\n> understanding the difference between the name and traverse_path members of\n> the struct traverse_info structure. name only stores a single tree entry\n> while traverse_path actually includes the full name from root. The method we\n> are editing also has an additional struct name_entry that fills in the tree\n> entry on top of the traverse_path, which explains how this worked to depth\n> two, but not depth three.\n\nThanks for the detailed explanation.  I looked around for similar\npotential problems elsewhere, but only noted that the comment at the\ntop of the function is also wrong and should be updated (as I\ncommented on Patch 2).  After you fix the comment similarly, feel free\nto add my\n\nReviewed-by: Elijah Newren <newren@gmail.com>\n"},{"id":"443150","messageId":"166c107e-1884-ca53-fa46-12b39e6af13f@gmail.com","threadId":"57029","inReplyTo":"CABPp-BEs8_F7SKsxDMd8p03oMn9M3H47=hrPP=AzxsacUkGg_A@mail.gmail.com","subject":"Re: [PATCH 2/2] unpack-trees: use traverse_path instead of name","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-06T13:59:07Z","receivedAt":"2021-12-06T13:59:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/4/2021 12:42 AM, Elijah Newren wrote:\n> On Fri, Dec 3, 2021 at 6:55 PM Derrick Stolee via GitGitGadget\n> <gitgitgadget@gmail.com> wrote:\n\n>> @@ -1243,11 +1243,11 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n>>         assert(S_ISSPARSEDIR(ce->ce_mode));\n>>         assert(ce->name[ce->ce_namelen - 1] == '/');\n>>\n>> -       if (info->namelen)\n>> -               return ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n>> -                      ce->name[info->namelen] == '/' &&\n>> -                      !strncmp(ce->name, info->name, info->namelen) &&\n>> -                      !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n>> +       if (info->pathlen)\n>> +               return ce->ce_namelen == info->pathlen + p->pathlen + 1 &&\n>> +                      ce->name[info->pathlen - 1] == '/' &&\n>> +                      !strncmp(ce->name, info->traverse_path, info->pathlen) &&\n>> +                      !strncmp(ce->name + info->pathlen, p->path, p->pathlen);\n>>         return ce->ce_namelen == p->pathlen + 1 &&\n>>                !strncmp(ce->name, p->path, p->pathlen);\n>>  }\n>> --\n> \n> The comment at the beginning of this function (not shown in this\n> patch) is now stale and misleading; it should be corrected too.\n \nWill do! Thanks for catching that.\n\nThanks,\n-Stolee\n"},{"id":"443152","messageId":"pull.1092.v2.git.1638799837.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":"pull.1092.git.1638586534.gitgitgadget@gmail.com","subject":"[PATCH v2 0/2] Sparse Index: fix a checkout bug with deep sparse-checkout patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-06T14:10:35Z","receivedAt":"2021-12-06T14:10:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This week, we rolled out the sparse index to a large internal monorepo. We\ngot two very similar bug reports that dealt with a strange error that\ninvolved the same set of paths. One was during git pull (pull was a red\nherring) and the other was git checkout. The git checkout case gave enough\nof a reproduction to debug deep into unpack-trees.c and find the problem.\n\nThis bug dates back to 523506d (unpack-trees: unpack sparse directory\nentries, 2021-07-14). The reason we didn't hit this before is because it\nrequires the following:\n\n 1. The sparse-checkout definition needs to have recursive inclusion of deep\n    folders (depth 3 or more).\n 2. Adjacent to those deep folders, we need a deep sparse directory entry\n    that receives changes.\n 3. In this particular repo, deep directories are only added to the\n    sparse-checkout in rare occasions and those adjacent folders are rarely\n    updated. They happened to update this week and hit our sparse index\n    dogfooders in surprising ways.\n\nThe first patch adds a test that fails without the fix. It requires\nmodifying our test data to make adjacent, deep sparse directory entries\npossible. It's a rather simple test after we have that data change.\n\nThe second patch includes the actual fix. It's really just an error of not\nunderstanding the difference between the name and traverse_path members of\nthe struct traverse_info structure. name only stores a single tree entry\nwhile traverse_path actually includes the full name from root. The method we\nare editing also has an additional struct name_entry that fills in the tree\nentry on top of the traverse_path, which explains how this worked to depth\ntwo, but not depth three.\n\n\nUpdate in v2\n============\n\n * Fixed the comment describing the sparse_dir_matches_path() method.\n\nThanks, -Stolee\n\nDerrick Stolee (2):\n  t1092: add deeper changes during a checkout\n  unpack-trees: use traverse_path instead of name\n\n t/t1092-sparse-checkout-compatibility.sh | 16 +++++++++++++++-\n unpack-trees.c                           | 14 ++++++++------\n 2 files changed, 23 insertions(+), 7 deletions(-)\n\n\nbase-commit: cd3e606211bb1cf8bc57f7d76bab98cc17a150bc\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1092%2Fderrickstolee%2Fsparse-index%2Fcheckout-bug-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1092/derrickstolee/sparse-index/checkout-bug-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1092\n\nRange-diff vs v1:\n\n 1:  ba05d7d4149 = 1:  ba05d7d4149 t1092: add deeper changes during a checkout\n 2:  c9142199656 ! 2:  aa37168dcb4 unpack-trees: use traverse_path instead of name\n     @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'add, commit, chec\n       \ttest_sparse_match git sparse-checkout set deep/deeper1/deepest &&\n      \n       ## unpack-trees.c ##\n     +@@ unpack-trees.c: static int find_cache_pos(struct traverse_info *info,\n     + \n     + /*\n     +  * Given a sparse directory entry 'ce', compare ce->name to\n     +- * info->name + '/' + p->path + '/' if info->name is non-empty.\n     ++ * info->traverse_path + p->path + '/' if info->traverse_path\n     ++ * is non-empty.\n     ++ *\n     +  * Compare ce->name to p->path + '/' otherwise. Note that\n     +  * ce->name must end in a trailing '/' because it is a sparse\n     +  * directory entry.\n      @@ unpack-trees.c: static int sparse_dir_matches_path(const struct cache_entry *ce,\n       \tassert(S_ISSPARSEDIR(ce->ce_mode));\n       \tassert(ce->name[ce->ce_namelen - 1] == '/');\n\n-- \ngitgitgadget\n"},{"id":"443151","messageId":"ba05d7d4149bc4e9e931aa225cd1fc013796d7b3.1638799837.git.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":"pull.1092.v2.git.1638799837.gitgitgadget@gmail.com","subject":"[PATCH v2 1/2] t1092: add deeper changes during a checkout","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-06T14:10:36Z","receivedAt":"2021-12-06T14:10:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nExtend the repository data in the setup of t1092 to include more\ndirectories within two parent directories. This reproduces a bug found\nby users of the sparse index feature with suitably-complicated\nsparse-checkout definitions.\n\nAdd a failing test that fails in its first 'git checkout deepest' run in\nthe sparse index case with this error:\n\n  error: Your local changes to the following files would be overwritten by checkout:\n          deep/deeper1/deepest2/a\n          deep/deeper1/deepest3/a\n  Please commit your changes or stash them before you switch branches.\n  Aborting\n\nThe next change will fix this error, and that fix will make it clear why\nthe extra depth is necessary for revealing this bug. The assignment of\nthe sparse-checkout definition to include deep/deeper1/deepest as a\nsibling directory is important to ensure that deep/deeper1 is not a\nsparse directory entry, but deep/deeper1/deepest2 is.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t1092-sparse-checkout-compatibility.sh | 16 +++++++++++++++-\n 1 file changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 16fbd2c6db9..e6aef40e9b3 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -19,6 +19,8 @@ test_expect_success 'setup' '\n \t\tmkdir folder1 folder2 deep x &&\n \t\tmkdir deep/deeper1 deep/deeper2 deep/before deep/later &&\n \t\tmkdir deep/deeper1/deepest &&\n+\t\tmkdir deep/deeper1/deepest2 &&\n+\t\tmkdir deep/deeper1/deepest3 &&\n \t\techo \"after deeper1\" >deep/e &&\n \t\techo \"after deepest\" >deep/deeper1/e &&\n \t\tcp a folder1 &&\n@@ -30,7 +32,9 @@ test_expect_success 'setup' '\n \t\tcp a deep/deeper2 &&\n \t\tcp a deep/later &&\n \t\tcp a deep/deeper1/deepest &&\n-\t\tcp -r deep/deeper1/deepest deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest2 &&\n+\t\tcp a deep/deeper1/deepest3 &&\n+\t\tcp -r deep/deeper1/ deep/deeper2 &&\n \t\tmkdir deep/deeper1/0 &&\n \t\tmkdir deep/deeper1/0/0 &&\n \t\ttouch deep/deeper1/0/1 &&\n@@ -126,6 +130,8 @@ test_expect_success 'setup' '\n \n \t\tgit checkout -b deepest base &&\n \t\techo \"updated deepest\" >deep/deeper1/deepest/a &&\n+\t\techo \"updated deepest2\" >deep/deeper1/deepest2/a &&\n+\t\techo \"updated deepest3\" >deep/deeper1/deepest3/a &&\n \t\tgit commit -a -m \"update deepest\" &&\n \n \t\tgit checkout -f base &&\n@@ -301,6 +307,14 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n+test_expect_failure 'deep changes during checkout' '\n+\tinit_repos &&\n+\n+\ttest_sparse_match git sparse-checkout set deep/deeper1/deepest &&\n+\ttest_all_match git checkout deepest &&\n+\ttest_all_match git checkout base\n+'\n+\n test_expect_success 'add outside sparse cone' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"443153","messageId":"aa37168dcb47c0144e9468e2d317b8b8efd20e46.1638799837.git.gitgitgadget@gmail.com","threadId":"57029","inReplyTo":"pull.1092.v2.git.1638799837.gitgitgadget@gmail.com","subject":"[PATCH v2 2/2] unpack-trees: use traverse_path instead of name","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-06T14:10:37Z","receivedAt":"2021-12-06T14:10:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe sparse_dir_matches_path() method compares a cache entry that is a\nsparse directory entry against a 'struct traverse_info *info' and a\n'struct name_entry *p' to see if the cache entry has exactly the right\nname for those other inputs.\n\nThis method was introduced in 523506d (unpack-trees: unpack sparse\ndirectory entries, 2021-07-14), but included a significant mistake. The\npath comparisons used 'info->name' instead of 'info->traverse_path'.\nSince 'info->name' only stores a single tree entry name while\n'info->traverse_path' stores the full path from root, this method does\nnot work when 'info' is in a subdirectory of a directory. Replacing the\nright strings and their corresponding lengths make the method work\nproperly.\n\nThe previous change included a failing test that exposes this issue.\nThat test now passes. The critical detail is that as we go deep into\nunpack_trees(), the logic for merging a sparse directory entry with a\ntree entry during 'git checkout' relies on this\nsparse_dir_matches_path() in order to avoid calling\ntraverse_trees_recursive() during unpack_callback() in this hunk:\n\n\tif (!is_sparse_directory_entry(src[0], names, info) &&\n\t    traverse_trees_recursive(n, dirmask, mask & ~dirmask,\n\t\t\t\t\t    names, info) < 0) {\n\t\treturn -1;\n\t}\n\nFor deep paths, the short-circuit never occurred and\ntraverse_trees_recursive() was being called incorrectly and that was\ncausing other strange issues. Specifically, the error message from the\nnow-passing test previously included this:\n\n      error: Your local changes to the following files would be overwritten by checkout:\n              deep/deeper1/deepest2/a\n              deep/deeper1/deepest3/a\n      Please commit your changes or stash them before you switch branches.\n      Aborting\n\nThese messages occurred because the 'current' cache entry in\ntwoway_merge() was showing as NULL because the index did not contain\nentries for the paths contained within the sparse directory entries. We\ninstead had 'oldtree' given as the entry at HEAD and 'newtree' as the\nentry in the target tree. This led to reject_merge() listing these\npaths.\n\nNow that sparse_dir_matches_path() works the same for deep paths as it\ndoes for shallow depths, the rest of the logic kicks in to properly\nhandle modifying the sparse directory entries as designed.\n\nReported-by: Gustave Granroth <gus.gran@gmail.com>\nReported-by: Mike Marcelais <michmarc@exchange.microsoft.com>\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n t/t1092-sparse-checkout-compatibility.sh |  2 +-\n unpack-trees.c                           | 14 ++++++++------\n 2 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex e6aef40e9b3..f04a02c6b20 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -307,7 +307,7 @@ test_expect_success 'add, commit, checkout' '\n \ttest_all_match git checkout -\n '\n \n-test_expect_failure 'deep changes during checkout' '\n+test_expect_success 'deep changes during checkout' '\n \tinit_repos &&\n \n \ttest_sparse_match git sparse-checkout set deep/deeper1/deepest &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 89ca95ce90b..d2363b44ec3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1231,7 +1231,9 @@ static int find_cache_pos(struct traverse_info *info,\n \n /*\n  * Given a sparse directory entry 'ce', compare ce->name to\n- * info->name + '/' + p->path + '/' if info->name is non-empty.\n+ * info->traverse_path + p->path + '/' if info->traverse_path\n+ * is non-empty.\n+ *\n  * Compare ce->name to p->path + '/' otherwise. Note that\n  * ce->name must end in a trailing '/' because it is a sparse\n  * directory entry.\n@@ -1243,11 +1245,11 @@ static int sparse_dir_matches_path(const struct cache_entry *ce,\n \tassert(S_ISSPARSEDIR(ce->ce_mode));\n \tassert(ce->name[ce->ce_namelen - 1] == '/');\n \n-\tif (info->namelen)\n-\t\treturn ce->ce_namelen == info->namelen + p->pathlen + 2 &&\n-\t\t       ce->name[info->namelen] == '/' &&\n-\t\t       !strncmp(ce->name, info->name, info->namelen) &&\n-\t\t       !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen);\n+\tif (info->pathlen)\n+\t\treturn ce->ce_namelen == info->pathlen + p->pathlen + 1 &&\n+\t\t       ce->name[info->pathlen - 1] == '/' &&\n+\t\t       !strncmp(ce->name, info->traverse_path, info->pathlen) &&\n+\t\t       !strncmp(ce->name + info->pathlen, p->path, p->pathlen);\n \treturn ce->ce_namelen == p->pathlen + 1 &&\n \t       !strncmp(ce->name, p->path, p->pathlen);\n }\n-- \ngitgitgadget\n"}]}