{"thread":{"id":"57464","subject":"[PATCH 0/7] Sparse index: integrate with 'read-tree'","startedAt":"2022-02-23T18:25:24Z","lastAt":"2022-03-04T18:47:10Z","messageCount":48,"participants":["Victoria Dye via GitGitGadget","Derrick Stolee","Victoria Dye","Elijah Newren","Ævar Arnfjörð Bjarmason","Glen Choo"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"449295","messageId":"pull.1157.git.1645640717.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":null,"subject":"[PATCH 0/7] Sparse index: integrate with 'read-tree'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:09Z","receivedAt":"2022-02-23T18:25:24Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Like previous integrations [1] [2], this series allows 'git read-tree' to\noperate using a sparse index.\n\nThe first two patches are bugfixes for issues found while implementing the\n'read-tree' integration:\n\n * The first (patch 1/7) fixes an edge case in which a repo with no in-cone\n files or directories would have its root collapsed into a sparse\n directory; the fix ensures the root directory is never collapsed into a\n sparse directory.\n * The second (patch 2/7) corrects the 'git status' reporting of changes\n nested inside the subdirectory of a sparse directory, ensuring that the\n modified file (not the subdirectory) is correctly reported as having\n changes.\n\nThe remainder of the series focuses on utilizing the sparse index in 'git\nread-tree'. After some baseline behavior-establishing tests (patch 3/7),\nsparse index usage is trivially enabled (patch 4/7) for 'read-tree' except:\n\n * usage with '--prefix'\n * two- and three-way merge\n\nThese cases require additional changes in order to work as expected (i.e.,\noutwardly matching non-sparse index sparse-checkout). For the former, the\nsparse index can be enabled as long as the index is expanded when the prefix\nis a directory outside the sparse cone (patch 5/7). For the latter, sparse\ndirectories that cannot be trivially merged must have their contents merged\nfile-by-file, done by recursively traversing the trees represented by the\nsparse directories (patches 6/7 & 7/7).\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/\n[2]\nhttps://lore.kernel.org/git/pull.1048.v6.git.1638201164.gitgitgadget@gmail.com/\n\nVictoria Dye (7):\n sparse-index: prevent repo root from becoming sparse\n status: fix nested sparse directory diff in sparse index\n read-tree: expand sparse checkout test coverage\n read-tree: integrate with sparse index\n read-tree: narrow scope of index expansion for '--prefix'\n read-tree: make two-way merge sparse-aware\n read-tree: make three-way merge sparse-aware\n\n builtin/read-tree.c | 10 +-\n dir.c | 7 +-\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 129 +++++++++++++++++++++++\n unpack-trees.c | 121 ++++++++++++++++++++-\n wt-status.c | 9 ++\n 6 files changed, 268 insertions(+), 9 deletions(-)\n\n\nbase-commit: e6ebfd0e8cbbd10878070c8a356b5ad1b3ca464e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1157%2Fvdye%2Fsparse%2Fread-tree-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1157/vdye/sparse/read-tree-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1157\n-- \ngitgitgadget\n"},{"id":"449296","messageId":"90da1f9f33a1383ba199f9b11e9890bc67e56edf.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 1/7] sparse-index: prevent repo root from becoming sparse","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:10Z","receivedAt":"2022-02-23T18:25:25Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nPrevent the repository root from being collapsed into a sparse directory by\ntreating an empty path as \"inside the sparse-checkout\". When collapsing a\nsparse index (e.g. in 'git sparse-checkout reapply'), the root directory\ntypically could not become a sparse directory due to the presence of in-cone\nroot-level files and directories. However, if no such in-cone files or\ndirectories were present, there was no explicit check signaling that the\n\"repository root path\" (an empty string, in the case of\n'convert_to_sparse(...)') was in-cone, and a sparse directory index entry\nwould be created from the repository root directory.\n\nThe documentation in Documentation/git-sparse-checkout.txt explicitly states\nthat the files in the root directory are expected to be in-cone for a\ncone-mode sparse-checkout. Collapsing the root into a sparse directory entry\nviolates that assumption, as sparse directory entries are expected to be\noutside the sparse cone and have SKIP_WORKTREE enabled. This invalid state\nin turn causes issues with commands that interact with the index, e.g.\n'git status'.\n\nTreating an empty (root) path as in-cone prevents the creation of a root\nsparse directory in 'convert_to_sparse(...)'. Because the repository root is\notherwise never compared with sparse patterns (in both cone-mode and\nnon-cone sparse-checkouts), the new check does not cause additional changes\nto how sparse patterns are applied.\n\nSigned-off-by: Victoria Dye \n---\n dir.c | 7 ++++---\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d91295f2bcd..f1cbef23a8a 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1463,10 +1463,11 @@ static int path_in_sparse_checkout_1(const char *path,\n \tconst char *end, *slash;\n \n \t/*\n-\t * We default to accepting a path if there are no patterns or\n-\t * they are of the wrong type.\n+\t * We default to accepting a path if the path is empty, there are no\n+\t * patterns, or the patterns are of the wrong type.\n \t */\n-\tif (init_sparse_checkout_patterns(istate) ||\n+\tif (!strlen(path) ||\n+\t init_sparse_checkout_patterns(istate) ||\n \t (require_cone_mode &&\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n \t\treturn 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f3a059e5af5..9ef7cd80885 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -244,6 +244,24 @@ test_expect_success 'expanded in-memory index matches full index' '\n \ttest_sparse_match git ls-files --stage\n '\n \n+test_expect_success 'root directory cannot be sparse' '\n+\tinit_repos &&\n+\n+\t# Remove all in-cone files and directories from the index, collapse index\n+\t# with `git sparse-checkout reapply`\n+\tgit -C sparse-index rm -r . &&\n+\tgit -C sparse-index sparse-checkout reapply &&\n+\n+\t# Verify sparse directories still present, root directory is not sparse\n+\tcat >expect <<-EOF &&\n+\tfolder1/\n+\tfolder2/\n+\tx/\n+\tEOF\n+\tgit -C sparse-index ls-files --sparse >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'status with options' '\n \tinit_repos &&\n \ttest_sparse_match ls &&\n-- \ngitgitgadget\n\n"},{"id":"449297","messageId":"c21c9b9be348640b7519ecc454331d7f1493228a.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 2/7] status: fix nested sparse directory diff in sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:11Z","receivedAt":"2022-02-23T18:25:29Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\nthe 'recursive' flag, 'git status' could report index changes incorrectly\nwhen the following conditions were met:\n\n* sparse index is enabled\n* there is a difference between the index and HEAD in a file inside a\n *subdirectory* of a sparse directory\n* the sparse directory index entry is *not* expanded in-core\n\nIn this scenario, 'git status' would not recurse into the sparse directory's\nsubdirectories to identify which file contained the difference between the\nindex and HEAD. Instead, it would report the immediate subdirectory itself\nas \"modified\".\n\nExample:\n\n$ git init\n$ mkdir -p sparse/sub\n$ echo test >sparse/sub/foo\n$ git add .\n$ git commit -m \"commit 1\"\n$ echo somethingelse >sparse/sub/foo\n$ git add .\n$ git commit -a -m \"commit 2\"\n$ git sparse-checkout set --cone --sparse-index 'sparse'\n$ git reset --soft HEAD~1\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub\n\nThe 'recursive' diff option in 'wt_status_collect_changes_index' corrects\nthis by indicating that 'git status' should recurse into sparse directories\nto find modified files. Given the same repository setup as the example\nabove, the corrected result of `git status` is:\n\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub/foo\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 7 +++++++\n wt-status.c | 9 +++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9ef7cd80885..b1dcaa0e642 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -278,6 +278,13 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status with diff in unexpanded sparse directory' '\n+\tinit_repos &&\n+\ttest_all_match git checkout rename-base &&\n+\ttest_all_match git reset --soft rename-out-to-out &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_expect_success 'status reports sparse-checkout' '\n \tinit_repos &&\n \tgit -C sparse-checkout status >full &&\ndiff --git a/wt-status.c b/wt-status.c\nindex 335e723a71e..4a5b9beeca1 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -651,6 +651,15 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trev.diffopt.detect_rename = s->detect_rename >= 0 ? s->detect_rename : rev.diffopt.detect_rename;\n \trev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;\n \trev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n+\n+\t/*\n+\t * The `recursive` option must be enabled to show differences in files\n+\t * *more than* one level deep in a sparse directory index entry (e.g., given\n+\t * sparse directory 'sparse-dir/', reporting a difference in the file\n+\t * 'sparse-dir/another-dir/my-file').\n+\t */\n+\trev.diffopt.flags.recursive = 1;\n+\n \tcopy_pathspec(&rev.prune_data, &s->pathspec);\n \trun_diff_index(&rev, 1);\n \tobject_array_clear(&rev.pending);\n-- \ngitgitgadget\n\n"},{"id":"449298","messageId":"5ee193bfa8733b55d74cb0b8e1d8944be52984c2.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 4/7] read-tree: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:13Z","receivedAt":"2022-02-23T18:25:41Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of sparse index in 'git read-tree'. The integration in this patch\nis limited only to usage of 'read-tree' that does not need additional\nfunctional changes for the sparse index to behave as expected (i.e., produce\nthe same user-facing results as a non-sparse index sparse-checkout). To\nensure no unexpected behavior occurs, the index is explicitly expanded when:\n\n* '--no-sparse-checkout' is specified (because it disables sparse-checkout)\n* '--prefix' is specified (if the prefix is inside a sparse directory, the\n prefixed tree cannot be properly traversed)\n* two or more arguments are specified ('twoway_merge' and\n 'threeway_merge' do not yet support merging sparse directories)\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 21 +++++++++++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++++\n 2 files changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 2109c4c9e5c..c2fdbc2657f 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -160,8 +160,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \targc = parse_options(argc, argv, cmd_prefix, read_tree_options,\n \t\t\t read_tree_usage, 0);\n \n-\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n-\n \tprefix_set = opts.prefix ? 1 : 0;\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n@@ -169,6 +167,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.reset)\n \t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n+\n \t/*\n \t * NEEDSWORK\n \t *\n@@ -210,6 +213,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n+\t/* TODO: audit sparse index behavior in unpack_trees */\n+\tif (opts.skip_sparse_checkout || opts.prefix)\n+\t\tensure_full_index(&the_index);\n+\n \tif (opts.merge) {\n \t\tswitch (stage - 1) {\n \t\tcase 0:\n@@ -219,11 +226,21 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n+\t\t\t/*\n+\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n+\t\t\t/*\n+\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9d58da4e925..ae44451a0a9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1407,6 +1407,17 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'sparse index is not expanded: read-tree' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded checkout -b test-branch update-folder1 &&\n+\tfor MERGE_TREES in \"update-folder2\"\n+\tdo\n+\t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n+\t\tensure_not_expanded reset --hard HEAD || return 1\n+\tdone\n+'\n+\n test_expect_success 'ls-files' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"449299","messageId":"ac42ae21d4a1134b866b4be650f28b319b75d7ba.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 3/7] read-tree: expand sparse checkout test coverage","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:12Z","receivedAt":"2022-02-23T18:25:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\nemphasis is placed on interactions with files outside the sparse cone, e.g.\nmerges with out-of-cone conflicts.\n\nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 2a7106b9495..382716cfca9 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -117,6 +117,7 @@ test_perf_on_all git diff\n test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n+test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b1dcaa0e642..9d58da4e925 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n \ttest_cmp expect sparse-checkout-out\n '\n \n+test_expect_success 'read-tree --merge with files outside sparse definition' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-branch update-folder1 &&\n+\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n+\t\t\t \"update-folder1 update-folder2\" \\\n+\t\t\t \"update-folder2\"\n+\tdo\n+\t\t# Clean up and remove on-disk files\n+\t\ttest_all_match git reset --hard HEAD &&\n+\t\ttest_sparse_match git sparse-checkout reapply &&\n+\n+\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t\t# definition files will not exist on disk for sparse checkouts\n+\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_path_is_missing sparse-checkout/folder2 &&\n+\t\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\t\ttest_all_match git read-tree --reset -u HEAD &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\n+\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n+\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n+\tdone\n+'\n+\n+test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n+\tinit_repos &&\n+\n+\t# Merge of multiple changes to same directory (but not same files) should\n+\t# succeed\n+\ttest_all_match git read-tree -mu base rename-base update-folder1 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match git read-tree -mu rename-base update-folder2 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match test_must_fail git read-tree -mu base update-folder1 rename-out-to-in &&\n+\ttest_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n+'\n+\n+test_expect_success 'read-tree --prefix outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# Cannot read-tree --prefix with a single argument when files exist within\n+\t# prefix\n+\ttest_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n+\n+\ttest_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n+\ttest_path_is_missing sparse-checkout/folder2 &&\n+\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\ttest_all_match git read-tree --reset -u HEAD &&\n+\ttest_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n+\ttest_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n+\ttest_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n+'\n+\n+test_expect_success 'read-tree --merge with directory-file conflicts' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-branch rename-base &&\n+\n+\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t# definition files will not exist on disk for sparse checkouts\n+\ttest_sparse_match git read-tree -mu rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_path_is_missing sparse-checkout/folder2 &&\n+\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\ttest_sparse_match git read-tree --reset -u HEAD &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\ttest_sparse_match git read-tree -mu --no-sparse-checkout rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_cmp sparse-checkout/folder2/0/1 sparse-index/folder2/0/1\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"449300","messageId":"bea482b6b28df8522aa64ab757616bbcdb84bbec.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:14Z","receivedAt":"2022-02-23T18:25:45Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nWhen 'git read-tree' is provided with a prefix, expand the index only if the\nprefix is equivalent to a sparse directory or contained within one. If the\nindex is not expanded in these cases, 'ce_in_traverse_path' will indicate\nthat the relevant sparse directory is not in the prefix/traverse path,\nskipping past it and not unpacking the appropriate tree(s).\n\nIf the prefix is in-cone, its sparse subdirectories (if any) will be\ntraversed correctly without index expansion.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 3 +--\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n unpack-trees.c | 30 ++++++++++++++++++++++++\n 3 files changed, 38 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex c2fdbc2657f..a7b7f822281 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n-\t/* TODO: audit sparse index behavior in unpack_trees */\n-\tif (opts.skip_sparse_checkout || opts.prefix)\n+\tif (opts.skip_sparse_checkout)\n \t\tensure_full_index(&the_index);\n \n \tif (opts.merge) {\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ae44451a0a9..a404be0a10f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\n-\tdone\n+\tdone &&\n+\n+\trm -rf sparse-index/deep/deeper2 &&\n+\tensure_not_expanded add . &&\n+\tensure_not_expanded commit -m \"test\" &&\n+\n+\tensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n '\n \n test_expect_success 'ls-files' '\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..dba122a02bb 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tsetup_standard_excludes(o->dir);\n \t}\n \n+\t/*\n+\t * If the prefix is equal to or contained within a sparse directory, the\n+\t * index needs to be expanded to traverse with the specified prefix.\n+\t */\n+\tif (o->prefix && o->src_index->sparse_index) {\n+\t\tint prefix_len = strlen(o->prefix);\n+\n+\t\twhile (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n+\t\t\tprefix_len--;\n+\n+\t\tif (prefix_len > 0) {\n+\t\t\tstruct strbuf ce_prefix = STRBUF_INIT;\n+\t\t\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n+\t\t\tstrbuf_add(&ce_prefix, o->prefix, prefix_len);\n+\t\t\tstrbuf_addch(&ce_prefix, '/');\n+\n+\t\t\t/*\n+\t\t\t * If the prefix is not inside the sparse cone, then the\n+\t\t\t * index is explicitly expanded if it is found as a sparse\n+\t\t\t * directory, or implicitly expanded (by 'index_name_pos')\n+\t\t\t * if the path is inside a sparse directory.\n+\t\t\t */\n+\t\t\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n+\t\t\t index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n+\t\t\t\tensure_full_index(o->src_index);\n+\n+\t\t\tstrbuf_release(&ce_prefix);\n+\t\t}\n+\t}\n+\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n-- \ngitgitgadget\n\n"},{"id":"449301","messageId":"9fdcab038b2962b7f954363e32d04591476cf219.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 6/7] read-tree: make two-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:15Z","receivedAt":"2022-02-23T18:25:47Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable two-way merge with 'git read-tree' without expanding the sparse\nindex. When in a sparse index, a two-way merge will trivially succeed as\nlong as there are not changes to the same sparse directory in multiple trees\n(i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\nconflicts, the merge will fail despite the possibility that individual files\ncould merge cleanly.\n\nIn order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\ndirectories are - rather than rejected - merged by traversing their\nassociated trees by OID. For each child of the sparse directory:\n\n1. Files are merged as normal (see Documentation/git-read-tree.txt for\n details).\n2. Subdirectories are treated as sparse directories and merged in\n 'twoway_merge'. If there are no conflicts, they are merged according to\n the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n is recursively traversed and merged.\n\nThis process allows sparse directories to be individually merged at the\nnecessary depth *without* expanding a full index.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 --\n t/t1092-sparse-checkout-compatibility.sh | 3 +-\n unpack-trees.c | 75 ++++++++++++++++++++++++\n 3 files changed, 77 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex a7b7f822281..5a421de2629 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -225,11 +225,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n-\t\t\t/*\n-\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a404be0a10f..d6f19682d65 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1411,7 +1411,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tinit_repos &&\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n-\tfor MERGE_TREES in \"update-folder2\"\n+\tfor MERGE_TREES in \"update-folder2\" \\\n+\t\t\t \"base update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dba122a02bb..a4ace53904e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n \treturn sparse_dir_matches_path(ce, info, name);\n }\n \n+static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n+{\n+\tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n+\tstruct unpack_trees_options *o = info->data;\n+\tint ret;\n+\n+\tassert(o->merge);\n+\n+\t/*\n+\t * Unlike in 'unpack_callback', where src[0] is derived from the index when\n+\t * merging, src[0] is a transient cache entry derived from the first tree\n+\t * provided. Create the temporary entry as if it came from a non-sparse index.\n+\t */\n+\tif (!is_null_oid(&names[0].oid)) {\n+\t\tsrc[0] = create_ce_entry(info, &names[0], 0,\n+\t\t\t\t\t&o->result, 1,\n+\t\t\t\t\tdirmask & (1ul << 0));\n+\t\tsrc[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t}\n+\n+\t/*\n+\t * 'unpack_single_entry' assumes that src[0] is derived directly from\n+\t * the index, rather than from an entry in 'names'. This is *not* true when\n+\t * merging a sparse directory, in which case names[0] is the \"index\" source\n+\t * entry. To match the expectations of 'unpack_single_entry', shift past the\n+\t * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n+\t * 'dirmask' accordingly.\n+\t */\n+\tret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n+\n+\tif (src[0])\n+\t\tdiscard_cache_entry(src[0]);\n+\n+\treturn ret >= 0 ? mask : -1;\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -2464,6 +2500,37 @@ static int merged_entry(const struct cache_entry *ce,\n \treturn 1;\n }\n \n+static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\tstruct tree_desc t[MAX_UNPACK_TREES + 1];\n+\tvoid * tree_bufs[MAX_UNPACK_TREES + 1];\n+\tstruct traverse_info info;\n+\tint i, ret;\n+\n+\t/*\n+\t * Create the tree traversal information for traversing into *only* the\n+\t * sparse directory.\n+\t */\n+\tsetup_traverse_info(&info, src[0]->name);\n+\tinfo.fn = unpack_sparse_callback;\n+\tinfo.data = o;\n+\tinfo.show_all_errors = o->show_all_errors;\n+\tinfo.pathspec = o->pathspec;\n+\n+\t/* Get the tree descriptors of the sparse directory in each of the merging trees */\n+\tfor (i = 0; i < n; i++)\n+\t\ttree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n+\t\t\t\t\t\t src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n+\n+\tret = traverse_trees(o->src_index, n, t, &info);\n+\n+\tfor (i = 0; i < n; i++)\n+\t\tfree(tree_bufs[i]);\n+\n+\treturn ret;\n+}\n+\n static int deleted_entry(const struct cache_entry *ce,\n \t\t\t const struct cache_entry *old,\n \t\t\t struct unpack_trees_options *o)\n@@ -2734,6 +2801,14 @@ int twoway_merge(const struct cache_entry * const *src,\n \t\t\t * reject the merge instead.\n \t\t\t */\n \t\t\treturn merged_entry(newtree, current, o);\n+\t\t} else if (S_ISSPARSEDIR(current->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * The sparse directories differ, but we don't know whether that's\n+\t\t\t * because of two different files in the directory being modified\n+\t\t\t * (can be trivially merged) or if there is a real file conflict.\n+\t\t\t * Merge the sparse directory by OID to compare file-by-file.\n+\t\t\t */\n+\t\t\treturn merged_sparse_dir(src, 3, o);\n \t\t} else\n \t\t\treturn reject_merge(current, o);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"449302","messageId":"1502e9acb326e03397311f02cf73921e89bbfc35.1645640717.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH 7/7] read-tree: make three-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T18:25:16Z","receivedAt":"2022-02-23T18:25:49Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of 'merged_sparse_dir' in 'threeway_merge'. As with two-way\nmerge, the contents of each conflicted sparse directory are merged without\nreferencing the index, avoiding sparse index expansion.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 -----\n t/t1092-sparse-checkout-compatibility.sh | 3 ++-\n unpack-trees.c | 16 ++++++++++++----\n 3 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 5a421de2629..dc2283fce53 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -230,11 +230,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n-\t\t\t/*\n-\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d6f19682d65..5447d314632 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1412,7 +1412,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n \tfor MERGE_TREES in \"update-folder2\" \\\n-\t\t\t \"base update-folder2\"\n+\t\t\t \"base update-folder2\" \\\n+\t\t\t \"base HEAD update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a4ace53904e..9e79cb13bd4 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2635,16 +2635,24 @@ int threeway_merge(const struct cache_entry * const *stages,\n \t */\n \t/* #14, #14ALT, #2ALT */\n \tif (remote && !df_conflict_head && head_match && !remote_match) {\n-\t\tif (index && !same(index, remote) && !same(index, head))\n-\t\t\treturn reject_merge(index, o);\n+\t\tif (index && !same(index, remote) && !same(index, head)) {\n+\t\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\t\telse\n+\t\t\t\treturn reject_merge(index, o);\n+\t\t}\n \t\treturn merged_entry(remote, index, o);\n \t}\n \t/*\n \t * If we have an entry in the index cache, then we want to\n \t * make sure that it matches head.\n \t */\n-\tif (index && !same(index, head))\n-\t\treturn reject_merge(index, o);\n+\tif (index && !same(index, head)) {\n+\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\telse\n+\t\t\treturn reject_merge(index, o);\n+\t}\n \n \tif (head) {\n \t\t/* #5ALT, #15 */\n-- \ngitgitgadget\n"},{"id":"449479","messageId":"ee861471-d870-f2e4-4f3b-3aea467d8415@github.com","threadId":"57464","inReplyTo":"90da1f9f33a1383ba199f9b11e9890bc67e56edf.1645640717.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] sparse-index: prevent repo root from becoming sparse","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:48:40Z","receivedAt":"2022-02-24T16:48:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2022 1:25 PM, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n\nAside: It looks like you have the same \"mailto:\" email addresses in the\nCC similar to a mistake I made. The root cause of my issue was that I\ncopy-pasted from the rendered description of another PR instead of\ncopying the plaintext available by clicking the \"Edit\" button. Perhaps\nthis is worth extending GGG to prevent this issue in the future.\n\n> Prevent the repository root from being collapsed into a sparse directory by\n> treating an empty path as \"inside the sparse-checkout\". When collapsing a\n> sparse index (e.g. in 'git sparse-checkout reapply'), the root directory\n> typically could not become a sparse directory due to the presence of in-cone\n> root-level files and directories. However, if no such in-cone files or\n> directories were present, there was no explicit check signaling that the\n> \"repository root path\" (an empty string, in the case of\n> 'convert_to_sparse(...)') was in-cone, and a sparse directory index entry\n> would be created from the repository root directory.\n> \n> The documentation in Documentation/git-sparse-checkout.txt explicitly states\n> that the files in the root directory are expected to be in-cone for a\n> cone-mode sparse-checkout. Collapsing the root into a sparse directory entry\n> violates that assumption, as sparse directory entries are expected to be\n> outside the sparse cone and have SKIP_WORKTREE enabled. This invalid state\n> in turn causes issues with commands that interact with the index, e.g.\n> 'git status'.\n> \n> Treating an empty (root) path as in-cone prevents the creation of a root\n> sparse directory in 'convert_to_sparse(...)'. Because the repository root is\n> otherwise never compared with sparse patterns (in both cone-mode and\n> non-cone sparse-checkouts), the new check does not cause additional changes\n> to how sparse patterns are applied.\n\nGood catch! I agree with everything said here.\n\n> +\tif (!strlen(path) ||\n\nInstead of a full strlen(), could we just check \"if (!*path ||\" to\nlook for the nul byte?\n\nThanks,\n-Stolee\n"},{"id":"449481","messageId":"2bc6fcb6-2d95-c962-3098-ca4e4e0afa00@github.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Sparse index: integrate with 'read-tree'","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-24T16:59:37Z","receivedAt":"2022-02-24T17:03:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2022 1:25 PM, Victoria Dye via GitGitGadget wrote:\n> Like previous integrations [1] [2], this series allows 'git read-tree' to\n> operate using a sparse index.\n\n(Fixed the CCs on this reply, too.)\n \n> The first two patches are bugfixes for issues found while implementing the\n> 'read-tree' integration:\n> \n> * The first (patch 1/7) fixes an edge case in which a repo with no in-cone\n> files or directories would have its root collapsed into a sparse\n> directory; the fix ensures the root directory is never collapsed into a\n> sparse directory.\n> * The second (patch 2/7) corrects the 'git status' reporting of changes\n> nested inside the subdirectory of a sparse directory, ensuring that the\n> modified file (not the subdirectory) is correctly reported as having\n> changes.\n\nThanks for these! I just found one tiny optimization in the first patch.\n\n> The remainder of the series focuses on utilizing the sparse index in 'git\n> read-tree'. After some baseline behavior-establishing tests (patch 3/7),\n> sparse index usage is trivially enabled (patch 4/7) for 'read-tree' except:\n> \n> * usage with '--prefix'\n> * two- and three-way merge\n> \n> These cases require additional changes in order to work as expected (i.e.,\n> outwardly matching non-sparse index sparse-checkout). For the former, the\n> sparse index can be enabled as long as the index is expanded when the prefix\n> is a directory outside the sparse cone (patch 5/7). For the latter, sparse\n> directories that cannot be trivially merged must have their contents merged\n> file-by-file, done by recursively traversing the trees represented by the\n> sparse directories (patches 6/7 & 7/7).\n\nI enjoyed reading these remaining patches. I'm impressed with how you\nconstructed these tests and patches to do the smallest amount of change\nper patch.\n\nI couldn't find any fault in these patches, but perhaps Elijah's deep\nexperience with merge machinery could help add confidence, especially\nfor patches 6 & 7.\n\nThanks,\n-Stolee\n"},{"id":"449514","messageId":"46428e01-ee6e-36c8-8bbd-1119d9ee2975@github.com","threadId":"57464","inReplyTo":"ee861471-d870-f2e4-4f3b-3aea467d8415@github.com","subject":"Re: [PATCH 1/7] sparse-index: prevent repo root from becoming sparse","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-24T21:42:50Z","receivedAt":"2022-02-24T21:42:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee wrote:\n> On 2/23/2022 1:25 PM, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye \n> \n> Aside: It looks like you have the same \"mailto:\" email addresses in the\n> CC similar to a mistake I made. The root cause of my issue was that I\n> copy-pasted from the rendered description of another PR instead of\n> copying the plaintext available by clicking the \"Edit\" button. Perhaps\n> this is worth extending GGG to prevent this issue in the future.\n> \n>> Prevent the repository root from being collapsed into a sparse directory by\n>> treating an empty path as \"inside the sparse-checkout\". When collapsing a\n>> sparse index (e.g. in 'git sparse-checkout reapply'), the root directory\n>> typically could not become a sparse directory due to the presence of in-cone\n>> root-level files and directories. However, if no such in-cone files or\n>> directories were present, there was no explicit check signaling that the\n>> \"repository root path\" (an empty string, in the case of\n>> 'convert_to_sparse(...)') was in-cone, and a sparse directory index entry\n>> would be created from the repository root directory.\n>>\n>> The documentation in Documentation/git-sparse-checkout.txt explicitly states\n>> that the files in the root directory are expected to be in-cone for a\n>> cone-mode sparse-checkout. Collapsing the root into a sparse directory entry\n>> violates that assumption, as sparse directory entries are expected to be\n>> outside the sparse cone and have SKIP_WORKTREE enabled. This invalid state\n>> in turn causes issues with commands that interact with the index, e.g.\n>> 'git status'.\n>>\n>> Treating an empty (root) path as in-cone prevents the creation of a root\n>> sparse directory in 'convert_to_sparse(...)'. Because the repository root is\n>> otherwise never compared with sparse patterns (in both cone-mode and\n>> non-cone sparse-checkouts), the new check does not cause additional changes\n>> to how sparse patterns are applied.\n> \n> Good catch! I agree with everything said here.\n> \n>> +\tif (!strlen(path) ||\n> \n> Instead of a full strlen(), could we just check \"if (!*path ||\" to\n> look for the nul byte?\n> \n> Thanks,\n> -Stolee\n\nI'll reroll with corrected CC's & make the `strlen` -> `*path` change.\nThanks!\n"},{"id":"449516","messageId":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.git.1645640717.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Sparse index: integrate with 'read-tree'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:26Z","receivedAt":"2022-02-24T22:34:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Like previous integrations [1] [2], this series allows 'git read-tree' to\noperate using a sparse index.\n\nThe first two patches are bugfixes for issues found while implementing the\n'read-tree' integration:\n\n * The first (patch 1/7) fixes an edge case in which a repo with no in-cone\n files or directories would have its root collapsed into a sparse\n directory; the fix ensures the root directory is never collapsed into a\n sparse directory.\n * The second (patch 2/7) corrects the 'git status' reporting of changes\n nested inside the subdirectory of a sparse directory, ensuring that the\n modified file (not the subdirectory) is correctly reported as having\n changes.\n\nThe remainder of the series focuses on utilizing the sparse index in 'git\nread-tree'. After some baseline behavior-establishing tests (patch 3/7),\nsparse index usage is trivially enabled (patch 4/7) for 'read-tree' except:\n\n * usage with '--prefix'\n * two- and three-way merge\n\nThese cases require additional changes in order to work as expected (i.e.,\noutwardly matching non-sparse index sparse-checkout). For the former, the\nsparse index can be enabled as long as the index is expanded when the prefix\nis a directory outside the sparse cone (patch 5/7). For the latter, sparse\ndirectories that cannot be trivially merged must have their contents merged\nfile-by-file, done by recursively traversing the trees represented by the\nsparse directories (patches 6/7 & 7/7).\n\n\nChanges since V1\n================\n\n * switched an empty string check from '!strlen(path)' to the\n slightly-less-expensive '!*path'\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/\n[2]\nhttps://lore.kernel.org/git/pull.1048.v6.git.1638201164.gitgitgadget@gmail.com/\n\nVictoria Dye (7):\n sparse-index: prevent repo root from becoming sparse\n status: fix nested sparse directory diff in sparse index\n read-tree: expand sparse checkout test coverage\n read-tree: integrate with sparse index\n read-tree: narrow scope of index expansion for '--prefix'\n read-tree: make two-way merge sparse-aware\n read-tree: make three-way merge sparse-aware\n\n builtin/read-tree.c | 10 +-\n dir.c | 7 +-\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 129 +++++++++++++++++++++++\n unpack-trees.c | 121 ++++++++++++++++++++-\n wt-status.c | 9 ++\n 6 files changed, 268 insertions(+), 9 deletions(-)\n\n\nbase-commit: e6ebfd0e8cbbd10878070c8a356b5ad1b3ca464e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1157%2Fvdye%2Fsparse%2Fread-tree-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1157/vdye/sparse/read-tree-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1157\n\nRange-diff vs v1:\n\n 1: 90da1f9f33a ! 1: 744668eeece sparse-index: prevent repo root from becoming sparse\n @@ Commit message\n non-cone sparse-checkouts), the new check does not cause additional changes\n to how sparse patterns are applied.\n \n + Helped-by: Derrick Stolee \n Signed-off-by: Victoria Dye \n \n ## dir.c ##\n @@ dir.c: static int path_in_sparse_checkout_1(const char *path,\n +\t * patterns, or the patterns are of the wrong type.\n \t */\n -\tif (init_sparse_checkout_patterns(istate) ||\n -+\tif (!strlen(path) ||\n ++\tif (!*path ||\n +\t init_sparse_checkout_patterns(istate) ||\n \t (require_cone_mode &&\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n 2: c21c9b9be34 = 2: f0cff03b95d status: fix nested sparse directory diff in sparse index\n 3: ac42ae21d4a = 3: ffe0b6aff2b read-tree: expand sparse checkout test coverage\n 4: 5ee193bfa87 = 4: cb7e0cf419c read-tree: integrate with sparse index\n 5: bea482b6b28 = 5: 4f05fa70209 read-tree: narrow scope of index expansion for '--prefix'\n 6: 9fdcab038b2 = 6: 94c2aad2f93 read-tree: make two-way merge sparse-aware\n 7: 1502e9acb32 = 7: c4080e99d6e read-tree: make three-way merge sparse-aware\n\n-- \ngitgitgadget\n"},{"id":"449517","messageId":"744668eeece68a8cfe98b91c50d1cadb6988b993.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] sparse-index: prevent repo root from becoming sparse","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:27Z","receivedAt":"2022-02-24T22:34:45Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nPrevent the repository root from being collapsed into a sparse directory by\ntreating an empty path as \"inside the sparse-checkout\". When collapsing a\nsparse index (e.g. in 'git sparse-checkout reapply'), the root directory\ntypically could not become a sparse directory due to the presence of in-cone\nroot-level files and directories. However, if no such in-cone files or\ndirectories were present, there was no explicit check signaling that the\n\"repository root path\" (an empty string, in the case of\n'convert_to_sparse(...)') was in-cone, and a sparse directory index entry\nwould be created from the repository root directory.\n\nThe documentation in Documentation/git-sparse-checkout.txt explicitly states\nthat the files in the root directory are expected to be in-cone for a\ncone-mode sparse-checkout. Collapsing the root into a sparse directory entry\nviolates that assumption, as sparse directory entries are expected to be\noutside the sparse cone and have SKIP_WORKTREE enabled. This invalid state\nin turn causes issues with commands that interact with the index, e.g.\n'git status'.\n\nTreating an empty (root) path as in-cone prevents the creation of a root\nsparse directory in 'convert_to_sparse(...)'. Because the repository root is\notherwise never compared with sparse patterns (in both cone-mode and\nnon-cone sparse-checkouts), the new check does not cause additional changes\nto how sparse patterns are applied.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n dir.c | 7 ++++---\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d91295f2bcd..a136377eb49 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1463,10 +1463,11 @@ static int path_in_sparse_checkout_1(const char *path,\n \tconst char *end, *slash;\n \n \t/*\n-\t * We default to accepting a path if there are no patterns or\n-\t * they are of the wrong type.\n+\t * We default to accepting a path if the path is empty, there are no\n+\t * patterns, or the patterns are of the wrong type.\n \t */\n-\tif (init_sparse_checkout_patterns(istate) ||\n+\tif (!*path ||\n+\t init_sparse_checkout_patterns(istate) ||\n \t (require_cone_mode &&\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n \t\treturn 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f3a059e5af5..9ef7cd80885 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -244,6 +244,24 @@ test_expect_success 'expanded in-memory index matches full index' '\n \ttest_sparse_match git ls-files --stage\n '\n \n+test_expect_success 'root directory cannot be sparse' '\n+\tinit_repos &&\n+\n+\t# Remove all in-cone files and directories from the index, collapse index\n+\t# with `git sparse-checkout reapply`\n+\tgit -C sparse-index rm -r . &&\n+\tgit -C sparse-index sparse-checkout reapply &&\n+\n+\t# Verify sparse directories still present, root directory is not sparse\n+\tcat >expect <<-EOF &&\n+\tfolder1/\n+\tfolder2/\n+\tx/\n+\tEOF\n+\tgit -C sparse-index ls-files --sparse >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'status with options' '\n \tinit_repos &&\n \ttest_sparse_match ls &&\n-- \ngitgitgadget\n\n"},{"id":"449518","messageId":"f0cff03b95d574dff414a63325a0f1c6d2d1ff96.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] status: fix nested sparse directory diff in sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:28Z","receivedAt":"2022-02-24T22:34:46Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\nthe 'recursive' flag, 'git status' could report index changes incorrectly\nwhen the following conditions were met:\n\n* sparse index is enabled\n* there is a difference between the index and HEAD in a file inside a\n *subdirectory* of a sparse directory\n* the sparse directory index entry is *not* expanded in-core\n\nIn this scenario, 'git status' would not recurse into the sparse directory's\nsubdirectories to identify which file contained the difference between the\nindex and HEAD. Instead, it would report the immediate subdirectory itself\nas \"modified\".\n\nExample:\n\n$ git init\n$ mkdir -p sparse/sub\n$ echo test >sparse/sub/foo\n$ git add .\n$ git commit -m \"commit 1\"\n$ echo somethingelse >sparse/sub/foo\n$ git add .\n$ git commit -a -m \"commit 2\"\n$ git sparse-checkout set --cone --sparse-index 'sparse'\n$ git reset --soft HEAD~1\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub\n\nThe 'recursive' diff option in 'wt_status_collect_changes_index' corrects\nthis by indicating that 'git status' should recurse into sparse directories\nto find modified files. Given the same repository setup as the example\nabove, the corrected result of `git status` is:\n\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub/foo\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 7 +++++++\n wt-status.c | 9 +++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9ef7cd80885..b1dcaa0e642 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -278,6 +278,13 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status with diff in unexpanded sparse directory' '\n+\tinit_repos &&\n+\ttest_all_match git checkout rename-base &&\n+\ttest_all_match git reset --soft rename-out-to-out &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_expect_success 'status reports sparse-checkout' '\n \tinit_repos &&\n \tgit -C sparse-checkout status >full &&\ndiff --git a/wt-status.c b/wt-status.c\nindex 335e723a71e..4a5b9beeca1 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -651,6 +651,15 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trev.diffopt.detect_rename = s->detect_rename >= 0 ? s->detect_rename : rev.diffopt.detect_rename;\n \trev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;\n \trev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n+\n+\t/*\n+\t * The `recursive` option must be enabled to show differences in files\n+\t * *more than* one level deep in a sparse directory index entry (e.g., given\n+\t * sparse directory 'sparse-dir/', reporting a difference in the file\n+\t * 'sparse-dir/another-dir/my-file').\n+\t */\n+\trev.diffopt.flags.recursive = 1;\n+\n \tcopy_pathspec(&rev.prune_data, &s->pathspec);\n \trun_diff_index(&rev, 1);\n \tobject_array_clear(&rev.pending);\n-- \ngitgitgadget\n\n"},{"id":"449519","messageId":"4f05fa70209768fb20284bd4018a1364567cad5a.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:31Z","receivedAt":"2022-02-24T22:34:50Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nWhen 'git read-tree' is provided with a prefix, expand the index only if the\nprefix is equivalent to a sparse directory or contained within one. If the\nindex is not expanded in these cases, 'ce_in_traverse_path' will indicate\nthat the relevant sparse directory is not in the prefix/traverse path,\nskipping past it and not unpacking the appropriate tree(s).\n\nIf the prefix is in-cone, its sparse subdirectories (if any) will be\ntraversed correctly without index expansion.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 3 +--\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n unpack-trees.c | 30 ++++++++++++++++++++++++\n 3 files changed, 38 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex c2fdbc2657f..a7b7f822281 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n-\t/* TODO: audit sparse index behavior in unpack_trees */\n-\tif (opts.skip_sparse_checkout || opts.prefix)\n+\tif (opts.skip_sparse_checkout)\n \t\tensure_full_index(&the_index);\n \n \tif (opts.merge) {\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex ae44451a0a9..a404be0a10f 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\n-\tdone\n+\tdone &&\n+\n+\trm -rf sparse-index/deep/deeper2 &&\n+\tensure_not_expanded add . &&\n+\tensure_not_expanded commit -m \"test\" &&\n+\n+\tensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n '\n \n test_expect_success 'ls-files' '\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..dba122a02bb 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tsetup_standard_excludes(o->dir);\n \t}\n \n+\t/*\n+\t * If the prefix is equal to or contained within a sparse directory, the\n+\t * index needs to be expanded to traverse with the specified prefix.\n+\t */\n+\tif (o->prefix && o->src_index->sparse_index) {\n+\t\tint prefix_len = strlen(o->prefix);\n+\n+\t\twhile (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n+\t\t\tprefix_len--;\n+\n+\t\tif (prefix_len > 0) {\n+\t\t\tstruct strbuf ce_prefix = STRBUF_INIT;\n+\t\t\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n+\t\t\tstrbuf_add(&ce_prefix, o->prefix, prefix_len);\n+\t\t\tstrbuf_addch(&ce_prefix, '/');\n+\n+\t\t\t/*\n+\t\t\t * If the prefix is not inside the sparse cone, then the\n+\t\t\t * index is explicitly expanded if it is found as a sparse\n+\t\t\t * directory, or implicitly expanded (by 'index_name_pos')\n+\t\t\t * if the path is inside a sparse directory.\n+\t\t\t */\n+\t\t\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n+\t\t\t index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n+\t\t\t\tensure_full_index(o->src_index);\n+\n+\t\t\tstrbuf_release(&ce_prefix);\n+\t\t}\n+\t}\n+\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n-- \ngitgitgadget\n\n"},{"id":"449520","messageId":"ffe0b6aff2baee238f77ae57561a62ea5473321f.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:29Z","receivedAt":"2022-02-24T22:34:52Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\nemphasis is placed on interactions with files outside the sparse cone, e.g.\nmerges with out-of-cone conflicts.\n\nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 2a7106b9495..382716cfca9 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -117,6 +117,7 @@ test_perf_on_all git diff\n test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n+test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b1dcaa0e642..9d58da4e925 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n \ttest_cmp expect sparse-checkout-out\n '\n \n+test_expect_success 'read-tree --merge with files outside sparse definition' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-branch update-folder1 &&\n+\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n+\t\t\t \"update-folder1 update-folder2\" \\\n+\t\t\t \"update-folder2\"\n+\tdo\n+\t\t# Clean up and remove on-disk files\n+\t\ttest_all_match git reset --hard HEAD &&\n+\t\ttest_sparse_match git sparse-checkout reapply &&\n+\n+\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t\t# definition files will not exist on disk for sparse checkouts\n+\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_path_is_missing sparse-checkout/folder2 &&\n+\t\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\t\ttest_all_match git read-tree --reset -u HEAD &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\n+\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n+\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n+\tdone\n+'\n+\n+test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n+\tinit_repos &&\n+\n+\t# Merge of multiple changes to same directory (but not same files) should\n+\t# succeed\n+\ttest_all_match git read-tree -mu base rename-base update-folder1 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match git read-tree -mu rename-base update-folder2 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match test_must_fail git read-tree -mu base update-folder1 rename-out-to-in &&\n+\ttest_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n+'\n+\n+test_expect_success 'read-tree --prefix outside sparse definition' '\n+\tinit_repos &&\n+\n+\t# Cannot read-tree --prefix with a single argument when files exist within\n+\t# prefix\n+\ttest_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n+\n+\ttest_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n+\ttest_path_is_missing sparse-checkout/folder2 &&\n+\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\ttest_all_match git read-tree --reset -u HEAD &&\n+\ttest_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n+\ttest_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n+\ttest_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n+'\n+\n+test_expect_success 'read-tree --merge with directory-file conflicts' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-branch rename-base &&\n+\n+\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t# definition files will not exist on disk for sparse checkouts\n+\ttest_sparse_match git read-tree -mu rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_path_is_missing sparse-checkout/folder2 &&\n+\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\ttest_sparse_match git read-tree --reset -u HEAD &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\ttest_sparse_match git read-tree -mu --no-sparse-checkout rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_cmp sparse-checkout/folder2/0/1 sparse-index/folder2/0/1\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"449521","messageId":"c4080e99d6ee030d204653e1d6ddc926c7e70eda.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] read-tree: make three-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:33Z","receivedAt":"2022-02-24T22:34:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of 'merged_sparse_dir' in 'threeway_merge'. As with two-way\nmerge, the contents of each conflicted sparse directory are merged without\nreferencing the index, avoiding sparse index expansion.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 -----\n t/t1092-sparse-checkout-compatibility.sh | 3 ++-\n unpack-trees.c | 16 ++++++++++++----\n 3 files changed, 14 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 5a421de2629..dc2283fce53 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -230,11 +230,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n-\t\t\t/*\n-\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d6f19682d65..5447d314632 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1412,7 +1412,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n \tfor MERGE_TREES in \"update-folder2\" \\\n-\t\t\t \"base update-folder2\"\n+\t\t\t \"base update-folder2\" \\\n+\t\t\t \"base HEAD update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a4ace53904e..9e79cb13bd4 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2635,16 +2635,24 @@ int threeway_merge(const struct cache_entry * const *stages,\n \t */\n \t/* #14, #14ALT, #2ALT */\n \tif (remote && !df_conflict_head && head_match && !remote_match) {\n-\t\tif (index && !same(index, remote) && !same(index, head))\n-\t\t\treturn reject_merge(index, o);\n+\t\tif (index && !same(index, remote) && !same(index, head)) {\n+\t\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\t\telse\n+\t\t\t\treturn reject_merge(index, o);\n+\t\t}\n \t\treturn merged_entry(remote, index, o);\n \t}\n \t/*\n \t * If we have an entry in the index cache, then we want to\n \t * make sure that it matches head.\n \t */\n-\tif (index && !same(index, head))\n-\t\treturn reject_merge(index, o);\n+\tif (index && !same(index, head)) {\n+\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\telse\n+\t\t\treturn reject_merge(index, o);\n+\t}\n \n \tif (head) {\n \t\t/* #5ALT, #15 */\n-- \ngitgitgadget\n"},{"id":"449522","messageId":"cb7e0cf419c02dfe9689ed2f2f448fbb098c7d6b.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] read-tree: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:30Z","receivedAt":"2022-02-24T22:34:56Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of sparse index in 'git read-tree'. The integration in this patch\nis limited only to usage of 'read-tree' that does not need additional\nfunctional changes for the sparse index to behave as expected (i.e., produce\nthe same user-facing results as a non-sparse index sparse-checkout). To\nensure no unexpected behavior occurs, the index is explicitly expanded when:\n\n* '--no-sparse-checkout' is specified (because it disables sparse-checkout)\n* '--prefix' is specified (if the prefix is inside a sparse directory, the\n prefixed tree cannot be properly traversed)\n* two or more arguments are specified ('twoway_merge' and\n 'threeway_merge' do not yet support merging sparse directories)\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 21 +++++++++++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++++\n 2 files changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 2109c4c9e5c..c2fdbc2657f 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -160,8 +160,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \targc = parse_options(argc, argv, cmd_prefix, read_tree_options,\n \t\t\t read_tree_usage, 0);\n \n-\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n-\n \tprefix_set = opts.prefix ? 1 : 0;\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n@@ -169,6 +167,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.reset)\n \t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n+\n \t/*\n \t * NEEDSWORK\n \t *\n@@ -210,6 +213,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n+\t/* TODO: audit sparse index behavior in unpack_trees */\n+\tif (opts.skip_sparse_checkout || opts.prefix)\n+\t\tensure_full_index(&the_index);\n+\n \tif (opts.merge) {\n \t\tswitch (stage - 1) {\n \t\tcase 0:\n@@ -219,11 +226,21 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n+\t\t\t/*\n+\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n+\t\t\t/*\n+\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9d58da4e925..ae44451a0a9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1407,6 +1407,17 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'sparse index is not expanded: read-tree' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded checkout -b test-branch update-folder1 &&\n+\tfor MERGE_TREES in \"update-folder2\"\n+\tdo\n+\t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n+\t\tensure_not_expanded reset --hard HEAD || return 1\n+\tdone\n+'\n+\n test_expect_success 'ls-files' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"449523","messageId":"94c2aad2f93f61d1e7731017c720faebb7cd2467.1645742073.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] read-tree: make two-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T22:34:32Z","receivedAt":"2022-02-24T22:34:58Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable two-way merge with 'git read-tree' without expanding the sparse\nindex. When in a sparse index, a two-way merge will trivially succeed as\nlong as there are not changes to the same sparse directory in multiple trees\n(i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\nconflicts, the merge will fail despite the possibility that individual files\ncould merge cleanly.\n\nIn order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\ndirectories are - rather than rejected - merged by traversing their\nassociated trees by OID. For each child of the sparse directory:\n\n1. Files are merged as normal (see Documentation/git-read-tree.txt for\n details).\n2. Subdirectories are treated as sparse directories and merged in\n 'twoway_merge'. If there are no conflicts, they are merged according to\n the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n is recursively traversed and merged.\n\nThis process allows sparse directories to be individually merged at the\nnecessary depth *without* expanding a full index.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 --\n t/t1092-sparse-checkout-compatibility.sh | 3 +-\n unpack-trees.c | 75 ++++++++++++++++++++++++\n 3 files changed, 77 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex a7b7f822281..5a421de2629 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -225,11 +225,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n-\t\t\t/*\n-\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex a404be0a10f..d6f19682d65 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1411,7 +1411,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tinit_repos &&\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n-\tfor MERGE_TREES in \"update-folder2\"\n+\tfor MERGE_TREES in \"update-folder2\" \\\n+\t\t\t \"base update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard HEAD || return 1\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dba122a02bb..a4ace53904e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n \treturn sparse_dir_matches_path(ce, info, name);\n }\n \n+static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n+{\n+\tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n+\tstruct unpack_trees_options *o = info->data;\n+\tint ret;\n+\n+\tassert(o->merge);\n+\n+\t/*\n+\t * Unlike in 'unpack_callback', where src[0] is derived from the index when\n+\t * merging, src[0] is a transient cache entry derived from the first tree\n+\t * provided. Create the temporary entry as if it came from a non-sparse index.\n+\t */\n+\tif (!is_null_oid(&names[0].oid)) {\n+\t\tsrc[0] = create_ce_entry(info, &names[0], 0,\n+\t\t\t\t\t&o->result, 1,\n+\t\t\t\t\tdirmask & (1ul << 0));\n+\t\tsrc[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t}\n+\n+\t/*\n+\t * 'unpack_single_entry' assumes that src[0] is derived directly from\n+\t * the index, rather than from an entry in 'names'. This is *not* true when\n+\t * merging a sparse directory, in which case names[0] is the \"index\" source\n+\t * entry. To match the expectations of 'unpack_single_entry', shift past the\n+\t * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n+\t * 'dirmask' accordingly.\n+\t */\n+\tret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n+\n+\tif (src[0])\n+\t\tdiscard_cache_entry(src[0]);\n+\n+\treturn ret >= 0 ? mask : -1;\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -2464,6 +2500,37 @@ static int merged_entry(const struct cache_entry *ce,\n \treturn 1;\n }\n \n+static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\tstruct tree_desc t[MAX_UNPACK_TREES + 1];\n+\tvoid * tree_bufs[MAX_UNPACK_TREES + 1];\n+\tstruct traverse_info info;\n+\tint i, ret;\n+\n+\t/*\n+\t * Create the tree traversal information for traversing into *only* the\n+\t * sparse directory.\n+\t */\n+\tsetup_traverse_info(&info, src[0]->name);\n+\tinfo.fn = unpack_sparse_callback;\n+\tinfo.data = o;\n+\tinfo.show_all_errors = o->show_all_errors;\n+\tinfo.pathspec = o->pathspec;\n+\n+\t/* Get the tree descriptors of the sparse directory in each of the merging trees */\n+\tfor (i = 0; i < n; i++)\n+\t\ttree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n+\t\t\t\t\t\t src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n+\n+\tret = traverse_trees(o->src_index, n, t, &info);\n+\n+\tfor (i = 0; i < n; i++)\n+\t\tfree(tree_bufs[i]);\n+\n+\treturn ret;\n+}\n+\n static int deleted_entry(const struct cache_entry *ce,\n \t\t\t const struct cache_entry *old,\n \t\t\t struct unpack_trees_options *o)\n@@ -2734,6 +2801,14 @@ int twoway_merge(const struct cache_entry * const *src,\n \t\t\t * reject the merge instead.\n \t\t\t */\n \t\t\treturn merged_entry(newtree, current, o);\n+\t\t} else if (S_ISSPARSEDIR(current->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * The sparse directories differ, but we don't know whether that's\n+\t\t\t * because of two different files in the directory being modified\n+\t\t\t * (can be trivially merged) or if there is a real file conflict.\n+\t\t\t * Merge the sparse directory by OID to compare file-by-file.\n+\t\t\t */\n+\t\t\treturn merged_sparse_dir(src, 3, o);\n \t\t} else\n \t\t\treturn reject_merge(current, o);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"449552","messageId":"CABPp-BGLG15g1UYanaNy=zM320DEYWW52xKNRDA_87mcVXWhYQ@mail.gmail.com","threadId":"57464","inReplyTo":"f0cff03b95d574dff414a63325a0f1c6d2d1ff96.1645742073.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] status: fix nested sparse directory diff in sparse index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-25T07:45:40Z","receivedAt":"2022-02-25T07:45:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\n\nPerhaps \"Set the 'recursive' diff option flag in\n'wt_status_collect_changes_index(...)'\" ? There's no function\nargument named 'recursive' in wt_status_collect_changes_index() before\nor after your changes, which is what the wording led me to think of.\n\n> the 'recursive' flag, 'git status' could report index changes incorrectly\n> when the following conditions were met:\n>\n> * sparse index is enabled\n> * there is a difference between the index and HEAD in a file inside a\n> *subdirectory* of a sparse directory\n> * the sparse directory index entry is *not* expanded in-core\n>\n> In this scenario, 'git status' would not recurse into the sparse directory's\n> subdirectories to identify which file contained the difference between the\n> index and HEAD. Instead, it would report the immediate subdirectory itself\n> as \"modified\".\n>\n> Example:\n>\n> $ git init\n> $ mkdir -p sparse/sub\n> $ echo test >sparse/sub/foo\n> $ git add .\n> $ git commit -m \"commit 1\"\n> $ echo somethingelse >sparse/sub/foo\n> $ git add .\n> $ git commit -a -m \"commit 2\"\n> $ git sparse-checkout set --cone --sparse-index 'sparse'\n> $ git reset --soft HEAD~1\n> $ git status\n> On branch master\n> You are in a sparse checkout.\n>\n> Changes to be committed:\n> (use \"git restore --staged ...\" to unstage)\n> modified: sparse/sub\n>\n> The 'recursive' diff option in 'wt_status_collect_changes_index' corrects\n> this by indicating that 'git status' should recurse into sparse directories\n> to find modified files. Given the same repository setup as the example\n> above, the corrected result of `git status` is:\n>\n> $ git status\n> On branch master\n> You are in a sparse checkout.\n>\n> Changes to be committed:\n> (use \"git restore --staged ...\" to unstage)\n> modified: sparse/sub/foo\n>\n> Signed-off-by: Victoria Dye \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 7 +++++++\n> wt-status.c | 9 +++++++++\n> 2 files changed, 16 insertions(+)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 9ef7cd80885..b1dcaa0e642 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -278,6 +278,13 @@ test_expect_success 'status with options' '\n> test_all_match git status --porcelain=v2 -uno\n> '\n>\n> +test_expect_success 'status with diff in unexpanded sparse directory' '\n> + init_repos &&\n> + test_all_match git checkout rename-base &&\n> + test_all_match git reset --soft rename-out-to-out &&\n> + test_all_match git status --porcelain=v2\n> +'\n> +\n> test_expect_success 'status reports sparse-checkout' '\n> init_repos &&\n> git -C sparse-checkout status >full &&\n> diff --git a/wt-status.c b/wt-status.c\n> index 335e723a71e..4a5b9beeca1 100644\n> --- a/wt-status.c\n> +++ b/wt-status.c\n> @@ -651,6 +651,15 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n> rev.diffopt.detect_rename = s->detect_rename >= 0 ? s->detect_rename : rev.diffopt.detect_rename;\n> rev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;\n> rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n> +\n> + /*\n> + * The `recursive` option must be enabled to show differences in files\n> + * *more than* one level deep in a sparse directory index entry (e.g., given\n> + * sparse directory 'sparse-dir/', reporting a difference in the file\n> + * 'sparse-dir/another-dir/my-file').\n> + */\n> + rev.diffopt.flags.recursive = 1;\n\nKind of clever, and makes sense.\n\nI'm wondering if there's an alternate wording that might be helpful\nhere or in the commit message, that instead of just saying the\n'recursive' option is necessary, perhaps says a little bit about why\nit helps. In particular, the diff machinery, by default, is not\nrecursive and stops at comparing the first level of trees. (See e.g.\nthe -r option in diff-tree, it's just that it's turned on by default\nin 'git diff' and by the -p option in 'git log'.) I'm guessing the\nrecursive option never needed to be turned on previously within\nwt-status, due to something about the nature of the index only holding\nfiles previously. Now, however, the sparse index changes that. (And\nit also suggests that perhaps we should look to see if other commands\nrun the diff machinery without the recursive flag, and see if they\nneed it now due to sparse indices.)\n\nGranted, I'm not totally sure how to work these facts in (in part\nbecause I don't know how comparison to the index normally avoids the\nneed for the recursive flag), and maybe what you have is fine. Just\nthought I'd point it out since I wasn't aware of the non-recursive\nnature of the diff machinery until I started doing things with\ndiff-tree.\n\n\n> +\n> copy_pathspec(&rev.prune_data, &s->pathspec);\n> run_diff_index(&rev, 1);\n> object_array_clear(&rev.pending);\n> --\n> gitgitgadget\n"},{"id":"449555","messageId":"CABPp-BHx8gFh717bovO6wCo0RA058=YjNtHhRme+Rxh8GOnxbQ@mail.gmail.com","threadId":"57464","inReplyTo":"4f05fa70209768fb20284bd4018a1364567cad5a.1645742073.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-25T08:38:26Z","receivedAt":"2022-02-25T08:38:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> When 'git read-tree' is provided with a prefix, expand the index only if the\n> prefix is equivalent to a sparse directory or contained within one. If the\n> index is not expanded in these cases, 'ce_in_traverse_path' will indicate\n> that the relevant sparse directory is not in the prefix/traverse path,\n> skipping past it and not unpacking the appropriate tree(s).\n>\n> If the prefix is in-cone, its sparse subdirectories (if any) will be\n> traversed correctly without index expansion.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/read-tree.c | 3 +--\n> t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n> unpack-trees.c | 30 ++++++++++++++++++++++++\n> 3 files changed, 38 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index c2fdbc2657f..a7b7f822281 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> if (opts.merge && !opts.index_only)\n> setup_work_tree();\n>\n> - /* TODO: audit sparse index behavior in unpack_trees */\n> - if (opts.skip_sparse_checkout || opts.prefix)\n> + if (opts.skip_sparse_checkout)\n> ensure_full_index(&the_index);\n>\n> if (opts.merge) {\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index ae44451a0a9..a404be0a10f 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n> do\n> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> ensure_not_expanded reset --hard HEAD || return 1\n> - done\n> + done &&\n> +\n> + rm -rf sparse-index/deep/deeper2 &&\n> + ensure_not_expanded add . &&\n> + ensure_not_expanded commit -m \"test\" &&\n> +\n> + ensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n> '\n>\n> test_expect_success 'ls-files' '\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 360844bda3a..dba122a02bb 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> setup_standard_excludes(o->dir);\n> }\n>\n> + /*\n> + * If the prefix is equal to or contained within a sparse directory, the\n> + * index needs to be expanded to traverse with the specified prefix.\n> + */\n> + if (o->prefix && o->src_index->sparse_index) {\n> + int prefix_len = strlen(o->prefix);\n> +\n> + while (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n> + prefix_len--;\n> +\n> + if (prefix_len > 0) {\n\nIs this condition check necessary? If we want some safety check here,\ncould it instead be something like\n\n if (prefix_len <= 0)\n BUG(\"Broken prefix passed to unpack_trees\");\n\nand then dedent the following code? (Or are callers allowed to not\nsanitize their input before passing to unpack_trees(), meaning that we\nshould use a die() rather than a BUG()?)\n\nTo test this idea, near the top of unpack_trees(), I added:\n if (o->prefix)\n assert(*o->prefix && *o->prefix != '/');\nand reran all tests. They all ran without hitting that assertion. FWIW.\n\n> + struct strbuf ce_prefix = STRBUF_INIT;\n> + strbuf_grow(&ce_prefix, prefix_len + 1);\n> + strbuf_add(&ce_prefix, o->prefix, prefix_len);\n> + strbuf_addch(&ce_prefix, '/');\n> +\n> + /*\n> + * If the prefix is not inside the sparse cone, then the\n> + * index is explicitly expanded if it is found as a sparse\n> + * directory, or implicitly expanded (by 'index_name_pos')\n> + * if the path is inside a sparse directory.\n> + */\n> + if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n> + index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n\nstyle nit: Can you rewrap both the comments and the code at 80 characters?\n\nIt took me a bit of playing and testing to understand these two lines.\nThe comment helps, but it's still a bit dense to unpack; somehow I\ndidn't understand that the comment was referring to index_name_pos()'s\ncall to ensure_full_index(). Once I understood that, it all looks\ngood.\n\n\n> + ensure_full_index(o->src_index);\n> +\n> + strbuf_release(&ce_prefix);\n> + }\n> + }\n> +\n> if (!core_apply_sparse_checkout || !o->update)\n> o->skip_sparse_checkout = 1;\n> if (!o->skip_sparse_checkout && !o->pl) {\n> --\n> gitgitgadget\n"},{"id":"449634","messageId":"686e9c8e-7c5b-7cbc-4772-92b9754caa07@github.com","threadId":"57464","inReplyTo":"CABPp-BHx8gFh717bovO6wCo0RA058=YjNtHhRme+Rxh8GOnxbQ@mail.gmail.com","subject":"Re: [PATCH v2 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-25T20:25:04Z","receivedAt":"2022-02-25T20:25:09Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> When 'git read-tree' is provided with a prefix, expand the index only if the\n>> prefix is equivalent to a sparse directory or contained within one. If the\n>> index is not expanded in these cases, 'ce_in_traverse_path' will indicate\n>> that the relevant sparse directory is not in the prefix/traverse path,\n>> skipping past it and not unpacking the appropriate tree(s).\n>>\n>> If the prefix is in-cone, its sparse subdirectories (if any) will be\n>> traversed correctly without index expansion.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/read-tree.c | 3 +--\n>> t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n>> unpack-trees.c | 30 ++++++++++++++++++++++++\n>> 3 files changed, 38 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>> index c2fdbc2657f..a7b7f822281 100644\n>> --- a/builtin/read-tree.c\n>> +++ b/builtin/read-tree.c\n>> @@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>> if (opts.merge && !opts.index_only)\n>> setup_work_tree();\n>>\n>> - /* TODO: audit sparse index behavior in unpack_trees */\n>> - if (opts.skip_sparse_checkout || opts.prefix)\n>> + if (opts.skip_sparse_checkout)\n>> ensure_full_index(&the_index);\n>>\n>> if (opts.merge) {\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index ae44451a0a9..a404be0a10f 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n>> do\n>> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n>> ensure_not_expanded reset --hard HEAD || return 1\n>> - done\n>> + done &&\n>> +\n>> + rm -rf sparse-index/deep/deeper2 &&\n>> + ensure_not_expanded add . &&\n>> + ensure_not_expanded commit -m \"test\" &&\n>> +\n>> + ensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n>> '\n>>\n>> test_expect_success 'ls-files' '\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index 360844bda3a..dba122a02bb 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> setup_standard_excludes(o->dir);\n>> }\n>>\n>> + /*\n>> + * If the prefix is equal to or contained within a sparse directory, the\n>> + * index needs to be expanded to traverse with the specified prefix.\n>> + */\n>> + if (o->prefix && o->src_index->sparse_index) {\n>> + int prefix_len = strlen(o->prefix);\n>> +\n>> + while (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n>> + prefix_len--;\n>> +\n>> + if (prefix_len > 0) {\n> \n> Is this condition check necessary? If we want some safety check here,\n> could it instead be something like\n> \n> if (prefix_len <= 0)\n> BUG(\"Broken prefix passed to unpack_trees\");\n> \n\nThis condition was intended to skip unnecessary computation for the\n(probably unlikely, but still technically valid) case where the prefix is\nthe repo root (e.g., '--prefix=/') - because the repo root is represented\nwith only directory separator(s), `prefix_len` would be 0 after removing\ntrailing '/'. In that scenario, the index won't need to be expanded, so we\ndon't need to go looking in the index for that path. \n\nNone of that is particularly clear from reading the patch, though, so I'll\nadd a comment & test covering it explicitly.\n\n> and then dedent the following code? (Or are callers allowed to not\n> sanitize their input before passing to unpack_trees(), meaning that we\n> should use a die() rather than a BUG()?)\n> \n> To test this idea, near the top of unpack_trees(), I added:\n> if (o->prefix)\n> assert(*o->prefix && *o->prefix != '/');\n> and reran all tests. They all ran without hitting that assertion. FWIW.\n> \n>> + struct strbuf ce_prefix = STRBUF_INIT;\n>> + strbuf_grow(&ce_prefix, prefix_len + 1);\n>> + strbuf_add(&ce_prefix, o->prefix, prefix_len);\n>> + strbuf_addch(&ce_prefix, '/');\n>> +\n>> + /*\n>> + * If the prefix is not inside the sparse cone, then the\n>> + * index is explicitly expanded if it is found as a sparse\n>> + * directory, or implicitly expanded (by 'index_name_pos')\n>> + * if the path is inside a sparse directory.\n>> + */\n>> + if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n>> + index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n> \n> style nit: Can you rewrap both the comments and the code at 80 characters?\n> \n\nI couldn't think of a way to wrap the condition that wouldn't make it more\ndifficult to read. The best I could come up with was:\n\n\t\t\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, \n\t\t\t\t\t\t\t o->src_index) &&\n\t\t\t index_name_pos(o->src_index, \n\t\t\t\t\t ce_prefix.buf, \n\t\t\t\t\t ce_prefix.len) >= 0)\n\t\t\t\tensure_full_index(o->src_index);\n\n\nwhich, to me, is a bit hard to parse. Alternatively, though, I can move the\nprefix-checking logic into its own function (kind of like\n'pathspec_needs_expanded_index(...)' in [1]), in which case I won't need to\nchange the current wrapping to keep it under 80 characters.\n\n[1] https://lore.kernel.org/git/822d7344587f698e73abba1ca726c3a905f7b403.1638201164.git.gitgitgadget@gmail.com/\n\n> It took me a bit of playing and testing to understand these two lines.\n> The comment helps, but it's still a bit dense to unpack; somehow I\n> didn't understand that the comment was referring to index_name_pos()'s\n> call to ensure_full_index(). Once I understood that, it all looks\n> good.\n> \n\nSorry about that, I'll revise to make that clearer.\n\n> \n>> + ensure_full_index(o->src_index);\n>> +\n>> + strbuf_release(&ce_prefix);\n>> + }\n>> + }\n>> +\n>> if (!core_apply_sparse_checkout || !o->update)\n>> o->skip_sparse_checkout = 1;\n>> if (!o->skip_sparse_checkout && !o->pl) {\n>> --\n>> gitgitgadget\n\n"},{"id":"449661","messageId":"CABPp-BE6n2-v7-91b-aXOsMod6d3dA4zZE4nNyzxac+tVpaaCw@mail.gmail.com","threadId":"57464","inReplyTo":"686e9c8e-7c5b-7cbc-4772-92b9754caa07@github.com","subject":"Re: [PATCH v2 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T07:52:25Z","receivedAt":"2022-02-26T07:52:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Victoria,\n\nOn Fri, Feb 25, 2022 at 12:25 PM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n> >> When 'git read-tree' is provided with a prefix, expand the index only if the\n> >> prefix is equivalent to a sparse directory or contained within one. If the\n> >> index is not expanded in these cases, 'ce_in_traverse_path' will indicate\n> >> that the relevant sparse directory is not in the prefix/traverse path,\n> >> skipping past it and not unpacking the appropriate tree(s).\n> >>\n> >> If the prefix is in-cone, its sparse subdirectories (if any) will be\n> >> traversed correctly without index expansion.\n> >>\n> >> Signed-off-by: Victoria Dye \n> >> ---\n> >> builtin/read-tree.c | 3 +--\n> >> t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n> >> unpack-trees.c | 30 ++++++++++++++++++++++++\n> >> 3 files changed, 38 insertions(+), 3 deletions(-)\n> >>\n> >> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> >> index c2fdbc2657f..a7b7f822281 100644\n> >> --- a/builtin/read-tree.c\n> >> +++ b/builtin/read-tree.c\n> >> @@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> >> if (opts.merge && !opts.index_only)\n> >> setup_work_tree();\n> >>\n> >> - /* TODO: audit sparse index behavior in unpack_trees */\n> >> - if (opts.skip_sparse_checkout || opts.prefix)\n> >> + if (opts.skip_sparse_checkout)\n> >> ensure_full_index(&the_index);\n> >>\n> >> if (opts.merge) {\n> >> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> >> index ae44451a0a9..a404be0a10f 100755\n> >> --- a/t/t1092-sparse-checkout-compatibility.sh\n> >> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> >> @@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n> >> do\n> >> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> >> ensure_not_expanded reset --hard HEAD || return 1\n> >> - done\n> >> + done &&\n> >> +\n> >> + rm -rf sparse-index/deep/deeper2 &&\n> >> + ensure_not_expanded add . &&\n> >> + ensure_not_expanded commit -m \"test\" &&\n> >> +\n> >> + ensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n> >> '\n> >>\n> >> test_expect_success 'ls-files' '\n> >> diff --git a/unpack-trees.c b/unpack-trees.c\n> >> index 360844bda3a..dba122a02bb 100644\n> >> --- a/unpack-trees.c\n> >> +++ b/unpack-trees.c\n> >> @@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> >> setup_standard_excludes(o->dir);\n> >> }\n> >>\n> >> + /*\n> >> + * If the prefix is equal to or contained within a sparse directory, the\n> >> + * index needs to be expanded to traverse with the specified prefix.\n> >> + */\n> >> + if (o->prefix && o->src_index->sparse_index) {\n> >> + int prefix_len = strlen(o->prefix);\n> >> +\n> >> + while (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n> >> + prefix_len--;\n> >> +\n> >> + if (prefix_len > 0) {\n> >\n> > Is this condition check necessary? If we want some safety check here,\n> > could it instead be something like\n> >\n> > if (prefix_len <= 0)\n> > BUG(\"Broken prefix passed to unpack_trees\");\n> >\n>\n> This condition was intended to skip unnecessary computation for the\n> (probably unlikely, but still technically valid) case where the prefix is\n> the repo root (e.g., '--prefix=/') - because the repo root is represented\n> with only directory separator(s), `prefix_len` would be 0 after removing\n> trailing '/'. In that scenario, the index won't need to be expanded, so we\n> don't need to go looking in the index for that path.\n\nWow, that doesn't result in an error?!? That surprises me. I never\neven thought to test such a thing. Clearly, the following related\ncommand does give an error:\n\n git read-tree --prefix=/toplevel/ $TREE\n\n(namely, \"error: invalid path '/toplevel/subpath'\")\n\nwhereas, the reason a single slash is accepted it because git is\ntrying to be forgiving and treat the following two commands the same:\n\n git read-tree --prefix=subdir/ $TREE\n git read-tree --prefix=subdir $TREE\n\ni.e. it tries to allow the trailing slash to be optional. And, by\nimplementation quirk, making a trailing slash be optional turns out to\nmean that --prefix=/ is treated the same as no prefix at all, because\nempty string prefix just happens to give the same behavior as NULL\nprefix.\n\nI think we should just throw an error if prefix starts with '/'.\nunpack_trees() can make it be a BUG() (and at the beginning of the\nfunction rather than down at this point and only inside some\nconditional). builtin/read-tree.c, the only thing that ever sets\nprefix in unpack_trees_options, should die() if it's passed something\nthat starts with a '/'. Having paths start with a '/' is antithetical\nto how \"prefix\" is used throughout the codebase, though I guess I can\nsee people making that mistake if they are used to gitignore-style\npatterns instead.\n\n> None of that is particularly clear from reading the patch, though, so I'll\n> add a comment & test covering it explicitly.\n>\n> > and then dedent the following code? (Or are callers allowed to not\n> > sanitize their input before passing to unpack_trees(), meaning that we\n> > should use a die() rather than a BUG()?)\n> >\n> > To test this idea, near the top of unpack_trees(), I added:\n> > if (o->prefix)\n> > assert(*o->prefix && *o->prefix != '/');\n> > and reran all tests. They all ran without hitting that assertion. FWIW.\n> >\n> >> + struct strbuf ce_prefix = STRBUF_INIT;\n> >> + strbuf_grow(&ce_prefix, prefix_len + 1);\n> >> + strbuf_add(&ce_prefix, o->prefix, prefix_len);\n> >> + strbuf_addch(&ce_prefix, '/');\n> >> +\n> >> + /*\n> >> + * If the prefix is not inside the sparse cone, then the\n> >> + * index is explicitly expanded if it is found as a sparse\n> >> + * directory, or implicitly expanded (by 'index_name_pos')\n> >> + * if the path is inside a sparse directory.\n> >> + */\n> >> + if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n> >> + index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n> >\n> > style nit: Can you rewrap both the comments and the code at 80 characters?\n> >\n>\n> I couldn't think of a way to wrap the condition that wouldn't make it more\n> difficult to read. The best I could come up with was:\n>\n> if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf,\n> o->src_index) &&\n> index_name_pos(o->src_index,\n> ce_prefix.buf,\n> ce_prefix.len) >= 0)\n> ensure_full_index(o->src_index);\n>\n>\n> which, to me, is a bit hard to parse. Alternatively, though, I can move the\n> prefix-checking logic into its own function (kind of like\n> 'pathspec_needs_expanded_index(...)' in [1]), in which case I won't need to\n> change the current wrapping to keep it under 80 characters.\n>\n> [1] https://lore.kernel.org/git/822d7344587f698e73abba1ca726c3a905f7b403.1638201164.git.gitgitgadget@gmail.com/\n>\n> > It took me a bit of playing and testing to understand these two lines.\n> > The comment helps, but it's still a bit dense to unpack; somehow I\n> > didn't understand that the comment was referring to index_name_pos()'s\n> > call to ensure_full_index(). Once I understood that, it all looks\n> > good.\n> >\n>\n> Sorry about that, I'll revise to make that clearer.\n\nThanks. :-)\n\n> >\n> >> + ensure_full_index(o->src_index);\n> >> +\n> >> + strbuf_release(&ce_prefix);\n> >> + }\n> >> + }\n> >> +\n> >> if (!core_apply_sparse_checkout || !o->update)\n> >> o->skip_sparse_checkout = 1;\n> >> if (!o->skip_sparse_checkout && !o->pl) {\n> >> --\n> >> gitgitgadget\n"},{"id":"449662","messageId":"CABPp-BHihsVQZWTE4ppOcFyk8-eVa+zZ1MhkssiTByxjPO4kcg@mail.gmail.com","threadId":"57464","inReplyTo":"9fdcab038b2962b7f954363e32d04591476cf219.1645640717.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/7] read-tree: make two-way merge sparse-aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T08:05:01Z","receivedAt":"2022-02-26T08:05:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Feb 23, 2022 at 4:09 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Enable two-way merge with 'git read-tree' without expanding the sparse\n> index. When in a sparse index, a two-way merge will trivially succeed as\n> long as there are not changes to the same sparse directory in multiple trees\n> (i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\n> conflicts, the merge will fail despite the possibility that individual files\n> could merge cleanly.\n>\n> In order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\n> directories are - rather than rejected - merged by traversing their\n> associated trees by OID. For each child of the sparse directory:\n>\n> 1. Files are merged as normal (see Documentation/git-read-tree.txt for\n> details).\n> 2. Subdirectories are treated as sparse directories and merged in\n> 'twoway_merge'. If there are no conflicts, they are merged according to\n> the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n> is recursively traversed and merged.\n>\n> This process allows sparse directories to be individually merged at the\n> necessary depth *without* expanding a full index.\n\nThe idea of merging directory-level entries turns out to be\nproblematic _if_ rename detection is involved, but read-tree-style\nmerges are only trivial merges that ignore rename detection. As such,\nthis idea is perfectly reasonable, and is a good way to go. Nicely\ndone.\n\nMostly the patch looks good. There's one thing I'm wondering about, though...\n\n>\n> Signed-off-by: Victoria Dye \n> ---\n> builtin/read-tree.c | 5 --\n> t/t1092-sparse-checkout-compatibility.sh | 3 +-\n> unpack-trees.c | 75 ++++++++++++++++++++++++\n> 3 files changed, 77 insertions(+), 6 deletions(-)\n>\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index a7b7f822281..5a421de2629 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -225,11 +225,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> opts.fn = opts.prefix ? bind_merge : oneway_merge;\n> break;\n> case 2:\n> - /*\n> - * TODO: update twoway_merge to handle edit/edit conflicts in\n> - * sparse directories.\n> - */\n> - ensure_full_index(&the_index);\n> opts.fn = twoway_merge;\n> opts.initial_checkout = is_cache_unborn();\n> break;\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index a404be0a10f..d6f19682d65 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -1411,7 +1411,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n> init_repos &&\n>\n> ensure_not_expanded checkout -b test-branch update-folder1 &&\n> - for MERGE_TREES in \"update-folder2\"\n> + for MERGE_TREES in \"update-folder2\" \\\n> + \"base update-folder2\"\n> do\n> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> ensure_not_expanded reset --hard HEAD || return 1\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index dba122a02bb..a4ace53904e 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n> return sparse_dir_matches_path(ce, info, name);\n> }\n>\n> +static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n> +{\n> + struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> + struct unpack_trees_options *o = info->data;\n> + int ret;\n> +\n> + assert(o->merge);\n> +\n> + /*\n> + * Unlike in 'unpack_callback', where src[0] is derived from the index when\n> + * merging, src[0] is a transient cache entry derived from the first tree\n> + * provided. Create the temporary entry as if it came from a non-sparse index.\n> + */\n> + if (!is_null_oid(&names[0].oid)) {\n> + src[0] = create_ce_entry(info, &names[0], 0,\n> + &o->result, 1,\n> + dirmask & (1ul << 0));\n> + src[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n> + }\n> +\n> + /*\n> + * 'unpack_single_entry' assumes that src[0] is derived directly from\n> + * the index, rather than from an entry in 'names'. This is *not* true when\n> + * merging a sparse directory, in which case names[0] is the \"index\" source\n> + * entry. To match the expectations of 'unpack_single_entry', shift past the\n> + * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n> + * 'dirmask' accordingly.\n> + */\n> + ret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n\nSo, you're passing one less entry to unpack_single_entry() when you've\ntraversed into a sparse directory...won't the traversal at the next\nsubdirectory deeper then also pass one less entry to\nunpack_single_entry(), so after recursing a directory or two, you only\nhave one directory left and it won't conflict with anything so it just\nuses that remaining tree? (Or maybe it passes the wrong number of\narguments into twoway_merge()?) Did I miss something in the logic\nsomewhere that avoids that issue? It'd be nice to test it out, which\nbrings me to...\n\n> +\n> + if (src[0])\n> + discard_cache_entry(src[0]);\n> +\n> + return ret >= 0 ? mask : -1;\n> +}\n> +\n> /*\n> * Note that traverse_by_cache_tree() duplicates some logic in this function\n> * without actually calling it. If you change the logic here you may need to\n> @@ -2464,6 +2500,37 @@ static int merged_entry(const struct cache_entry *ce,\n> return 1;\n> }\n>\n> +static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n> + struct unpack_trees_options *o)\n> +{\n> + struct tree_desc t[MAX_UNPACK_TREES + 1];\n> + void * tree_bufs[MAX_UNPACK_TREES + 1];\n> + struct traverse_info info;\n> + int i, ret;\n> +\n> + /*\n> + * Create the tree traversal information for traversing into *only* the\n> + * sparse directory.\n> + */\n> + setup_traverse_info(&info, src[0]->name);\n> + info.fn = unpack_sparse_callback;\n> + info.data = o;\n> + info.show_all_errors = o->show_all_errors;\n> + info.pathspec = o->pathspec;\n> +\n> + /* Get the tree descriptors of the sparse directory in each of the merging trees */\n> + for (i = 0; i < n; i++)\n> + tree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n> + src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n> +\n> + ret = traverse_trees(o->src_index, n, t, &info);\n> +\n> + for (i = 0; i < n; i++)\n> + free(tree_bufs[i]);\n> +\n> + return ret;\n> +}\n> +\n> static int deleted_entry(const struct cache_entry *ce,\n> const struct cache_entry *old,\n> struct unpack_trees_options *o)\n> @@ -2734,6 +2801,14 @@ int twoway_merge(const struct cache_entry * const *src,\n> * reject the merge instead.\n> */\n> return merged_entry(newtree, current, o);\n> + } else if (S_ISSPARSEDIR(current->ce_mode)) {\n> + /*\n> + * The sparse directories differ, but we don't know whether that's\n> + * because of two different files in the directory being modified\n> + * (can be trivially merged) or if there is a real file conflict.\n> + * Merge the sparse directory by OID to compare file-by-file.\n> + */\n> + return merged_sparse_dir(src, 3, o);\n> } else\n> return reject_merge(current, o);\n> }\n> --\n> gitgitgadget\n\nIt would be nice to have a couple of tests. In particular, one\ndesigned to see what happens when we need to traverse into\nsubdirectories of sparse directory entries and paths different between\nthe two trees being merged.\n"},{"id":"449663","messageId":"CABPp-BF4jtEzyUBEymhFJxaGOMRaUs-XyBdr89Fs7E_ehnwWMg@mail.gmail.com","threadId":"57464","inReplyTo":"ffe0b6aff2baee238f77ae57561a62ea5473321f.1645742073.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T08:41:28Z","receivedAt":"2022-02-26T08:41:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> From: Victoria Dye \n>\n> Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\n> emphasis is placed on interactions with files outside the sparse cone, e.g.\n> merges with out-of-cone conflicts.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n> 2 files changed, 86 insertions(+)\n>\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index 2a7106b9495..382716cfca9 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -117,6 +117,7 @@ test_perf_on_all git diff\n> test_perf_on_all git diff --cached\n> test_perf_on_all git blame $SPARSE_CONE/a\n> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> +test_perf_on_all git read-tree -mu HEAD\n> test_perf_on_all git checkout-index -f --all\n> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index b1dcaa0e642..9d58da4e925 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n> test_cmp expect sparse-checkout-out\n> '\n>\n> +test_expect_success 'read-tree --merge with files outside sparse definition' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b test-branch update-folder1 &&\n> + for MERGE_TREES in \"base HEAD update-folder2\" \\\n> + \"update-folder1 update-folder2\" \\\n> + \"update-folder2\"\n> + do\n> + # Clean up and remove on-disk files\n> + test_all_match git reset --hard HEAD &&\n> + test_sparse_match git sparse-checkout reapply &&\n> +\n> + # Although the index matches, without --no-sparse-checkout, outside-of-\n> + # definition files will not exist on disk for sparse checkouts\n> + test_all_match git read-tree -mu $MERGE_TREES &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_path_is_missing sparse-checkout/folder2 &&\n> + test_path_is_missing sparse-index/folder2 &&\n> +\n> + test_all_match git read-tree --reset -u HEAD &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n> + test_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n> + done\n> +'\n> +\n> +test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n> + init_repos &&\n> +\n> + # Merge of multiple changes to same directory (but not same files) should\n> + # succeed\n> + test_all_match git read-tree -mu base rename-base update-folder1 &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset --hard &&\n> +\n> + test_all_match git read-tree -mu rename-base update-folder2 &&\n> + test_all_match git status --porcelain=v2 &&\n> +\n> + test_all_match git reset --hard &&\n> +\n> + test_all_match test_must_fail git read-tree -mu base update-folder1 rename-out-to-in &&\n> + test_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n> +'\n> +\n> +test_expect_success 'read-tree --prefix outside sparse definition' '\n> + init_repos &&\n> +\n> + # Cannot read-tree --prefix with a single argument when files exist within\n> + # prefix\n\nGiven the comments in the cover letter about --prefix needing special\nwork, it's not clear to me whether the below is expected behavior or\ncurrent-but-buggy behavior that you are testing and documenting.\nCould you clarify?\n\n> + test_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n> +\n> + test_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n> + test_path_is_missing sparse-checkout/folder2 &&\n> + test_path_is_missing sparse-index/folder2 &&\n> +\n> + test_all_match git read-tree --reset -u HEAD &&\n> + test_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n> + test_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n> + test_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n> +'\n> +\n> +test_expect_success 'read-tree --merge with directory-file conflicts' '\n> + init_repos &&\n> +\n> + test_all_match git checkout -b test-branch rename-base &&\n> +\n> + # Although the index matches, without --no-sparse-checkout, outside-of-\n> + # definition files will not exist on disk for sparse checkouts\n> + test_sparse_match git read-tree -mu rename-out-to-out &&\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_path_is_missing sparse-checkout/folder2 &&\n> + test_path_is_missing sparse-index/folder2 &&\n> +\n> + test_sparse_match git read-tree --reset -u HEAD &&\n> + test_sparse_match git status --porcelain=v2 &&\n> +\n> + test_sparse_match git read-tree -mu --no-sparse-checkout rename-out-to-out &&\n> + test_sparse_match git status --porcelain=v2 &&\n> + test_cmp sparse-checkout/folder2/0/1 sparse-index/folder2/0/1\n> +'\n> +\n> test_expect_success 'merge, cherry-pick, and rebase' '\n> init_repos &&\n>\n> --\n> gitgitgadget\n>\n"},{"id":"449664","messageId":"CABPp-BHu2sDFiDU=xGEr-MzqbtGwcwu+_m3b3gMWri561JyqWA@mail.gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/7] Sparse index: integrate with 'read-tree'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T08:46:50Z","receivedAt":"2022-02-26T08:47:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> Like previous integrations [1] [2], this series allows 'git read-tree' to\n> operate using a sparse index.\n>\n> The first two patches are bugfixes for issues found while implementing the\n> 'read-tree' integration:\n>\n> * The first (patch 1/7) fixes an edge case in which a repo with no in-cone\n> files or directories would have its root collapsed into a sparse\n> directory; the fix ensures the root directory is never collapsed into a\n> sparse directory.\n> * The second (patch 2/7) corrects the 'git status' reporting of changes\n> nested inside the subdirectory of a sparse directory, ensuring that the\n> modified file (not the subdirectory) is correctly reported as having\n> changes.\n>\n> The remainder of the series focuses on utilizing the sparse index in 'git\n> read-tree'. After some baseline behavior-establishing tests (patch 3/7),\n> sparse index usage is trivially enabled (patch 4/7) for 'read-tree' except:\n>\n> * usage with '--prefix'\n> * two- and three-way merge\n>\n> These cases require additional changes in order to work as expected (i.e.,\n> outwardly matching non-sparse index sparse-checkout). For the former, the\n> sparse index can be enabled as long as the index is expanded when the prefix\n> is a directory outside the sparse cone (patch 5/7). For the latter, sparse\n> directories that cannot be trivially merged must have their contents merged\n> file-by-file, done by recursively traversing the trees represented by the\n> sparse directories (patches 6/7 & 7/7).\n>\n>\n> Changes since V1\n> ================\n>\n> * switched an empty string check from '!strlen(path)' to the\n> slightly-less-expensive '!*path'\n\nI've read over the series. It was a nice read, well motivated, and\nsplit up rather nicely. I only had a few small commetns.\n\nI think it'd be nice to insert another patch into the series that\nthrows an error if the argument to --prefix starts with a '/'. That\nwould also allow you to simplify patch 5/7 a little.\n\nPatch 6/7 has the right idea, but I'm worried about one part of it; a\ntest would go a long way towards verifying whether that aspect is\nhandled correctly or whether my concern is warranted.\n\nI had a couple smaller comments on some of the other patches.\n\nOverall, nicely done.\n\n\n> Thanks!\n>\n> * Victoria\n>\n> [1]\n> https://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/\n> [2]\n> https://lore.kernel.org/git/pull.1048.v6.git.1638201164.gitgitgadget@gmail.com/\n>\n> Victoria Dye (7):\n> sparse-index: prevent repo root from becoming sparse\n> status: fix nested sparse directory diff in sparse index\n> read-tree: expand sparse checkout test coverage\n> read-tree: integrate with sparse index\n> read-tree: narrow scope of index expansion for '--prefix'\n> read-tree: make two-way merge sparse-aware\n> read-tree: make three-way merge sparse-aware\n>\n> builtin/read-tree.c | 10 +-\n> dir.c | 7 +-\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1092-sparse-checkout-compatibility.sh | 129 +++++++++++++++++++++++\n> unpack-trees.c | 121 ++++++++++++++++++++-\n> wt-status.c | 9 ++\n> 6 files changed, 268 insertions(+), 9 deletions(-)\n>\n>\n> base-commit: e6ebfd0e8cbbd10878070c8a356b5ad1b3ca464e\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1157%2Fvdye%2Fsparse%2Fread-tree-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1157/vdye/sparse/read-tree-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1157\n>\n> Range-diff vs v1:\n>\n> 1: 90da1f9f33a ! 1: 744668eeece sparse-index: prevent repo root from becoming sparse\n> @@ Commit message\n> non-cone sparse-checkouts), the new check does not cause additional changes\n> to how sparse patterns are applied.\n>\n> + Helped-by: Derrick Stolee \n> Signed-off-by: Victoria Dye \n>\n> ## dir.c ##\n> @@ dir.c: static int path_in_sparse_checkout_1(const char *path,\n> + * patterns, or the patterns are of the wrong type.\n> */\n> - if (init_sparse_checkout_patterns(istate) ||\n> -+ if (!strlen(path) ||\n> ++ if (!*path ||\n> + init_sparse_checkout_patterns(istate) ||\n> (require_cone_mode &&\n> !istate->sparse_checkout_patterns->use_cone_patterns))\n> 2: c21c9b9be34 = 2: f0cff03b95d status: fix nested sparse directory diff in sparse index\n> 3: ac42ae21d4a = 3: ffe0b6aff2b read-tree: expand sparse checkout test coverage\n> 4: 5ee193bfa87 = 4: cb7e0cf419c read-tree: integrate with sparse index\n> 5: bea482b6b28 = 5: 4f05fa70209 read-tree: narrow scope of index expansion for '--prefix'\n> 6: 9fdcab038b2 = 6: 94c2aad2f93 read-tree: make two-way merge sparse-aware\n> 7: 1502e9acb32 = 7: c4080e99d6e read-tree: make three-way merge sparse-aware\n>\n> --\n> gitgitgadget\n"},{"id":"449793","messageId":"dc47f12b-8724-22ef-ed2c-096badfafd76@github.com","threadId":"57464","inReplyTo":"CABPp-BHihsVQZWTE4ppOcFyk8-eVa+zZ1MhkssiTByxjPO4kcg@mail.gmail.com","subject":"Re: [PATCH 6/7] read-tree: make two-way merge sparse-aware","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-28T18:04:41Z","receivedAt":"2022-02-28T18:24:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Wed, Feb 23, 2022 at 4:09 PM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Enable two-way merge with 'git read-tree' without expanding the sparse\n>> index. When in a sparse index, a two-way merge will trivially succeed as\n>> long as there are not changes to the same sparse directory in multiple trees\n>> (i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\n>> conflicts, the merge will fail despite the possibility that individual files\n>> could merge cleanly.\n>>\n>> In order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\n>> directories are - rather than rejected - merged by traversing their\n>> associated trees by OID. For each child of the sparse directory:\n>>\n>> 1. Files are merged as normal (see Documentation/git-read-tree.txt for\n>> details).\n>> 2. Subdirectories are treated as sparse directories and merged in\n>> 'twoway_merge'. If there are no conflicts, they are merged according to\n>> the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n>> is recursively traversed and merged.\n>>\n>> This process allows sparse directories to be individually merged at the\n>> necessary depth *without* expanding a full index.\n> \n> The idea of merging directory-level entries turns out to be\n> problematic _if_ rename detection is involved, but read-tree-style\n> merges are only trivial merges that ignore rename detection. As such,\n> this idea is perfectly reasonable, and is a good way to go. Nicely\n> done.\n> \n> Mostly the patch looks good. There's one thing I'm wondering about, though...\n> \n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> builtin/read-tree.c | 5 --\n>> t/t1092-sparse-checkout-compatibility.sh | 3 +-\n>> unpack-trees.c | 75 ++++++++++++++++++++++++\n>> 3 files changed, 77 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>> index a7b7f822281..5a421de2629 100644\n>> --- a/builtin/read-tree.c\n>> +++ b/builtin/read-tree.c\n>> @@ -225,11 +225,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>> opts.fn = opts.prefix ? bind_merge : oneway_merge;\n>> break;\n>> case 2:\n>> - /*\n>> - * TODO: update twoway_merge to handle edit/edit conflicts in\n>> - * sparse directories.\n>> - */\n>> - ensure_full_index(&the_index);\n>> opts.fn = twoway_merge;\n>> opts.initial_checkout = is_cache_unborn();\n>> break;\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index a404be0a10f..d6f19682d65 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -1411,7 +1411,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n>> init_repos &&\n>>\n>> ensure_not_expanded checkout -b test-branch update-folder1 &&\n>> - for MERGE_TREES in \"update-folder2\"\n>> + for MERGE_TREES in \"update-folder2\" \\\n>> + \"base update-folder2\"\n>> do\n>> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n>> ensure_not_expanded reset --hard HEAD || return 1\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index dba122a02bb..a4ace53904e 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n>> return sparse_dir_matches_path(ce, info, name);\n>> }\n>>\n>> +static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n>> +{\n>> + struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n>> + struct unpack_trees_options *o = info->data;\n>> + int ret;\n>> +\n>> + assert(o->merge);\n>> +\n>> + /*\n>> + * Unlike in 'unpack_callback', where src[0] is derived from the index when\n>> + * merging, src[0] is a transient cache entry derived from the first tree\n>> + * provided. Create the temporary entry as if it came from a non-sparse index.\n>> + */\n>> + if (!is_null_oid(&names[0].oid)) {\n>> + src[0] = create_ce_entry(info, &names[0], 0,\n>> + &o->result, 1,\n>> + dirmask & (1ul << 0));\n>> + src[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n>> + }\n>> +\n>> + /*\n>> + * 'unpack_single_entry' assumes that src[0] is derived directly from\n>> + * the index, rather than from an entry in 'names'. This is *not* true when\n>> + * merging a sparse directory, in which case names[0] is the \"index\" source\n>> + * entry. To match the expectations of 'unpack_single_entry', shift past the\n>> + * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n>> + * 'dirmask' accordingly.\n>> + */\n>> + ret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n> \n> So, you're passing one less entry to unpack_single_entry() when you've\n> traversed into a sparse directory...won't the traversal at the next\n> subdirectory deeper then also pass one less entry to\n> unpack_single_entry(), so after recursing a directory or two, you only\n> have one directory left and it won't conflict with anything so it just\n> uses that remaining tree? (Or maybe it passes the wrong number of\n> arguments into twoway_merge()?) Did I miss something in the logic\n> somewhere that avoids that issue? It'd be nice to test it out, which\n> brings me to...\n> \n\nThe answer itself is pretty straightforward (`merged_sparse_dir(...)` is\ncalled with `n = 3`, which is +1 to the `n` propagated throughout\n`unpack_trees(...)` for a two-way merge), but I'd like to take a more\nin-depth approach answering \"why\" in case it helps with review and/or anyone\nreading along.\n\nSuppose you are performing a two-way merge, e.g. with the command `git\nread-tree my-base other-commit`. The repo contains the following files:\n\n.\n├── bar\n│ └── f1\n├── baz\n│ ├── deep\n│ │ └── a\n│ └── f2\n├── foo\n└── foo1\n\nAdditionally:\n\n1. 'other-commit' is identical to the index\n2. 'baz/' is a sparse directory\n\nWith those assumptions in mind, below are excerpts of the execution path for\nmerging a file, a non-sparse directory, and a sparse directory (with values\nfor arguments indicated where appropriate):\n\nMERGING FILE 'foo'\n------------------\nunpack_trees(len = 2, t = [my-base, other-commit], ...)\n-> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n -> unpack_callback(n = 2, ..., names = [my-base:foo, other-commit:foo], ...)\n -> unpack_single_entry(n = 2, ..., src = [INDEX:foo, NULL, NULL], \n\t\t\t names = [my-base:foo, other-commit:foo], ...)\n -> call_unpack_fn(src = [INDEX:foo, my-base:foo, other-commit:foo], ...)\n -> twoway_merge(src = [INDEX:foo, my-base:foo, other-commit:foo] ...)\n -> unpack_callback(n = 2, ..., names = [my-base:foo1, other-commit:foo1], ...)\n ...\n\nMERGING NON-SPARSE DIRECTORY 'bar/'\n----------------------------------\nunpack_trees(len = 2, t = [my-base, other-commit], ...)\n-> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n -> unpack_callback(n = 2, ..., names = [my-base:bar/, other-commit:bar/], ...)\n -> unpack_single_entry(n = 2, ..., src = [NULL, NULL, NULL], \n\t\t\t names = [my-base:bar/, other-commit:bar/], ...)\n -> traverse_trees_recursive(n = 2, ..., names = [my-base:bar/, other-commit:bar/], ...)\n -> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n -> unpack_callback(n = 2, ..., names = [my-base:bar/f1, other-commit:bar/f1], ...)\n ...\n\nMERGING SPARSE DIRECTORY 'baz/'\n------------------------------\nunpack_trees(len = 2, t = [my-base, other-commit], ...)\n-> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n -> unpack_callback(n = 2, ..., names = [my-base:baz/, other-commit:baz/], ...)\n -> unpack_single_entry(n = 2, ..., src = [INDEX:baz/, NULL, NULL], \n\t\t\t names = [my-base:baz/, other-commit:baz/], ...)\n -> call_unpack_fn(src = [INDEX:baz/, my-base:baz/, other-commit:baz/], ...)\n -> twoway_merge(src = [INDEX:baz/, my-base:baz/, other-commit:baz/] ...)\n -> unpack_callback(n = 2, ..., names = [my-base:foo, other-commit:foo], ...)\n ...\n\nNote that, throughout this process, `n = 2` despite the fact that\n`twoway_merge` is actually called with *three* source entries, because\n`src[0]` is the traversed entry *as it appears in the index*. Additionally,\nbecause there are no differences between the index and 'other-commit',\n'baz/' is merged without conflict in basically the same way as 'foo'.\n\nNow suppose you update the the index to differ from 'other-commit':\n\n1. 'baz/deep/a' is different between the index and 'my-base', same between\n 'my-base' and 'other-commit'\n2. 'baz/f2' is different between 'my-base' and 'other-commit', same between\n the index and 'my-base'\n\nBefore this patch, `twoway_merge` would reject the merge entirely because\nthe sparse directory 'baz/' has three different tree OIDs between the index,\n'my-base' , and 'other-commit'. However, these changes *should* be mergeable\nas long as you merge 'baz/deep/' separately from 'baz/f2'.\n\nThe way we do this is by traversing the 'baz/' index entry the same way we\ntraverse 'my-base' and 'other-commit' so we can circumvent populating\n`src[0]` from the index. The execution path looks something like this (when\nstarted at the sparse directory's `twoway_merge`):\n\nMERGING CONFLICT IN SPARSE 'baz/'\n---------------------------------\ntwoway_merge(src = [INDEX:baz/, my-base:baz/, other-commit:baz/] ...)\n-> merged_sparse_dir(src = [INDEX:baz/, my-base:baz/, other-commit:baz/], n = 3, ...)\n -> traverse_trees(..., n = 3, t = [INDEX^{tree}, my-base, other-commit], ...)\n -> unpack_sparse_callback(n = 3, ..., \n\t\t\t names = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/])\n -> unpack_single_entry(n = 2, ..., src = [INDEX^{tree}:baz/deep/, NULL, NULL], \n\t\t\t names = [my-base:baz/deep/, other-commit:baz/deep/], ...)\n -> call_unpack_fn(src = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/], ...)\n -> twoway_merge(src = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/] ...)\n -> unpack_sparse_callback(n = 3, ..., \n\t\t\t names = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2])\n -> unpack_single_entry(n = 2, ..., src = [INDEX^{tree}:baz/f2, NULL, NULL], \n\t\t\t names = [my-base:baz/f2, other-commit:baz/f2], ...)\n -> call_unpack_fn(src = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2], ...)\n -> twoway_merge(src = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2] ...)\n\nHere, `unpack_sparse_callback` transforms its inputs (from `traverse_trees`)\ninto the those needed by `unpack_single_entry`. Unlike `unpack_callback`,\nwhich extracts its `src[0]` from the index, `unpack_sparse_callback` creates\n`src[0]` from the first tree in `names`. Then, because `unpack_single_entry`\nexpects `n = 2` with a populated `src[0]` and the two remaining trees in\n`names`, it subtracts 1 from its value of `n` and shifts `names` (along with\nother arguments like `mask` and `dirmask`). As you can see, this could\ncontinue recursing without losing trees, since the first tree in the\n`merged_sparse_dir` path is basically storage for the index as the traversal\ncontinues.\n\nPlease let me know if I can clarify anything - this explanation more-or-less\ncorresponds to how I understood the problem while solving it, but it's still\npretty dense and there could be details I'm not conveying well or taking for\ngranted.\n\n>> +\n>> + if (src[0])\n>> + discard_cache_entry(src[0]);\n>> +\n>> + return ret >= 0 ? mask : -1;\n>> +}\n>> +\n>> /*\n>> * Note that traverse_by_cache_tree() duplicates some logic in this function\n>> * without actually calling it. If you change the logic here you may need to\n>> @@ -2464,6 +2500,37 @@ static int merged_entry(const struct cache_entry *ce,\n>> return 1;\n>> }\n>>\n>> +static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n>> + struct unpack_trees_options *o)\n>> +{\n>> + struct tree_desc t[MAX_UNPACK_TREES + 1];\n>> + void * tree_bufs[MAX_UNPACK_TREES + 1];\n>> + struct traverse_info info;\n>> + int i, ret;\n>> +\n>> + /*\n>> + * Create the tree traversal information for traversing into *only* the\n>> + * sparse directory.\n>> + */\n>> + setup_traverse_info(&info, src[0]->name);\n>> + info.fn = unpack_sparse_callback;\n>> + info.data = o;\n>> + info.show_all_errors = o->show_all_errors;\n>> + info.pathspec = o->pathspec;\n>> +\n>> + /* Get the tree descriptors of the sparse directory in each of the merging trees */\n>> + for (i = 0; i < n; i++)\n>> + tree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n>> + src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n>> +\n>> + ret = traverse_trees(o->src_index, n, t, &info);\n>> +\n>> + for (i = 0; i < n; i++)\n>> + free(tree_bufs[i]);\n>> +\n>> + return ret;\n>> +}\n>> +\n>> static int deleted_entry(const struct cache_entry *ce,\n>> const struct cache_entry *old,\n>> struct unpack_trees_options *o)\n>> @@ -2734,6 +2801,14 @@ int twoway_merge(const struct cache_entry * const *src,\n>> * reject the merge instead.\n>> */\n>> return merged_entry(newtree, current, o);\n>> + } else if (S_ISSPARSEDIR(current->ce_mode)) {\n>> + /*\n>> + * The sparse directories differ, but we don't know whether that's\n>> + * because of two different files in the directory being modified\n>> + * (can be trivially merged) or if there is a real file conflict.\n>> + * Merge the sparse directory by OID to compare file-by-file.\n>> + */\n>> + return merged_sparse_dir(src, 3, o);\n>> } else\n>> return reject_merge(current, o);\n>> }\n>> --\n>> gitgitgadget\n> \n> It would be nice to have a couple of tests. In particular, one\n> designed to see what happens when we need to traverse into\n> subdirectories of sparse directory entries and paths different between\n> the two trees being merged.\n\nThere were supposed to be \"ensure_not_expanded\" tests added in this patch,\nbut they ended up in [7/7] - I'll move them back in my next version.\nAdditionally, the 't1092' test 'read-tree --merge with edit/edit conflicts\nin sparse directories' contains examples of merges that require recursing\ninto sparse directories (added back in [3/7] to establish expected behavior\nbefore changing `read-tree`).\n"},{"id":"449795","messageId":"472bdb5f-4bf2-4d62-3dd5-bb6ba4f4162a@github.com","threadId":"57464","inReplyTo":"CABPp-BF4jtEzyUBEymhFJxaGOMRaUs-XyBdr89Fs7E_ehnwWMg@mail.gmail.com","subject":"Re: [PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-28T18:14:53Z","receivedAt":"2022-02-28T18:33:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\n>> emphasis is placed on interactions with files outside the sparse cone, e.g.\n>> merges with out-of-cone conflicts.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> t/perf/p2000-sparse-operations.sh | 1 +\n>> t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n>> 2 files changed, 86 insertions(+)\n>>\n>> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n>> index 2a7106b9495..382716cfca9 100755\n>> --- a/t/perf/p2000-sparse-operations.sh\n>> +++ b/t/perf/p2000-sparse-operations.sh\n>> @@ -117,6 +117,7 @@ test_perf_on_all git diff\n>> test_perf_on_all git diff --cached\n>> test_perf_on_all git blame $SPARSE_CONE/a\n>> test_perf_on_all git blame $SPARSE_CONE/f3/a\n>> +test_perf_on_all git read-tree -mu HEAD\n>> test_perf_on_all git checkout-index -f --all\n>> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index b1dcaa0e642..9d58da4e925 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n>> test_cmp expect sparse-checkout-out\n>> '\n>>\n>> +test_expect_success 'read-tree --merge with files outside sparse definition' '\n>> + init_repos &&\n>> +\n>> + test_all_match git checkout -b test-branch update-folder1 &&\n>> + for MERGE_TREES in \"base HEAD update-folder2\" \\\n>> + \"update-folder1 update-folder2\" \\\n>> + \"update-folder2\"\n>> + do\n>> + # Clean up and remove on-disk files\n>> + test_all_match git reset --hard HEAD &&\n>> + test_sparse_match git sparse-checkout reapply &&\n>> +\n>> + # Although the index matches, without --no-sparse-checkout, outside-of-\n>> + # definition files will not exist on disk for sparse checkouts\n>> + test_all_match git read-tree -mu $MERGE_TREES &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_path_is_missing sparse-checkout/folder2 &&\n>> + test_path_is_missing sparse-index/folder2 &&\n>> +\n>> + test_all_match git read-tree --reset -u HEAD &&\n>> + test_all_match git status --porcelain=v2 &&\n>> +\n>> + test_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n>> + test_all_match git status --porcelain=v2 &&\n>> + test_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n>> + test_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n>> + done\n>> +'\n>> +\n>> +test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n>> + init_repos &&\n>> +\n>> + # Merge of multiple changes to same directory (but not same files) should\n>> + # succeed\n>> + test_all_match git read-tree -mu base rename-base update-folder1 &&\n>> + test_all_match git status --porcelain=v2 &&\n>> +\n>> + test_all_match git reset --hard &&\n>> +\n>> + test_all_match git read-tree -mu rename-base update-folder2 &&\n>> + test_all_match git status --porcelain=v2 &&\n>> +\n>> + test_all_match git reset --hard &&\n>> +\n>> + test_all_match test_must_fail git read-tree -mu base update-folder1 rename-out-to-in &&\n>> + test_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n>> +'\n>> +\n>> +test_expect_success 'read-tree --prefix outside sparse definition' '\n>> + init_repos &&\n>> +\n>> + # Cannot read-tree --prefix with a single argument when files exist within\n>> + # prefix\n> \n> Given the comments in the cover letter about --prefix needing special\n> work, it's not clear to me whether the below is expected behavior or\n> current-but-buggy behavior that you are testing and documenting.\n> Could you clarify?\n> \n\nIt's expected - per 'Documentation/git-read-tree.txt':\n\n\tThe command will refuse to overwrite entries that already\n\texisted in the original index file.\n\nI'll update the comment to make that clearer. Thanks!\n\n>> + test_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n>> +\n>> + test_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n>> + test_path_is_missing sparse-checkout/folder2 &&\n>> + test_path_is_missing sparse-index/folder2 &&\n>> +\n>> + test_all_match git read-tree --reset -u HEAD &&\n>> + test_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n>> + test_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n>> + test_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n>> +'\n>> +\n>> +test_expect_success 'read-tree --merge with directory-file conflicts' '\n>> + init_repos &&\n>> +\n>> + test_all_match git checkout -b test-branch rename-base &&\n>> +\n>> + # Although the index matches, without --no-sparse-checkout, outside-of-\n>> + # definition files will not exist on disk for sparse checkouts\n>> + test_sparse_match git read-tree -mu rename-out-to-out &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_path_is_missing sparse-checkout/folder2 &&\n>> + test_path_is_missing sparse-index/folder2 &&\n>> +\n>> + test_sparse_match git read-tree --reset -u HEAD &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> +\n>> + test_sparse_match git read-tree -mu --no-sparse-checkout rename-out-to-out &&\n>> + test_sparse_match git status --porcelain=v2 &&\n>> + test_cmp sparse-checkout/folder2/0/1 sparse-index/folder2/0/1\n>> +'\n>> +\n>> test_expect_success 'merge, cherry-pick, and rebase' '\n>> init_repos &&\n>>\n>> --\n>> gitgitgadget\n>>\n\n"},{"id":"449798","messageId":"51bd953d-7a57-9643-f4d3-c0e7c0dc9bf6@github.com","threadId":"57464","inReplyTo":"CABPp-BE6n2-v7-91b-aXOsMod6d3dA4zZE4nNyzxac+tVpaaCw@mail.gmail.com","subject":"Re: [PATCH v2 5/7] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-28T18:44:51Z","receivedAt":"2022-02-28T18:45:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> Hi Victoria,\n> \n> On Fri, Feb 25, 2022 at 12:25 PM Victoria Dye wrote:\n>>\n>> Elijah Newren wrote:\n>>> On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n>>> wrote:\n>>>>\n>>>> From: Victoria Dye \n>>>>\n>>>> When 'git read-tree' is provided with a prefix, expand the index only if the\n>>>> prefix is equivalent to a sparse directory or contained within one. If the\n>>>> index is not expanded in these cases, 'ce_in_traverse_path' will indicate\n>>>> that the relevant sparse directory is not in the prefix/traverse path,\n>>>> skipping past it and not unpacking the appropriate tree(s).\n>>>>\n>>>> If the prefix is in-cone, its sparse subdirectories (if any) will be\n>>>> traversed correctly without index expansion.\n>>>>\n>>>> Signed-off-by: Victoria Dye \n>>>> ---\n>>>> builtin/read-tree.c | 3 +--\n>>>> t/t1092-sparse-checkout-compatibility.sh | 8 ++++++-\n>>>> unpack-trees.c | 30 ++++++++++++++++++++++++\n>>>> 3 files changed, 38 insertions(+), 3 deletions(-)\n>>>>\n>>>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>>>> index c2fdbc2657f..a7b7f822281 100644\n>>>> --- a/builtin/read-tree.c\n>>>> +++ b/builtin/read-tree.c\n>>>> @@ -213,8 +213,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>>>> if (opts.merge && !opts.index_only)\n>>>> setup_work_tree();\n>>>>\n>>>> - /* TODO: audit sparse index behavior in unpack_trees */\n>>>> - if (opts.skip_sparse_checkout || opts.prefix)\n>>>> + if (opts.skip_sparse_checkout)\n>>>> ensure_full_index(&the_index);\n>>>>\n>>>> if (opts.merge) {\n>>>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>>>> index ae44451a0a9..a404be0a10f 100755\n>>>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>>>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>>>> @@ -1415,7 +1415,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n>>>> do\n>>>> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n>>>> ensure_not_expanded reset --hard HEAD || return 1\n>>>> - done\n>>>> + done &&\n>>>> +\n>>>> + rm -rf sparse-index/deep/deeper2 &&\n>>>> + ensure_not_expanded add . &&\n>>>> + ensure_not_expanded commit -m \"test\" &&\n>>>> +\n>>>> + ensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n>>>> '\n>>>>\n>>>> test_expect_success 'ls-files' '\n>>>> diff --git a/unpack-trees.c b/unpack-trees.c\n>>>> index 360844bda3a..dba122a02bb 100644\n>>>> --- a/unpack-trees.c\n>>>> +++ b/unpack-trees.c\n>>>> @@ -1739,6 +1739,36 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>>>> setup_standard_excludes(o->dir);\n>>>> }\n>>>>\n>>>> + /*\n>>>> + * If the prefix is equal to or contained within a sparse directory, the\n>>>> + * index needs to be expanded to traverse with the specified prefix.\n>>>> + */\n>>>> + if (o->prefix && o->src_index->sparse_index) {\n>>>> + int prefix_len = strlen(o->prefix);\n>>>> +\n>>>> + while (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n>>>> + prefix_len--;\n>>>> +\n>>>> + if (prefix_len > 0) {\n>>>\n>>> Is this condition check necessary? If we want some safety check here,\n>>> could it instead be something like\n>>>\n>>> if (prefix_len <= 0)\n>>> BUG(\"Broken prefix passed to unpack_trees\");\n>>>\n>>\n>> This condition was intended to skip unnecessary computation for the\n>> (probably unlikely, but still technically valid) case where the prefix is\n>> the repo root (e.g., '--prefix=/') - because the repo root is represented\n>> with only directory separator(s), `prefix_len` would be 0 after removing\n>> trailing '/'. In that scenario, the index won't need to be expanded, so we\n>> don't need to go looking in the index for that path.\n> \n> Wow, that doesn't result in an error?!? That surprises me. I never\n> even thought to test such a thing. Clearly, the following related\n> command does give an error:\n> \n> git read-tree --prefix=/toplevel/ $TREE\n> \n> (namely, \"error: invalid path '/toplevel/subpath'\")\n> \n> whereas, the reason a single slash is accepted it because git is\n> trying to be forgiving and treat the following two commands the same:\n> \n> git read-tree --prefix=subdir/ $TREE\n> git read-tree --prefix=subdir $TREE\n> \n> i.e. it tries to allow the trailing slash to be optional. And, by\n> implementation quirk, making a trailing slash be optional turns out to\n> mean that --prefix=/ is treated the same as no prefix at all, because\n> empty string prefix just happens to give the same behavior as NULL\n> prefix.\n> \n\nIf by \"NULL prefix\" you mean \"the --prefix option isn't specified\", the\nbehavior isn't *quite* the same as with an empty prefix, since `git\nread-tree $TREE` will completely overwrite the index without regard for\nwhat's already there, whereas `git read-tree --prefix= $TREE` \"merges\"\n`$TREE` into the current index (failing if any entries in `$TREE` already\nexist in the index). I still think it's an extremely niche usage, but can\nalso imagine a couple instances where it might be useful (e.g., reading from\nan orphan branch).\n\n> I think we should just throw an error if prefix starts with '/'.\n> unpack_trees() can make it be a BUG() (and at the beginning of the\n> function rather than down at this point and only inside some\n> conditional). builtin/read-tree.c, the only thing that ever sets\n> prefix in unpack_trees_options, should die() if it's passed something\n> that starts with a '/'. Having paths start with a '/' is antithetical\n> to how \"prefix\" is used throughout the codebase, though I guess I can\n> see people making that mistake if they are used to gitignore-style\n> patterns instead.\n> \n\nI think this makes sense (especially because it doesn't cause a loss of\nfunctionality - a user can still specify '--prefix=' for the repo root if\nthey really want to). I'll include a patch for it in my next re-roll.\n\n>> None of that is particularly clear from reading the patch, though, so I'll\n>> add a comment & test covering it explicitly.\n>>\n>>> and then dedent the following code? (Or are callers allowed to not\n>>> sanitize their input before passing to unpack_trees(), meaning that we\n>>> should use a die() rather than a BUG()?)\n>>>\n>>> To test this idea, near the top of unpack_trees(), I added:\n>>> if (o->prefix)\n>>> assert(*o->prefix && *o->prefix != '/');\n>>> and reran all tests. They all ran without hitting that assertion. FWIW.\n>>>\n>>>> + struct strbuf ce_prefix = STRBUF_INIT;\n>>>> + strbuf_grow(&ce_prefix, prefix_len + 1);\n>>>> + strbuf_add(&ce_prefix, o->prefix, prefix_len);\n>>>> + strbuf_addch(&ce_prefix, '/');\n>>>> +\n>>>> + /*\n>>>> + * If the prefix is not inside the sparse cone, then the\n>>>> + * index is explicitly expanded if it is found as a sparse\n>>>> + * directory, or implicitly expanded (by 'index_name_pos')\n>>>> + * if the path is inside a sparse directory.\n>>>> + */\n>>>> + if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n>>>> + index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n>>>\n>>> style nit: Can you rewrap both the comments and the code at 80 characters?\n>>>\n>>\n>> I couldn't think of a way to wrap the condition that wouldn't make it more\n>> difficult to read. The best I could come up with was:\n>>\n>> if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf,\n>> o->src_index) &&\n>> index_name_pos(o->src_index,\n>> ce_prefix.buf,\n>> ce_prefix.len) >= 0)\n>> ensure_full_index(o->src_index);\n>>\n>>\n>> which, to me, is a bit hard to parse. Alternatively, though, I can move the\n>> prefix-checking logic into its own function (kind of like\n>> 'pathspec_needs_expanded_index(...)' in [1]), in which case I won't need to\n>> change the current wrapping to keep it under 80 characters.\n>>\n>> [1] https://lore.kernel.org/git/822d7344587f698e73abba1ca726c3a905f7b403.1638201164.git.gitgitgadget@gmail.com/\n>>\n>>> It took me a bit of playing and testing to understand these two lines.\n>>> The comment helps, but it's still a bit dense to unpack; somehow I\n>>> didn't understand that the comment was referring to index_name_pos()'s\n>>> call to ensure_full_index(). Once I understood that, it all looks\n>>> good.\n>>>\n>>\n>> Sorry about that, I'll revise to make that clearer.\n> \n> Thanks. :-)\n> \n>>>\n>>>> + ensure_full_index(o->src_index);\n>>>> +\n>>>> + strbuf_release(&ce_prefix);\n>>>> + }\n>>>> + }\n>>>> +\n>>>> if (!core_apply_sparse_checkout || !o->update)\n>>>> o->skip_sparse_checkout = 1;\n>>>> if (!o->skip_sparse_checkout && !o->pl) {\n>>>> --\n>>>> gitgitgadget\n\n"},{"id":"449817","messageId":"220301.861qzm37qt.gmgdl@evledraar.gmail.com","threadId":"57464","inReplyTo":"ffe0b6aff2baee238f77ae57561a62ea5473321f.1645742073.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T23:09:40Z","receivedAt":"2022-02-28T23:13:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 24 2022, Victoria Dye via GitGitGadget wrote:\n\n> From: Victoria Dye \n>\n> Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\n> emphasis is placed on interactions with files outside the sparse cone, e.g.\n> merges with out-of-cone conflicts.\n>\n> Signed-off-by: Victoria Dye \n> ---\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n> 2 files changed, 86 insertions(+)\n>\n> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n> index 2a7106b9495..382716cfca9 100755\n> --- a/t/perf/p2000-sparse-operations.sh\n> +++ b/t/perf/p2000-sparse-operations.sh\n> @@ -117,6 +117,7 @@ test_perf_on_all git diff\n> test_perf_on_all git diff --cached\n> test_perf_on_all git blame $SPARSE_CONE/a\n> test_perf_on_all git blame $SPARSE_CONE/f3/a\n> +test_perf_on_all git read-tree -mu HEAD\n> test_perf_on_all git checkout-index -f --all\n> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n> \n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index b1dcaa0e642..9d58da4e925 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n> \ttest_cmp expect sparse-checkout-out\n> '\n> \n> +test_expect_success 'read-tree --merge with files outside sparse definition' '\n> +\tinit_repos &&\n> +\n> +\ttest_all_match git checkout -b test-branch update-folder1 &&\n> +\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n> +\t\t\t \"update-folder1 update-folder2\" \\\n> +\t\t\t \"update-folder2\"\n> +\tdo\n> +\t\t# Clean up and remove on-disk files\n> +\t\ttest_all_match git reset --hard HEAD &&\n> +\t\ttest_sparse_match git sparse-checkout reapply &&\n> +\n> +\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n> +\t\t# definition files will not exist on disk for sparse checkouts\n> +\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n> +\t\ttest_all_match git status --porcelain=v2 &&\n> +\t\ttest_path_is_missing sparse-checkout/folder2 &&\n> +\t\ttest_path_is_missing sparse-index/folder2 &&\n> +\n> +\t\ttest_all_match git read-tree --reset -u HEAD &&\n> +\t\ttest_all_match git status --porcelain=v2 &&\n> +\n> +\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n> +\t\ttest_all_match git status --porcelain=v2 &&\n> +\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n> +\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n> +\tdone\n> +'\n\nNit: Isn't this nicer/easier by unrolling the for-loop to the top-level, i.e.:\n\nfor MERGE_TREES in \"base HEAD update-folder2\" [...]\ndo\n\ttest_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n\t\tinit_repos &&\n\t\ttest_when_finished \"test_all_match git reset --hard HEAD\" &&\n ...\n\t'\ndone\n\nIt makes failures easier to reason about since you see which for-loop\niteration you're in right away, and can e.g. pick one with --run.\n\nAnd we can do the cleanup in test_when_finished instead of at the start\nof every loop.\n"},{"id":"449818","messageId":"7cc515a0-2690-3e54-de87-8e065095dae7@github.com","threadId":"57464","inReplyTo":"CABPp-BGLG15g1UYanaNy=zM320DEYWW52xKNRDA_87mcVXWhYQ@mail.gmail.com","subject":"Re: [PATCH v2 2/7] status: fix nested sparse directory diff in sparse index","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-28T23:17:48Z","receivedAt":"2022-02-28T23:17:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Elijah Newren wrote:\n> On Thu, Feb 24, 2022 at 2:34 PM Victoria Dye via GitGitGadget\n> wrote:\n>>\n>> From: Victoria Dye \n>>\n>> Add the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\n> \n> Perhaps \"Set the 'recursive' diff option flag in\n> 'wt_status_collect_changes_index(...)'\" ? There's no function\n> argument named 'recursive' in wt_status_collect_changes_index() before\n> or after your changes, which is what the wording led me to think of.\n> \n\nRe-reading that now, I agree. Will fix!\n\n>> the 'recursive' flag, 'git status' could report index changes incorrectly\n>> when the following conditions were met:\n>>\n>> * sparse index is enabled\n>> * there is a difference between the index and HEAD in a file inside a\n>> *subdirectory* of a sparse directory\n>> * the sparse directory index entry is *not* expanded in-core\n>>\n>> In this scenario, 'git status' would not recurse into the sparse directory's\n>> subdirectories to identify which file contained the difference between the\n>> index and HEAD. Instead, it would report the immediate subdirectory itself\n>> as \"modified\".\n>>\n>> Example:\n>>\n>> $ git init\n>> $ mkdir -p sparse/sub\n>> $ echo test >sparse/sub/foo\n>> $ git add .\n>> $ git commit -m \"commit 1\"\n>> $ echo somethingelse >sparse/sub/foo\n>> $ git add .\n>> $ git commit -a -m \"commit 2\"\n>> $ git sparse-checkout set --cone --sparse-index 'sparse'\n>> $ git reset --soft HEAD~1\n>> $ git status\n>> On branch master\n>> You are in a sparse checkout.\n>>\n>> Changes to be committed:\n>> (use \"git restore --staged ...\" to unstage)\n>> modified: sparse/sub\n>>\n>> The 'recursive' diff option in 'wt_status_collect_changes_index' corrects\n>> this by indicating that 'git status' should recurse into sparse directories\n>> to find modified files. Given the same repository setup as the example\n>> above, the corrected result of `git status` is:\n>>\n>> $ git status\n>> On branch master\n>> You are in a sparse checkout.\n>>\n>> Changes to be committed:\n>> (use \"git restore --staged ...\" to unstage)\n>> modified: sparse/sub/foo\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> t/t1092-sparse-checkout-compatibility.sh | 7 +++++++\n>> wt-status.c | 9 +++++++++\n>> 2 files changed, 16 insertions(+)\n>>\n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index 9ef7cd80885..b1dcaa0e642 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -278,6 +278,13 @@ test_expect_success 'status with options' '\n>> test_all_match git status --porcelain=v2 -uno\n>> '\n>>\n>> +test_expect_success 'status with diff in unexpanded sparse directory' '\n>> + init_repos &&\n>> + test_all_match git checkout rename-base &&\n>> + test_all_match git reset --soft rename-out-to-out &&\n>> + test_all_match git status --porcelain=v2\n>> +'\n>> +\n>> test_expect_success 'status reports sparse-checkout' '\n>> init_repos &&\n>> git -C sparse-checkout status >full &&\n>> diff --git a/wt-status.c b/wt-status.c\n>> index 335e723a71e..4a5b9beeca1 100644\n>> --- a/wt-status.c\n>> +++ b/wt-status.c\n>> @@ -651,6 +651,15 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n>> rev.diffopt.detect_rename = s->detect_rename >= 0 ? s->detect_rename : rev.diffopt.detect_rename;\n>> rev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;\n>> rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n>> +\n>> + /*\n>> + * The `recursive` option must be enabled to show differences in files\n>> + * *more than* one level deep in a sparse directory index entry (e.g., given\n>> + * sparse directory 'sparse-dir/', reporting a difference in the file\n>> + * 'sparse-dir/another-dir/my-file').\n>> + */\n>> + rev.diffopt.flags.recursive = 1;\n> \n> Kind of clever, and makes sense.\n> \n> I'm wondering if there's an alternate wording that might be helpful\n> here or in the commit message, that instead of just saying the\n> 'recursive' option is necessary, perhaps says a little bit about why\n> it helps. In particular, the diff machinery, by default, is not\n> recursive and stops at comparing the first level of trees. (See e.g.\n> the -r option in diff-tree, it's just that it's turned on by default\n> in 'git diff' and by the -p option in 'git log'.) I'm guessing the\n> recursive option never needed to be turned on previously within\n> wt-status, due to something about the nature of the index only holding\n> files previously. Now, however, the sparse index changes that. (And\n> it also suggests that perhaps we should look to see if other commands\n> run the diff machinery without the recursive flag, and see if they\n> need it now due to sparse indices.)\n> \n> Granted, I'm not totally sure how to work these facts in (in part\n> because I don't know how comparison to the index normally avoids the\n> need for the recursive flag), and maybe what you have is fine. Just\n> thought I'd point it out since I wasn't aware of the non-recursive\n> nature of the diff machinery until I started doing things with\n> diff-tree.\n>\n\nAll good points - I'll reword the commit message and code comment to explain\nthat 1) diff is not recursive by default, and 2) why the diff of a sparse\ndirectory requires `recursive = 1`.\n \n> \n>> +\n>> copy_pathspec(&rev.prune_data, &s->pathspec);\n>> run_diff_index(&rev, 1);\n>> object_array_clear(&rev.pending);\n>> --\n>> gitgitgadget\n\n"},{"id":"449819","messageId":"509e860c-5bda-b92f-96c5-39f9a54f1e9b@github.com","threadId":"57464","inReplyTo":"220301.861qzm37qt.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-02-28T23:27:00Z","receivedAt":"2022-02-28T23:27:05Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Feb 24 2022, Victoria Dye via GitGitGadget wrote:\n> \n>> From: Victoria Dye \n>>\n>> Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\n>> emphasis is placed on interactions with files outside the sparse cone, e.g.\n>> merges with out-of-cone conflicts.\n>>\n>> Signed-off-by: Victoria Dye \n>> ---\n>> t/perf/p2000-sparse-operations.sh | 1 +\n>> t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n>> 2 files changed, 86 insertions(+)\n>>\n>> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n>> index 2a7106b9495..382716cfca9 100755\n>> --- a/t/perf/p2000-sparse-operations.sh\n>> +++ b/t/perf/p2000-sparse-operations.sh\n>> @@ -117,6 +117,7 @@ test_perf_on_all git diff\n>> test_perf_on_all git diff --cached\n>> test_perf_on_all git blame $SPARSE_CONE/a\n>> test_perf_on_all git blame $SPARSE_CONE/f3/a\n>> +test_perf_on_all git read-tree -mu HEAD\n>> test_perf_on_all git checkout-index -f --all\n>> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n>> \n>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>> index b1dcaa0e642..9d58da4e925 100755\n>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>> @@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n>> \ttest_cmp expect sparse-checkout-out\n>> '\n>> \n>> +test_expect_success 'read-tree --merge with files outside sparse definition' '\n>> +\tinit_repos &&\n>> +\n>> +\ttest_all_match git checkout -b test-branch update-folder1 &&\n>> +\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n>> +\t\t\t \"update-folder1 update-folder2\" \\\n>> +\t\t\t \"update-folder2\"\n>> +\tdo\n>> +\t\t# Clean up and remove on-disk files\n>> +\t\ttest_all_match git reset --hard HEAD &&\n>> +\t\ttest_sparse_match git sparse-checkout reapply &&\n>> +\n>> +\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n>> +\t\t# definition files will not exist on disk for sparse checkouts\n>> +\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n>> +\t\ttest_all_match git status --porcelain=v2 &&\n>> +\t\ttest_path_is_missing sparse-checkout/folder2 &&\n>> +\t\ttest_path_is_missing sparse-index/folder2 &&\n>> +\n>> +\t\ttest_all_match git read-tree --reset -u HEAD &&\n>> +\t\ttest_all_match git status --porcelain=v2 &&\n>> +\n>> +\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n>> +\t\ttest_all_match git status --porcelain=v2 &&\n>> +\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n>> +\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n>> +\tdone\n>> +'\n> \n> Nit: Isn't this nicer/easier by unrolling the for-loop to the top-level, i.e.:\n> \n> for MERGE_TREES in \"base HEAD update-folder2\" [...]\n> do\n> \ttest_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n> \t\tinit_repos &&\n> \t\ttest_when_finished \"test_all_match git reset --hard HEAD\" &&\n> ...\n> \t'\n> done\n> \n> It makes failures easier to reason about since you see which for-loop\n> iteration you're in right away, and can e.g. pick one with --run.\n> \n\nI like how this separates the test cases (while not adding any\nredundant/copied code). I'll update in the next version, thanks!\n\n> And we can do the cleanup in test_when_finished instead of at the start\n> of every loop.\n\nBecause `init_repos` completely resets the test repos, this actually lets me\nremove the extra cleanup steps completely.\n"},{"id":"449821","messageId":"220301.86sfs21ri8.gmgdl@evledraar.gmail.com","threadId":"57464","inReplyTo":"509e860c-5bda-b92f-96c5-39f9a54f1e9b@github.com","subject":"Re: [PATCH v2 3/7] read-tree: expand sparse checkout test coverage","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T23:46:40Z","receivedAt":"2022-02-28T23:49:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Victoria Dye wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Thu, Feb 24 2022, Victoria Dye via GitGitGadget wrote:\n>> \n>>> From: Victoria Dye \n>>>\n>>> Add tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\n>>> emphasis is placed on interactions with files outside the sparse cone, e.g.\n>>> merges with out-of-cone conflicts.\n>>>\n>>> Signed-off-by: Victoria Dye \n>>> ---\n>>> t/perf/p2000-sparse-operations.sh | 1 +\n>>> t/t1092-sparse-checkout-compatibility.sh | 85 ++++++++++++++++++++++++\n>>> 2 files changed, 86 insertions(+)\n>>>\n>>> diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\n>>> index 2a7106b9495..382716cfca9 100755\n>>> --- a/t/perf/p2000-sparse-operations.sh\n>>> +++ b/t/perf/p2000-sparse-operations.sh\n>>> @@ -117,6 +117,7 @@ test_perf_on_all git diff\n>>> test_perf_on_all git diff --cached\n>>> test_perf_on_all git blame $SPARSE_CONE/a\n>>> test_perf_on_all git blame $SPARSE_CONE/f3/a\n>>> +test_perf_on_all git read-tree -mu HEAD\n>>> test_perf_on_all git checkout-index -f --all\n>>> test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n>>> \n>>> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n>>> index b1dcaa0e642..9d58da4e925 100755\n>>> --- a/t/t1092-sparse-checkout-compatibility.sh\n>>> +++ b/t/t1092-sparse-checkout-compatibility.sh\n>>> @@ -819,6 +819,91 @@ test_expect_success 'update-index --cacheinfo' '\n>>> \ttest_cmp expect sparse-checkout-out\n>>> '\n>>> \n>>> +test_expect_success 'read-tree --merge with files outside sparse definition' '\n>>> +\tinit_repos &&\n>>> +\n>>> +\ttest_all_match git checkout -b test-branch update-folder1 &&\n>>> +\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n>>> +\t\t\t \"update-folder1 update-folder2\" \\\n>>> +\t\t\t \"update-folder2\"\n>>> +\tdo\n>>> +\t\t# Clean up and remove on-disk files\n>>> +\t\ttest_all_match git reset --hard HEAD &&\n>>> +\t\ttest_sparse_match git sparse-checkout reapply &&\n>>> +\n>>> +\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n>>> +\t\t# definition files will not exist on disk for sparse checkouts\n>>> +\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n>>> +\t\ttest_all_match git status --porcelain=v2 &&\n>>> +\t\ttest_path_is_missing sparse-checkout/folder2 &&\n>>> +\t\ttest_path_is_missing sparse-index/folder2 &&\n>>> +\n>>> +\t\ttest_all_match git read-tree --reset -u HEAD &&\n>>> +\t\ttest_all_match git status --porcelain=v2 &&\n>>> +\n>>> +\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n>>> +\t\ttest_all_match git status --porcelain=v2 &&\n>>> +\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n>>> +\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n>>> +\tdone\n>>> +'\n>> \n>> Nit: Isn't this nicer/easier by unrolling the for-loop to the top-level, i.e.:\n>> \n>> for MERGE_TREES in \"base HEAD update-folder2\" [...]\n>> do\n>> \ttest_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n>> \t\tinit_repos &&\n>> \t\ttest_when_finished \"test_all_match git reset --hard HEAD\" &&\n>> ...\n>> \t'\n>> done\n>> \n>> It makes failures easier to reason about since you see which for-loop\n>> iteration you're in right away, and can e.g. pick one with --run.\n>> \n>\n> I like how this separates the test cases (while not adding any\n> redundant/copied code). I'll update in the next version, thanks!\n>\n>> And we can do the cleanup in test_when_finished instead of at the start\n>> of every loop.\n\nSounds good!\n\nNote for we eval the body of the test into existence, but\n*not* the description. So:\n\n for x in [...] test_expect_success \"$x\" '$x'\n\nWorks to expand \"$x\" in both cases, but not:\n\n for x in [...] test_expect_success '$x' '$x'\n\nAnd you don't need to do:\n\n for x in [...] test_expect_success \"$x\" \"$x\"\n\nWhich is handy as double-quoting the body is often a hassle with\nescaping stuff.\n\nI only think I got that wrong the first 1, 2, 3.... etc. times I used\nthis pattern, so I thought I'd mention it :)\n\n> Because `init_repos` completely resets the test repos, this actually lets me\n> remove the extra cleanup steps completely.\n\n\\o/\n"},{"id":"449847","messageId":"CABPp-BH4jf6z5umYpWB_A+gokUtDkjkiLOahB3FKed_Pe4sZ2g@mail.gmail.com","threadId":"57464","inReplyTo":"dc47f12b-8724-22ef-ed2c-096badfafd76@github.com","subject":"Re: [PATCH 6/7] read-tree: make two-way merge sparse-aware","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-03-01T02:56:58Z","receivedAt":"2022-03-01T02:57:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 28, 2022 at 10:04 AM Victoria Dye wrote:\n>\n> Elijah Newren wrote:\n> > On Wed, Feb 23, 2022 at 4:09 PM Victoria Dye via GitGitGadget\n> > wrote:\n> >>\n> >> From: Victoria Dye \n> >>\n> >> Enable two-way merge with 'git read-tree' without expanding the sparse\n> >> index. When in a sparse index, a two-way merge will trivially succeed as\n> >> long as there are not changes to the same sparse directory in multiple trees\n> >> (i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\n> >> conflicts, the merge will fail despite the possibility that individual files\n> >> could merge cleanly.\n> >>\n> >> In order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\n> >> directories are - rather than rejected - merged by traversing their\n> >> associated trees by OID. For each child of the sparse directory:\n> >>\n> >> 1. Files are merged as normal (see Documentation/git-read-tree.txt for\n> >> details).\n> >> 2. Subdirectories are treated as sparse directories and merged in\n> >> 'twoway_merge'. If there are no conflicts, they are merged according to\n> >> the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n> >> is recursively traversed and merged.\n> >>\n> >> This process allows sparse directories to be individually merged at the\n> >> necessary depth *without* expanding a full index.\n> >\n> > The idea of merging directory-level entries turns out to be\n> > problematic _if_ rename detection is involved, but read-tree-style\n> > merges are only trivial merges that ignore rename detection. As such,\n> > this idea is perfectly reasonable, and is a good way to go. Nicely\n> > done.\n> >\n> > Mostly the patch looks good. There's one thing I'm wondering about, though...\n> >\n> >>\n> >> Signed-off-by: Victoria Dye \n> >> ---\n> >> builtin/read-tree.c | 5 --\n> >> t/t1092-sparse-checkout-compatibility.sh | 3 +-\n> >> unpack-trees.c | 75 ++++++++++++++++++++++++\n> >> 3 files changed, 77 insertions(+), 6 deletions(-)\n> >>\n> >> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> >> index a7b7f822281..5a421de2629 100644\n> >> --- a/builtin/read-tree.c\n> >> +++ b/builtin/read-tree.c\n> >> @@ -225,11 +225,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> >> opts.fn = opts.prefix ? bind_merge : oneway_merge;\n> >> break;\n> >> case 2:\n> >> - /*\n> >> - * TODO: update twoway_merge to handle edit/edit conflicts in\n> >> - * sparse directories.\n> >> - */\n> >> - ensure_full_index(&the_index);\n> >> opts.fn = twoway_merge;\n> >> opts.initial_checkout = is_cache_unborn();\n> >> break;\n> >> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> >> index a404be0a10f..d6f19682d65 100755\n> >> --- a/t/t1092-sparse-checkout-compatibility.sh\n> >> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> >> @@ -1411,7 +1411,8 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n> >> init_repos &&\n> >>\n> >> ensure_not_expanded checkout -b test-branch update-folder1 &&\n> >> - for MERGE_TREES in \"update-folder2\"\n> >> + for MERGE_TREES in \"update-folder2\" \\\n> >> + \"base update-folder2\"\n> >> do\n> >> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> >> ensure_not_expanded reset --hard HEAD || return 1\n> >> diff --git a/unpack-trees.c b/unpack-trees.c\n> >> index dba122a02bb..a4ace53904e 100644\n> >> --- a/unpack-trees.c\n> >> +++ b/unpack-trees.c\n> >> @@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n> >> return sparse_dir_matches_path(ce, info, name);\n> >> }\n> >>\n> >> +static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n> >> +{\n> >> + struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n> >> + struct unpack_trees_options *o = info->data;\n> >> + int ret;\n> >> +\n> >> + assert(o->merge);\n> >> +\n> >> + /*\n> >> + * Unlike in 'unpack_callback', where src[0] is derived from the index when\n> >> + * merging, src[0] is a transient cache entry derived from the first tree\n> >> + * provided. Create the temporary entry as if it came from a non-sparse index.\n> >> + */\n> >> + if (!is_null_oid(&names[0].oid)) {\n> >> + src[0] = create_ce_entry(info, &names[0], 0,\n> >> + &o->result, 1,\n> >> + dirmask & (1ul << 0));\n> >> + src[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n> >> + }\n> >> +\n> >> + /*\n> >> + * 'unpack_single_entry' assumes that src[0] is derived directly from\n> >> + * the index, rather than from an entry in 'names'. This is *not* true when\n> >> + * merging a sparse directory, in which case names[0] is the \"index\" source\n> >> + * entry. To match the expectations of 'unpack_single_entry', shift past the\n> >> + * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n> >> + * 'dirmask' accordingly.\n> >> + */\n> >> + ret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n> >\n> > So, you're passing one less entry to unpack_single_entry() when you've\n> > traversed into a sparse directory...won't the traversal at the next\n> > subdirectory deeper then also pass one less entry to\n> > unpack_single_entry(), so after recursing a directory or two, you only\n> > have one directory left and it won't conflict with anything so it just\n> > uses that remaining tree? (Or maybe it passes the wrong number of\n> > arguments into twoway_merge()?) Did I miss something in the logic\n> > somewhere that avoids that issue? It'd be nice to test it out, which\n> > brings me to...\n> >\n>\n> The answer itself is pretty straightforward (`merged_sparse_dir(...)` is\n> called with `n = 3`, which is +1 to the `n` propagated throughout\n> `unpack_trees(...)` for a two-way merge), but I'd like to take a more\n> in-depth approach answering \"why\" in case it helps with review and/or anyone\n> reading along.\n>\n> Suppose you are performing a two-way merge, e.g. with the command `git\n> read-tree my-base other-commit`. The repo contains the following files:\n>\n> .\n> ├── bar\n> │ └── f1\n> ├── baz\n> │ ├── deep\n> │ │ └── a\n> │ └── f2\n> ├── foo\n> └── foo1\n>\n> Additionally:\n>\n> 1. 'other-commit' is identical to the index\n> 2. 'baz/' is a sparse directory\n>\n> With those assumptions in mind, below are excerpts of the execution path for\n> merging a file, a non-sparse directory, and a sparse directory (with values\n> for arguments indicated where appropriate):\n>\n> MERGING FILE 'foo'\n> ------------------\n> unpack_trees(len = 2, t = [my-base, other-commit], ...)\n> -> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:foo, other-commit:foo], ...)\n> -> unpack_single_entry(n = 2, ..., src = [INDEX:foo, NULL, NULL],\n> names = [my-base:foo, other-commit:foo], ...)\n> -> call_unpack_fn(src = [INDEX:foo, my-base:foo, other-commit:foo], ...)\n> -> twoway_merge(src = [INDEX:foo, my-base:foo, other-commit:foo] ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:foo1, other-commit:foo1], ...)\n> ...\n>\n> MERGING NON-SPARSE DIRECTORY 'bar/'\n> ----------------------------------\n> unpack_trees(len = 2, t = [my-base, other-commit], ...)\n> -> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:bar/, other-commit:bar/], ...)\n> -> unpack_single_entry(n = 2, ..., src = [NULL, NULL, NULL],\n> names = [my-base:bar/, other-commit:bar/], ...)\n> -> traverse_trees_recursive(n = 2, ..., names = [my-base:bar/, other-commit:bar/], ...)\n> -> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:bar/f1, other-commit:bar/f1], ...)\n> ...\n>\n> MERGING SPARSE DIRECTORY 'baz/'\n> ------------------------------\n> unpack_trees(len = 2, t = [my-base, other-commit], ...)\n> -> traverse_trees(..., n = 2, t = [my-base, other-commit], ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:baz/, other-commit:baz/], ...)\n> -> unpack_single_entry(n = 2, ..., src = [INDEX:baz/, NULL, NULL],\n> names = [my-base:baz/, other-commit:baz/], ...)\n> -> call_unpack_fn(src = [INDEX:baz/, my-base:baz/, other-commit:baz/], ...)\n> -> twoway_merge(src = [INDEX:baz/, my-base:baz/, other-commit:baz/] ...)\n> -> unpack_callback(n = 2, ..., names = [my-base:foo, other-commit:foo], ...)\n> ...\n>\n> Note that, throughout this process, `n = 2` despite the fact that\n> `twoway_merge` is actually called with *three* source entries, because\n> `src[0]` is the traversed entry *as it appears in the index*. Additionally,\n> because there are no differences between the index and 'other-commit',\n> 'baz/' is merged without conflict in basically the same way as 'foo'.\n>\n> Now suppose you update the the index to differ from 'other-commit':\n>\n> 1. 'baz/deep/a' is different between the index and 'my-base', same between\n> 'my-base' and 'other-commit'\n> 2. 'baz/f2' is different between 'my-base' and 'other-commit', same between\n> the index and 'my-base'\n>\n> Before this patch, `twoway_merge` would reject the merge entirely because\n> the sparse directory 'baz/' has three different tree OIDs between the index,\n> 'my-base' , and 'other-commit'. However, these changes *should* be mergeable\n> as long as you merge 'baz/deep/' separately from 'baz/f2'.\n>\n> The way we do this is by traversing the 'baz/' index entry the same way we\n> traverse 'my-base' and 'other-commit' so we can circumvent populating\n> `src[0]` from the index. The execution path looks something like this (when\n> started at the sparse directory's `twoway_merge`):\n>\n> MERGING CONFLICT IN SPARSE 'baz/'\n> ---------------------------------\n> twoway_merge(src = [INDEX:baz/, my-base:baz/, other-commit:baz/] ...)\n> -> merged_sparse_dir(src = [INDEX:baz/, my-base:baz/, other-commit:baz/], n = 3, ...)\n> -> traverse_trees(..., n = 3, t = [INDEX^{tree}, my-base, other-commit], ...)\n> -> unpack_sparse_callback(n = 3, ...,\n> names = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/])\n> -> unpack_single_entry(n = 2, ..., src = [INDEX^{tree}:baz/deep/, NULL, NULL],\n> names = [my-base:baz/deep/, other-commit:baz/deep/], ...)\n> -> call_unpack_fn(src = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/], ...)\n> -> twoway_merge(src = [INDEX^{tree}:baz/deep/, my-base:baz/deep/, other-commit:baz/deep/] ...)\n> -> unpack_sparse_callback(n = 3, ...,\n> names = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2])\n> -> unpack_single_entry(n = 2, ..., src = [INDEX^{tree}:baz/f2, NULL, NULL],\n> names = [my-base:baz/f2, other-commit:baz/f2], ...)\n> -> call_unpack_fn(src = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2], ...)\n> -> twoway_merge(src = [INDEX^{tree}:baz/f2, my-base:baz/f2, other-commit:baz/f2] ...)\n>\n> Here, `unpack_sparse_callback` transforms its inputs (from `traverse_trees`)\n> into the those needed by `unpack_single_entry`. Unlike `unpack_callback`,\n> which extracts its `src[0]` from the index, `unpack_sparse_callback` creates\n> `src[0]` from the first tree in `names`. Then, because `unpack_single_entry`\n> expects `n = 2` with a populated `src[0]` and the two remaining trees in\n> `names`, it subtracts 1 from its value of `n` and shifts `names` (along with\n> other arguments like `mask` and `dirmask`). As you can see, this could\n> continue recursing without losing trees, since the first tree in the\n> `merged_sparse_dir` path is basically storage for the index as the traversal\n> continues.\n>\n> Please let me know if I can clarify anything - this explanation more-or-less\n> corresponds to how I understood the problem while solving it, but it's still\n> pretty dense and there could be details I'm not conveying well or taking for\n> granted.\n\nThanks for taking the time to explain this. For some reason, I tend\nto get confused by the extra index entry. I blame the fact that I\nmostly only ever looked at threeway_merge() (since it was the one used\nby merge-recursive.c), and there we get to assume that the HEAD tree\nis equal to the index always, so there really isn't a separate fourth\nargument from the index to worry about. So the combination of having\nto look at twoway_merge() as well as the special index entry for some\nreason was throwing me a bit. This clears it up nicely.\n\n(Once upon a time we only claimed but didn't enforce that index was\nequal to the HEAD tree for the three way merging, but that was an\nabject failure; if you're curious about it, see the fun commit message\nat 9822175d2b (\"Ensure index matches head before invoking merge\nmachinery, round N\", 2019-08-17) explaining it.)\n\n> >> +\n> >> + if (src[0])\n> >> + discard_cache_entry(src[0]);\n> >> +\n> >> + return ret >= 0 ? mask : -1;\n> >> +}\n> >> +\n> >> /*\n> >> * Note that traverse_by_cache_tree() duplicates some logic in this function\n> >> * without actually calling it. If you change the logic here you may need to\n> >> @@ -2464,6 +2500,37 @@ static int merged_entry(const struct cache_entry *ce,\n> >> return 1;\n> >> }\n> >>\n> >> +static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n> >> + struct unpack_trees_options *o)\n> >> +{\n> >> + struct tree_desc t[MAX_UNPACK_TREES + 1];\n> >> + void * tree_bufs[MAX_UNPACK_TREES + 1];\n> >> + struct traverse_info info;\n> >> + int i, ret;\n> >> +\n> >> + /*\n> >> + * Create the tree traversal information for traversing into *only* the\n> >> + * sparse directory.\n> >> + */\n> >> + setup_traverse_info(&info, src[0]->name);\n> >> + info.fn = unpack_sparse_callback;\n> >> + info.data = o;\n> >> + info.show_all_errors = o->show_all_errors;\n> >> + info.pathspec = o->pathspec;\n> >> +\n> >> + /* Get the tree descriptors of the sparse directory in each of the merging trees */\n> >> + for (i = 0; i < n; i++)\n> >> + tree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n> >> + src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n> >> +\n> >> + ret = traverse_trees(o->src_index, n, t, &info);\n> >> +\n> >> + for (i = 0; i < n; i++)\n> >> + free(tree_bufs[i]);\n> >> +\n> >> + return ret;\n> >> +}\n> >> +\n> >> static int deleted_entry(const struct cache_entry *ce,\n> >> const struct cache_entry *old,\n> >> struct unpack_trees_options *o)\n> >> @@ -2734,6 +2801,14 @@ int twoway_merge(const struct cache_entry * const *src,\n> >> * reject the merge instead.\n> >> */\n> >> return merged_entry(newtree, current, o);\n> >> + } else if (S_ISSPARSEDIR(current->ce_mode)) {\n> >> + /*\n> >> + * The sparse directories differ, but we don't know whether that's\n> >> + * because of two different files in the directory being modified\n> >> + * (can be trivially merged) or if there is a real file conflict.\n> >> + * Merge the sparse directory by OID to compare file-by-file.\n> >> + */\n> >> + return merged_sparse_dir(src, 3, o);\n> >> } else\n> >> return reject_merge(current, o);\n> >> }\n> >> --\n> >> gitgitgadget\n> >\n> > It would be nice to have a couple of tests. In particular, one\n> > designed to see what happens when we need to traverse into\n> > subdirectories of sparse directory entries and paths different between\n> > the two trees being merged.\n>\n> There were supposed to be \"ensure_not_expanded\" tests added in this patch,\n> but they ended up in [7/7] - I'll move them back in my next version.\n\nAwesome.\n\n> Additionally, the 't1092' test 'read-tree --merge with edit/edit conflicts\n> in sparse directories' contains examples of merges that require recursing\n> into sparse directories (added back in [3/7] to establish expected behavior\n> before changing `read-tree`).\n\nAny chance you could reference the testcase from 3/7 in your commit\nmessage just so other reviewers or our future selves can find it more\neasily?\n"},{"id":"450006","messageId":"744668eeece68a8cfe98b91c50d1cadb6988b993.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] sparse-index: prevent repo root from becoming sparse","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:24Z","receivedAt":"2022-03-01T20:26:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nPrevent the repository root from being collapsed into a sparse directory by\ntreating an empty path as \"inside the sparse-checkout\". When collapsing a\nsparse index (e.g. in 'git sparse-checkout reapply'), the root directory\ntypically could not become a sparse directory due to the presence of in-cone\nroot-level files and directories. However, if no such in-cone files or\ndirectories were present, there was no explicit check signaling that the\n\"repository root path\" (an empty string, in the case of\n'convert_to_sparse(...)') was in-cone, and a sparse directory index entry\nwould be created from the repository root directory.\n\nThe documentation in Documentation/git-sparse-checkout.txt explicitly states\nthat the files in the root directory are expected to be in-cone for a\ncone-mode sparse-checkout. Collapsing the root into a sparse directory entry\nviolates that assumption, as sparse directory entries are expected to be\noutside the sparse cone and have SKIP_WORKTREE enabled. This invalid state\nin turn causes issues with commands that interact with the index, e.g.\n'git status'.\n\nTreating an empty (root) path as in-cone prevents the creation of a root\nsparse directory in 'convert_to_sparse(...)'. Because the repository root is\notherwise never compared with sparse patterns (in both cone-mode and\nnon-cone sparse-checkouts), the new check does not cause additional changes\nto how sparse patterns are applied.\n\nHelped-by: Derrick Stolee \nSigned-off-by: Victoria Dye \n---\n dir.c | 7 ++++---\n t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++\n 2 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d91295f2bcd..a136377eb49 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1463,10 +1463,11 @@ static int path_in_sparse_checkout_1(const char *path,\n \tconst char *end, *slash;\n \n \t/*\n-\t * We default to accepting a path if there are no patterns or\n-\t * they are of the wrong type.\n+\t * We default to accepting a path if the path is empty, there are no\n+\t * patterns, or the patterns are of the wrong type.\n \t */\n-\tif (init_sparse_checkout_patterns(istate) ||\n+\tif (!*path ||\n+\t init_sparse_checkout_patterns(istate) ||\n \t (require_cone_mode &&\n \t !istate->sparse_checkout_patterns->use_cone_patterns))\n \t\treturn 1;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f3a059e5af5..9ef7cd80885 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -244,6 +244,24 @@ test_expect_success 'expanded in-memory index matches full index' '\n \ttest_sparse_match git ls-files --stage\n '\n \n+test_expect_success 'root directory cannot be sparse' '\n+\tinit_repos &&\n+\n+\t# Remove all in-cone files and directories from the index, collapse index\n+\t# with `git sparse-checkout reapply`\n+\tgit -C sparse-index rm -r . &&\n+\tgit -C sparse-index sparse-checkout reapply &&\n+\n+\t# Verify sparse directories still present, root directory is not sparse\n+\tcat >expect <<-EOF &&\n+\tfolder1/\n+\tfolder2/\n+\tx/\n+\tEOF\n+\tgit -C sparse-index ls-files --sparse >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'status with options' '\n \tinit_repos &&\n \ttest_sparse_match ls &&\n-- \ngitgitgadget\n\n"},{"id":"450007","messageId":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v2.git.1645742073.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] Sparse index: integrate with 'read-tree'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:23Z","receivedAt":"2022-03-01T20:26:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Like previous integrations [1] [2], this series allows 'git read-tree' to\noperate using a sparse index.\n\nThe first three patches are bugfixes for issues found while implementing the\n'read-tree' integration:\n\n * The first (patch [1/8]) fixes an edge case in which a repo with no\n in-cone files or directories would have its root collapsed into a sparse\n directory; the fix ensures the root directory is never collapsed into a\n sparse directory.\n * The second (patch [2/8]) corrects the 'git status' reporting of changes\n nested inside the subdirectory of a sparse directory, ensuring that the\n modified file (not the subdirectory) is correctly reported as having\n changes.\n * The third (patch [3/8]) explicitly disallows the somewhat ambiguous case\n of using 'git read-tree --prefix=/' to represent the repo root (it could\n be interpreted as a leading slash - even though it's actually trailing\n slash - which is not allowed for prefixes). Now, an error specifying that\n prefixes cannot begin with '/' guides users more directly to the correct\n syntax. If a user does want to specify the repo root as their prefix,\n that can still be done with 'git read-tree --prefix='\n\nThe remainder of the series focuses on utilizing the sparse index in 'git\nread-tree'. After some baseline behavior-establishing tests (patch [4/8]),\nsparse index usage is trivially enabled (patch [5/8]) for 'read-tree'\nexcept:\n\n * usage with '--prefix'\n * two- and three-way merge\n\nThese cases require additional changes in order to work as expected (i.e.,\noutwardly matching non-sparse index sparse-checkout). For the former, the\nsparse index can be enabled as long as the index is expanded when the prefix\nis a directory outside the sparse cone (patch [6/8]). For the latter, sparse\ndirectories that cannot be trivially merged must have their contents merged\nfile-by-file, done by recursively traversing the trees represented by the\nsparse directories (patches [7/8] & [8/8]).\n\n\nChanges since V2\n================\n\n * revised/clarified explanation for why the 'recursive' flag is necessary\n in 'git status' internal diff in both the commit message for [2/8] and\n the code comments\n * added patch [3/8] to disallow '/' as a prefix\n * moved the loop over different 'read-tree' scenarios outside the test in\n t1092 test 'read-tree --merge with files outside sparse definition',\n separating the tests into individually-runnable and traceable cases\n * improved test case clarity and variety in 'read-tree --prefix' tests\n * removed redundant arg to 'git reset' in 'sparse index is not expanded:\n read-tree'\n * pointed out pre-existing testing that covers '--prefix' in [6/8] commit\n message\n * moved prefix-dependent index expansion logic into function\n 'update_sparsity_for_prefix', improved explanation for function behavior\n in comments\n * pointed out pre-existing testing that covers two- and three-way merge in\n [7/8] & [8/8] commit messages, respectively\n * added test cases for \"ensure not expanded\" tests of read-tree for both\n two- and three-way merge\n * fixed issue with three-way merge implementation (if the \"no merge\" cases\n described in 't/t1000-read-tree-m-3way.sh' are reached, sparse\n directories should be recursively merged), testing added in the \"ensure\n not expanded\" cases\n\n\nChanges since V1\n================\n\n * switched an empty string check from '!strlen(path)' to the\n slightly-less-expensive '!*path'\n\nThanks!\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/\n[2]\nhttps://lore.kernel.org/git/pull.1048.v6.git.1638201164.gitgitgadget@gmail.com/\n\nVictoria Dye (8):\n sparse-index: prevent repo root from becoming sparse\n status: fix nested sparse directory diff in sparse index\n read-tree: explicitly disallow prefixes with a leading '/'\n read-tree: expand sparse checkout test coverage\n read-tree: integrate with sparse index\n read-tree: narrow scope of index expansion for '--prefix'\n read-tree: make two-way merge sparse-aware\n read-tree: make three-way merge sparse-aware\n\n builtin/read-tree.c | 14 ++-\n dir.c | 7 +-\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1003-read-tree-prefix.sh | 10 ++\n t/t1092-sparse-checkout-compatibility.sh | 133 ++++++++++++++++++++\n unpack-trees.c | 147 +++++++++++++++++++++--\n wt-status.c | 9 ++\n 7 files changed, 308 insertions(+), 13 deletions(-)\n\n\nbase-commit: e6ebfd0e8cbbd10878070c8a356b5ad1b3ca464e\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1157%2Fvdye%2Fsparse%2Fread-tree-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1157/vdye/sparse/read-tree-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1157\n\nRange-diff vs v2:\n\n 1: 744668eeece = 1: 744668eeece sparse-index: prevent repo root from becoming sparse\n 2: f0cff03b95d ! 2: 26f4d30bd95 status: fix nested sparse directory diff in sparse index\n @@ Metadata\n ## Commit message ##\n status: fix nested sparse directory diff in sparse index\n \n - Add the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\n - the 'recursive' flag, 'git status' could report index changes incorrectly\n - when the following conditions were met:\n + Enable the 'recursive' diff option for the diff executed as part of 'git\n + status'. Without the 'recursive' enabled, 'git status' reports index\n + changes incorrectly when the following conditions were met:\n \n * sparse index is enabled\n * there is a difference between the index and HEAD in a file inside a\n *subdirectory* of a sparse directory\n * the sparse directory index entry is *not* expanded in-core\n \n - In this scenario, 'git status' would not recurse into the sparse directory's\n - subdirectories to identify which file contained the difference between the\n - index and HEAD. Instead, it would report the immediate subdirectory itself\n - as \"modified\".\n + Because it is not recursive by default, the diff in 'git status' reports\n + changes only at the level of files and directories that are immediate\n + children of a sparse directory, rather than recursing into directories with\n + changes to identify the modified file(s). As a result, 'git status' reports\n + the immediate subdirectory itself as \"modified\".\n \n Example:\n \n @@ Commit message\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub\n \n - The 'recursive' diff option in 'wt_status_collect_changes_index' corrects\n - this by indicating that 'git status' should recurse into sparse directories\n - to find modified files. Given the same repository setup as the example\n - above, the corrected result of `git status` is:\n + Enabling the 'recursive' diff option in 'wt_status_collect_changes_index'\n + corrects this issue by allowing the diff to recurse into subdirectories of\n + sparse directories to find modified files. Given the same repository setup\n + as the example above, the corrected result of `git status` is:\n \n $ git status\n On branch master\n @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n \trev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n +\n +\t/*\n -+\t * The `recursive` option must be enabled to show differences in files\n -+\t * *more than* one level deep in a sparse directory index entry (e.g., given\n -+\t * sparse directory 'sparse-dir/', reporting a difference in the file\n -+\t * 'sparse-dir/another-dir/my-file').\n ++\t * The `recursive` option must be enabled to allow the diff to recurse\n ++\t * into subdirectories of sparse directory index entries. If it is not\n ++\t * enabled, a subdirectory containing file(s) with changes is reported\n ++\t * as \"modified\", rather than the modified files themselves.\n +\t */\n +\trev.diffopt.flags.recursive = 1;\n +\n -: ----------- > 3: e48a281a4d3 read-tree: explicitly disallow prefixes with a leading '/'\n 3: ffe0b6aff2b ! 4: 90ebcb7b8ff read-tree: expand sparse checkout test coverage\n @@ Commit message\n emphasis is placed on interactions with files outside the sparse cone, e.g.\n merges with out-of-cone conflicts.\n \n + Helped-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Victoria Dye \n \n ## t/perf/p2000-sparse-operations.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n \ttest_cmp expect sparse-checkout-out\n '\n \n -+test_expect_success 'read-tree --merge with files outside sparse definition' '\n -+\tinit_repos &&\n -+\n -+\ttest_all_match git checkout -b test-branch update-folder1 &&\n -+\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n -+\t\t\t \"update-folder1 update-folder2\" \\\n -+\t\t\t \"update-folder2\"\n -+\tdo\n -+\t\t# Clean up and remove on-disk files\n -+\t\ttest_all_match git reset --hard HEAD &&\n -+\t\ttest_sparse_match git sparse-checkout reapply &&\n ++for MERGE_TREES in \"base HEAD update-folder2\" \\\n ++\t\t \"update-folder1 update-folder2\" \\\n ++\t\t \"update-folder2\"\n ++do\n ++\ttest_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n ++\t\tinit_repos &&\n +\n +\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n +\t\t# definition files will not exist on disk for sparse checkouts\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n +\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n +\t\ttest_all_match git status --porcelain=v2 &&\n +\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n -+\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n -+\tdone\n -+'\n ++\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a\n ++\n ++\t'\n ++done\n +\n +test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n +\tinit_repos &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n +\ttest_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n +'\n +\n -+test_expect_success 'read-tree --prefix outside sparse definition' '\n ++test_expect_success 'read-tree --prefix' '\n +\tinit_repos &&\n +\n -+\t# Cannot read-tree --prefix with a single argument when files exist within\n -+\t# prefix\n -+\ttest_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n ++\t# If files differing between the index and target exist\n ++\t# inside the prefix, `read-tree --prefix` should fail\n ++\ttest_all_match test_must_fail git read-tree --prefix=deep/ deepest &&\n ++\ttest_all_match test_must_fail git read-tree --prefix=folder1/ update-folder1 &&\n +\n -+\ttest_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n -+\ttest_path_is_missing sparse-checkout/folder2 &&\n -+\ttest_path_is_missing sparse-index/folder2 &&\n ++\t# If no differing index entries exist matching the prefix,\n ++\t# `read-tree --prefix` updates the index successfully\n ++\ttest_all_match git rm -rf deep/deeper1/deepest/ &&\n ++\ttest_all_match git read-tree --prefix=deep/deeper1/deepest -u deepest &&\n ++\ttest_all_match git status --porcelain=v2 &&\n ++\n ++\ttest_all_match git rm -rf --sparse folder1/ &&\n ++\ttest_all_match git read-tree --prefix=folder1/ -u update-folder1 &&\n ++\ttest_all_match git status --porcelain=v2 &&\n +\n -+\ttest_all_match git read-tree --reset -u HEAD &&\n -+\ttest_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n -+\ttest_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n -+\ttest_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n ++\ttest_all_match git rm -rf --sparse folder2/0 &&\n ++\ttest_all_match git read-tree --prefix=folder2/0/ -u rename-out-to-out &&\n ++\ttest_all_match git status --porcelain=v2\n +'\n +\n +test_expect_success 'read-tree --merge with directory-file conflicts' '\n 4: cb7e0cf419c ! 5: 015618a9f29 read-tree: integrate with sparse index\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\tfor MERGE_TREES in \"update-folder2\"\n +\tdo\n +\t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n -+\t\tensure_not_expanded reset --hard HEAD || return 1\n ++\t\tensure_not_expanded reset --hard || return 1\n +\tdone\n +'\n +\n 5: 4f05fa70209 ! 6: 1a9365c3bc5 read-tree: narrow scope of index expansion for '--prefix'\n @@ Commit message\n If the prefix is in-cone, its sparse subdirectories (if any) will be\n traversed correctly without index expansion.\n \n + The behavior of 'git read-tree' with prefixes 1) inside of cone, 2) equal to\n + a sparse directory, and 3) inside a sparse directory are all tested as part\n + of the 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --prefix',\n + ensuring that the sparse index case works the way it did prior to this\n + change as well as matching non-sparse index sparse-checkout.\n +\n + Helped-by: Elijah Newren \n Signed-off-by: Victoria Dye \n \n ## builtin/read-tree.c ##\n @@ t/t1092-sparse-checkout-compatibility.sh\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: read-tree' '\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n - \t\tensure_not_expanded reset --hard HEAD || return 1\n + \t\tensure_not_expanded reset --hard || return 1\n -\tdone\n +\tdone &&\n +\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n test_expect_success 'ls-files' '\n \n ## unpack-trees.c ##\n -@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n - \t\tsetup_standard_excludes(o->dir);\n - \t}\n +@@ unpack-trees.c: static void populate_from_existing_patterns(struct unpack_trees_options *o,\n + \t\to->pl = pl;\n + }\n \n -+\t/*\n -+\t * If the prefix is equal to or contained within a sparse directory, the\n -+\t * index needs to be expanded to traverse with the specified prefix.\n -+\t */\n -+\tif (o->prefix && o->src_index->sparse_index) {\n -+\t\tint prefix_len = strlen(o->prefix);\n ++static void update_sparsity_for_prefix(const char *prefix,\n ++\t\t\t\t struct index_state *istate)\n ++{\n ++\tint prefix_len = strlen(prefix);\n ++\tstruct strbuf ce_prefix = STRBUF_INIT;\n +\n -+\t\twhile (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n -+\t\t\tprefix_len--;\n ++\tif (!istate->sparse_index)\n ++\t\treturn;\n +\n -+\t\tif (prefix_len > 0) {\n -+\t\t\tstruct strbuf ce_prefix = STRBUF_INIT;\n -+\t\t\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n -+\t\t\tstrbuf_add(&ce_prefix, o->prefix, prefix_len);\n -+\t\t\tstrbuf_addch(&ce_prefix, '/');\n ++\twhile (prefix_len > 0 && prefix[prefix_len - 1] == '/')\n ++\t\tprefix_len--;\n +\n -+\t\t\t/*\n -+\t\t\t * If the prefix is not inside the sparse cone, then the\n -+\t\t\t * index is explicitly expanded if it is found as a sparse\n -+\t\t\t * directory, or implicitly expanded (by 'index_name_pos')\n -+\t\t\t * if the path is inside a sparse directory.\n -+\t\t\t */\n -+\t\t\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n -+\t\t\t index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n -+\t\t\t\tensure_full_index(o->src_index);\n ++\tif (prefix_len <= 0)\n ++\t\tBUG(\"Invalid prefix passed to update_sparsity_for_prefix\");\n +\n -+\t\t\tstrbuf_release(&ce_prefix);\n -+\t\t}\n -+\t}\n ++\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n ++\tstrbuf_add(&ce_prefix, prefix, prefix_len);\n ++\tstrbuf_addch(&ce_prefix, '/');\n ++\n ++\t/*\n ++\t * If the prefix points to a sparse directory or a path inside a sparse\n ++\t * directory, the index should be expanded. This is accomplished in one\n ++\t * of two ways:\n ++\t * - if the prefix is inside a sparse directory, it will be expanded by\n ++\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n ++\t * - if the prefix matches an existing sparse directory entry,\n ++\t * 'index_name_pos(...)' will return its index position, triggering\n ++\t * the 'ensure_full_index(...)' below.\n ++\t */\n ++\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n ++\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n ++\t\tensure_full_index(istate);\n ++\n ++\tstrbuf_release(&ce_prefix);\n ++}\n + \n + static int verify_absent(const struct cache_entry *,\n + \t\t\t enum unpack_trees_error_types,\n +@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n + \t\tsetup_standard_excludes(o->dir);\n + \t}\n + \n ++\tif (o->prefix)\n ++\t\tupdate_sparsity_for_prefix(o->prefix, o->src_index);\n +\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n 6: 94c2aad2f93 ! 7: 71bec3ddad1 read-tree: make two-way merge sparse-aware\n @@ Commit message\n This process allows sparse directories to be individually merged at the\n necessary depth *without* expanding a full index.\n \n + The 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --merge with\n + edit/edit conflicts in sparse directories' tests two-way merges with 1)\n + changes inside sparse directories that do not conflict and 2) changes that\n + do conflict (with the correct file(s) reported in the error message).\n + Additionally, add two-way merge cases to 'sparse index is not expanded:\n + read-tree' to confirm that the index is not expanded regardless of whether\n + edit/edit conflicts are present in a sparse directory.\n +\n Signed-off-by: Victoria Dye \n \n ## builtin/read-tree.c ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n -\tfor MERGE_TREES in \"update-folder2\"\n -+\tfor MERGE_TREES in \"update-folder2\" \\\n -+\t\t\t \"base update-folder2\"\n ++\tfor MERGE_TREES in \"base update-folder2\" \\\n ++\t\t\t \"base rename-base\" \\\n ++\t\t\t \"update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n - \t\tensure_not_expanded reset --hard HEAD || return 1\n + \t\tensure_not_expanded reset --hard || return 1\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: static int is_sparse_directory_entry(struct cache_entry *ce,\n 7: c4080e99d6e ! 8: 6bc11325dd1 read-tree: make three-way merge sparse-aware\n @@ Commit message\n merge, the contents of each conflicted sparse directory are merged without\n referencing the index, avoiding sparse index expansion.\n \n + As with two-way merge, the 't/t1092-sparse-checkout-compatibility.sh' test\n + 'read-tree --merge with edit/edit conflicts in sparse directories' confirms\n + that three-way merges with edit/edit changes (both with and without\n + conflicts) inside a sparse directory result in the correct index state or\n + error message. To ensure the index is not unnecessarily expanded, add\n + three-way merge cases to 'sparse index is not expanded: read-tree'.\n +\n Signed-off-by: Victoria Dye \n \n ## builtin/read-tree.c ##\n @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: read-tree' '\n + \tinit_repos &&\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n - \tfor MERGE_TREES in \"update-folder2\" \\\n --\t\t\t \"base update-folder2\"\n +-\tfor MERGE_TREES in \"base update-folder2\" \\\n ++\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n ++\t\t\t \"base HEAD rename-base\" \\\n +\t\t\t \"base update-folder2\" \\\n -+\t\t\t \"base HEAD update-folder2\"\n + \t\t\t \"base rename-base\" \\\n + \t\t\t \"update-folder2\"\n \tdo\n - \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n - \t\tensure_not_expanded reset --hard HEAD || return 1\n \n ## unpack-trees.c ##\n @@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n @@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n \n \tif (head) {\n \t\t/* #5ALT, #15 */\n +@@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n + \n + \t}\n + \n +-\t/* Below are \"no merge\" cases, which require that the index be\n +-\t * up-to-date to avoid the files getting overwritten with\n +-\t * conflict resolution files.\n +-\t */\n ++\t/* Handle \"no merge\" cases (see t/t1000-read-tree-m-3way.sh) */\n + \tif (index) {\n ++\t\t/*\n ++\t\t * If we've reached the \"no merge\" cases and we're merging\n ++\t\t * a sparse directory, we may have an \"edit/edit\" conflict that\n ++\t\t * can be resolved by individually merging directory contents.\n ++\t\t */\n ++\t\tif (S_ISSPARSEDIR(index->ce_mode))\n ++\t\t\treturn merged_sparse_dir(stages, 4, o);\n ++\n ++\t\t/*\n ++\t\t * If we're not merging a sparse directory, ensure the index is\n ++\t\t * up-to-date to avoid files getting overwritten with conflict\n ++\t\t * resolution files\n ++\t\t */\n + \t\tif (verify_uptodate(index, o))\n + \t\t\treturn -1;\n + \t}\n\n-- \ngitgitgadget\n"},{"id":"450008","messageId":"26f4d30bd9546cf940ddd6827ddfc271c99b6665.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] status: fix nested sparse directory diff in sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:25Z","receivedAt":"2022-03-01T20:26:20Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable the 'recursive' diff option for the diff executed as part of 'git\nstatus'. Without the 'recursive' enabled, 'git status' reports index\nchanges incorrectly when the following conditions were met:\n\n* sparse index is enabled\n* there is a difference between the index and HEAD in a file inside a\n *subdirectory* of a sparse directory\n* the sparse directory index entry is *not* expanded in-core\n\nBecause it is not recursive by default, the diff in 'git status' reports\nchanges only at the level of files and directories that are immediate\nchildren of a sparse directory, rather than recursing into directories with\nchanges to identify the modified file(s). As a result, 'git status' reports\nthe immediate subdirectory itself as \"modified\".\n\nExample:\n\n$ git init\n$ mkdir -p sparse/sub\n$ echo test >sparse/sub/foo\n$ git add .\n$ git commit -m \"commit 1\"\n$ echo somethingelse >sparse/sub/foo\n$ git add .\n$ git commit -a -m \"commit 2\"\n$ git sparse-checkout set --cone --sparse-index 'sparse'\n$ git reset --soft HEAD~1\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub\n\nEnabling the 'recursive' diff option in 'wt_status_collect_changes_index'\ncorrects this issue by allowing the diff to recurse into subdirectories of\nsparse directories to find modified files. Given the same repository setup\nas the example above, the corrected result of `git status` is:\n\n$ git status\nOn branch master\nYou are in a sparse checkout.\n\nChanges to be committed:\n (use \"git restore --staged ...\" to unstage)\n modified: sparse/sub/foo\n\nSigned-off-by: Victoria Dye \n---\n t/t1092-sparse-checkout-compatibility.sh | 7 +++++++\n wt-status.c | 9 +++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9ef7cd80885..b1dcaa0e642 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -278,6 +278,13 @@ test_expect_success 'status with options' '\n \ttest_all_match git status --porcelain=v2 -uno\n '\n \n+test_expect_success 'status with diff in unexpanded sparse directory' '\n+\tinit_repos &&\n+\ttest_all_match git checkout rename-base &&\n+\ttest_all_match git reset --soft rename-out-to-out &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n test_expect_success 'status reports sparse-checkout' '\n \tinit_repos &&\n \tgit -C sparse-checkout status >full &&\ndiff --git a/wt-status.c b/wt-status.c\nindex 335e723a71e..7da8bbe261a 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -651,6 +651,15 @@ static void wt_status_collect_changes_index(struct wt_status *s)\n \trev.diffopt.detect_rename = s->detect_rename >= 0 ? s->detect_rename : rev.diffopt.detect_rename;\n \trev.diffopt.rename_limit = s->rename_limit >= 0 ? s->rename_limit : rev.diffopt.rename_limit;\n \trev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n+\n+\t/*\n+\t * The `recursive` option must be enabled to allow the diff to recurse\n+\t * into subdirectories of sparse directory index entries. If it is not\n+\t * enabled, a subdirectory containing file(s) with changes is reported\n+\t * as \"modified\", rather than the modified files themselves.\n+\t */\n+\trev.diffopt.flags.recursive = 1;\n+\n \tcopy_pathspec(&rev.prune_data, &s->pathspec);\n \trun_diff_index(&rev, 1);\n \tobject_array_clear(&rev.pending);\n-- \ngitgitgadget\n\n"},{"id":"450009","messageId":"e48a281a4d3db0a04c0609fcb8658e4fcc797210.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] read-tree: explicitly disallow prefixes with a leading '/'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:26Z","receivedAt":"2022-03-01T20:26:21Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nExit with an error if a prefix provided to `git read-tree --prefix` begins\nwith '/'. In most cases, prefixes like this result in an \"invalid path\"\nerror; however, the repository root would be interpreted as valid when\nspecified as '--prefix=/'. This is due to leniency around trailing directory\nseparators on prefixes (e.g., allowing both '--prefix=my-dir' and\n'--prefix=my-dir/') - the '/' in the prefix is actually the *trailing*\nslash, although it could be misinterpreted as a *leading* slash.\n\nTo remove the confusing repo root-as-'/' case and make it clear that\nprefixes should not begin with '/', exit with an error if the first\ncharacter of the provided prefix is '/'.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 4 ++++\n t/t1003-read-tree-prefix.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 2109c4c9e5c..c1a401971c2 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -166,6 +166,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n \n+\t/* Prefix should not start with a directory separator */\n+\tif (opts.prefix && opts.prefix[0] == '/')\n+\t\tdie(\"Invalid prefix, prefix cannot start with '/'\");\n+\n \tif (opts.reset)\n \t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n \ndiff --git a/t/t1003-read-tree-prefix.sh b/t/t1003-read-tree-prefix.sh\nindex e0db2066f31..c860c08ecb4 100755\n--- a/t/t1003-read-tree-prefix.sh\n+++ b/t/t1003-read-tree-prefix.sh\n@@ -25,4 +25,14 @@ test_expect_success 'read-tree --prefix' '\n \tcmp expect actual\n '\n \n+test_expect_success 'read-tree --prefix with leading slash exits with error' '\n+\tgit rm -rf . &&\n+\ttest_must_fail git read-tree --prefix=/two/ $tree &&\n+\tgit read-tree --prefix=two/ $tree &&\n+\n+\tgit rm -rf . &&\n+\ttest_must_fail git read-tree --prefix=/ $tree &&\n+\tgit read-tree --prefix= $tree\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450010","messageId":"6bc11325dd14a67121cf5f5a8e20d83e1d7f4047.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] read-tree: make three-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:31Z","receivedAt":"2022-03-01T20:26:25Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of 'merged_sparse_dir' in 'threeway_merge'. As with two-way\nmerge, the contents of each conflicted sparse directory are merged without\nreferencing the index, avoiding sparse index expansion.\n\nAs with two-way merge, the 't/t1092-sparse-checkout-compatibility.sh' test\n'read-tree --merge with edit/edit conflicts in sparse directories' confirms\nthat three-way merges with edit/edit changes (both with and without\nconflicts) inside a sparse directory result in the correct index state or\nerror message. To ensure the index is not unnecessarily expanded, add\nthree-way merge cases to 'sparse index is not expanded: read-tree'.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 ----\n t/t1092-sparse-checkout-compatibility.sh | 4 ++-\n unpack-trees.c | 34 ++++++++++++++++++------\n 3 files changed, 29 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 9227f07ab15..9f1f33e9546 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -234,11 +234,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n-\t\t\t/*\n-\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 61dc2ea777c..8be8e3c06a6 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1413,7 +1413,9 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tinit_repos &&\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n-\tfor MERGE_TREES in \"base update-folder2\" \\\n+\tfor MERGE_TREES in \"base HEAD update-folder2\" \\\n+\t\t\t \"base HEAD rename-base\" \\\n+\t\t\t \"base update-folder2\" \\\n \t\t\t \"base rename-base\" \\\n \t\t\t \"update-folder2\"\n \tdo\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0c2a678cd6d..b876caca0d1 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -2643,16 +2643,24 @@ int threeway_merge(const struct cache_entry * const *stages,\n \t */\n \t/* #14, #14ALT, #2ALT */\n \tif (remote && !df_conflict_head && head_match && !remote_match) {\n-\t\tif (index && !same(index, remote) && !same(index, head))\n-\t\t\treturn reject_merge(index, o);\n+\t\tif (index && !same(index, remote) && !same(index, head)) {\n+\t\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\t\telse\n+\t\t\t\treturn reject_merge(index, o);\n+\t\t}\n \t\treturn merged_entry(remote, index, o);\n \t}\n \t/*\n \t * If we have an entry in the index cache, then we want to\n \t * make sure that it matches head.\n \t */\n-\tif (index && !same(index, head))\n-\t\treturn reject_merge(index, o);\n+\tif (index && !same(index, head)) {\n+\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\t\telse\n+\t\t\treturn reject_merge(index, o);\n+\t}\n \n \tif (head) {\n \t\t/* #5ALT, #15 */\n@@ -2714,11 +2722,21 @@ int threeway_merge(const struct cache_entry * const *stages,\n \n \t}\n \n-\t/* Below are \"no merge\" cases, which require that the index be\n-\t * up-to-date to avoid the files getting overwritten with\n-\t * conflict resolution files.\n-\t */\n+\t/* Handle \"no merge\" cases (see t/t1000-read-tree-m-3way.sh) */\n \tif (index) {\n+\t\t/*\n+\t\t * If we've reached the \"no merge\" cases and we're merging\n+\t\t * a sparse directory, we may have an \"edit/edit\" conflict that\n+\t\t * can be resolved by individually merging directory contents.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(index->ce_mode))\n+\t\t\treturn merged_sparse_dir(stages, 4, o);\n+\n+\t\t/*\n+\t\t * If we're not merging a sparse directory, ensure the index is\n+\t\t * up-to-date to avoid files getting overwritten with conflict\n+\t\t * resolution files\n+\t\t */\n \t\tif (verify_uptodate(index, o))\n \t\t\treturn -1;\n \t}\n-- \ngitgitgadget\n"},{"id":"450011","messageId":"1a9365c3bc5b810a60593612bfba97a8b0c1d657.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:29Z","receivedAt":"2022-03-01T20:26:28Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nWhen 'git read-tree' is provided with a prefix, expand the index only if the\nprefix is equivalent to a sparse directory or contained within one. If the\nindex is not expanded in these cases, 'ce_in_traverse_path' will indicate\nthat the relevant sparse directory is not in the prefix/traverse path,\nskipping past it and not unpacking the appropriate tree(s).\n\nIf the prefix is in-cone, its sparse subdirectories (if any) will be\ntraversed correctly without index expansion.\n\nThe behavior of 'git read-tree' with prefixes 1) inside of cone, 2) equal to\na sparse directory, and 3) inside a sparse directory are all tested as part\nof the 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --prefix',\nensuring that the sparse index case works the way it did prior to this\nchange as well as matching non-sparse index sparse-checkout.\n\nHelped-by: Elijah Newren \nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 3 +-\n t/t1092-sparse-checkout-compatibility.sh | 8 ++++-\n unpack-trees.c | 38 ++++++++++++++++++++++++\n 3 files changed, 46 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 0a52cab7752..ec6d038242a 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -217,8 +217,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n-\t/* TODO: audit sparse index behavior in unpack_trees */\n-\tif (opts.skip_sparse_checkout || opts.prefix)\n+\tif (opts.skip_sparse_checkout)\n \t\tensure_full_index(&the_index);\n \n \tif (opts.merge) {\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 86241b01a59..d98558f3238 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1417,7 +1417,13 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard || return 1\n-\tdone\n+\tdone &&\n+\n+\trm -rf sparse-index/deep/deeper2 &&\n+\tensure_not_expanded add . &&\n+\tensure_not_expanded commit -m \"test\" &&\n+\n+\tensure_not_expanded read-tree --prefix=deep/deeper2 -u deepest\n '\n \n test_expect_success 'ls-files' '\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 360844bda3a..f3667d85ec5 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1693,6 +1693,41 @@ static void populate_from_existing_patterns(struct unpack_trees_options *o,\n \t\to->pl = pl;\n }\n \n+static void update_sparsity_for_prefix(const char *prefix,\n+\t\t\t\t struct index_state *istate)\n+{\n+\tint prefix_len = strlen(prefix);\n+\tstruct strbuf ce_prefix = STRBUF_INIT;\n+\n+\tif (!istate->sparse_index)\n+\t\treturn;\n+\n+\twhile (prefix_len > 0 && prefix[prefix_len - 1] == '/')\n+\t\tprefix_len--;\n+\n+\tif (prefix_len <= 0)\n+\t\tBUG(\"Invalid prefix passed to update_sparsity_for_prefix\");\n+\n+\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n+\tstrbuf_add(&ce_prefix, prefix, prefix_len);\n+\tstrbuf_addch(&ce_prefix, '/');\n+\n+\t/*\n+\t * If the prefix points to a sparse directory or a path inside a sparse\n+\t * directory, the index should be expanded. This is accomplished in one\n+\t * of two ways:\n+\t * - if the prefix is inside a sparse directory, it will be expanded by\n+\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n+\t * - if the prefix matches an existing sparse directory entry,\n+\t * 'index_name_pos(...)' will return its index position, triggering\n+\t * the 'ensure_full_index(...)' below.\n+\t */\n+\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n+\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n+\t\tensure_full_index(istate);\n+\n+\tstrbuf_release(&ce_prefix);\n+}\n \n static int verify_absent(const struct cache_entry *,\n \t\t\t enum unpack_trees_error_types,\n@@ -1739,6 +1774,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tsetup_standard_excludes(o->dir);\n \t}\n \n+\tif (o->prefix)\n+\t\tupdate_sparsity_for_prefix(o->prefix, o->src_index);\n+\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n-- \ngitgitgadget\n\n"},{"id":"450012","messageId":"71bec3ddad1a086ba9b40ce6b69ad0ad9371cc75.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] read-tree: make two-way merge sparse-aware","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:30Z","receivedAt":"2022-03-01T20:26:30Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable two-way merge with 'git read-tree' without expanding the sparse\nindex. When in a sparse index, a two-way merge will trivially succeed as\nlong as there are not changes to the same sparse directory in multiple trees\n(i.e., sparse directory-level \"edit-edit\" conflicts). If there are such\nconflicts, the merge will fail despite the possibility that individual files\ncould merge cleanly.\n\nIn order to resolve these \"edit-edit\" conflicts, \"conflicted\" sparse\ndirectories are - rather than rejected - merged by traversing their\nassociated trees by OID. For each child of the sparse directory:\n\n1. Files are merged as normal (see Documentation/git-read-tree.txt for\n details).\n2. Subdirectories are treated as sparse directories and merged in\n 'twoway_merge'. If there are no conflicts, they are merged according to\n the rules in Documentation/git-read-tree.txt; otherwise, the subdirectory\n is recursively traversed and merged.\n\nThis process allows sparse directories to be individually merged at the\nnecessary depth *without* expanding a full index.\n\nThe 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --merge with\nedit/edit conflicts in sparse directories' tests two-way merges with 1)\nchanges inside sparse directories that do not conflict and 2) changes that\ndo conflict (with the correct file(s) reported in the error message).\nAdditionally, add two-way merge cases to 'sparse index is not expanded:\nread-tree' to confirm that the index is not expanded regardless of whether\nedit/edit conflicts are present in a sparse directory.\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 5 --\n t/t1092-sparse-checkout-compatibility.sh | 4 +-\n unpack-trees.c | 75 ++++++++++++++++++++++++\n 3 files changed, 78 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex ec6d038242a..9227f07ab15 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -229,11 +229,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n-\t\t\t/*\n-\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n-\t\t\t * sparse directories.\n-\t\t\t */\n-\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d98558f3238..61dc2ea777c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1413,7 +1413,9 @@ test_expect_success 'sparse index is not expanded: read-tree' '\n \tinit_repos &&\n \n \tensure_not_expanded checkout -b test-branch update-folder1 &&\n-\tfor MERGE_TREES in \"update-folder2\"\n+\tfor MERGE_TREES in \"base update-folder2\" \\\n+\t\t\t \"base rename-base\" \\\n+\t\t\t \"update-folder2\"\n \tdo\n \t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n \t\tensure_not_expanded reset --hard || return 1\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f3667d85ec5..0c2a678cd6d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1360,6 +1360,42 @@ static int is_sparse_directory_entry(struct cache_entry *ce,\n \treturn sparse_dir_matches_path(ce, info, name);\n }\n \n+static int unpack_sparse_callback(int n, unsigned long mask, unsigned long dirmask, struct name_entry *names, struct traverse_info *info)\n+{\n+\tstruct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, };\n+\tstruct unpack_trees_options *o = info->data;\n+\tint ret;\n+\n+\tassert(o->merge);\n+\n+\t/*\n+\t * Unlike in 'unpack_callback', where src[0] is derived from the index when\n+\t * merging, src[0] is a transient cache entry derived from the first tree\n+\t * provided. Create the temporary entry as if it came from a non-sparse index.\n+\t */\n+\tif (!is_null_oid(&names[0].oid)) {\n+\t\tsrc[0] = create_ce_entry(info, &names[0], 0,\n+\t\t\t\t\t&o->result, 1,\n+\t\t\t\t\tdirmask & (1ul << 0));\n+\t\tsrc[0]->ce_flags |= (CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t}\n+\n+\t/*\n+\t * 'unpack_single_entry' assumes that src[0] is derived directly from\n+\t * the index, rather than from an entry in 'names'. This is *not* true when\n+\t * merging a sparse directory, in which case names[0] is the \"index\" source\n+\t * entry. To match the expectations of 'unpack_single_entry', shift past the\n+\t * \"index\" tree (i.e., names[0]) and adjust 'names', 'n', 'mask', and\n+\t * 'dirmask' accordingly.\n+\t */\n+\tret = unpack_single_entry(n - 1, mask >> 1, dirmask >> 1, src, names + 1, info);\n+\n+\tif (src[0])\n+\t\tdiscard_cache_entry(src[0]);\n+\n+\treturn ret >= 0 ? mask : -1;\n+}\n+\n /*\n * Note that traverse_by_cache_tree() duplicates some logic in this function\n * without actually calling it. If you change the logic here you may need to\n@@ -2472,6 +2508,37 @@ static int merged_entry(const struct cache_entry *ce,\n \treturn 1;\n }\n \n+static int merged_sparse_dir(const struct cache_entry * const *src, int n,\n+\t\t\t struct unpack_trees_options *o)\n+{\n+\tstruct tree_desc t[MAX_UNPACK_TREES + 1];\n+\tvoid * tree_bufs[MAX_UNPACK_TREES + 1];\n+\tstruct traverse_info info;\n+\tint i, ret;\n+\n+\t/*\n+\t * Create the tree traversal information for traversing into *only* the\n+\t * sparse directory.\n+\t */\n+\tsetup_traverse_info(&info, src[0]->name);\n+\tinfo.fn = unpack_sparse_callback;\n+\tinfo.data = o;\n+\tinfo.show_all_errors = o->show_all_errors;\n+\tinfo.pathspec = o->pathspec;\n+\n+\t/* Get the tree descriptors of the sparse directory in each of the merging trees */\n+\tfor (i = 0; i < n; i++)\n+\t\ttree_bufs[i] = fill_tree_descriptor(o->src_index->repo, &t[i],\n+\t\t\t\t\t\t src[i] && !is_null_oid(&src[i]->oid) ? &src[i]->oid : NULL);\n+\n+\tret = traverse_trees(o->src_index, n, t, &info);\n+\n+\tfor (i = 0; i < n; i++)\n+\t\tfree(tree_bufs[i]);\n+\n+\treturn ret;\n+}\n+\n static int deleted_entry(const struct cache_entry *ce,\n \t\t\t const struct cache_entry *old,\n \t\t\t struct unpack_trees_options *o)\n@@ -2742,6 +2809,14 @@ int twoway_merge(const struct cache_entry * const *src,\n \t\t\t * reject the merge instead.\n \t\t\t */\n \t\t\treturn merged_entry(newtree, current, o);\n+\t\t} else if (S_ISSPARSEDIR(current->ce_mode)) {\n+\t\t\t/*\n+\t\t\t * The sparse directories differ, but we don't know whether that's\n+\t\t\t * because of two different files in the directory being modified\n+\t\t\t * (can be trivially merged) or if there is a real file conflict.\n+\t\t\t * Merge the sparse directory by OID to compare file-by-file.\n+\t\t\t */\n+\t\t\treturn merged_sparse_dir(src, 3, o);\n \t\t} else\n \t\t\treturn reject_merge(current, o);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"450013","messageId":"015618a9f292fc6eb20b21d40619cade343a9a0d.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] read-tree: integrate with sparse index","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:28Z","receivedAt":"2022-03-01T20:26:33Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nEnable use of sparse index in 'git read-tree'. The integration in this patch\nis limited only to usage of 'read-tree' that does not need additional\nfunctional changes for the sparse index to behave as expected (i.e., produce\nthe same user-facing results as a non-sparse index sparse-checkout). To\nensure no unexpected behavior occurs, the index is explicitly expanded when:\n\n* '--no-sparse-checkout' is specified (because it disables sparse-checkout)\n* '--prefix' is specified (if the prefix is inside a sparse directory, the\n prefixed tree cannot be properly traversed)\n* two or more arguments are specified ('twoway_merge' and\n 'threeway_merge' do not yet support merging sparse directories)\n\nSigned-off-by: Victoria Dye \n---\n builtin/read-tree.c | 21 +++++++++++++++++++--\n t/t1092-sparse-checkout-compatibility.sh | 11 +++++++++++\n 2 files changed, 30 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex c1a401971c2..0a52cab7752 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -160,8 +160,6 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \targc = parse_options(argc, argv, cmd_prefix, read_tree_options,\n \t\t\t read_tree_usage, 0);\n \n-\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n-\n \tprefix_set = opts.prefix ? 1 : 0;\n \tif (1 < opts.merge + opts.reset + prefix_set)\n \t\tdie(\"Which one? -m, --reset, or --prefix?\");\n@@ -173,6 +171,11 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.reset)\n \t\topts.reset = UNPACK_RESET_OVERWRITE_UNTRACKED;\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n+\thold_locked_index(&lock_file, LOCK_DIE_ON_ERROR);\n+\n \t/*\n \t * NEEDSWORK\n \t *\n@@ -214,6 +217,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \tif (opts.merge && !opts.index_only)\n \t\tsetup_work_tree();\n \n+\t/* TODO: audit sparse index behavior in unpack_trees */\n+\tif (opts.skip_sparse_checkout || opts.prefix)\n+\t\tensure_full_index(&the_index);\n+\n \tif (opts.merge) {\n \t\tswitch (stage - 1) {\n \t\tcase 0:\n@@ -223,11 +230,21 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \t\t\topts.fn = opts.prefix ? bind_merge : oneway_merge;\n \t\t\tbreak;\n \t\tcase 2:\n+\t\t\t/*\n+\t\t\t * TODO: update twoway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = twoway_merge;\n \t\t\topts.initial_checkout = is_cache_unborn();\n \t\t\tbreak;\n \t\tcase 3:\n \t\tdefault:\n+\t\t\t/*\n+\t\t\t * TODO: update threeway_merge to handle edit/edit conflicts in\n+\t\t\t * sparse directories.\n+\t\t\t */\n+\t\t\tensure_full_index(&the_index);\n \t\t\topts.fn = threeway_merge;\n \t\t\tbreak;\n \t\t}\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 9bb5aeb979c..86241b01a59 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1409,6 +1409,17 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'sparse index is not expanded: read-tree' '\n+\tinit_repos &&\n+\n+\tensure_not_expanded checkout -b test-branch update-folder1 &&\n+\tfor MERGE_TREES in \"update-folder2\"\n+\tdo\n+\t\tensure_not_expanded read-tree -mu $MERGE_TREES &&\n+\t\tensure_not_expanded reset --hard || return 1\n+\tdone\n+'\n+\n test_expect_success 'ls-files' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"450014","messageId":"90ebcb7b8ff4b4f1ba09abcbe636d639fa597e74.1646166271.git.gitgitgadget@gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] read-tree: expand sparse checkout test coverage","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T20:24:27Z","receivedAt":"2022-03-01T20:26:34Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdd tests focused on how 'git read-tree' behaves in sparse checkouts. Extra\nemphasis is placed on interactions with files outside the sparse cone, e.g.\nmerges with out-of-cone conflicts.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Victoria Dye \n---\n t/perf/p2000-sparse-operations.sh | 1 +\n t/t1092-sparse-checkout-compatibility.sh | 87 ++++++++++++++++++++++++\n 2 files changed, 88 insertions(+)\n\ndiff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh\nindex 2a7106b9495..382716cfca9 100755\n--- a/t/perf/p2000-sparse-operations.sh\n+++ b/t/perf/p2000-sparse-operations.sh\n@@ -117,6 +117,7 @@ test_perf_on_all git diff\n test_perf_on_all git diff --cached\n test_perf_on_all git blame $SPARSE_CONE/a\n test_perf_on_all git blame $SPARSE_CONE/f3/a\n+test_perf_on_all git read-tree -mu HEAD\n test_perf_on_all git checkout-index -f --all\n test_perf_on_all git update-index --add --remove $SPARSE_CONE/a\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex b1dcaa0e642..9bb5aeb979c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -819,6 +819,93 @@ test_expect_success 'update-index --cacheinfo' '\n \ttest_cmp expect sparse-checkout-out\n '\n \n+for MERGE_TREES in \"base HEAD update-folder2\" \\\n+\t\t \"update-folder1 update-folder2\" \\\n+\t\t \"update-folder2\"\n+do\n+\ttest_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n+\t\tinit_repos &&\n+\n+\t\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t\t# definition files will not exist on disk for sparse checkouts\n+\t\ttest_all_match git read-tree -mu $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_path_is_missing sparse-checkout/folder2 &&\n+\t\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\t\ttest_all_match git read-tree --reset -u HEAD &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\n+\t\ttest_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n+\t\ttest_all_match git status --porcelain=v2 &&\n+\t\ttest_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n+\t\ttest_cmp sparse-checkout/folder2/a full-checkout/folder2/a\n+\n+\t'\n+done\n+\n+test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n+\tinit_repos &&\n+\n+\t# Merge of multiple changes to same directory (but not same files) should\n+\t# succeed\n+\ttest_all_match git read-tree -mu base rename-base update-folder1 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match git read-tree -mu rename-base update-folder2 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git reset --hard &&\n+\n+\ttest_all_match test_must_fail git read-tree -mu base update-folder1 rename-out-to-in &&\n+\ttest_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n+'\n+\n+test_expect_success 'read-tree --prefix' '\n+\tinit_repos &&\n+\n+\t# If files differing between the index and target exist\n+\t# inside the prefix, `read-tree --prefix` should fail\n+\ttest_all_match test_must_fail git read-tree --prefix=deep/ deepest &&\n+\ttest_all_match test_must_fail git read-tree --prefix=folder1/ update-folder1 &&\n+\n+\t# If no differing index entries exist matching the prefix,\n+\t# `read-tree --prefix` updates the index successfully\n+\ttest_all_match git rm -rf deep/deeper1/deepest/ &&\n+\ttest_all_match git read-tree --prefix=deep/deeper1/deepest -u deepest &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git rm -rf --sparse folder1/ &&\n+\ttest_all_match git read-tree --prefix=folder1/ -u update-folder1 &&\n+\ttest_all_match git status --porcelain=v2 &&\n+\n+\ttest_all_match git rm -rf --sparse folder2/0 &&\n+\ttest_all_match git read-tree --prefix=folder2/0/ -u rename-out-to-out &&\n+\ttest_all_match git status --porcelain=v2\n+'\n+\n+test_expect_success 'read-tree --merge with directory-file conflicts' '\n+\tinit_repos &&\n+\n+\ttest_all_match git checkout -b test-branch rename-base &&\n+\n+\t# Although the index matches, without --no-sparse-checkout, outside-of-\n+\t# definition files will not exist on disk for sparse checkouts\n+\ttest_sparse_match git read-tree -mu rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_path_is_missing sparse-checkout/folder2 &&\n+\ttest_path_is_missing sparse-index/folder2 &&\n+\n+\ttest_sparse_match git read-tree --reset -u HEAD &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\n+\ttest_sparse_match git read-tree -mu --no-sparse-checkout rename-out-to-out &&\n+\ttest_sparse_match git status --porcelain=v2 &&\n+\ttest_cmp sparse-checkout/folder2/0/1 sparse-index/folder2/0/1\n+'\n+\n test_expect_success 'merge, cherry-pick, and rebase' '\n \tinit_repos &&\n \n-- \ngitgitgadget\n\n"},{"id":"450061","messageId":"CABPp-BHxt2cM0E47eYTn=+H-FYJEQ54O7x2TAbqawwFMWAGYgA@mail.gmail.com","threadId":"57464","inReplyTo":"pull.1157.v3.git.1646166271.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/8] Sparse index: integrate with 'read-tree'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-03-02T07:22:06Z","receivedAt":"2022-03-02T07:22:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 1, 2022 at 12:24 PM Victoria Dye via GitGitGadget\n wrote:\n>\n> Like previous integrations [1] [2], this series allows 'git read-tree' to\n> operate using a sparse index.\n>\n> The first three patches are bugfixes for issues found while implementing the\n> 'read-tree' integration:\n>\n> * The first (patch [1/8]) fixes an edge case in which a repo with no\n> in-cone files or directories would have its root collapsed into a sparse\n> directory; the fix ensures the root directory is never collapsed into a\n> sparse directory.\n> * The second (patch [2/8]) corrects the 'git status' reporting of changes\n> nested inside the subdirectory of a sparse directory, ensuring that the\n> modified file (not the subdirectory) is correctly reported as having\n> changes.\n> * The third (patch [3/8]) explicitly disallows the somewhat ambiguous case\n> of using 'git read-tree --prefix=/' to represent the repo root (it could\n> be interpreted as a leading slash - even though it's actually trailing\n> slash - which is not allowed for prefixes). Now, an error specifying that\n> prefixes cannot begin with '/' guides users more directly to the correct\n> syntax. If a user does want to specify the repo root as their prefix,\n> that can still be done with 'git read-tree --prefix='\n>\n> The remainder of the series focuses on utilizing the sparse index in 'git\n> read-tree'. After some baseline behavior-establishing tests (patch [4/8]),\n> sparse index usage is trivially enabled (patch [5/8]) for 'read-tree'\n> except:\n>\n> * usage with '--prefix'\n> * two- and three-way merge\n>\n> These cases require additional changes in order to work as expected (i.e.,\n> outwardly matching non-sparse index sparse-checkout). For the former, the\n> sparse index can be enabled as long as the index is expanded when the prefix\n> is a directory outside the sparse cone (patch [6/8]). For the latter, sparse\n> directories that cannot be trivially merged must have their contents merged\n> file-by-file, done by recursively traversing the trees represented by the\n> sparse directories (patches [7/8] & [8/8]).\n>\n>\n> Changes since V2\n> ================\n>\n> * revised/clarified explanation for why the 'recursive' flag is necessary\n> in 'git status' internal diff in both the commit message for [2/8] and\n> the code comments\n> * added patch [3/8] to disallow '/' as a prefix\n> * moved the loop over different 'read-tree' scenarios outside the test in\n> t1092 test 'read-tree --merge with files outside sparse definition',\n> separating the tests into individually-runnable and traceable cases\n> * improved test case clarity and variety in 'read-tree --prefix' tests\n> * removed redundant arg to 'git reset' in 'sparse index is not expanded:\n> read-tree'\n> * pointed out pre-existing testing that covers '--prefix' in [6/8] commit\n> message\n> * moved prefix-dependent index expansion logic into function\n> 'update_sparsity_for_prefix', improved explanation for function behavior\n> in comments\n> * pointed out pre-existing testing that covers two- and three-way merge in\n> [7/8] & [8/8] commit messages, respectively\n> * added test cases for \"ensure not expanded\" tests of read-tree for both\n> two- and three-way merge\n> * fixed issue with three-way merge implementation (if the \"no merge\" cases\n> described in 't/t1000-read-tree-m-3way.sh' are reached, sparse\n> directories should be recursively merged), testing added in the \"ensure\n> not expanded\" cases\n\nI've read through all of these. This round is:\n\nReviewed-by: Elijah Newren \n\n>\n>\n> Changes since V1\n> ================\n>\n> * switched an empty string check from '!strlen(path)' to the\n> slightly-less-expensive '!*path'\n>\n> Thanks!\n>\n> * Victoria\n>\n> [1]\n> https://lore.kernel.org/git/pull.1109.v2.git.1641924306.gitgitgadget@gmail.com/\n> [2]\n> https://lore.kernel.org/git/pull.1048.v6.git.1638201164.gitgitgadget@gmail.com/\n>\n> Victoria Dye (8):\n> sparse-index: prevent repo root from becoming sparse\n> status: fix nested sparse directory diff in sparse index\n> read-tree: explicitly disallow prefixes with a leading '/'\n> read-tree: expand sparse checkout test coverage\n> read-tree: integrate with sparse index\n> read-tree: narrow scope of index expansion for '--prefix'\n> read-tree: make two-way merge sparse-aware\n> read-tree: make three-way merge sparse-aware\n>\n> builtin/read-tree.c | 14 ++-\n> dir.c | 7 +-\n> t/perf/p2000-sparse-operations.sh | 1 +\n> t/t1003-read-tree-prefix.sh | 10 ++\n> t/t1092-sparse-checkout-compatibility.sh | 133 ++++++++++++++++++++\n> unpack-trees.c | 147 +++++++++++++++++++++--\n> wt-status.c | 9 ++\n> 7 files changed, 308 insertions(+), 13 deletions(-)\n>\n>\n> base-commit: e6ebfd0e8cbbd10878070c8a356b5ad1b3ca464e\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1157%2Fvdye%2Fsparse%2Fread-tree-v3\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1157/vdye/sparse/read-tree-v3\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1157\n>\n> Range-diff vs v2:\n>\n> 1: 744668eeece = 1: 744668eeece sparse-index: prevent repo root from becoming sparse\n> 2: f0cff03b95d ! 2: 26f4d30bd95 status: fix nested sparse directory diff in sparse index\n> @@ Metadata\n> ## Commit message ##\n> status: fix nested sparse directory diff in sparse index\n>\n> - Add the 'recursive' flag to 'wt_status_collect_changes_index(...)'. Without\n> - the 'recursive' flag, 'git status' could report index changes incorrectly\n> - when the following conditions were met:\n> + Enable the 'recursive' diff option for the diff executed as part of 'git\n> + status'. Without the 'recursive' enabled, 'git status' reports index\n> + changes incorrectly when the following conditions were met:\n>\n> * sparse index is enabled\n> * there is a difference between the index and HEAD in a file inside a\n> *subdirectory* of a sparse directory\n> * the sparse directory index entry is *not* expanded in-core\n>\n> - In this scenario, 'git status' would not recurse into the sparse directory's\n> - subdirectories to identify which file contained the difference between the\n> - index and HEAD. Instead, it would report the immediate subdirectory itself\n> - as \"modified\".\n> + Because it is not recursive by default, the diff in 'git status' reports\n> + changes only at the level of files and directories that are immediate\n> + children of a sparse directory, rather than recursing into directories with\n> + changes to identify the modified file(s). As a result, 'git status' reports\n> + the immediate subdirectory itself as \"modified\".\n>\n> Example:\n>\n> @@ Commit message\n> (use \"git restore --staged ...\" to unstage)\n> modified: sparse/sub\n>\n> - The 'recursive' diff option in 'wt_status_collect_changes_index' corrects\n> - this by indicating that 'git status' should recurse into sparse directories\n> - to find modified files. Given the same repository setup as the example\n> - above, the corrected result of `git status` is:\n> + Enabling the 'recursive' diff option in 'wt_status_collect_changes_index'\n> + corrects this issue by allowing the diff to recurse into subdirectories of\n> + sparse directories to find modified files. Given the same repository setup\n> + as the example above, the corrected result of `git status` is:\n>\n> $ git status\n> On branch master\n> @@ wt-status.c: static void wt_status_collect_changes_index(struct wt_status *s)\n> rev.diffopt.rename_score = s->rename_score >= 0 ? s->rename_score : rev.diffopt.rename_score;\n> +\n> + /*\n> -+ * The `recursive` option must be enabled to show differences in files\n> -+ * *more than* one level deep in a sparse directory index entry (e.g., given\n> -+ * sparse directory 'sparse-dir/', reporting a difference in the file\n> -+ * 'sparse-dir/another-dir/my-file').\n> ++ * The `recursive` option must be enabled to allow the diff to recurse\n> ++ * into subdirectories of sparse directory index entries. If it is not\n> ++ * enabled, a subdirectory containing file(s) with changes is reported\n> ++ * as \"modified\", rather than the modified files themselves.\n> + */\n> + rev.diffopt.flags.recursive = 1;\n> +\n> -: ----------- > 3: e48a281a4d3 read-tree: explicitly disallow prefixes with a leading '/'\n> 3: ffe0b6aff2b ! 4: 90ebcb7b8ff read-tree: expand sparse checkout test coverage\n> @@ Commit message\n> emphasis is placed on interactions with files outside the sparse cone, e.g.\n> merges with out-of-cone conflicts.\n>\n> + Helped-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Victoria Dye \n>\n> ## t/perf/p2000-sparse-operations.sh ##\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n> test_cmp expect sparse-checkout-out\n> '\n>\n> -+test_expect_success 'read-tree --merge with files outside sparse definition' '\n> -+ init_repos &&\n> -+\n> -+ test_all_match git checkout -b test-branch update-folder1 &&\n> -+ for MERGE_TREES in \"base HEAD update-folder2\" \\\n> -+ \"update-folder1 update-folder2\" \\\n> -+ \"update-folder2\"\n> -+ do\n> -+ # Clean up and remove on-disk files\n> -+ test_all_match git reset --hard HEAD &&\n> -+ test_sparse_match git sparse-checkout reapply &&\n> ++for MERGE_TREES in \"base HEAD update-folder2\" \\\n> ++ \"update-folder1 update-folder2\" \\\n> ++ \"update-folder2\"\n> ++do\n> ++ test_expect_success \"'read-tree -mu $MERGE_TREES' with files outside sparse definition\" '\n> ++ init_repos &&\n> +\n> + # Although the index matches, without --no-sparse-checkout, outside-of-\n> + # definition files will not exist on disk for sparse checkouts\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n> + test_all_match git read-tree -mu --no-sparse-checkout $MERGE_TREES &&\n> + test_all_match git status --porcelain=v2 &&\n> + test_cmp sparse-checkout/folder2/a sparse-index/folder2/a &&\n> -+ test_cmp sparse-checkout/folder2/a full-checkout/folder2/a || return 1\n> -+ done\n> -+'\n> ++ test_cmp sparse-checkout/folder2/a full-checkout/folder2/a\n> ++\n> ++ '\n> ++done\n> +\n> +test_expect_success 'read-tree --merge with edit/edit conflicts in sparse directories' '\n> + init_repos &&\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'update-index --ca\n> + test_all_match test_must_fail git read-tree -mu rename-out-to-in update-folder1\n> +'\n> +\n> -+test_expect_success 'read-tree --prefix outside sparse definition' '\n> ++test_expect_success 'read-tree --prefix' '\n> + init_repos &&\n> +\n> -+ # Cannot read-tree --prefix with a single argument when files exist within\n> -+ # prefix\n> -+ test_all_match test_must_fail git read-tree --prefix=folder1/ -u update-folder1 &&\n> ++ # If files differing between the index and target exist\n> ++ # inside the prefix, `read-tree --prefix` should fail\n> ++ test_all_match test_must_fail git read-tree --prefix=deep/ deepest &&\n> ++ test_all_match test_must_fail git read-tree --prefix=folder1/ update-folder1 &&\n> +\n> -+ test_all_match git read-tree --prefix=folder2/0 -u rename-base &&\n> -+ test_path_is_missing sparse-checkout/folder2 &&\n> -+ test_path_is_missing sparse-index/folder2 &&\n> ++ # If no differing index entries exist matching the prefix,\n> ++ # `read-tree --prefix` updates the index successfully\n> ++ test_all_match git rm -rf deep/deeper1/deepest/ &&\n> ++ test_all_match git read-tree --prefix=deep/deeper1/deepest -u deepest &&\n> ++ test_all_match git status --porcelain=v2 &&\n> ++\n> ++ test_all_match git rm -rf --sparse folder1/ &&\n> ++ test_all_match git read-tree --prefix=folder1/ -u update-folder1 &&\n> ++ test_all_match git status --porcelain=v2 &&\n> +\n> -+ test_all_match git read-tree --reset -u HEAD &&\n> -+ test_all_match git read-tree --prefix=folder2/0 -u --no-sparse-checkout rename-base &&\n> -+ test_cmp sparse-checkout/folder2/0/a sparse-index/folder2/0/a &&\n> -+ test_cmp sparse-checkout/folder2/0/a full-checkout/folder2/0/a\n> ++ test_all_match git rm -rf --sparse folder2/0 &&\n> ++ test_all_match git read-tree --prefix=folder2/0/ -u rename-out-to-out &&\n> ++ test_all_match git status --porcelain=v2\n> +'\n> +\n> +test_expect_success 'read-tree --merge with directory-file conflicts' '\n> 4: cb7e0cf419c ! 5: 015618a9f29 read-tree: integrate with sparse index\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> + for MERGE_TREES in \"update-folder2\"\n> + do\n> + ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> -+ ensure_not_expanded reset --hard HEAD || return 1\n> ++ ensure_not_expanded reset --hard || return 1\n> + done\n> +'\n> +\n> 5: 4f05fa70209 ! 6: 1a9365c3bc5 read-tree: narrow scope of index expansion for '--prefix'\n> @@ Commit message\n> If the prefix is in-cone, its sparse subdirectories (if any) will be\n> traversed correctly without index expansion.\n>\n> + The behavior of 'git read-tree' with prefixes 1) inside of cone, 2) equal to\n> + a sparse directory, and 3) inside a sparse directory are all tested as part\n> + of the 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --prefix',\n> + ensuring that the sparse index case works the way it did prior to this\n> + change as well as matching non-sparse index sparse-checkout.\n> +\n> + Helped-by: Elijah Newren \n> Signed-off-by: Victoria Dye \n>\n> ## builtin/read-tree.c ##\n> @@ t/t1092-sparse-checkout-compatibility.sh\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: read-tree' '\n> do\n> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> - ensure_not_expanded reset --hard HEAD || return 1\n> + ensure_not_expanded reset --hard || return 1\n> - done\n> + done &&\n> +\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> test_expect_success 'ls-files' '\n>\n> ## unpack-trees.c ##\n> -@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> - setup_standard_excludes(o->dir);\n> - }\n> +@@ unpack-trees.c: static void populate_from_existing_patterns(struct unpack_trees_options *o,\n> + o->pl = pl;\n> + }\n>\n> -+ /*\n> -+ * If the prefix is equal to or contained within a sparse directory, the\n> -+ * index needs to be expanded to traverse with the specified prefix.\n> -+ */\n> -+ if (o->prefix && o->src_index->sparse_index) {\n> -+ int prefix_len = strlen(o->prefix);\n> ++static void update_sparsity_for_prefix(const char *prefix,\n> ++ struct index_state *istate)\n> ++{\n> ++ int prefix_len = strlen(prefix);\n> ++ struct strbuf ce_prefix = STRBUF_INIT;\n> +\n> -+ while (prefix_len > 0 && o->prefix[prefix_len - 1] == '/')\n> -+ prefix_len--;\n> ++ if (!istate->sparse_index)\n> ++ return;\n> +\n> -+ if (prefix_len > 0) {\n> -+ struct strbuf ce_prefix = STRBUF_INIT;\n> -+ strbuf_grow(&ce_prefix, prefix_len + 1);\n> -+ strbuf_add(&ce_prefix, o->prefix, prefix_len);\n> -+ strbuf_addch(&ce_prefix, '/');\n> ++ while (prefix_len > 0 && prefix[prefix_len - 1] == '/')\n> ++ prefix_len--;\n> +\n> -+ /*\n> -+ * If the prefix is not inside the sparse cone, then the\n> -+ * index is explicitly expanded if it is found as a sparse\n> -+ * directory, or implicitly expanded (by 'index_name_pos')\n> -+ * if the path is inside a sparse directory.\n> -+ */\n> -+ if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, o->src_index) &&\n> -+ index_name_pos(o->src_index, ce_prefix.buf, ce_prefix.len) >= 0)\n> -+ ensure_full_index(o->src_index);\n> ++ if (prefix_len <= 0)\n> ++ BUG(\"Invalid prefix passed to update_sparsity_for_prefix\");\n> +\n> -+ strbuf_release(&ce_prefix);\n> -+ }\n> -+ }\n> ++ strbuf_grow(&ce_prefix, prefix_len + 1);\n> ++ strbuf_add(&ce_prefix, prefix, prefix_len);\n> ++ strbuf_addch(&ce_prefix, '/');\n> ++\n> ++ /*\n> ++ * If the prefix points to a sparse directory or a path inside a sparse\n> ++ * directory, the index should be expanded. This is accomplished in one\n> ++ * of two ways:\n> ++ * - if the prefix is inside a sparse directory, it will be expanded by\n> ++ * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n> ++ * - if the prefix matches an existing sparse directory entry,\n> ++ * 'index_name_pos(...)' will return its index position, triggering\n> ++ * the 'ensure_full_index(...)' below.\n> ++ */\n> ++ if (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n> ++ index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n> ++ ensure_full_index(istate);\n> ++\n> ++ strbuf_release(&ce_prefix);\n> ++}\n> +\n> + static int verify_absent(const struct cache_entry *,\n> + enum unpack_trees_error_types,\n> +@@ unpack-trees.c: int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> + setup_standard_excludes(o->dir);\n> + }\n> +\n> ++ if (o->prefix)\n> ++ update_sparsity_for_prefix(o->prefix, o->src_index);\n> +\n> if (!core_apply_sparse_checkout || !o->update)\n> o->skip_sparse_checkout = 1;\n> 6: 94c2aad2f93 ! 7: 71bec3ddad1 read-tree: make two-way merge sparse-aware\n> @@ Commit message\n> This process allows sparse directories to be individually merged at the\n> necessary depth *without* expanding a full index.\n>\n> + The 't/t1092-sparse-checkout-compatibility.sh' test 'read-tree --merge with\n> + edit/edit conflicts in sparse directories' tests two-way merges with 1)\n> + changes inside sparse directories that do not conflict and 2) changes that\n> + do conflict (with the correct file(s) reported in the error message).\n> + Additionally, add two-way merge cases to 'sparse index is not expanded:\n> + read-tree' to confirm that the index is not expanded regardless of whether\n> + edit/edit conflicts are present in a sparse directory.\n> +\n> Signed-off-by: Victoria Dye \n>\n> ## builtin/read-tree.c ##\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n>\n> ensure_not_expanded checkout -b test-branch update-folder1 &&\n> - for MERGE_TREES in \"update-folder2\"\n> -+ for MERGE_TREES in \"update-folder2\" \\\n> -+ \"base update-folder2\"\n> ++ for MERGE_TREES in \"base update-folder2\" \\\n> ++ \"base rename-base\" \\\n> ++ \"update-folder2\"\n> do\n> ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> - ensure_not_expanded reset --hard HEAD || return 1\n> + ensure_not_expanded reset --hard || return 1\n>\n> ## unpack-trees.c ##\n> @@ unpack-trees.c: static int is_sparse_directory_entry(struct cache_entry *ce,\n> 7: c4080e99d6e ! 8: 6bc11325dd1 read-tree: make three-way merge sparse-aware\n> @@ Commit message\n> merge, the contents of each conflicted sparse directory are merged without\n> referencing the index, avoiding sparse index expansion.\n>\n> + As with two-way merge, the 't/t1092-sparse-checkout-compatibility.sh' test\n> + 'read-tree --merge with edit/edit conflicts in sparse directories' confirms\n> + that three-way merges with edit/edit changes (both with and without\n> + conflicts) inside a sparse directory result in the correct index state or\n> + error message. To ensure the index is not unnecessarily expanded, add\n> + three-way merge cases to 'sparse index is not expanded: read-tree'.\n> +\n> Signed-off-by: Victoria Dye \n>\n> ## builtin/read-tree.c ##\n> @@ builtin/read-tree.c: int cmd_read_tree(int argc, const char **argv, const char *\n>\n> ## t/t1092-sparse-checkout-compatibility.sh ##\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: read-tree' '\n> + init_repos &&\n>\n> ensure_not_expanded checkout -b test-branch update-folder1 &&\n> - for MERGE_TREES in \"update-folder2\" \\\n> -- \"base update-folder2\"\n> +- for MERGE_TREES in \"base update-folder2\" \\\n> ++ for MERGE_TREES in \"base HEAD update-folder2\" \\\n> ++ \"base HEAD rename-base\" \\\n> + \"base update-folder2\" \\\n> -+ \"base HEAD update-folder2\"\n> + \"base rename-base\" \\\n> + \"update-folder2\"\n> do\n> - ensure_not_expanded read-tree -mu $MERGE_TREES &&\n> - ensure_not_expanded reset --hard HEAD || return 1\n>\n> ## unpack-trees.c ##\n> @@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n> @@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n>\n> if (head) {\n> /* #5ALT, #15 */\n> +@@ unpack-trees.c: int threeway_merge(const struct cache_entry * const *stages,\n> +\n> + }\n> +\n> +- /* Below are \"no merge\" cases, which require that the index be\n> +- * up-to-date to avoid the files getting overwritten with\n> +- * conflict resolution files.\n> +- */\n> ++ /* Handle \"no merge\" cases (see t/t1000-read-tree-m-3way.sh) */\n> + if (index) {\n> ++ /*\n> ++ * If we've reached the \"no merge\" cases and we're merging\n> ++ * a sparse directory, we may have an \"edit/edit\" conflict that\n> ++ * can be resolved by individually merging directory contents.\n> ++ */\n> ++ if (S_ISSPARSEDIR(index->ce_mode))\n> ++ return merged_sparse_dir(stages, 4, o);\n> ++\n> ++ /*\n> ++ * If we're not merging a sparse directory, ensure the index is\n> ++ * up-to-date to avoid files getting overwritten with conflict\n> ++ * resolution files\n> ++ */\n> + if (verify_uptodate(index, o))\n> + return -1;\n> + }\n>\n> --\n> gitgitgadget\n"},{"id":"450101","messageId":"7f16a81e-d806-57f9-51c4-ff9462c388e5@github.com","threadId":"57464","inReplyTo":"CABPp-BHxt2cM0E47eYTn=+H-FYJEQ54O7x2TAbqawwFMWAGYgA@mail.gmail.com","subject":"Re: [PATCH v3 0/8] Sparse index: integrate with 'read-tree'","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-02T13:40:46Z","receivedAt":"2022-03-02T13:40:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/2/2022 2:22 AM, Elijah Newren wrote:\n> On Tue, Mar 1, 2022 at 12:24 PM Victoria Dye via GitGitGadget\n> wrote:\n\n> I've read through all of these. This round is:\n> \n> Reviewed-by: Elijah Newren \n\nThanks to your and Ævar's extra eyes, this series has\nonly improved since I reviewed v1.\n\nThanks,\n-Stolee\n"},{"id":"450295","messageId":"kl6lczj2exbn.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57464","inReplyTo":"1a9365c3bc5b810a60593612bfba97a8b0c1d657.1646166271.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-03T17:54:20Z","receivedAt":"2022-03-03T17:54:26Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"Victoria Dye via GitGitGadget\" writes:\n\n> From: Victoria Dye \n> +static void update_sparsity_for_prefix(const char *prefix,\n> +\t\t\t\t struct index_state *istate)\n> +{\n> +\tint prefix_len = strlen(prefix);\n> +\tstruct strbuf ce_prefix = STRBUF_INIT;\n> +\n> +\tif (!istate->sparse_index)\n> +\t\treturn;\n> +\n> +\twhile (prefix_len > 0 && prefix[prefix_len - 1] == '/')\n> +\t\tprefix_len--;\n> +\n> +\tif (prefix_len <= 0)\n> +\t\tBUG(\"Invalid prefix passed to update_sparsity_for_prefix\");\n> +\n> +\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n> +\tstrbuf_add(&ce_prefix, prefix, prefix_len);\n> +\tstrbuf_addch(&ce_prefix, '/');\n> +\n> +\t/*\n> +\t * If the prefix points to a sparse directory or a path inside a sparse\n> +\t * directory, the index should be expanded. This is accomplished in one\n> +\t * of two ways:\n> +\t * - if the prefix is inside a sparse directory, it will be expanded by\n> +\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n> +\t * - if the prefix matches an existing sparse directory entry,\n> +\t * 'index_name_pos(...)' will return its index position, triggering\n> +\t * the 'ensure_full_index(...)' below.\n> +\t */\n> +\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n> +\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n> +\t\tensure_full_index(istate);\n> +\n> +\tstrbuf_release(&ce_prefix);\n> +}\n\nHm, I don't think I follow the rationale for having two different ways\nof expanding the index:\n\n- If the prefix is inside a sparse directory, we should expand the\n index.\n- If the prefix matches a sparse directory entry, we should expand the\n index.\n\nSo it seems like distinguishing between the two cases with\nindex_name_pos(...) isn't necessary. I've attached a diff that does\nexactly this, and it passes t1092-sparse-checkout-compatibility.sh as\nfar as I can tell. I've also amended the comment in a way that makes\nmore sense to me, but I'm not 100% sure if it's accurate.\n\nI'm also a little averse to using a side effect of index_name_pos() to\nachieve what we really want, so I'd prefer to get rid of the call if we\ncan :)\n\n----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b876caca0d..5b07055605 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1749,17 +1749,11 @@ static void update_sparsity_for_prefix(const char *prefix,\n \tstrbuf_addch(&ce_prefix, '/');\n \n \t/*\n-\t * If the prefix points to a sparse directory or a path inside a sparse\n-\t * directory, the index should be expanded. This is accomplished in one\n-\t * of two ways:\n-\t * - if the prefix is inside a sparse directory, it will be expanded by\n-\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n-\t * - if the prefix matches an existing sparse directory entry,\n-\t * 'index_name_pos(...)' will return its index position, triggering\n-\t * the 'ensure_full_index(...)' below.\n+\t * If the prefix points to a sparse directory or a path inside a\n+\t * sparse directory (not within the sparse patterns), the index\n+\t * should be expanded.\n \t */\n-\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n-\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n+\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate))\n \t\tensure_full_index(istate);\n \n \tstrbuf_release(&ce_prefix);\n"},{"id":"450306","messageId":"5353fb06-09e4-77bf-554c-1cd750158730@github.com","threadId":"57464","inReplyTo":"kl6lczj2exbn.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 6/8] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-03-03T21:19:06Z","receivedAt":"2022-03-03T21:19:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Glen Choo wrote:\n> \"Victoria Dye via GitGitGadget\" writes:\n> \n>> From: Victoria Dye \n>> +static void update_sparsity_for_prefix(const char *prefix,\n>> +\t\t\t\t struct index_state *istate)\n>> +{\n>> +\tint prefix_len = strlen(prefix);\n>> +\tstruct strbuf ce_prefix = STRBUF_INIT;\n>> +\n>> +\tif (!istate->sparse_index)\n>> +\t\treturn;\n>> +\n>> +\twhile (prefix_len > 0 && prefix[prefix_len - 1] == '/')\n>> +\t\tprefix_len--;\n>> +\n>> +\tif (prefix_len <= 0)\n>> +\t\tBUG(\"Invalid prefix passed to update_sparsity_for_prefix\");\n>> +\n>> +\tstrbuf_grow(&ce_prefix, prefix_len + 1);\n>> +\tstrbuf_add(&ce_prefix, prefix, prefix_len);\n>> +\tstrbuf_addch(&ce_prefix, '/');\n>> +\n>> +\t/*\n>> +\t * If the prefix points to a sparse directory or a path inside a sparse\n>> +\t * directory, the index should be expanded. This is accomplished in one\n>> +\t * of two ways:\n>> +\t * - if the prefix is inside a sparse directory, it will be expanded by\n>> +\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n>> +\t * - if the prefix matches an existing sparse directory entry,\n>> +\t * 'index_name_pos(...)' will return its index position, triggering\n>> +\t * the 'ensure_full_index(...)' below.\n>> +\t */\n>> +\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n>> +\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n>> +\t\tensure_full_index(istate);\n>> +\n>> +\tstrbuf_release(&ce_prefix);\n>> +}\n> \n> Hm, I don't think I follow the rationale for having two different ways\n> of expanding the index:\n> \n> - If the prefix is inside a sparse directory, we should expand the\n> index.\n> - If the prefix matches a sparse directory entry, we should expand the\n> index.\n> \n\nThere's an (admittedly subtle if you aren't familiar with sparse index)\ndistinction between a \"sparse directory\" index entry and \"a directory\noutside the sparse-checkout cone with SKIP_WORKTREE enabled on its entries\".\nUltimately, that's what necessitates the two checks but, as in [1], I want\nto use this as an opportunity to shed some light on what 'unpack_trees(...)'\ndoes.\n\nTaking a step back, why would index expansion matter when you pass a prefix\nto 'read-tree'? The answer lies in the tree traversal at the core of\n'unpack_trees(...)'; when a prefix is provided, 'unpack_trees' does the\nfollowing:\n\n1. Set the \"traversal prefix\" to the user-provided prefix.\n2. Call 'traverse_trees(...)', iterating through the child files/directories\n of the prefix directory (NOTE: this does *not* use the index - it finds\n the child entries by extracting them from the tree).\n3. For each child, call 'unpack_callback' which, among other things, looks\n for the child entry as it exists in the index to merge the input trees\n into it.\n\nThe problem with sparse directories arises in step 3. Suppose you have the\nfollowing repo:\n\n.\n├── bar\n│ └── f1\n├── baz\n│ ├── deep\n│ │ └── a\n│ └── f2\n├── foo\n└── foo1\n\n\nwhere directory 'baz/' is sparse, leading to an index that looks like this\n(per 'git ls-files -t --sparse'):\n\nH bar/f1\nS baz/\nH foo\nH foo1\n\nIf you provide 'baz/' as a prefix, 'unpack_callback(...)' tries to find\n'baz/deep/a' and 'baz/f2' in the index, but they won't be found because\nthey're \"wrapped\" in sparse directory 'baz/'. Ultimately, this leads to a\ncorrupted index with duplicate 'baz/' contents merged in:\n\nH bar/f1\nS baz/\nS baz/deep/a\nS baz/f2\nH foo\nH foo1\n\nThis explains why you need to expand the index at all, but not why you need\nto check that the prefix is not in the sparse cone *and* that it (or its\nparent directory) doesn't exist in the index. For that, an important bit of\ncontext is that you can have a sparse index with non-sparse directories\noutside the cone. This can happen in a number of ways (for example, running\n'git update-index --no-skip-worktree' on a file in a sparse directory), but\nthe important thing is that it is a completely valid state and not entirely\nuncommon. \n\nUsing our example above, suppose 'baz/' is partially expanded in the index,\nwith the following index contents:\n\nH bar/f1\nS baz/deep/\nS baz/f2\nH foo\nH foo1\n\nIf we use the prefix 'baz/' here, we actually traverse the trees properly:\n'baz/deep/' and 'baz/f2' will be found and merged - no index expansion\nneeded! But if we only checked '!path_in_cone_mode_sparse_checkout(...)', we\nwould have expanded the index because 'baz/' is outside the sparse cone. \n\nThis presents a problem because index expansion is *extremely* expensive -\nwe should avoid it whenever possible. That's where checking\n'index_name_pos(...)' comes in: if the directory is in the index as a sparse\ndirectory, the position is '>= 0' and 'ensure_full_index(...)' is called; if\nthe directory is inside an existing sparse directory, the position will be\n'< 0' but the index will be expanded implicitly. In every other case, we\navoid expanding the index and proceed with the merge as normal.\n\nHope this helps!\n\n[1] https://lore.kernel.org/git/dc47f12b-8724-22ef-ed2c-096badfafd76@github.com/\n\n> So it seems like distinguishing between the two cases with\n> index_name_pos(...) isn't necessary. I've attached a diff that does\n> exactly this, and it passes t1092-sparse-checkout-compatibility.sh as\n> far as I can tell. I've also amended the comment in a way that makes\n> more sense to me, but I'm not 100% sure if it's accurate.\n> \n> I'm also a little averse to using a side effect of index_name_pos() to\n> achieve what we really want, so I'd prefer to get rid of the call if we\n> can :)\n> \n> ----- >8 --------- >8 --------- >8 --------- >8 --------- >8 ----\n> \n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index b876caca0d..5b07055605 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1749,17 +1749,11 @@ static void update_sparsity_for_prefix(const char *prefix,\n> \tstrbuf_addch(&ce_prefix, '/');\n> \n> \t/*\n> -\t * If the prefix points to a sparse directory or a path inside a sparse\n> -\t * directory, the index should be expanded. This is accomplished in one\n> -\t * of two ways:\n> -\t * - if the prefix is inside a sparse directory, it will be expanded by\n> -\t * the 'ensure_full_index(...)' call in 'index_name_pos(...)'.\n> -\t * - if the prefix matches an existing sparse directory entry,\n> -\t * 'index_name_pos(...)' will return its index position, triggering\n> -\t * the 'ensure_full_index(...)' below.\n> +\t * If the prefix points to a sparse directory or a path inside a\n> +\t * sparse directory (not within the sparse patterns), the index\n> +\t * should be expanded.\n> \t */\n> -\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate) &&\n> -\t index_name_pos(istate, ce_prefix.buf, ce_prefix.len) >= 0)\n> +\tif (!path_in_cone_mode_sparse_checkout(ce_prefix.buf, istate))\n> \t\tensure_full_index(istate);\n> \n> \tstrbuf_release(&ce_prefix);\n\n"},{"id":"450431","messageId":"kl6lilst4kt3.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"57464","inReplyTo":"5353fb06-09e4-77bf-554c-1cd750158730@github.com","subject":"Re: [PATCH v3 6/8] read-tree: narrow scope of index expansion for '--prefix'","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-03-04T18:47:04Z","receivedAt":"2022-03-04T18:47:10Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Victoria Dye writes:\n> There's an (admittedly subtle if you aren't familiar with sparse index)\n> distinction between a \"sparse directory\" index entry and \"a directory\n> outside the sparse-checkout cone with SKIP_WORKTREE enabled on its entries\".\n> Ultimately, that's what necessitates the two checks but, as in [1], I want\n> to use this as an opportunity to shed some light on what 'unpack_trees(...)'\n> does.\n\nThanks! This explanation was really helpful.\n\n> Using our example above, suppose 'baz/' is partially expanded in the index,\n> with the following index contents:\n>\n> H bar/f1\n> S baz/deep/\n> S baz/f2\n> H foo\n> H foo1\n>\n> If we use the prefix 'baz/' here, we actually traverse the trees properly:\n> 'baz/deep/' and 'baz/f2' will be found and merged - no index expansion\n> needed! But if we only checked '!path_in_cone_mode_sparse_checkout(...)', we\n> would have expanded the index because 'baz/' is outside the sparse cone. \n\nIn particular, I didn't consider that a directory outside of the\nsparse-checkout cone could be partially expanded. This seems to be the\ncrux of it, which is that even if the path is outside of the\nsparse-checkout clone, we can still get correct behavior (without\nexpanding the index) if its entries are expanded...\n\n> This presents a problem because index expansion is *extremely* expensive -\n> we should avoid it whenever possible. That's where checking\n> 'index_name_pos(...)' comes in: if the directory is in the index as a sparse\n> directory, the position is '>= 0' and 'ensure_full_index(...)' is called; if\n> the directory is inside an existing sparse directory, the position will be\n> '< 0' but the index will be expanded implicitly. In every other case, we\n> avoid expanding the index and proceed with the merge as normal.\n\nand because of this, we don't always need to expand the index when the\npath is outside of the cone, so my suggested patch expands the index in\ntoo many cases.\n\nWhat I also didn't consider is that index_name_pos() doesn't _always_\nexpand the index, it only expands the index when the directory is\ninside a sparse directory entry.\n\nSo the side-effect of index_name_pos() is actually _exactly_ what we\nwant. Granted, it would be clearer if we had a function that did _only_\n'expand if path is inside a sparse directory entry', but I suppose it's\noverkill.\n\n(Purely optional suggestion) I wonder if we could add a test that can\ndistinguish between 'always expand if --prefix is outside of the cone'\nvs 'expand only if path is outside of cone AND inside a sparse directory\nentry'. The scenario you described sounds perfect as a test case, though\nI don't know how feasible it is to set up.\n"}]}