{"thread":{"id":"56917","subject":"[PATCH 1/2] fetch/pull: use the sparse index","startedAt":"2021-11-16T15:38:14Z","lastAt":"2021-12-22T23:56:45Z","messageCount":51,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Derrick Stolee","Elijah Newren","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"441332","messageId":"451056e1a77893374ec07509f25d35c11eae59b7.1637077083.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.git.1637077083.gitgitgadget@gmail.com","subject":"[PATCH 1/2] fetch/pull: use the sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T15:38:01Z","receivedAt":"2021-11-16T15:38:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git fetch' and 'git pull' commands parse the index in order to\ndetermine if submodules exist. Without command_requires_full_index=0,\nthis will expand a sparse index, causing slow performance even when\nthere is no new data to fetch.\n\nThe .gitmodules file will never be inside a sparse directory entry, and\neven if it was, the index_name_pos() method would expand the sparse\nindex if needed as we search for the path by name. These commands do not\niterate over the index, which is the typical thing we are careful about\nwhen integrating with the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 2 ++\n builtin/pull.c | 2 ++\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f7abbc31ff1..1696b7791d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1993,6 +1993,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tgit_config(git_fetch_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 127798ba84e..9f6174f4433 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -993,6 +993,8 @@ int cmd_pull(int argc, const char **argv, const char *prefix)\n \t\tset_reflog_message(argc, argv);\n \n \tgit_config(git_pull_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix, pull_options, pull_usage, 0);\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex d75f64aad56..f8a8dde60af 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -932,6 +932,16 @@ test_expect_success 'sparse index is not expanded: blame' '\n \tensure_not_expanded blame deep/deeper1/deepest/a\n '\n \n+test_expect_success 'sparse index is not expanded: fetch/pull' '\n+\tinit_repos &&\n+\n+\tgit -C sparse-index remote add full \"file://$(pwd)/full-checkout\" &&\n+\tensure_not_expanded fetch full &&\n+\tgit -C full-checkout commit --allow-empty -m \"for pull merge\" &&\n+\tgit -C sparse-index commit --allow-empty -m \"for pull merge\" &&\n+\tensure_not_expanded pull full base\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"441333","messageId":"pull.1080.git.1637077083.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":null,"subject":"[PATCH 0/2] Sparse index: fetch, pull, ls-files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T15:38:00Z","receivedAt":"2021-11-16T15:38:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ld/sparse-index-blame (merged with 'master' due to an\nunrelated build issue).\n\nHere are two relatively-simple patches that further the sparse index\nintegrations.\n\nDid you know that 'fetch' and 'pull' read the index? I didn't, or this would\nhave been an integration much earlier in the cycle. They read the index to\nlook for the .gitmodules file in case there are submodules that need to be\nfetched. Since looking for a file by name is already protected, we only need\nto disable 'command_requires_full_index' and we are done.\n\nThe 'ls-files' builtin is useful when debugging the index, and some scripts\nuse it, too. We are not changing the default behavior which expands a sparse\nindex in order to show all of the cached blobs. Instead, we add a '--sparse'\noption that allows us to see the sparse directory entries upon request.\nCombined with --debug, we can see a lot of index details, such as:\n\n$ git ls-files --debug --sparse\nLICENSE\n ctime: 1634910503:287405820\n mtime: 1634910503:287405820\n dev: 16777220 ino: 119325319\n uid: 501 gid: 20\n size: 1098 flags: 200000\nREADME.md\n ctime: 1634910503:288090279\n mtime: 1634910503:288090279\n dev: 16777220 ino: 119325320\n uid: 501 gid: 20\n size: 934 flags: 200000\nbin/index.js\n ctime: 1634910767:828434033\n mtime: 1634910767:828434033\n dev: 16777220 ino: 119325520\n uid: 501 gid: 20\n size: 7292 flags: 200000\nexamples/\n ctime: 0:0\n mtime: 0:0\n dev: 0 ino: 0\n uid: 0 gid: 0\n size: 0 flags: 40004000\npackage.json\n ctime: 1634910503:288676330\n mtime: 1634910503:288676330\n dev: 16777220 ino: 119325321\n uid: 501 gid: 20\n size: 680 flags: 200000\n\n\n(In this example, the 'examples/' directory is sparse.)\n\nThanks!\n\nDerrick Stolee (2):\n fetch/pull: use the sparse index\n ls-files: add --sparse option\n\n Documentation/git-ls-files.txt | 4 ++\n builtin/fetch.c | 2 +\n builtin/ls-files.c | 12 ++++-\n builtin/pull.c | 2 +\n t/t1092-sparse-checkout-compatibility.sh | 57 ++++++++++++++++++++++++\n 5 files changed, 75 insertions(+), 2 deletions(-)\n\n\nbase-commit: ce9c834d5de7026a5271e289ab4518e569464171\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1080%2Fderrickstolee%2Fsparse-index%2Ffetch-pull-ls-files-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1080/derrickstolee/sparse-index/fetch-pull-ls-files-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1080\n-- \ngitgitgadget\n"},{"id":"441334","messageId":"e42c0feec94de0e4869cda1fc6b28bd7055774e3.1637077083.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.git.1637077083.gitgitgadget@gmail.com","subject":"[PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-11-16T15:38:02Z","receivedAt":"2021-11-16T15:38:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExisting callers to 'git ls-files' are expecting file names, not\ndirectories. It is best to expand a sparse index to show all of the\ncontained files in this case.\n\nHowever, expert users may want to inspect the contents of the index\nitself including which directories are sparse. Add a --sparse option to\nallow users to request this information.\n\nDuring testing, I noticed that options such as --modified did not affect\nthe output when the files in question were outside the sparse-checkout\ndefinition. Tests are added to document this preexisting behavior and\nhow it remains unchanged with the sparse index and the --sparse option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ls-files.txt | 4 ++\n builtin/ls-files.c | 12 +++++-\n t/t1092-sparse-checkout-compatibility.sh | 47 ++++++++++++++++++++++++\n 3 files changed, 61 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex 6d11ab506b7..1c5d5f85ec5 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -187,6 +187,10 @@ Both the in the index (\"i/\")\n and in the working tree (\"w/\") are shown for regular files,\n followed by the (\"attr/\").\n \n+--sparse::\n+\tIf the index is sparse, show the sparse directories without expanding\n+\tto the contained files.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 031fef1bcaa..c151eb1fb77 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -37,6 +37,7 @@ static int debug_mode;\n static int show_eol;\n static int recurse_submodules;\n static int skipping_duplicates;\n+static int show_sparse_dirs;\n \n static const char *prefix;\n static int max_prefix_len;\n@@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\n+\tif (!show_sparse_dirs)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tOPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n \t\tOPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n \t\t\t N_(\"suppress duplicate entries\")),\n+\t\tOPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n+\t\t\t N_(\"show sparse directories in the presence of a sparse index\")),\n \t\tOPT_END()\n \t};\n \tint ret = 0;\n@@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(ls_files_usage, builtin_ls_files_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tprefix = cmd_prefix;\n \tif (prefix)\n \t\tprefix_len = strlen(prefix);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex f8a8dde60af..ffb6052ff60 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -814,6 +814,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt &&\n+\n+\t# ls-files expands on read, but does not write.\n+\trm trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index ls-files &&\n \ttest_region index ensure_full_index trace2.txt\n '\n \n@@ -838,6 +844,7 @@ test_expect_success 'sparse-index is not expanded' '\n \tinit_repos &&\n \n \tensure_not_expanded status &&\n+\tensure_not_expanded ls-files --sparse &&\n \tensure_not_expanded commit --allow-empty -m empty &&\n \techo >>sparse-index/a &&\n \tensure_not_expanded commit -a -m a &&\n@@ -942,6 +949,46 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'ls-files' '\n+\tinit_repos &&\n+\n+\t# Behavior agrees by default. Sparse index is expanded.\n+\ttest_all_match git ls-files &&\n+\n+\t# With --sparse, the sparse index data changes behavior.\n+\tgit -C sparse-index ls-files --sparse >sparse-index-out &&\n+\tgrep \"^folder1/\\$\" sparse-index-out &&\n+\tgrep \"^folder2/\\$\" sparse-index-out &&\n+\n+\t# With --sparse and no sparse index, nothing changes.\n+\tgit -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n+\tgrep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n+\t! grep \"/\\$\" sparse-checkout-out &&\n+\n+\twrite_script edit-content <<-\\EOF &&\n+\tmkdir folder1 &&\n+\techo content >>folder1/a\n+\tEOF\n+\trun_on_sparse ../edit-content &&\n+\n+\t# ls-files does not notice modified files whose\n+\t# cache entries are marked SKIP_WORKTREE.\n+\ttest_sparse_match git ls-files --modified &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\trun_on_sparse git sparse-checkout add folder1 &&\n+\ttest_sparse_match git ls-files --modified &&\n+\tgrep \"^folder1/a\\$\" sparse-checkout-out &&\n+\tgrep \"^folder1/a\\$\" sparse-index-out &&\n+\n+\t# Double-check index expansion\n+\tensure_not_expanded ls-files --sparse\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n"},{"id":"441426","messageId":"xmqqlf1n2krf.fsf@gitster.g","threadId":"56917","inReplyTo":"pull.1080.git.1637077083.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-17T09:29:08Z","receivedAt":"2021-11-17T09:29:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This is based on ld/sparse-index-blame (merged with 'master' due to an\n> unrelated build issue).\n>\n> Here are two relatively-simple patches that further the sparse index\n> integrations.\n>\n> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n> have been an integration much earlier in the cycle. They read the index to\n> look for the .gitmodules file in case there are submodules that need to be\n> fetched. Since looking for a file by name is already protected, we only need\n> to disable 'command_requires_full_index' and we are done.\n\nThis reminds me of one thing. Our strategy so far has been:\n\n - start with blindly calling ensure_full();\n\n - audit the code and adjust each code path that needs to walk to\n\n . keep walking the full index, but narrow the scope of\n ensure_full_index() if we can; or\n\n . stop assuming we need to walk the full index, but teach it to\n \"skip\" the tree-in-index that we are not interested in.\n\n - declare victory and drop the blind call to ensure_full().\n\nBut what makes sure, after all of the above happens, that no new\nchanges that assume it can walk the full index enters in the\ncodebase?\n\nIn other words, after \"fetch\" is declared \"sparse clean\" with patch\n[1/2], what effort from us will it take to stay clean?\n\nThanks.\n"},{"id":"441500","messageId":"08679a6f-51c1-0e48-9dcc-e7ac4c9a9c80@gmail.com","threadId":"56917","inReplyTo":"xmqqlf1n2krf.fsf@gitster.g","subject":"Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-17T15:28:22Z","receivedAt":"2021-11-17T15:28:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/17/2021 4:29 AM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> This is based on ld/sparse-index-blame (merged with 'master' due to an\n>> unrelated build issue).\n>>\n>> Here are two relatively-simple patches that further the sparse index\n>> integrations.\n>>\n>> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n>> have been an integration much earlier in the cycle. They read the index to\n>> look for the .gitmodules file in case there are submodules that need to be\n>> fetched. Since looking for a file by name is already protected, we only need\n>> to disable 'command_requires_full_index' and we are done.\n> \n> This reminds me of one thing. Our strategy so far has been:\n> \n> - start with blindly calling ensure_full();\n> \n> - audit the code and adjust each code path that needs to walk to\n> \n> . keep walking the full index, but narrow the scope of\n> ensure_full_index() if we can; or\n> \n> . stop assuming we need to walk the full index, but teach it to\n> \"skip\" the tree-in-index that we are not interested in.\n> \n> - declare victory and drop the blind call to ensure_full().\n\nThis is a fair summary of the approach.\n\n> But what makes sure, after all of the above happens, that no new\n> changes that assume it can walk the full index enters in the\n> codebase?\n> \n> In other words, after \"fetch\" is declared \"sparse clean\" with patch\n> [1/2], what effort from us will it take to stay clean?\n\nThe tests in t1092 that use the \"ensure_not_expanded\" helper are\nintended to be regression tests that would start failing if the\nsparse index starts expanding in a new way. I think this is what\nyou mean by staying \"sparse clean\".\n\nThe rest of the tests are attempting to verify that the behavior\nis correct when the sparse index is enabled, and that is hopefully\na more obvious situation when something goes wrong. We've tried to\ncreate a robust set of tests here, but I'm sure we will add new\nones as more users start using it. (We will soon have a large set\nof real users of the sparse index, and that should be informative.)\n\nDo you see a test gap that would be prudent to address?\n\nOne direction I could see is that as new features are contributed\nthat change how the index is used, these features are not\nautomatically tested with sparse-checkout and the sparse index.\nIn this case, we will need to increase our awareness when reviewing\nsuch features to ensure that they could fit within the sparse index\nmodel (or are sufficiently protected by ensure_full_index() in their\nfirst version).\n\nI spent some time thinking about how we could add a\nGIT_TEST_SPARSE_INDEX mode to the test suite that would automatically\ntest the sparse index in the existing tests. However, the feature is\nonly enabled when in cone-mode sparse-checkout, so we would need to\nmodify the test repos in a way that some sparse directory entries\nexist but also don't collide with the test expectations. I never\nfound a way to get around that issue, which is why t1092 is such a\nbig test script.\n\nThanks,\n-Stolee\n"},{"id":"441667","messageId":"xmqqfsrtw1s2.fsf@gitster.g","threadId":"56917","inReplyTo":"08679a6f-51c1-0e48-9dcc-e7ac4c9a9c80@gmail.com","subject":"Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-11-18T22:13:17Z","receivedAt":"2021-11-18T22:13:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> But what makes sure, after all of the above happens, that no new\n>> changes that assume it can walk the full index enters in the\n>> codebase?\n>> \n>> In other words, after \"fetch\" is declared \"sparse clean\" with patch\n>> [1/2], what effort from us will it take to stay clean?\n>\n> The tests in t1092 that use the \"ensure_not_expanded\" helper are\n> intended to be regression tests that would start failing if the\n> sparse index starts expanding in a new way. I think this is what\n> you mean by staying \"sparse clean\".\n\nIt is more like how would we help a new piece of code, which does\nnot even consider the possibility that the in-core index might be\nsparse, and instead blindly walks the in-core index entries, doing\nsomething unexpected (to both the author of the new code and to\nthose who are aware of the sparse-index feature) when it sees an\n\"tree\" entry. We'll hopefully see a breakage and the developer who\nadded such a new piece of code can keep both halves?\n\n> One direction I could see is that as new features are contributed\n> that change how the index is used, these features are not\n> automatically tested with sparse-checkout and the sparse index.\n> In this case, we will need to increase our awareness when reviewing\n> such features to ensure that they could fit within the sparse index\n> model (or are sufficiently protected by ensure_full_index() in their\n> first version).\n\nYup, making developers aware of the issue is probably the good first\nthing to do.\n\n"},{"id":"442002","messageId":"CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com","threadId":"56917","inReplyTo":"e42c0feec94de0e4869cda1fc6b28bd7055774e3.1637077083.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-11-22T18:36:21Z","receivedAt":"2021-11-22T18:36:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Nov 16, 2021 at 7:38 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Existing callers to 'git ls-files' are expecting file names, not\n> directories. It is best to expand a sparse index to show all of the\n> contained files in this case.\n>\n> However, expert users may want to inspect the contents of the index\n> itself including which directories are sparse. Add a --sparse option to\n> allow users to request this information.\n>\n> During testing, I noticed that options such as --modified did not affect\n> the output when the files in question were outside the sparse-checkout\n> definition. Tests are added to document this preexisting behavior and\n> how it remains unchanged with the sparse index and the --sparse option.\n\nSure, 'git diff' and 'git status' don't show anything for such files\neither; we're being consistent. However, this feels like it's\nnormalizing an erroneous condition, possibly starting to cement one of\nthe bigger UI problems we have the sparse-checkout (and sparse-index):\n\nI assert that: Having tracked files be SKIP_WORKTREE despite having a\nfile present in the working directory is typically an *error*, and one\nthat we need to help users prevent/discover/recover-from.\n\n* I've seen users get into this condition by doing the equivalent of\neither 'git sparse-checkout disable' or 'git sparse-checkout set\n$NEW_PATHS' and then hit Ctrl-C in the middle of its operation.\n* Users also just occasionally copy files from other places (maybe\neven `git show REVISION:FILE >FILE`), and potentially edit.\n\nIn either of the above cases:\n* There is no command provided for discovering these files; not diff,\nstatus, ls-files, or anything\n* There is no command provided for correcting the problem; not clean,\nnot reset, or anything. They have to do it by hand\n* There are ample opportunities for the error to propagate and grow,\ndue to the fact that these files will not be reported or updated.[1]\n\n[1] For example, the user may first either switch to another branch or\nresets to some other commit. Neither of those operations update these\nerroneous present-despite-SKIP_WORKTREE files. And there will still\nbe nothing that reports them. But now the files are out-of-date with\nrespect to the new commit. Now if the user disables or changes the\nsparse checkout, they suddenly have several \"changed\" files in their\nworking tree -- which they didn't touch. In the best case, they run\ndiff or status or something and notice the files and then correct it,\nand just get perplexed at where the changes came from. But with\nenough users, some percentage of them are just going to commit (some\nof) those changes. When someone else asks why they modified those\nfiles, they'll (correctly, as it turns out) claim they never did and\nthat no program on their system did. They might even think those\nfiles weren't part of the commit and claim that git modified the\ncommit behind their back, which would be wrong, but there won't be any\nreasonable logs to check to prove what happened.\n\n\nI know this issue is out of scope for your series here, but the\naddition of testcases that purposely set up an erroneous condition\nmakes it feel like we're trying to normalize that situation and not\ntreat it like an error, and are perhap undercutting future attempts to\ntry to fix it. I'd rather have it explicitly stated that we're\nsetting up an erroneous condition in our tests, in order to make sure\nwe handle it as best as we can so far -- in a manner in line with diff\nand grep -- but also note that later solutions to this deeper issue\nmay affect one or more of these commands.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-ls-files.txt | 4 ++\n> builtin/ls-files.c | 12 +++++-\n> t/t1092-sparse-checkout-compatibility.sh | 47 ++++++++++++++++++++++++\n> 3 files changed, 61 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\n> index 6d11ab506b7..1c5d5f85ec5 100644\n> --- a/Documentation/git-ls-files.txt\n> +++ b/Documentation/git-ls-files.txt\n> @@ -187,6 +187,10 @@ Both the in the index (\"i/\")\n> and in the working tree (\"w/\") are shown for regular files,\n> followed by the (\"attr/\").\n>\n> +--sparse::\n> + If the index is sparse, show the sparse directories without expanding\n> + to the contained files.\n> +\n> \\--::\n> Do not interpret any more arguments as options.\n>\n> diff --git a/builtin/ls-files.c b/builtin/ls-files.c\n> index 031fef1bcaa..c151eb1fb77 100644\n> --- a/builtin/ls-files.c\n> +++ b/builtin/ls-files.c\n> @@ -37,6 +37,7 @@ static int debug_mode;\n> static int show_eol;\n> static int recurse_submodules;\n> static int skipping_duplicates;\n> +static int show_sparse_dirs;\n>\n> static const char *prefix;\n> static int max_prefix_len;\n> @@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n>\n> if (!(show_cached || show_stage || show_deleted || show_modified))\n> return;\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(repo->index);\n> +\n> + if (!show_sparse_dirs)\n> + ensure_full_index(repo->index);\n> +\n> for (i = 0; i < repo->index->cache_nr; i++) {\n> const struct cache_entry *ce = repo->index->cache[i];\n> struct stat st;\n> @@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> OPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n> OPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n> N_(\"suppress duplicate entries\")),\n> + OPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n> + N_(\"show sparse directories in the presence of a sparse index\")),\n> OPT_END()\n> };\n> int ret = 0;\n> @@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> usage_with_options(ls_files_usage, builtin_ls_files_options);\n>\n> + prepare_repo_settings(the_repository);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> prefix = cmd_prefix;\n> if (prefix)\n> prefix_len = strlen(prefix);\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index f8a8dde60af..ffb6052ff60 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -814,6 +814,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> git -C sparse-index reset -- folder1/a &&\n> test_region index convert_to_sparse trace2.txt &&\n> + test_region index ensure_full_index trace2.txt &&\n> +\n> + # ls-files expands on read, but does not write.\n> + rm trace2.txt &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git -C sparse-index ls-files &&\n> test_region index ensure_full_index trace2.txt\n> '\n>\n> @@ -838,6 +844,7 @@ test_expect_success 'sparse-index is not expanded' '\n> init_repos &&\n>\n> ensure_not_expanded status &&\n> + ensure_not_expanded ls-files --sparse &&\n\nDo we also want a\n ensure_not_expanded ls-files &&\n? We don't expect ls-files to write a new index file in any scenario, right?\n\n> ensure_not_expanded commit --allow-empty -m empty &&\n> echo >>sparse-index/a &&\n> ensure_not_expanded commit -a -m a &&\n> @@ -942,6 +949,46 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n> ensure_not_expanded pull full base\n> '\n>\n> +test_expect_success 'ls-files' '\n> + init_repos &&\n> +\n> + # Behavior agrees by default. Sparse index is expanded.\n> + test_all_match git ls-files &&\n> +\n> + # With --sparse, the sparse index data changes behavior.\n> + git -C sparse-index ls-files --sparse >sparse-index-out &&\n> + grep \"^folder1/\\$\" sparse-index-out &&\n> + grep \"^folder2/\\$\" sparse-index-out &&\n> +\n> + # With --sparse and no sparse index, nothing changes.\n> + git -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n> + grep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n> + ! grep \"/\\$\" sparse-checkout-out &&\n\nI'd be tempted to split the test up to this point from the rest of the test.\n\n> +\n> + write_script edit-content <<-\\EOF &&\n> + mkdir folder1 &&\n> + echo content >>folder1/a\n> + EOF\n> + run_on_sparse ../edit-content &&\n\nAs above, since folder1/a is a tracked file, I'd rather we explicitly\ncalled out that we're intentionally setting up an erroneous condition.\n\n> +\n> + # ls-files does not notice modified files whose\n> + # cache entries are marked SKIP_WORKTREE.\n\n...nor does diff, status, etc., as per my lengthy comment from the\nbeginning of the email.\n\n> + test_sparse_match git ls-files --modified &&\n> + test_must_be_empty sparse-checkout-out &&\n> + test_must_be_empty sparse-index-out &&\n> +\n> + git -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n> + test_must_be_empty sparse-index-out &&\n> +\n> + run_on_sparse git sparse-checkout add folder1 &&\n> + test_sparse_match git ls-files --modified &&\n> + grep \"^folder1/a\\$\" sparse-checkout-out &&\n> + grep \"^folder1/a\\$\" sparse-index-out &&\n> +\n> + # Double-check index expansion\n> + ensure_not_expanded ls-files --sparse\n> +'\n> +\n> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> # in this scenario, but it shouldn't.\n> test_expect_success 'reset mixed and checkout orphan' '\n> --\n> gitgitgadget\n\n\nOther than my concerns about careful messages with erroneous\nconditions (and the minor question about also having a\nensure_not_expanded without the --sparse flag), the patch looks good\nto me.\n"},{"id":"442015","messageId":"191b2db8-1be4-d6a4-1cf6-eaefaf373dd2@gmail.com","threadId":"56917","inReplyTo":"CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-11-22T19:44:20Z","receivedAt":"2021-11-22T19:44:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nOn 11/22/2021 1:36 PM, Elijah Newren wrote:\n> On Tue, Nov 16, 2021 at 7:38 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> Existing callers to 'git ls-files' are expecting file names, not\n>> directories. It is best to expand a sparse index to show all of the\n>> contained files in this case.\n>>\n>> However, expert users may want to inspect the contents of the index\n>> itself including which directories are sparse. Add a --sparse option to\n>> allow users to request this information.\n>>\n>> During testing, I noticed that options such as --modified did not affect\n>> the output when the files in question were outside the sparse-checkout\n>> definition. Tests are added to document this preexisting behavior and\n>> how it remains unchanged with the sparse index and the --sparse option.\n> \n> Sure, 'git diff' and 'git status' don't show anything for such files\n> either; we're being consistent. However, this feels like it's\n> normalizing an erroneous condition, possibly starting to cement one of\n> the bigger UI problems we have the sparse-checkout (and sparse-index):\n> \n> I assert that: Having tracked files be SKIP_WORKTREE despite having a\n> file present in the working directory is typically an *error*, and one\n> that we need to help users prevent/discover/recover-from.\n\nI do think we need to tread carefully here, since some users use\n'git update-index --skip-worktree' to mark a file in their\nworktree as \"ignored\" so they can change it from HEAD without it\never being staged. This is usually independent of the sparse-\ncheckout feature, so you might want to focus your efforts to paths\nthat exist in the worktree but are outside of the sparse-checkout\npatterns.\n\n> * I've seen users get into this condition by doing the equivalent of\n> either 'git sparse-checkout disable' or 'git sparse-checkout set\n> $NEW_PATHS' and then hit Ctrl-C in the middle of its operation.\n> * Users also just occasionally copy files from other places (maybe\n> even `git show REVISION:FILE >FILE`), and potentially edit.\n> \n> In either of the above cases:\n> * There is no command provided for discovering these files; not diff,\n> status, ls-files, or anything\n> * There is no command provided for correcting the problem; not clean,\n> not reset, or anything. They have to do it by hand\n> * There are ample opportunities for the error to propagate and grow,\n> due to the fact that these files will not be reported or updated.[1]\n> \n> [1] For example, the user may first either switch to another branch or\n> resets to some other commit. Neither of those operations update these\n> erroneous present-despite-SKIP_WORKTREE files. And there will still\n> be nothing that reports them. But now the files are out-of-date with\n> respect to the new commit. Now if the user disables or changes the\n> sparse checkout, they suddenly have several \"changed\" files in their\n> working tree -- which they didn't touch. In the best case, they run\n> diff or status or something and notice the files and then correct it,\n> and just get perplexed at where the changes came from. But with\n> enough users, some percentage of them are just going to commit (some\n> of) those changes. When someone else asks why they modified those\n> files, they'll (correctly, as it turns out) claim they never did and\n> that no program on their system did. They might even think those\n> files weren't part of the commit and claim that git modified the\n> commit behind their back, which would be wrong, but there won't be any\n> reasonable logs to check to prove what happened.\n> \n> \n> I know this issue is out of scope for your series here, but the\n> addition of testcases that purposely set up an erroneous condition\n> makes it feel like we're trying to normalize that situation and not\n> treat it like an error, and are perhap undercutting future attempts to\n> try to fix it. I'd rather have it explicitly stated that we're\n> setting up an erroneous condition in our tests, in order to make sure\n> we handle it as best as we can so far -- in a manner in line with diff\n> and grep -- but also note that later solutions to this deeper issue\n> may affect one or more of these commands.\n\nI appreciate the attention to this scenario, but I also think\nthat a patch series whose goal is to change how we react when\nin this case already needs to make a case for changing the\nbehavior. Having tests that exist documenting the previous\nbehavior form a good basis for \"it did this before, but now it\nwill do this,\" which hopefully further justifies the change.\n\nPersonally, I think of _every_ test as \"this is what the code\ndoes right now\" and the purpose of a test is to show that we\ndon't change things _unintentionally_. Every test can be changed\nif there is sufficient evidence that it should.\n\n>> @@ -838,6 +844,7 @@ test_expect_success 'sparse-index is not expanded' '\n>> init_repos &&\n>>\n>> ensure_not_expanded status &&\n>> + ensure_not_expanded ls-files --sparse &&\n> \n> Do we also want a\n> ensure_not_expanded ls-files &&\n> ? We don't expect ls-files to write a new index file in any scenario, right?\n\nWhen the index is sparse, 'git ls-files' will expand before\nlisting all of the contents, to avoid listing a sparse\ndirectory entry. One way to avoid ensure_full_index() would\nbe to do trickery with read_tree_at() whenever we find a\nsparse directory entry and use the callback to output what\nls-files would have written. However, that's pretty much the\nsame amount of work as what ensure_full_index() is doing, so\nthere is likely little benefit to complicating the code for\nthis reason.\n\n>> ensure_not_expanded commit --allow-empty -m empty &&\n>> echo >>sparse-index/a &&\n>> ensure_not_expanded commit -a -m a &&\n>> @@ -942,6 +949,46 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n>> ensure_not_expanded pull full base\n>> '\n>>\n>> +test_expect_success 'ls-files' '\n>> + init_repos &&\n>> +\n>> + # Behavior agrees by default. Sparse index is expanded.\n>> + test_all_match git ls-files &&\n>> +\n>> + # With --sparse, the sparse index data changes behavior.\n>> + git -C sparse-index ls-files --sparse >sparse-index-out &&\n>> + grep \"^folder1/\\$\" sparse-index-out &&\n>> + grep \"^folder2/\\$\" sparse-index-out &&\n>> +\n>> + # With --sparse and no sparse index, nothing changes.\n>> + git -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n>> + grep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n>> + ! grep \"/\\$\" sparse-checkout-out &&\n> \n> I'd be tempted to split the test up to this point from the rest of the test.\n\nEvery instance of \"init_repos\" adds to the cost of this test script,\nso when possible I'd err on the side of grouping them. It also keeps\ntest able to isolate with \"--run=1,\".\n\n>> +\n>> + write_script edit-content <<-\\EOF &&\n>> + mkdir folder1 &&\n>> + echo content >>folder1/a\n>> + EOF\n>> + run_on_sparse ../edit-content &&\n> \n> As above, since folder1/a is a tracked file, I'd rather we explicitly\n> called out that we're intentionally setting up an erroneous condition.\n> \n>> +\n>> + # ls-files does not notice modified files whose\n>> + # cache entries are marked SKIP_WORKTREE.\n> \n> ...nor does diff, status, etc., as per my lengthy comment from the\n> beginning of the email.\n\nI think the existence of this comment points out that \"this is\nsuch a strange situation that it requires explanation.\" A slight\ncomment change such as\n\n\t# In this strange situation where a tracked file\n\t# exists in the worktree but is marked with the\n\t# SKIP_WORKTREE bit in the index, Git ignores the\n\t# worktree contents.\n\nThat both points out that this case is unusual, but also that\nls-files isn't the only command that does this.\n> Other than my concerns about careful messages with erroneous\n> conditions (and the minor question about also having a\n> ensure_not_expanded without the --sparse flag), the patch looks good\n> to me.\n \nThanks for your careful read!\n-Stolee\n"},{"id":"442084","messageId":"211123.86lf1fwrq5.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"pull.1080.git.1637077083.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] Sparse index: fetch, pull, ls-files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-23T01:57:33Z","receivedAt":"2021-11-23T02:06:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 16 2021, Derrick Stolee via GitGitGadget wrote:\n\n> The 'ls-files' builtin is useful when debugging the index, and some scripts\n> use it, too. We are not changing the default behavior which expands a sparse\n> index in order to show all of the cached blobs. Instead, we add a '--sparse'\n> option that allows us to see the sparse directory entries upon request.\n> Combined with --debug, we can see a lot of index details, such as:\n>\n> $ git ls-files --debug --sparse\n> LICENSE\n> ctime: 1634910503:287405820\n> mtime: 1634910503:287405820\n> dev: 16777220 ino: 119325319\n> uid: 501 gid: 20\n> size: 1098 flags: 200000\n> README.md\n> ctime: 1634910503:288090279\n> mtime: 1634910503:288090279\n> dev: 16777220 ino: 119325320\n> uid: 501 gid: 20\n> size: 934 flags: 200000\n> bin/index.js\n> ctime: 1634910767:828434033\n> mtime: 1634910767:828434033\n> dev: 16777220 ino: 119325520\n> uid: 501 gid: 20\n> size: 7292 flags: 200000\n> examples/\n> ctime: 0:0\n> mtime: 0:0\n> dev: 0 ino: 0\n> uid: 0 gid: 0\n> size: 0 flags: 40004000\n> package.json\n> ctime: 1634910503:288676330\n> mtime: 1634910503:288676330\n> dev: 16777220 ino: 119325321\n> uid: 501 gid: 20\n> size: 680 flags: 200000\n>\n>\n> (In this example, the 'examples/' directory is sparse.)\n>\n> Thanks!\n\nThat's useful, and seems to be closing the same feature gap that the RFC\nseries I sent back in March[1], but at the time you went with adding\nthis ability to t/helper/test-read-cache.c.\n\nWith your series diffing the two with the data used in your new test\nshows that we had this with the helper beore:\n \n diff -u <(t/helper/test-tool -C sparse-index read-cache --table) <(t/helper/test-tool -C sparse-index read-cache --table --expand)\n --- /dev/fd/63 2021-11-23 02:57:00.980651400 +0100\n +++ /dev/fd/62 2021-11-23 02:57:00.980651400 +0100\n @@ -13,9 +13,13 @@\n 100644 blob cebd1739abee3e524d72ca9f51465a94d5a71daf e\n 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 folder1-\n 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 folder1.x\n -040000 tree e0f2d30b633eb781d675fedd78808135103fe1a0 folder1/\n +100644 blob 8b137891791fe96927ad78e64b0aad7bded08bdc folder1/0/0/0\n +100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 folder1/0/1\n +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85 folder1/a\n 100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 folder10\n -040000 tree 123706f6fc38949628eaf0483edbf97ba21123ae folder2/\n +100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 folder2/0/0/0\n +100644 blob 8b137891791fe96927ad78e64b0aad7bded08bdc folder2/0/1\n +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85 folder2/a\n 100644 blob 18912c9a915632d7b3344ec25f349dc8b4b9bf27 g\n -040000 tree aaff74984cccd156a469afa7d9ab10e4777beb24 x/\n +100644 blob 78981922613b2afb6025042ff6bd878ac1994e85 x/a\n 100644 blob 2d8c856aebeb20da61bd5112d6fa46ff3f56a9e8 z\n\nWhich we can now get out of ls-files:\n \n $ diff -u <(git -C sparse-index ls-files --stage --sparse) <(git -C sparse-index ls-files --stage)\n --- /dev/fd/63 2021-11-23 02:55:13.329613255 +0100\n +++ /dev/fd/62 2021-11-23 02:55:13.329613255 +0100\n @@ -13,9 +13,13 @@\n 100644 cebd1739abee3e524d72ca9f51465a94d5a71daf 0 e\n 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 folder1-\n 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 folder1.x\n -040000 e0f2d30b633eb781d675fedd78808135103fe1a0 0 folder1/\n +100644 8b137891791fe96927ad78e64b0aad7bded08bdc 0 folder1/0/0/0\n +100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 folder1/0/1\n +100644 78981922613b2afb6025042ff6bd878ac1994e85 0 folder1/a\n 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 folder10\n -040000 123706f6fc38949628eaf0483edbf97ba21123ae 0 folder2/\n +100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0 folder2/0/0/0\n +100644 8b137891791fe96927ad78e64b0aad7bded08bdc 0 folder2/0/1\n +100644 78981922613b2afb6025042ff6bd878ac1994e85 0 folder2/a\n 100644 18912c9a915632d7b3344ec25f349dc8b4b9bf27 0 g\n -040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n +100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n 100644 2d8c856aebeb20da61bd5112d6fa46ff3f56a9e8 0 z\n\nSo that's neat, but unless I'm wrong about there still being some\nspecial-sauce in t/helper/test-read-cache.c that's needed this series\nseems incomplete without migrating the existing test users of it to this\nnew ls-files --sparse, followed by a cherry-pick (or equivalent) of [2].\n\nIs there any reason we wouldn't use ls-files instead of the test-tool in\nthe tests now? I understood from you at the time that the reason for the\nthat mode of the test-tool existing was an interim state until ls-files\nlearned to emit this sort of output, but that you wanted to add that\nlater. It seems we've arrived at that \"later\" :)\n\n1. https://lore.kernel.org/git/20210317132814.30175-1-avarab@gmail.com/\n2. https://lore.kernel.org/git/20210317132814.30175-5-avarab@gmail.com/\n"},{"id":"442086","messageId":"211123.86h7c3wrg2.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"e42c0feec94de0e4869cda1fc6b28bd7055774e3.1637077083.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-11-23T02:07:08Z","receivedAt":"2021-11-23T02:12:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 16 2021, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> +test_expect_success 'ls-files' '\n> +\tinit_repos &&\n> +\n> +\t# Behavior agrees by default. Sparse index is expanded.\n> +\ttest_all_match git ls-files &&\n> +\n> +\t# With --sparse, the sparse index data changes behavior.\n> +\tgit -C sparse-index ls-files --sparse >sparse-index-out &&\n> +\tgrep \"^folder1/\\$\" sparse-index-out &&\n> +\tgrep \"^folder2/\\$\" sparse-index-out &&\n> +\n> +\t# With --sparse and no sparse index, nothing changes.\n> +\tgit -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n> +\tgrep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n> +\t! grep \"/\\$\" sparse-checkout-out &&\n\nI think all of this would be much clearer both in terms of explaining\nthis change, and also for future test relability if it did away with the\nselective grepping, and simply ran tls-files with and without --sparse,\nand then test_cmp'd the full output (after munging away the OIDs).\n\nI.e. the sort of output that's in my just-sent reply to the CL:\nhttps://lore.kernel.org/git/211123.86lf1fwrq5.gmgdl@evledraar.gmail.com/\n\nWe really don't need to optimize for lines of tests added, and having\n~30 lines of plainly understood diff output is IMO preferrable to even 5\nlines of tricky positive & negative grep invocations that take some time\nto reason about and understand.\n\nI.e. something like:\n\n cat >expected <<-\\EOF &&\n 100644 blob OID e\n 100644 blob OID folder1-\n 100644 blob OID folder1.x\n -040000 tree OID folder1/\n +100644 blob OID folder1/0/0/0\n +100644 blob OID folder1/0/1\n +100644 blob OID folder1/a\n 100644 blob OID folder10\n -040000 tree OID folder2/\n +100644 blob OID folder2/0/0/0\n +100644 blob OID folder2/0/1\n +100644 blob OID folder2/a\n 100644 blob OID g\n -040000 tree OID x/\n +100644 blob OID x/a\n 100644 blob OID z\n EOF\n git [...] ls-files --sparse >actual.raw &&\n [munge away OIDs] actual &&\n test_cmp expected actual\n\nWould test everything you're trying to test here and more (would need 2x\nof those..), and would be easier to read & understand.\n"},{"id":"443466","messageId":"03a642fc-6310-1ea4-083e-9fe4530cf761@gmail.com","threadId":"56917","inReplyTo":"211123.86h7c3wrg2.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-08T15:14:02Z","receivedAt":"2021-12-08T15:14:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/22/2021 9:07 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Nov 16 2021, Derrick Stolee via GitGitGadget wrote:\n\nThings in the dependent topics are starting to simmer down, so I'm\nback revisiting this topic.\n\n>> From: Derrick Stolee \n>> [...]\n>> +test_expect_success 'ls-files' '\n>> +\tinit_repos &&\n>> +\n>> +\t# Behavior agrees by default. Sparse index is expanded.\n>> +\ttest_all_match git ls-files &&\n>> +\n>> +\t# With --sparse, the sparse index data changes behavior.\n>> +\tgit -C sparse-index ls-files --sparse >sparse-index-out &&\n>> +\tgrep \"^folder1/\\$\" sparse-index-out &&\n>> +\tgrep \"^folder2/\\$\" sparse-index-out &&\n>> +\n>> +\t# With --sparse and no sparse index, nothing changes.\n>> +\tgit -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n>> +\tgrep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n>> +\t! grep \"/\\$\" sparse-checkout-out &&\n> \n> I think all of this would be much clearer both in terms of explaining\n> this change, and also for future test relability if it did away with the\n> selective grepping, and simply ran tls-files with and without --sparse,\n> and then test_cmp'd the full output (after munging away the OIDs).\n> \n> I.e. the sort of output that's in my just-sent reply to the CL:\n> https://lore.kernel.org/git/211123.86lf1fwrq5.gmgdl@evledraar.gmail.com/\n> \n> We really don't need to optimize for lines of tests added, and having\n> ~30 lines of plainly understood diff output is IMO preferrable to even 5\n> lines of tricky positive & negative grep invocations that take some time\n> to reason about and understand.\n> \n> I.e. something like:\n> \n> cat >expected <<-\\EOF &&\n> 100644 blob OID e\n> 100644 blob OID folder1-\n> 100644 blob OID folder1.x\n> -040000 tree OID folder1/\n> +100644 blob OID folder1/0/0/0\n> +100644 blob OID folder1/0/1\n> +100644 blob OID folder1/a\n> 100644 blob OID folder10\n> -040000 tree OID folder2/\n> +100644 blob OID folder2/0/0/0\n> +100644 blob OID folder2/0/1\n> +100644 blob OID folder2/a\n> 100644 blob OID g\n> -040000 tree OID x/\n> +100644 blob OID x/a\n> 100644 blob OID z\n> EOF\n> git [...] ls-files --sparse >actual.raw &&\n> [munge away OIDs] actual &&\n> test_cmp expected actual\n> \n> Would test everything you're trying to test here and more (would need 2x\n> of those..), and would be easier to read & understand.\n\nI don't think it is that hard to understand \"I expect to see these\nlines and not these lines\" but I am open to more fully verifying\nthe full output and demonstrating the change that happens when the\nflag is added.\n\nTaking your idea and applying it to 'ls-files' (without --stage to\navoid OIDs which would change depending on the hash algorithm), the\nstart of the test looks like this:\n\ntest_expect_success 'ls-files' '\n\tinit_repos &&\n\n\t# Behavior agrees by default. Sparse index is expanded.\n\ttest_all_match git ls-files &&\n\n\t# With --sparse, the sparse index data changes behavior.\n\tgit -C sparse-index ls-files --stage >out &&\n\tgit -C sparse-index ls-files --stage --sparse >sparse &&\n\n\tcat >expect <<-\\EOF &&\n\t e\n\t folder1-\n\t folder1.x\n\t-folder1/0/0/0\n\t-folder1/0/1\n\t-folder1/a\n\t+folder1/\n\t folder10\n\t-folder2/0/0/0\n\t-folder2/0/1\n\t-folder2/a\n\t+folder2/\n\t g\n\t-x/a\n\t+x/\n\t z\n\tEOF\n\n\tdiff -u out sparse | tail -n 16 >actual &&\n\ttest_cmp expect actual\n'\n\nI can make similar adjustments throughout the test to match\nthis style.\n\nThanks,\n-Stolee\n"},{"id":"443467","messageId":"7dac5ebc-fe01-4041-4f1c-0641edc3776c@gmail.com","threadId":"56917","inReplyTo":"03a642fc-6310-1ea4-083e-9fe4530cf761@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-08T15:20:37Z","receivedAt":"2021-12-08T15:20:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2021 10:14 AM, Derrick Stolee wrote:\n\n> Taking your idea and applying it to 'ls-files' (without --stage to\n> avoid OIDs which would change depending on the hash algorithm), the\n> start of the test looks like this:\n> \n> test_expect_success 'ls-files' '\n> \tinit_repos &&\n> \n> \t# Behavior agrees by default. Sparse index is expanded.\n> \ttest_all_match git ls-files &&\n> \n> \t# With --sparse, the sparse index data changes behavior.\n> \tgit -C sparse-index ls-files --stage >out &&\n> \tgit -C sparse-index ls-files --stage --sparse >sparse &&\n\nOf course, I had a stale version here... \"--stage\" should not be here.\n \nThanks,\n-Stolee\n"},{"id":"443483","messageId":"CABPp-BGJJM757CoOPjP=XBK-cMMGJemaeruxXSN9TEGmk+NKvg@mail.gmail.com","threadId":"56917","inReplyTo":"03a642fc-6310-1ea4-083e-9fe4530cf761@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-08T17:04:18Z","receivedAt":"2021-12-08T17:04:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 7:14 AM Derrick Stolee wrote:\n>\n> On 11/22/2021 9:07 PM, Ævar Arnfjörð Bjarmason wrote:\n> >\n> > On Tue, Nov 16 2021, Derrick Stolee via GitGitGadget wrote:\n>\n> Things in the dependent topics are starting to simmer down, so I'm\n> back revisiting this topic.\n>\n> >> From: Derrick Stolee \n> >> [...]\n> >> +test_expect_success 'ls-files' '\n> >> + init_repos &&\n> >> +\n> >> + # Behavior agrees by default. Sparse index is expanded.\n> >> + test_all_match git ls-files &&\n> >> +\n> >> + # With --sparse, the sparse index data changes behavior.\n> >> + git -C sparse-index ls-files --sparse >sparse-index-out &&\n> >> + grep \"^folder1/\\$\" sparse-index-out &&\n> >> + grep \"^folder2/\\$\" sparse-index-out &&\n> >> +\n> >> + # With --sparse and no sparse index, nothing changes.\n> >> + git -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n> >> + grep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n> >> + ! grep \"/\\$\" sparse-checkout-out &&\n> >\n> > I think all of this would be much clearer both in terms of explaining\n> > this change, and also for future test relability if it did away with the\n> > selective grepping, and simply ran tls-files with and without --sparse,\n> > and then test_cmp'd the full output (after munging away the OIDs).\n> >\n> > I.e. the sort of output that's in my just-sent reply to the CL:\n> > https://lore.kernel.org/git/211123.86lf1fwrq5.gmgdl@evledraar.gmail.com/\n> >\n> > We really don't need to optimize for lines of tests added, and having\n> > ~30 lines of plainly understood diff output is IMO preferrable to even 5\n> > lines of tricky positive & negative grep invocations that take some time\n> > to reason about and understand.\n> >\n> > I.e. something like:\n> >\n> > cat >expected <<-\\EOF &&\n> > 100644 blob OID e\n> > 100644 blob OID folder1-\n> > 100644 blob OID folder1.x\n> > -040000 tree OID folder1/\n> > +100644 blob OID folder1/0/0/0\n> > +100644 blob OID folder1/0/1\n> > +100644 blob OID folder1/a\n> > 100644 blob OID folder10\n> > -040000 tree OID folder2/\n> > +100644 blob OID folder2/0/0/0\n> > +100644 blob OID folder2/0/1\n> > +100644 blob OID folder2/a\n> > 100644 blob OID g\n> > -040000 tree OID x/\n> > +100644 blob OID x/a\n> > 100644 blob OID z\n> > EOF\n> > git [...] ls-files --sparse >actual.raw &&\n> > [munge away OIDs] actual &&\n> > test_cmp expected actual\n> >\n> > Would test everything you're trying to test here and more (would need 2x\n> > of those..), and would be easier to read & understand.\n\nThe loss of checking for trees would be bad; the point of testing a\nsparse-index is that it has tree objects in it. However, the basic\nsuggestion inspired Stolee's variant below that does check for trees.\nSo...\n\n> I don't think it is that hard to understand \"I expect to see these\n> lines and not these lines\" but I am open to more fully verifying\n> the full output and demonstrating the change that happens when the\n> flag is added.\n>\n> Taking your idea and applying it to 'ls-files' (without --stage to\n> avoid OIDs which would change depending on the hash algorithm), the\n> start of the test looks like this:\n>\n> test_expect_success 'ls-files' '\n> init_repos &&\n>\n> # Behavior agrees by default. Sparse index is expanded.\n> test_all_match git ls-files &&\n>\n> # With --sparse, the sparse index data changes behavior.\n> git -C sparse-index ls-files --stage >out &&\n> git -C sparse-index ls-files --stage --sparse >sparse &&\n>\n> cat >expect <<-\\EOF &&\n> e\n> folder1-\n> folder1.x\n> -folder1/0/0/0\n> -folder1/0/1\n> -folder1/a\n> +folder1/\n> folder10\n> -folder2/0/0/0\n> -folder2/0/1\n> -folder2/a\n> +folder2/\n> g\n> -x/a\n> +x/\n> z\n> EOF\n>\n> diff -u out sparse | tail -n 16 >actual &&\n> test_cmp expect actual\n> '\n\nThis actually looks quite nice, though the magic '16' is kind of\nannoying. Could we get rid of that -- perhaps using something to rip\nout the diff header, or using comm instead?\n\nAlso, perhaps 'dense' rather than 'out'?\n"},{"id":"443492","messageId":"d1275b30-b9b0-a416-3300-9809d880eb55@gmail.com","threadId":"56917","inReplyTo":"CABPp-BGJJM757CoOPjP=XBK-cMMGJemaeruxXSN9TEGmk+NKvg@mail.gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-08T18:23:14Z","receivedAt":"2021-12-08T18:23:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2021 12:04 PM, Elijah Newren wrote:\n> On Wed, Dec 8, 2021 at 7:14 AM Derrick Stolee wrote:\n\n>> Taking your idea and applying it to 'ls-files' (without --stage to\n>> avoid OIDs which would change depending on the hash algorithm), the\n>> start of the test looks like this:\n>>\n>> test_expect_success 'ls-files' '\n>> init_repos &&\n>>\n>> # Behavior agrees by default. Sparse index is expanded.\n>> test_all_match git ls-files &&\n>>\n>> # With --sparse, the sparse index data changes behavior.\n>> git -C sparse-index ls-files --stage >out &&\n>> git -C sparse-index ls-files --stage --sparse >sparse &&\n>>\n>> cat >expect <<-\\EOF &&\n>> e\n>> folder1-\n>> folder1.x\n>> -folder1/0/0/0\n>> -folder1/0/1\n>> -folder1/a\n>> +folder1/\n>> folder10\n>> -folder2/0/0/0\n>> -folder2/0/1\n>> -folder2/a\n>> +folder2/\n>> g\n>> -x/a\n>> +x/\n>> z\n>> EOF\n>>\n>> diff -u out sparse | tail -n 16 >actual &&\n>> test_cmp expect actual\n>> '\n> \n> This actually looks quite nice, though the magic '16' is kind of\n> annoying. Could we get rid of that -- perhaps using something to rip\n> out the diff header, or using comm instead?\n\nWhat I really want is \"remove the first two lines of this file\"\nbut perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\nmagic number?\n\n> Also, perhaps 'dense' rather than 'out'?\n\nSounds good.\n\nThanks,\n-Stolee\n"},{"id":"443494","messageId":"CABPp-BEyvkRdxJoJZcNF1VoQBcJajq1CVUrNTfHnHG433Q=cNQ@mail.gmail.com","threadId":"56917","inReplyTo":"d1275b30-b9b0-a416-3300-9809d880eb55@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-08T18:36:09Z","receivedAt":"2021-12-08T18:36:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 10:23 AM Derrick Stolee wrote:\n>\n> On 12/8/2021 12:04 PM, Elijah Newren wrote:\n> > On Wed, Dec 8, 2021 at 7:14 AM Derrick Stolee wrote:\n>\n> >> Taking your idea and applying it to 'ls-files' (without --stage to\n> >> avoid OIDs which would change depending on the hash algorithm), the\n> >> start of the test looks like this:\n> >>\n> >> test_expect_success 'ls-files' '\n> >> init_repos &&\n> >>\n> >> # Behavior agrees by default. Sparse index is expanded.\n> >> test_all_match git ls-files &&\n> >>\n> >> # With --sparse, the sparse index data changes behavior.\n> >> git -C sparse-index ls-files --stage >out &&\n> >> git -C sparse-index ls-files --stage --sparse >sparse &&\n> >>\n> >> cat >expect <<-\\EOF &&\n> >> e\n> >> folder1-\n> >> folder1.x\n> >> -folder1/0/0/0\n> >> -folder1/0/1\n> >> -folder1/a\n> >> +folder1/\n> >> folder10\n> >> -folder2/0/0/0\n> >> -folder2/0/1\n> >> -folder2/a\n> >> +folder2/\n> >> g\n> >> -x/a\n> >> +x/\n> >> z\n> >> EOF\n> >>\n> >> diff -u out sparse | tail -n 16 >actual &&\n> >> test_cmp expect actual\n> >> '\n> >\n> > This actually looks quite nice, though the magic '16' is kind of\n> > annoying. Could we get rid of that -- perhaps using something to rip\n> > out the diff header, or using comm instead?\n>\n> What I really want is \"remove the first two lines of this file\"\n\nIs `tail -n +3` portable? Looks like we have five uses of tail -n +N\nin the testsuite, so it should be okay to use.\n\n> but perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\n> magic number?\n\nThat works too.\n\n> > Also, perhaps 'dense' rather than 'out'?\n>\n> Sounds good.\n>\n> Thanks,\n> -Stolee\n"},{"id":"443497","messageId":"c8e22885-1759-d3d9-3944-2d70c70960e2@gmail.com","threadId":"56917","inReplyTo":"CABPp-BEyvkRdxJoJZcNF1VoQBcJajq1CVUrNTfHnHG433Q=cNQ@mail.gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-08T19:06:30Z","receivedAt":"2021-12-08T19:06:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2021 1:36 PM, Elijah Newren wrote:\n> On Wed, Dec 8, 2021 at 10:23 AM Derrick Stolee wrote:\n>>\n>> On 12/8/2021 12:04 PM, Elijah Newren wrote:\n>>>\n>>> This actually looks quite nice, though the magic '16' is kind of\n>>> annoying. Could we get rid of that -- perhaps using something to rip\n>>> out the diff header, or using comm instead?\n>>\n>> What I really want is \"remove the first two lines of this file\"\n> \n> Is `tail -n +3` portable? Looks like we have five uses of tail -n +N\n> in the testsuite, so it should be okay to use.\n\nAh, that's the magic incantation. Sounds good.\n\n>> but perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\n>> magic number?\n> \n> That works too.\n\nIf the \"-n +X\" syntax works, then I'll opt for that.\n\nThanks!\n"},{"id":"443504","messageId":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.git.1637077083.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Sparse index: fetch, pull, ls-files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:50Z","receivedAt":"2021-12-08T19:40:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ld/sparse-index-blame (merged with 'master' due to an\nunrelated build issue).\n\nHere are two relatively-simple patches that further the sparse index\nintegrations.\n\nDid you know that 'fetch' and 'pull' read the index? I didn't, or this would\nhave been an integration much earlier in the cycle. They read the index to\nlook for the .gitmodules file in case there are submodules that need to be\nfetched. Since looking for a file by name is already protected, we only need\nto disable 'command_requires_full_index' and we are done.\n\nThe 'ls-files' builtin is useful when debugging the index, and some scripts\nuse it, too. We are not changing the default behavior which expands a sparse\nindex in order to show all of the cached blobs. Instead, we add a '--sparse'\noption that allows us to see the sparse directory entries upon request.\nCombined with --debug, we can see a lot of index details, such as:\n\n$ git ls-files --debug --sparse\nLICENSE\n ctime: 1634910503:287405820\n mtime: 1634910503:287405820\n dev: 16777220 ino: 119325319\n uid: 501 gid: 20\n size: 1098 flags: 200000\nREADME.md\n ctime: 1634910503:288090279\n mtime: 1634910503:288090279\n dev: 16777220 ino: 119325320\n uid: 501 gid: 20\n size: 934 flags: 200000\nbin/index.js\n ctime: 1634910767:828434033\n mtime: 1634910767:828434033\n dev: 16777220 ino: 119325520\n uid: 501 gid: 20\n size: 7292 flags: 200000\nexamples/\n ctime: 0:0\n mtime: 0:0\n dev: 0 ino: 0\n uid: 0 gid: 0\n size: 0 flags: 40004000\npackage.json\n ctime: 1634910503:288676330\n mtime: 1634910503:288676330\n dev: 16777220 ino: 119325321\n uid: 501 gid: 20\n size: 680 flags: 200000\n\n\n(In this example, the 'examples/' directory is sparse.)\n\nThanks!\n\n\nUpdates in v2\n=============\n\n * Rebased onto latest ld/sparse-index-blame without issue.\n * Updated the test to use diff-of-diffs instead of a sequence of greps.\n * Added patches that remove the use of 'test-tool read-cache --table' and\n its implementation.\n\nDerrick Stolee (5):\n fetch/pull: use the sparse index\n ls-files: add --sparse option\n t1092: replace 'read-cache --table' with 'ls-files --sparse'\n t1091/t3705: remove 'test-tool read-cache --table'\n test-read-cache: remove --table, --expand options\n\n Documentation/git-ls-files.txt | 4 +\n builtin/fetch.c | 2 +\n builtin/ls-files.c | 12 ++-\n builtin/pull.c | 2 +\n t/helper/test-read-cache.c | 64 ++---------\n t/t1091-sparse-checkout-builtin.sh | 25 ++++-\n t/t1092-sparse-checkout-compatibility.sh | 129 ++++++++++++++++++++---\n t/t3705-add-sparse-checkout.sh | 8 +-\n 8 files changed, 165 insertions(+), 81 deletions(-)\n\n\nbase-commit: 3fffe69d24e4ecc95246766f5396303a953695ff\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1080%2Fderrickstolee%2Fsparse-index%2Ffetch-pull-ls-files-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1080/derrickstolee/sparse-index/fetch-pull-ls-files-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1080\n\nRange-diff vs v1:\n\n 1: 451056e1a77 ! 1: f72001638d1 fetch/pull: use the sparse index\n @@ builtin/pull.c: int cmd_pull(int argc, const char **argv, const char *prefix)\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: blame' '\n - \tensure_not_expanded blame deep/deeper1/deepest/a\n + \tdone\n '\n \n +test_expect_success 'sparse index is not expanded: fetch/pull' '\n 2: e42c0feec94 ! 2: 58b5eca4835 ls-files: add --sparse option\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\ttest_all_match git ls-files &&\n +\n +\t# With --sparse, the sparse index data changes behavior.\n -+\tgit -C sparse-index ls-files --sparse >sparse-index-out &&\n -+\tgrep \"^folder1/\\$\" sparse-index-out &&\n -+\tgrep \"^folder2/\\$\" sparse-index-out &&\n ++\tgit -C sparse-index ls-files >dense &&\n ++\tgit -C sparse-index ls-files --sparse >sparse &&\n ++\n ++\tcat >expect <<-\\EOF &&\n ++\t@@ -13,13 +13,9 @@\n ++\t e\n ++\t folder1-\n ++\t folder1.x\n ++\t-folder1/0/0/0\n ++\t-folder1/0/1\n ++\t-folder1/a\n ++\t+folder1/\n ++\t folder10\n ++\t-folder2/0/0/0\n ++\t-folder2/0/1\n ++\t-folder2/a\n ++\t+folder2/\n ++\t g\n ++\t-x/a\n ++\t+x/\n ++\t z\n ++\tEOF\n ++\n ++\tdiff -u dense sparse | tail -n +3 >actual &&\n ++\ttest_cmp expect actual &&\n +\n +\t# With --sparse and no sparse index, nothing changes.\n -+\tgit -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n -+\tgrep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n -+\t! grep \"/\\$\" sparse-checkout-out &&\n ++\tgit -C sparse-checkout ls-files >dense &&\n ++\tgit -C sparse-checkout ls-files --sparse >sparse &&\n ++\ttest_cmp dense sparse &&\n +\n +\twrite_script edit-content <<-\\EOF &&\n +\tmkdir folder1 &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n +\ttest_must_be_empty sparse-index-out &&\n +\n -+\trun_on_sparse git sparse-checkout add folder1 &&\n ++\t# Add folder1 to the sparse-checkout cone and\n ++\t# check that ls-files shows the expanded files.\n ++\ttest_sparse_match git sparse-checkout add folder1 &&\n +\ttest_sparse_match git ls-files --modified &&\n -+\tgrep \"^folder1/a\\$\" sparse-checkout-out &&\n -+\tgrep \"^folder1/a\\$\" sparse-index-out &&\n +\n -+\t# Double-check index expansion\n ++\tgit -C sparse-index ls-files >dense &&\n ++\tgit -C sparse-index ls-files --sparse >sparse &&\n ++\n ++\tcat >expect <<-\\EOF &&\n ++\t@@ -17,9 +17,7 @@\n ++\t folder1/0/1\n ++\t folder1/a\n ++\t folder10\n ++\t-folder2/0/0/0\n ++\t-folder2/0/1\n ++\t-folder2/a\n ++\t+folder2/\n ++\t g\n ++\t-x/a\n ++\t+x/\n ++\t z\n ++\tEOF\n ++\n ++\tdiff -u dense sparse | tail -n +3 >actual &&\n ++\ttest_cmp expect actual &&\n ++\n ++\t# Double-check index expansion is avoided\n +\tensure_not_expanded ls-files --sparse\n +'\n +\n -: ----------- > 3: 5ffae2a03ae t1092: replace 'read-cache --table' with 'ls-files --sparse'\n -: ----------- > 4: b98e5e6d2bc t1091/t3705: remove 'test-tool read-cache --table'\n -: ----------- > 5: f31a24eeb9b test-read-cache: remove --table, --expand options\n\n-- \ngitgitgadget\n"},{"id":"443505","messageId":"f72001638d1d0865cea0f728e700a4148bf7b37f.1638992396.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] fetch/pull: use the sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:51Z","receivedAt":"2021-12-08T19:40:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git fetch' and 'git pull' commands parse the index in order to\ndetermine if submodules exist. Without command_requires_full_index=0,\nthis will expand a sparse index, causing slow performance even when\nthere is no new data to fetch.\n\nThe .gitmodules file will never be inside a sparse directory entry, and\neven if it was, the index_name_pos() method would expand the sparse\nindex if needed as we search for the path by name. These commands do not\niterate over the index, which is the typical thing we are careful about\nwhen integrating with the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 2 ++\n builtin/pull.c | 2 ++\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f7abbc31ff1..1696b7791d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1993,6 +1993,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tgit_config(git_fetch_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 1cfaf9f3436..7bce3bd80f9 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -994,6 +994,8 @@ int cmd_pull(int argc, const char **argv, const char *prefix)\n \t\tset_reflog_message(argc, argv);\n \n \tgit_config(git_pull_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix, pull_options, pull_usage, 0);\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 83971174ac3..3ba19ba1c14 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -964,6 +964,16 @@ test_expect_success 'sparse index is not expanded: blame' '\n \tdone\n '\n \n+test_expect_success 'sparse index is not expanded: fetch/pull' '\n+\tinit_repos &&\n+\n+\tgit -C sparse-index remote add full \"file://$(pwd)/full-checkout\" &&\n+\tensure_not_expanded fetch full &&\n+\tgit -C full-checkout commit --allow-empty -m \"for pull merge\" &&\n+\tgit -C sparse-index commit --allow-empty -m \"for pull merge\" &&\n+\tensure_not_expanded pull full base\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"443506","messageId":"58b5eca483502b0d45601490f6b4255f6419e6bd.1638992396.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] ls-files: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:52Z","receivedAt":"2021-12-08T19:40:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExisting callers to 'git ls-files' are expecting file names, not\ndirectories. It is best to expand a sparse index to show all of the\ncontained files in this case.\n\nHowever, expert users may want to inspect the contents of the index\nitself including which directories are sparse. Add a --sparse option to\nallow users to request this information.\n\nDuring testing, I noticed that options such as --modified did not affect\nthe output when the files in question were outside the sparse-checkout\ndefinition. Tests are added to document this preexisting behavior and\nhow it remains unchanged with the sparse index and the --sparse option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ls-files.txt | 4 ++\n builtin/ls-files.c | 12 +++-\n t/t1092-sparse-checkout-compatibility.sh | 90 ++++++++++++++++++++++++\n 3 files changed, 104 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex 6d11ab506b7..1c5d5f85ec5 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -187,6 +187,10 @@ Both the in the index (\"i/\")\n and in the working tree (\"w/\") are shown for regular files,\n followed by the (\"attr/\").\n \n+--sparse::\n+\tIf the index is sparse, show the sparse directories without expanding\n+\tto the contained files.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 031fef1bcaa..c151eb1fb77 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -37,6 +37,7 @@ static int debug_mode;\n static int show_eol;\n static int recurse_submodules;\n static int skipping_duplicates;\n+static int show_sparse_dirs;\n \n static const char *prefix;\n static int max_prefix_len;\n@@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\n+\tif (!show_sparse_dirs)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tOPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n \t\tOPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n \t\t\t N_(\"suppress duplicate entries\")),\n+\t\tOPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n+\t\t\t N_(\"show sparse directories in the presence of a sparse index\")),\n \t\tOPT_END()\n \t};\n \tint ret = 0;\n@@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(ls_files_usage, builtin_ls_files_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tprefix = cmd_prefix;\n \tif (prefix)\n \t\tprefix_len = strlen(prefix);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 3ba19ba1c14..bf2c6b169b9 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -802,6 +802,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt &&\n+\n+\t# ls-files expands on read, but does not write.\n+\trm trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index ls-files &&\n \ttest_region index ensure_full_index trace2.txt\n '\n \n@@ -826,6 +832,7 @@ test_expect_success 'sparse-index is not expanded' '\n \tinit_repos &&\n \n \tensure_not_expanded status &&\n+\tensure_not_expanded ls-files --sparse &&\n \tensure_not_expanded commit --allow-empty -m empty &&\n \techo >>sparse-index/a &&\n \tensure_not_expanded commit -a -m a &&\n@@ -974,6 +981,89 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'ls-files' '\n+\tinit_repos &&\n+\n+\t# Behavior agrees by default. Sparse index is expanded.\n+\ttest_all_match git ls-files &&\n+\n+\t# With --sparse, the sparse index data changes behavior.\n+\tgit -C sparse-index ls-files >dense &&\n+\tgit -C sparse-index ls-files --sparse >sparse &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t@@ -13,13 +13,9 @@\n+\t e\n+\t folder1-\n+\t folder1.x\n+\t-folder1/0/0/0\n+\t-folder1/0/1\n+\t-folder1/a\n+\t+folder1/\n+\t folder10\n+\t-folder2/0/0/0\n+\t-folder2/0/1\n+\t-folder2/a\n+\t+folder2/\n+\t g\n+\t-x/a\n+\t+x/\n+\t z\n+\tEOF\n+\n+\tdiff -u dense sparse | tail -n +3 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# With --sparse and no sparse index, nothing changes.\n+\tgit -C sparse-checkout ls-files >dense &&\n+\tgit -C sparse-checkout ls-files --sparse >sparse &&\n+\ttest_cmp dense sparse &&\n+\n+\twrite_script edit-content <<-\\EOF &&\n+\tmkdir folder1 &&\n+\techo content >>folder1/a\n+\tEOF\n+\trun_on_sparse ../edit-content &&\n+\n+\t# ls-files does not notice modified files whose\n+\t# cache entries are marked SKIP_WORKTREE.\n+\ttest_sparse_match git ls-files --modified &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\t# Add folder1 to the sparse-checkout cone and\n+\t# check that ls-files shows the expanded files.\n+\ttest_sparse_match git sparse-checkout add folder1 &&\n+\ttest_sparse_match git ls-files --modified &&\n+\n+\tgit -C sparse-index ls-files >dense &&\n+\tgit -C sparse-index ls-files --sparse >sparse &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t@@ -17,9 +17,7 @@\n+\t folder1/0/1\n+\t folder1/a\n+\t folder10\n+\t-folder2/0/0/0\n+\t-folder2/0/1\n+\t-folder2/a\n+\t+folder2/\n+\t g\n+\t-x/a\n+\t+x/\n+\t z\n+\tEOF\n+\n+\tdiff -u dense sparse | tail -n +3 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Double-check index expansion is avoided\n+\tensure_not_expanded ls-files --sparse\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"443507","messageId":"5ffae2a03ae3d00a1f128d7cf2a397081055a2fe.1638992396.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] t1092: replace 'read-cache --table' with 'ls-files --sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:53Z","receivedAt":"2021-12-08T19:40:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1092-sparse-checkout-compatibility.sh.\n\nThe important changes are due to the different output format. We need to\nuse the '--stage' output to get a file mode and OID, but it also\nincludes a stage value and drops the object type. This leads to some\ndifferences in how we handle looking for specific entries.\n\nSome places where we previously looked for no 'tree' entries, we can\ninstead directly compare the output across the repository with a sparse\nindex and the one without.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 29 +++++++++++-------------\n 1 file changed, 13 insertions(+), 16 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex bf2c6b169b9..4a6832ea3c5 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -200,45 +200,42 @@ test_sparse_unstaged () {\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set folder1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set deep/deeper1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep/deeper2 folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \t# Disabling the sparse-index removes tree entries with full ones\n \tgit -C sparse-index sparse-checkout init --no-sparse-index &&\n-\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\t! grep \"040000 tree\" cache &&\n-\ttest_sparse_match test-tool read-cache --table\n+\ttest_sparse_match git ls-files --stage --sparse\n '\n \n test_expect_success 'expanded in-memory index matches full index' '\n \tinit_repos &&\n-\ttest_sparse_match test-tool read-cache --expand --table\n+\ttest_sparse_match git ls-files --stage\n '\n \n test_expect_success 'status with options' '\n@@ -787,9 +784,9 @@ test_expect_success 'submodule handling' '\n \n \t# having a submodule prevents \"modules\" from collapse\n \ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\tgrep \"100644 blob .*\tmodules/a\" cache &&\n-\tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n+\tgrep \"100644 .*\tmodules/a\" cache &&\n+\tgrep \"160000 $(git -C initial-repo rev-parse HEAD) 0\tmodules/sub\" cache\n '\n \n # When working with a sparse index, some commands will need to expand the\n@@ -1079,13 +1076,13 @@ test_expect_success 'reset mixed and checkout orphan' '\n \t# the sparse checkouts skip \"adding\" the other side of\n \t# the conflict.\n \ttest_sparse_match git reset --mixed HEAD~1 &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2 &&\n \n \t# At this point, sparse-checkouts behave differently\n \t# from the full-checkout.\n \ttest_sparse_match git checkout --orphan new-branch &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2\n '\n \n-- \ngitgitgadget\n\n"},{"id":"443508","messageId":"b98e5e6d2bc921ea76d8b5419d1f00e9db30a55a.1638992396.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] t1091/t3705: remove 'test-tool read-cache --table'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:54Z","receivedAt":"2021-12-08T19:40:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1091-sparse-checkout-builtin.sh and\nt3705-add-sparse-checkout.sh.\n\nThe important changes are due to the different output format. In t3705,\nwWe need to use the '--stage' output to get a file mode and OID, but\nit also includes a stage value and drops the object type. This leads\nto some differences in how we handle looking for specific entries.\n\nIn t1091, the test focuses on enabling the sparse index, so we do not\nneed the --stage flag to demonstrate how the index changes, and instead\ncan use a diff.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 25 ++++++++++++++++++++-----\n t/t3705-add-sparse-checkout.sh | 8 ++++----\n 2 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 272ba1b566b..680e0043c36 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -212,12 +212,27 @@ test_expect_success 'sparse-index enabled and disabled' '\n \n \t\tgit -C repo sparse-checkout init --cone --sparse-index &&\n \t\ttest_cmp_config -C repo true index.sparse &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\tgrep \" tree \" cache &&\n-\n+\t\tgit -C repo ls-files --sparse >sparse &&\n \t\tgit -C repo sparse-checkout disable &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\t! grep \" tree \" cache &&\n+\t\tgit -C repo ls-files --sparse >full &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\t@@ -1,4 +1,7 @@\n+\t\t a\n+\t\t-deep/\n+\t\t-folder1/\n+\t\t-folder2/\n+\t\t+deep/a\n+\t\t+deep/deeper1/a\n+\t\t+deep/deeper1/deepest/a\n+\t\t+deep/deeper2/a\n+\t\t+folder1/a\n+\t\t+folder2/a\n+\t\tEOF\n+\n+\t\tdiff -u sparse full | tail -n +3 >actual &&\n+\t\ttest_cmp expect actual &&\n+\n \t\tgit -C repo config --list >config &&\n \t\t! grep index.sparse config\n \t)\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex f3143c92908..81f3384eeed 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -181,13 +181,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100644 .*sparse_entry\\$\" actual &&\n \n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100755 .*sparse_entry\\$\" actual &&\n \n \tgit reset &&\n \n-- \ngitgitgadget\n\n"},{"id":"443509","messageId":"f31a24eeb9b2a7e0aca5bd02a02ac95232337d81.1638992396.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] test-read-cache: remove --table, --expand options","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-08T19:39:55Z","receivedAt":"2021-12-08T19:40:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis commit effectively reverts 2782db3 (test-tool: don't force full\nindex, 2021-03-30) and e2df6c3 (test-read-cache: print cache entries\nwith --table, 2021-03-30) to remove the --table and --expand options\nfrom 'test-tool read-cache'. The previous changes already removed these\noptions from the test suite in favor of 'git ls-files --sparse'.\n\nThe initial thought of creating these options was to allow for tests to\nsee additional information with every cache entry. In particular, the\nobject type is still not mirrored in 'git ls-files'. Since sparse\ndirectory entries always end with a slash, the object type is not\ncritical to verify the sparse index is enabled. It was thought that it\nwould be helpful to have additional information, such as flags, but that\nwas not needed for the FS Monitor integration and hasn't been needed\nsince.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-cache.c | 64 ++++++--------------------------------\n 1 file changed, 10 insertions(+), 54 deletions(-)\n\ndiff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c\nindex 0d9f08931a1..b736ef16421 100644\n--- a/t/helper/test-read-cache.c\n+++ b/t/helper/test-read-cache.c\n@@ -1,83 +1,39 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n #include \"config.h\"\n-#include \"blob.h\"\n-#include \"commit.h\"\n-#include \"tree.h\"\n-#include \"sparse-index.h\"\n-\n-static void print_cache_entry(struct cache_entry *ce)\n-{\n-\tconst char *type;\n-\tprintf(\"%06o \", ce->ce_mode & 0177777);\n-\n-\tif (S_ISSPARSEDIR(ce->ce_mode))\n-\t\ttype = tree_type;\n-\telse if (S_ISGITLINK(ce->ce_mode))\n-\t\ttype = commit_type;\n-\telse\n-\t\ttype = blob_type;\n-\n-\tprintf(\"%s %s\\t%s\\n\",\n-\t type,\n-\t oid_to_hex(&ce->oid),\n-\t ce->name);\n-}\n-\n-static void print_cache(struct index_state *istate)\n-{\n-\tint i;\n-\tfor (i = 0; i < istate->cache_nr; i++)\n-\t\tprint_cache_entry(istate->cache[i]);\n-}\n \n int cmd__read_cache(int argc, const char **argv)\n {\n-\tstruct repository *r = the_repository;\n \tint i, cnt = 1;\n \tconst char *name = NULL;\n-\tint table = 0, expand = 0;\n \n \tinitialize_the_repository();\n \n-\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n-\t\tif (skip_prefix(*argv, \"--print-and-refresh=\", &name))\n-\t\t\tcontinue;\n-\t\tif (!strcmp(*argv, \"--table\"))\n-\t\t\ttable = 1;\n-\t\telse if (!strcmp(*argv, \"--expand\"))\n-\t\t\texpand = 1;\n+\tif (argc > 1 && skip_prefix(argv[1], \"--print-and-refresh=\", &name)) {\n+\t\targc--;\n+\t\targv++;\n \t}\n \n-\tif (argc == 1)\n-\t\tcnt = strtol(argv[0], NULL, 0);\n+\tif (argc == 2)\n+\t\tcnt = strtol(argv[1], NULL, 0);\n \tsetup_git_directory();\n \tgit_config(git_default_config, NULL);\n \n-\tprepare_repo_settings(r);\n-\tr->settings.command_requires_full_index = 0;\n-\n \tfor (i = 0; i < cnt; i++) {\n-\t\trepo_read_index(r);\n-\n-\t\tif (expand)\n-\t\t\tensure_full_index(r->index);\n-\n+\t\tread_cache();\n \t\tif (name) {\n \t\t\tint pos;\n \n-\t\t\trefresh_index(r->index, REFRESH_QUIET,\n+\t\t\trefresh_index(&the_index, REFRESH_QUIET,\n \t\t\t\t NULL, NULL, NULL);\n-\t\t\tpos = index_name_pos(r->index, name, strlen(name));\n+\t\t\tpos = index_name_pos(&the_index, name, strlen(name));\n \t\t\tif (pos < 0)\n \t\t\t\tdie(\"%s not in index\", name);\n \t\t\tprintf(\"%s is%s up to date\\n\", name,\n-\t\t\t ce_uptodate(r->index->cache[pos]) ? \"\" : \" not\");\n+\t\t\t ce_uptodate(the_index.cache[pos]) ? \"\" : \" not\");\n \t\t\twrite_file(name, \"%d\\n\", i);\n \t\t}\n-\t\tif (table)\n-\t\t\tprint_cache(r->index);\n-\t\tdiscard_index(r->index);\n+\t\tdiscard_cache();\n \t}\n \treturn 0;\n }\n-- \ngitgitgadget\n"},{"id":"443581","messageId":"CABPp-BF5kw69ioheMvRPoVso8FJf45Fdm_Cf-P7PfqWsqHz9Gw@mail.gmail.com","threadId":"56917","inReplyTo":"58b5eca483502b0d45601490f6b4255f6419e6bd.1638992396.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/5] ls-files: add --sparse option","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-09T05:08:59Z","receivedAt":"2021-12-09T05:09:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 11:39 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Existing callers to 'git ls-files' are expecting file names, not\n> directories. It is best to expand a sparse index to show all of the\n> contained files in this case.\n>\n> However, expert users may want to inspect the contents of the index\n> itself including which directories are sparse. Add a --sparse option to\n> allow users to request this information.\n>\n> During testing, I noticed that options such as --modified did not affect\n> the output when the files in question were outside the sparse-checkout\n> definition. Tests are added to document this preexisting behavior and\n> how it remains unchanged with the sparse index and the --sparse option.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-ls-files.txt | 4 ++\n> builtin/ls-files.c | 12 +++-\n> t/t1092-sparse-checkout-compatibility.sh | 90 ++++++++++++++++++++++++\n> 3 files changed, 104 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\n> index 6d11ab506b7..1c5d5f85ec5 100644\n> --- a/Documentation/git-ls-files.txt\n> +++ b/Documentation/git-ls-files.txt\n> @@ -187,6 +187,10 @@ Both the in the index (\"i/\")\n> and in the working tree (\"w/\") are shown for regular files,\n> followed by the (\"attr/\").\n>\n> +--sparse::\n> + If the index is sparse, show the sparse directories without expanding\n> + to the contained files.\n> +\n> \\--::\n> Do not interpret any more arguments as options.\n>\n> diff --git a/builtin/ls-files.c b/builtin/ls-files.c\n> index 031fef1bcaa..c151eb1fb77 100644\n> --- a/builtin/ls-files.c\n> +++ b/builtin/ls-files.c\n> @@ -37,6 +37,7 @@ static int debug_mode;\n> static int show_eol;\n> static int recurse_submodules;\n> static int skipping_duplicates;\n> +static int show_sparse_dirs;\n>\n> static const char *prefix;\n> static int max_prefix_len;\n> @@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n>\n> if (!(show_cached || show_stage || show_deleted || show_modified))\n> return;\n> - /* TODO: audit for interaction with sparse-index. */\n> - ensure_full_index(repo->index);\n> +\n> + if (!show_sparse_dirs)\n> + ensure_full_index(repo->index);\n> +\n> for (i = 0; i < repo->index->cache_nr; i++) {\n> const struct cache_entry *ce = repo->index->cache[i];\n> struct stat st;\n> @@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> OPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n> OPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n> N_(\"suppress duplicate entries\")),\n> + OPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n> + N_(\"show sparse directories in the presence of a sparse index\")),\n> OPT_END()\n> };\n> int ret = 0;\n> @@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> if (argc == 2 && !strcmp(argv[1], \"-h\"))\n> usage_with_options(ls_files_usage, builtin_ls_files_options);\n>\n> + prepare_repo_settings(the_repository);\n> + the_repository->settings.command_requires_full_index = 0;\n> +\n> prefix = cmd_prefix;\n> if (prefix)\n> prefix_len = strlen(prefix);\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index 3ba19ba1c14..bf2c6b169b9 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -802,6 +802,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> git -C sparse-index reset -- folder1/a &&\n> test_region index convert_to_sparse trace2.txt &&\n> + test_region index ensure_full_index trace2.txt &&\n> +\n> + # ls-files expands on read, but does not write.\n> + rm trace2.txt &&\n> + GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n> + git -C sparse-index ls-files &&\n> test_region index ensure_full_index trace2.txt\n> '\n>\n> @@ -826,6 +832,7 @@ test_expect_success 'sparse-index is not expanded' '\n> init_repos &&\n>\n> ensure_not_expanded status &&\n> + ensure_not_expanded ls-files --sparse &&\n> ensure_not_expanded commit --allow-empty -m empty &&\n> echo >>sparse-index/a &&\n> ensure_not_expanded commit -a -m a &&\n> @@ -974,6 +981,89 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n> ensure_not_expanded pull full base\n> '\n>\n> +test_expect_success 'ls-files' '\n> + init_repos &&\n> +\n> + # Behavior agrees by default. Sparse index is expanded.\n> + test_all_match git ls-files &&\n> +\n> + # With --sparse, the sparse index data changes behavior.\n> + git -C sparse-index ls-files >dense &&\n> + git -C sparse-index ls-files --sparse >sparse &&\n> +\n> + cat >expect <<-\\EOF &&\n> + @@ -13,13 +13,9 @@\n> + e\n> + folder1-\n> + folder1.x\n> + -folder1/0/0/0\n> + -folder1/0/1\n> + -folder1/a\n> + +folder1/\n> + folder10\n> + -folder2/0/0/0\n> + -folder2/0/1\n> + -folder2/a\n> + +folder2/\n> + g\n> + -x/a\n> + +x/\n> + z\n> + EOF\n> +\n> + diff -u dense sparse | tail -n +3 >actual &&\n> + test_cmp expect actual &&\n> +\n> + # With --sparse and no sparse index, nothing changes.\n> + git -C sparse-checkout ls-files >dense &&\n> + git -C sparse-checkout ls-files --sparse >sparse &&\n> + test_cmp dense sparse &&\n> +\n\nI still believe this next section deserves a comment; something like\n\n# Set up an erroneous condition: folder1/a present despite SKIP_WORKTREE.\n# We do this just to verify consistency between sparse-index and\nnon-sparse-index\n# with such files.\n\n> + write_script edit-content <<-\\EOF &&\n> + mkdir folder1 &&\n> + echo content >>folder1/a\n> + EOF\n> + run_on_sparse ../edit-content &&\n> +\n> + # ls-files does not notice modified files whose\n> + # cache entries are marked SKIP_WORKTREE.\n\nI also believe this still deserves a slight change, e.g.\n\n# ls-files does not currently notice modified files whose\n# cache entries are marked SKIP_WORKTREE; this\n# may change in the future, but we test here that\n# sparse-index and non-sparse-index at least match.\n\n\n> + test_sparse_match git ls-files --modified &&\n> + test_must_be_empty sparse-checkout-out &&\n> + test_must_be_empty sparse-index-out &&\n> +\n> + git -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n> + test_must_be_empty sparse-index-out &&\n> +\n> + # Add folder1 to the sparse-checkout cone and\n> + # check that ls-files shows the expanded files.\n> + test_sparse_match git sparse-checkout add folder1 &&\n> + test_sparse_match git ls-files --modified &&\n> +\n> + git -C sparse-index ls-files >dense &&\n> + git -C sparse-index ls-files --sparse >sparse &&\n> +\n> + cat >expect <<-\\EOF &&\n> + @@ -17,9 +17,7 @@\n> + folder1/0/1\n> + folder1/a\n> + folder10\n> + -folder2/0/0/0\n> + -folder2/0/1\n> + -folder2/a\n> + +folder2/\n> + g\n> + -x/a\n> + +x/\n> + z\n> + EOF\n> +\n> + diff -u dense sparse | tail -n +3 >actual &&\n> + test_cmp expect actual &&\n> +\n> + # Double-check index expansion is avoided\n> + ensure_not_expanded ls-files --sparse\n> +'\n> +\n> # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n> # in this scenario, but it shouldn't.\n> test_expect_success 'reset mixed and checkout orphan' '\n> --\n> gitgitgadget\n"},{"id":"443602","messageId":"CABPp-BGQh2qNHKGmxr7rUqzyCjVpY=o2QB9qv6S8arz-mVm51Q@mail.gmail.com","threadId":"56917","inReplyTo":"5ffae2a03ae3d00a1f128d7cf2a397081055a2fe.1638992396.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] t1092: replace 'read-cache --table' with 'ls-files --sparse'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-09T05:19:49Z","receivedAt":"2021-12-09T05:20:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 11:39 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Now that 'git ls-files --sparse' exists, we can use it to verify the\n> state of a sparse index instead of 'test-tool read-cache table'. Replace\n> these usages within t1092-sparse-checkout-compatibility.sh.\n>\n> The important changes are due to the different output format. We need to\n> use the '--stage' output to get a file mode and OID, but it also\n> includes a stage value and drops the object type. This leads to some\n> differences in how we handle looking for specific entries.\n>\n> Some places where we previously looked for no 'tree' entries, we can\n> instead directly compare the output across the repository with a sparse\n> index and the one without.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1092-sparse-checkout-compatibility.sh | 29 +++++++++++-------------\n> 1 file changed, 13 insertions(+), 16 deletions(-)\n>\n> diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\n> index bf2c6b169b9..4a6832ea3c5 100755\n> --- a/t/t1092-sparse-checkout-compatibility.sh\n> +++ b/t/t1092-sparse-checkout-compatibility.sh\n> @@ -200,45 +200,42 @@ test_sparse_unstaged () {\n> test_expect_success 'sparse-index contents' '\n> init_repos &&\n>\n> - test-tool -C sparse-index read-cache --table >cache &&\n> + git -C sparse-index ls-files --sparse --stage >cache &&\n> for dir in folder1 folder2 x\n> do\n> TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n> - grep \"040000 tree $TREE $dir/\" cache \\\n> + grep \"040000 $TREE 0 $dir/\" cache \\\n> || return 1\n> done &&\n>\n> git -C sparse-index sparse-checkout set folder1 &&\n>\n> - test-tool -C sparse-index read-cache --table >cache &&\n> + git -C sparse-index ls-files --sparse --stage >cache &&\n> for dir in deep folder2 x\n> do\n> TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n> - grep \"040000 tree $TREE $dir/\" cache \\\n> + grep \"040000 $TREE 0 $dir/\" cache \\\n> || return 1\n> done &&\n>\n> git -C sparse-index sparse-checkout set deep/deeper1 &&\n>\n> - test-tool -C sparse-index read-cache --table >cache &&\n> + git -C sparse-index ls-files --sparse --stage >cache &&\n> for dir in deep/deeper2 folder1 folder2 x\n> do\n> TREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n> - grep \"040000 tree $TREE $dir/\" cache \\\n> + grep \"040000 $TREE 0 $dir/\" cache \\\n> || return 1\n> done &&\n>\n> # Disabling the sparse-index removes tree entries with full ones\n\nNot the fault of this patch, but perhaps worth fixing \"removes\" ->\n\"replaces\" while you're making fixes in this area?\n\n\n> git -C sparse-index sparse-checkout init --no-sparse-index &&\n> -\n> - test-tool -C sparse-index read-cache --table >cache &&\n> - ! grep \"040000 tree\" cache &&\n> - test_sparse_match test-tool read-cache --table\n> + test_sparse_match git ls-files --stage --sparse\n> '\n>\n> test_expect_success 'expanded in-memory index matches full index' '\n> init_repos &&\n> - test_sparse_match test-tool read-cache --expand --table\n> + test_sparse_match git ls-files --stage\n> '\n>\n> test_expect_success 'status with options' '\n> @@ -787,9 +784,9 @@ test_expect_success 'submodule handling' '\n>\n> # having a submodule prevents \"modules\" from collapse\n> test_sparse_match git sparse-checkout set deep/deeper1 &&\n> - test-tool -C sparse-index read-cache --table >cache &&\n> - grep \"100644 blob .* modules/a\" cache &&\n> - grep \"160000 commit $(git -C initial-repo rev-parse HEAD) modules/sub\" cache\n> + git -C sparse-index ls-files --sparse --stage >cache &&\n> + grep \"100644 .* modules/a\" cache &&\n> + grep \"160000 $(git -C initial-repo rev-parse HEAD) 0 modules/sub\" cache\n> '\n>\n> # When working with a sparse index, some commands will need to expand the\n> @@ -1079,13 +1076,13 @@ test_expect_success 'reset mixed and checkout orphan' '\n> # the sparse checkouts skip \"adding\" the other side of\n> # the conflict.\n> test_sparse_match git reset --mixed HEAD~1 &&\n> - test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git ls-files --stage &&\n> test_sparse_match git status --porcelain=v2 &&\n>\n> # At this point, sparse-checkouts behave differently\n> # from the full-checkout.\n> test_sparse_match git checkout --orphan new-branch &&\n> - test_sparse_match test-tool read-cache --table --expand &&\n> + test_sparse_match git ls-files --stage &&\n> test_sparse_match git status --porcelain=v2\n> '\n>\n> --\n> gitgitgadget\n>\n"},{"id":"443603","messageId":"CABPp-BHkdyMD1RVptCgROnd=1pw_xvhOj4b1pvYFRmukqyvbNQ@mail.gmail.com","threadId":"56917","inReplyTo":"b98e5e6d2bc921ea76d8b5419d1f00e9db30a55a.1638992396.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/5] t1091/t3705: remove 'test-tool read-cache --table'","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-09T05:20:08Z","receivedAt":"2021-12-09T05:20:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 11:40 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Now that 'git ls-files --sparse' exists, we can use it to verify the\n> state of a sparse index instead of 'test-tool read-cache table'. Replace\n> these usages within t1091-sparse-checkout-builtin.sh and\n> t3705-add-sparse-checkout.sh.\n>\n> The important changes are due to the different output format. In t3705,\n> wWe need to use the '--stage' output to get a file mode and OID, but\n\ns/wWe/we/\n\n\n> it also includes a stage value and drops the object type. This leads\n> to some differences in how we handle looking for specific entries.\n>\n> In t1091, the test focuses on enabling the sparse index, so we do not\n> need the --stage flag to demonstrate how the index changes, and instead\n> can use a diff.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1091-sparse-checkout-builtin.sh | 25 ++++++++++++++++++++-----\n> t/t3705-add-sparse-checkout.sh | 8 ++++----\n> 2 files changed, 24 insertions(+), 9 deletions(-)\n>\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 272ba1b566b..680e0043c36 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -212,12 +212,27 @@ test_expect_success 'sparse-index enabled and disabled' '\n>\n> git -C repo sparse-checkout init --cone --sparse-index &&\n> test_cmp_config -C repo true index.sparse &&\n> - test-tool -C repo read-cache --table >cache &&\n> - grep \" tree \" cache &&\n> -\n> + git -C repo ls-files --sparse >sparse &&\n> git -C repo sparse-checkout disable &&\n> - test-tool -C repo read-cache --table >cache &&\n> - ! grep \" tree \" cache &&\n> + git -C repo ls-files --sparse >full &&\n> +\n> + cat >expect <<-\\EOF &&\n> + @@ -1,4 +1,7 @@\n> + a\n> + -deep/\n> + -folder1/\n> + -folder2/\n> + +deep/a\n> + +deep/deeper1/a\n> + +deep/deeper1/deepest/a\n> + +deep/deeper2/a\n> + +folder1/a\n> + +folder2/a\n> + EOF\n> +\n> + diff -u sparse full | tail -n +3 >actual &&\n> + test_cmp expect actual &&\n> +\n> git -C repo config --list >config &&\n> ! grep index.sparse config\n> )\n> diff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\n> index f3143c92908..81f3384eeed 100755\n> --- a/t/t3705-add-sparse-checkout.sh\n> +++ b/t/t3705-add-sparse-checkout.sh\n> @@ -181,13 +181,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n> # Avoid munging CRLFs to avoid an error message\n> git -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n> test_must_be_empty stderr &&\n> - test-tool read-cache --table >actual &&\n> - grep \"^100644 blob.*sparse_entry\\$\" actual &&\n> + git ls-files --stage >actual &&\n> + grep \"^100644 .*sparse_entry\\$\" actual &&\n>\n> git add --sparse --chmod=+x sparse_entry 2>stderr &&\n> test_must_be_empty stderr &&\n> - test-tool read-cache --table >actual &&\n> - grep \"^100755 blob.*sparse_entry\\$\" actual &&\n> + git ls-files --stage >actual &&\n> + grep \"^100755 .*sparse_entry\\$\" actual &&\n>\n> git reset &&\n>\n> --\n> gitgitgadget\n"},{"id":"443604","messageId":"CABPp-BH1y1DccvqW58WBui7kwj0Z2HrvzrRoQhdG3YLJkGX=KA@mail.gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/5] Sparse index: fetch, pull, ls-files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-09T05:23:43Z","receivedAt":"2021-12-09T05:23:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 8, 2021 at 11:39 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is based on ld/sparse-index-blame (merged with 'master' due to an\n> unrelated build issue).\n>\n> Here are two relatively-simple patches that further the sparse index\n> integrations.\n>\n> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n> have been an integration much earlier in the cycle. They read the index to\n> look for the .gitmodules file in case there are submodules that need to be\n> fetched. Since looking for a file by name is already protected, we only need\n> to disable 'command_requires_full_index' and we are done.\n>\n> The 'ls-files' builtin is useful when debugging the index, and some scripts\n> use it, too. We are not changing the default behavior which expands a sparse\n> index in order to show all of the cached blobs. Instead, we add a '--sparse'\n> option that allows us to see the sparse directory entries upon request.\n> Combined with --debug, we can see a lot of index details, such as:\n>\n> $ git ls-files --debug --sparse\n> LICENSE\n> ctime: 1634910503:287405820\n> mtime: 1634910503:287405820\n> dev: 16777220 ino: 119325319\n> uid: 501 gid: 20\n> size: 1098 flags: 200000\n> README.md\n> ctime: 1634910503:288090279\n> mtime: 1634910503:288090279\n> dev: 16777220 ino: 119325320\n> uid: 501 gid: 20\n> size: 934 flags: 200000\n> bin/index.js\n> ctime: 1634910767:828434033\n> mtime: 1634910767:828434033\n> dev: 16777220 ino: 119325520\n> uid: 501 gid: 20\n> size: 7292 flags: 200000\n> examples/\n> ctime: 0:0\n> mtime: 0:0\n> dev: 0 ino: 0\n> uid: 0 gid: 0\n> size: 0 flags: 40004000\n> package.json\n> ctime: 1634910503:288676330\n> mtime: 1634910503:288676330\n> dev: 16777220 ino: 119325321\n> uid: 501 gid: 20\n> size: 680 flags: 200000\n>\n>\n> (In this example, the 'examples/' directory is sparse.)\n>\n> Thanks!\n>\n>\n> Updates in v2\n> =============\n>\n> * Rebased onto latest ld/sparse-index-blame without issue.\n> * Updated the test to use diff-of-diffs instead of a sequence of greps.\n> * Added patches that remove the use of 'test-tool read-cache --table' and\n> its implementation.\n\nI still think a couple things in patch 2 deserve some comments about\nthe expectations. Other than that, though, the series reads nicely\nand I was only able to spot a few other very minor items.\n\n> Derrick Stolee (5):\n> fetch/pull: use the sparse index\n> ls-files: add --sparse option\n> t1092: replace 'read-cache --table' with 'ls-files --sparse'\n> t1091/t3705: remove 'test-tool read-cache --table'\n> test-read-cache: remove --table, --expand options\n>\n> Documentation/git-ls-files.txt | 4 +\n> builtin/fetch.c | 2 +\n> builtin/ls-files.c | 12 ++-\n> builtin/pull.c | 2 +\n> t/helper/test-read-cache.c | 64 ++---------\n> t/t1091-sparse-checkout-builtin.sh | 25 ++++-\n> t/t1092-sparse-checkout-compatibility.sh | 129 ++++++++++++++++++++---\n> t/t3705-add-sparse-checkout.sh | 8 +-\n> 8 files changed, 165 insertions(+), 81 deletions(-)\n>\n>\n> base-commit: 3fffe69d24e4ecc95246766f5396303a953695ff\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1080%2Fderrickstolee%2Fsparse-index%2Ffetch-pull-ls-files-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1080/derrickstolee/sparse-index/fetch-pull-ls-files-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1080\n>\n> Range-diff vs v1:\n>\n> 1: 451056e1a77 ! 1: f72001638d1 fetch/pull: use the sparse index\n> @@ builtin/pull.c: int cmd_pull(int argc, const char **argv, const char *prefix)\n>\n> ## t/t1092-sparse-checkout-compatibility.sh ##\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is not expanded: blame' '\n> - ensure_not_expanded blame deep/deeper1/deepest/a\n> + done\n> '\n>\n> +test_expect_success 'sparse index is not expanded: fetch/pull' '\n> 2: e42c0feec94 ! 2: 58b5eca4835 ls-files: add --sparse option\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> + test_all_match git ls-files &&\n> +\n> + # With --sparse, the sparse index data changes behavior.\n> -+ git -C sparse-index ls-files --sparse >sparse-index-out &&\n> -+ grep \"^folder1/\\$\" sparse-index-out &&\n> -+ grep \"^folder2/\\$\" sparse-index-out &&\n> ++ git -C sparse-index ls-files >dense &&\n> ++ git -C sparse-index ls-files --sparse >sparse &&\n> ++\n> ++ cat >expect <<-\\EOF &&\n> ++ @@ -13,13 +13,9 @@\n> ++ e\n> ++ folder1-\n> ++ folder1.x\n> ++ -folder1/0/0/0\n> ++ -folder1/0/1\n> ++ -folder1/a\n> ++ +folder1/\n> ++ folder10\n> ++ -folder2/0/0/0\n> ++ -folder2/0/1\n> ++ -folder2/a\n> ++ +folder2/\n> ++ g\n> ++ -x/a\n> ++ +x/\n> ++ z\n> ++ EOF\n> ++\n> ++ diff -u dense sparse | tail -n +3 >actual &&\n> ++ test_cmp expect actual &&\n> +\n> + # With --sparse and no sparse index, nothing changes.\n> -+ git -C sparse-checkout ls-files --sparse >sparse-checkout-out &&\n> -+ grep \"^folder1/0/0/0\\$\" sparse-checkout-out &&\n> -+ ! grep \"/\\$\" sparse-checkout-out &&\n> ++ git -C sparse-checkout ls-files >dense &&\n> ++ git -C sparse-checkout ls-files --sparse >sparse &&\n> ++ test_cmp dense sparse &&\n> +\n> + write_script edit-content <<-\\EOF &&\n> + mkdir folder1 &&\n> @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n> + git -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n> + test_must_be_empty sparse-index-out &&\n> +\n> -+ run_on_sparse git sparse-checkout add folder1 &&\n> ++ # Add folder1 to the sparse-checkout cone and\n> ++ # check that ls-files shows the expanded files.\n> ++ test_sparse_match git sparse-checkout add folder1 &&\n> + test_sparse_match git ls-files --modified &&\n> -+ grep \"^folder1/a\\$\" sparse-checkout-out &&\n> -+ grep \"^folder1/a\\$\" sparse-index-out &&\n> +\n> -+ # Double-check index expansion\n> ++ git -C sparse-index ls-files >dense &&\n> ++ git -C sparse-index ls-files --sparse >sparse &&\n> ++\n> ++ cat >expect <<-\\EOF &&\n> ++ @@ -17,9 +17,7 @@\n> ++ folder1/0/1\n> ++ folder1/a\n> ++ folder10\n> ++ -folder2/0/0/0\n> ++ -folder2/0/1\n> ++ -folder2/a\n> ++ +folder2/\n> ++ g\n> ++ -x/a\n> ++ +x/\n> ++ z\n> ++ EOF\n> ++\n> ++ diff -u dense sparse | tail -n +3 >actual &&\n> ++ test_cmp expect actual &&\n> ++\n> ++ # Double-check index expansion is avoided\n> + ensure_not_expanded ls-files --sparse\n> +'\n> +\n> -: ----------- > 3: 5ffae2a03ae t1092: replace 'read-cache --table' with 'ls-files --sparse'\n> -: ----------- > 4: b98e5e6d2bc t1091/t3705: remove 'test-tool read-cache --table'\n> -: ----------- > 5: f31a24eeb9b test-read-cache: remove --table, --expand options\n>\n> --\n> gitgitgadget\n"},{"id":"443646","messageId":"211209.86v8zydjjl.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"c8e22885-1759-d3d9-3944-2d70c70960e2@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-09T12:50:53Z","receivedAt":"2021-12-09T12:55:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Dec 08 2021, Derrick Stolee wrote:\n\n> On 12/8/2021 1:36 PM, Elijah Newren wrote:\n>> On Wed, Dec 8, 2021 at 10:23 AM Derrick Stolee wrote:\n>>>\n>>> On 12/8/2021 12:04 PM, Elijah Newren wrote:\n>>>>\n>>>> This actually looks quite nice, though the magic '16' is kind of\n>>>> annoying. Could we get rid of that -- perhaps using something to rip\n>>>> out the diff header, or using comm instead?\n>>>\n>>> What I really want is \"remove the first two lines of this file\"\n>> \n>> Is `tail -n +3` portable? Looks like we have five uses of tail -n +N\n>> in the testsuite, so it should be okay to use.\n>\n> Ah, that's the magic incantation. Sounds good.\n>\n>>> but perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\n>>> magic number?\n>> \n>> That works too.\n>\n> If the \"-n +X\" syntax works, then I'll opt for that.\n\nI think it should be per\nhttps://pubs.opengroup.org/onlinepubs/009695299/utilities/tail.html\n\nBut isn't that \"diff -u\" non-portable, per GIT_TEST_CMP. I.e. sometimes\nwe'll fall back on \"cmp\" etc.\n\nJust pre-munging both \"a\" and \"b\" and doing a:\n\n test_cmp a b\n\nShould work around that & make it portable.\n\nI realize that's also a problem with my initial example, usually we'd\ncompare expected/actual, not the diff between the two.\n\nBut if that's more readable we could also just use \"git diff --no-index\na b >actual\" and \"test_cmp\" that v.s. \"expected\".\n\n \n"},{"id":"443801","messageId":"e2a39196-5ee6-fbe7-a24c-c14c809f707d@gmail.com","threadId":"56917","inReplyTo":"CABPp-BF5kw69ioheMvRPoVso8FJf45Fdm_Cf-P7PfqWsqHz9Gw@mail.gmail.com","subject":"Re: [PATCH v2 2/5] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-10T13:51:53Z","receivedAt":"2021-12-10T13:51:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/9/2021 12:08 AM, Elijah Newren wrote:\n> On Wed, Dec 8, 2021 at 11:39 AM Derrick Stolee via GitGitGadget\n\n>> + # With --sparse and no sparse index, nothing changes.\n>> + git -C sparse-checkout ls-files >dense &&\n>> + git -C sparse-checkout ls-files --sparse >sparse &&\n>> + test_cmp dense sparse &&\n>> +\n> \n> I still believe this next section deserves a comment; something like\n> \n> # Set up an erroneous condition: folder1/a present despite SKIP_WORKTREE.\n> # We do this just to verify consistency between sparse-index and\n> non-sparse-index\n> # with such files.\n> \n>> + write_script edit-content <<-\\EOF &&\n>> + mkdir folder1 &&\n>> + echo content >>folder1/a\n>> + EOF\n>> + run_on_sparse ../edit-content &&\n>> +\n>> + # ls-files does not notice modified files whose\n>> + # cache entries are marked SKIP_WORKTREE.\n> \n> I also believe this still deserves a slight change, e.g.\n> \n> # ls-files does not currently notice modified files whose\n> # cache entries are marked SKIP_WORKTREE; this\n> # may change in the future, but we test here that\n> # sparse-index and non-sparse-index at least match.\n> \n> \n>> + test_sparse_match git ls-files --modified &&\n>> + test_must_be_empty sparse-checkout-out &&\n>> + test_must_be_empty sparse-index-out &&\n\nOk. Sorry I missed these in the gap between v1 and v2.\n\nThanks,\n-Stolee\n"},{"id":"443802","messageId":"745e79e3-521d-58bd-2ddf-1da4e0f6fe26@gmail.com","threadId":"56917","inReplyTo":"211209.86v8zydjjl.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-10T13:57:29Z","receivedAt":"2021-12-10T13:57:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/9/2021 7:50 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Dec 08 2021, Derrick Stolee wrote:\n> \n>> On 12/8/2021 1:36 PM, Elijah Newren wrote:\n>>> On Wed, Dec 8, 2021 at 10:23 AM Derrick Stolee wrote:\n>>>>\n>>>> On 12/8/2021 12:04 PM, Elijah Newren wrote:\n>>>>>\n>>>>> This actually looks quite nice, though the magic '16' is kind of\n>>>>> annoying. Could we get rid of that -- perhaps using something to rip\n>>>>> out the diff header, or using comm instead?\n>>>>\n>>>> What I really want is \"remove the first two lines of this file\"\n>>>\n>>> Is `tail -n +3` portable? Looks like we have five uses of tail -n +N\n>>> in the testsuite, so it should be okay to use.\n>>\n>> Ah, that's the magic incantation. Sounds good.\n>>\n>>>> but perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\n>>>> magic number?\n>>>\n>>> That works too.\n>>\n>> If the \"-n +X\" syntax works, then I'll opt for that.\n> \n> I think it should be per\n> https://pubs.opengroup.org/onlinepubs/009695299/utilities/tail.html\n> \n> But isn't that \"diff -u\" non-portable, per GIT_TEST_CMP. I.e. sometimes\n> we'll fall back on \"cmp\" etc.\n\nYou're talking about this hunk, right?\n\nif test -z \"$GIT_TEST_CMP\"\nthen\n\tif test -n \"$GIT_TEST_CMP_USE_COPIED_CONTEXT\"\n\tthen\n\t\tGIT_TEST_CMP=\"$DIFF -c\"\n\telse\n\t\tGIT_TEST_CMP=\"$DIFF -u\"\n\tfi\nfi\n\nThis only switches from \"diff -u\" to \"diff -c\" if the\nGIT_TEST_CMP_USE_COPIED_CONTEXT variable is set, but it is not set\nby default. Thus, we are using \"diff -u\" by default throughout.\n\nPlease let me know if I'm misreading this.\n\nThanks,\n-Stolee\n"},{"id":"443808","messageId":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v2.git.1638992395.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:07Z","receivedAt":"2021-12-10T15:13:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ld/sparse-index-blame (merged with 'master' due to an\nunrelated build issue).\n\nHere are two relatively-simple patches that further the sparse index\nintegrations.\n\nDid you know that 'fetch' and 'pull' read the index? I didn't, or this would\nhave been an integration much earlier in the cycle. They read the index to\nlook for the .gitmodules file in case there are submodules that need to be\nfetched. Since looking for a file by name is already protected, we only need\nto disable 'command_requires_full_index' and we are done.\n\nThe 'ls-files' builtin is useful when debugging the index, and some scripts\nuse it, too. We are not changing the default behavior which expands a sparse\nindex in order to show all of the cached blobs. Instead, we add a '--sparse'\noption that allows us to see the sparse directory entries upon request.\nCombined with --debug, we can see a lot of index details, such as:\n\n$ git ls-files --debug --sparse\nLICENSE\n ctime: 1634910503:287405820\n mtime: 1634910503:287405820\n dev: 16777220 ino: 119325319\n uid: 501 gid: 20\n size: 1098 flags: 200000\nREADME.md\n ctime: 1634910503:288090279\n mtime: 1634910503:288090279\n dev: 16777220 ino: 119325320\n uid: 501 gid: 20\n size: 934 flags: 200000\nbin/index.js\n ctime: 1634910767:828434033\n mtime: 1634910767:828434033\n dev: 16777220 ino: 119325520\n uid: 501 gid: 20\n size: 7292 flags: 200000\nexamples/\n ctime: 0:0\n mtime: 0:0\n dev: 0 ino: 0\n uid: 0 gid: 0\n size: 0 flags: 40004000\npackage.json\n ctime: 1634910503:288676330\n mtime: 1634910503:288676330\n dev: 16777220 ino: 119325321\n uid: 501 gid: 20\n size: 680 flags: 200000\n\n\n(In this example, the 'examples/' directory is sparse.)\n\nThanks!\n\n\nUpdates in v2\n=============\n\n * Rebased onto latest ld/sparse-index-blame without issue.\n * Updated the test to use diff-of-diffs instead of a sequence of greps.\n * Added patches that remove the use of 'test-tool read-cache --table' and\n its implementation.\n\n\nUpdates in v3\n=============\n\n * Fixed typo in commit message.\n * Added comments around doing strange things in an ls-files test.\n * Fixed adjacent typo in a test comment.\n\nDerrick Stolee (5):\n fetch/pull: use the sparse index\n ls-files: add --sparse option\n t1092: replace 'read-cache --table' with 'ls-files --sparse'\n t1091/t3705: remove 'test-tool read-cache --table'\n test-read-cache: remove --table, --expand options\n\n Documentation/git-ls-files.txt | 4 +\n builtin/fetch.c | 2 +\n builtin/ls-files.c | 12 +-\n builtin/pull.c | 2 +\n t/helper/test-read-cache.c | 64 ++---------\n t/t1091-sparse-checkout-builtin.sh | 25 ++++-\n t/t1092-sparse-checkout-compatibility.sh | 137 ++++++++++++++++++++---\n t/t3705-add-sparse-checkout.sh | 8 +-\n 8 files changed, 172 insertions(+), 82 deletions(-)\n\n\nbase-commit: 3fffe69d24e4ecc95246766f5396303a953695ff\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1080%2Fderrickstolee%2Fsparse-index%2Ffetch-pull-ls-files-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1080/derrickstolee/sparse-index/fetch-pull-ls-files-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1080\n\nRange-diff vs v2:\n\n 1: f72001638d1 = 1: f72001638d1 fetch/pull: use the sparse index\n 2: 58b5eca4835 ! 2: b81174ba54b ls-files: add --sparse option\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\tgit -C sparse-checkout ls-files --sparse >sparse &&\n +\ttest_cmp dense sparse &&\n +\n ++\t# Set up a strange condition of having a file edit\n ++\t# outside of the sparse-checkout cone. This is just\n ++\t# to verify that sparse-checkout and sparse-index\n ++\t# behave the same in this case.\n +\twrite_script edit-content <<-\\EOF &&\n +\tmkdir folder1 &&\n +\techo content >>folder1/a\n +\tEOF\n +\trun_on_sparse ../edit-content &&\n +\n -+\t# ls-files does not notice modified files whose\n -+\t# cache entries are marked SKIP_WORKTREE.\n ++\t# ls-files does not currently notice modified files whose\n ++\t# cache entries are marked SKIP_WORKTREE. This may change\n ++\t# in the future, but here we test that sparse index does\n ++\t# not accidentally create a change of behavior.\n +\ttest_sparse_match git ls-files --modified &&\n +\ttest_must_be_empty sparse-checkout-out &&\n +\ttest_must_be_empty sparse-index-out &&\n 3: 5ffae2a03ae ! 3: 2a6a1c5a39c t1092: replace 'read-cache --table' with 'ls-files --sparse'\n @@ t/t1092-sparse-checkout-compatibility.sh: test_sparse_unstaged () {\n \t\t\t|| return 1\n \tdone &&\n \n - \t# Disabling the sparse-index removes tree entries with full ones\n +-\t# Disabling the sparse-index removes tree entries with full ones\n ++\t# Disabling the sparse-index replaces tree entries with full ones\n \tgit -C sparse-index sparse-checkout init --no-sparse-index &&\n -\n -\ttest-tool -C sparse-index read-cache --table >cache &&\n 4: b98e5e6d2bc ! 4: f0143686754 t1091/t3705: remove 'test-tool read-cache --table'\n @@ Commit message\n t3705-add-sparse-checkout.sh.\n \n The important changes are due to the different output format. In t3705,\n - wWe need to use the '--stage' output to get a file mode and OID, but\n + we need to use the '--stage' output to get a file mode and OID, but\n it also includes a stage value and drops the object type. This leads\n to some differences in how we handle looking for specific entries.\n \n 5: f31a24eeb9b = 5: 9227dc54165 test-read-cache: remove --table, --expand options\n\n-- \ngitgitgadget\n"},{"id":"443809","messageId":"f72001638d1d0865cea0f728e700a4148bf7b37f.1639149192.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v3 1/5] fetch/pull: use the sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:08Z","receivedAt":"2021-12-10T15:13:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git fetch' and 'git pull' commands parse the index in order to\ndetermine if submodules exist. Without command_requires_full_index=0,\nthis will expand a sparse index, causing slow performance even when\nthere is no new data to fetch.\n\nThe .gitmodules file will never be inside a sparse directory entry, and\neven if it was, the index_name_pos() method would expand the sparse\nindex if needed as we search for the path by name. These commands do not\niterate over the index, which is the typical thing we are careful about\nwhen integrating with the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 2 ++\n builtin/pull.c | 2 ++\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex f7abbc31ff1..1696b7791d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1993,6 +1993,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tgit_config(git_fetch_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 1cfaf9f3436..7bce3bd80f9 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -994,6 +994,8 @@ int cmd_pull(int argc, const char **argv, const char *prefix)\n \t\tset_reflog_message(argc, argv);\n \n \tgit_config(git_pull_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix, pull_options, pull_usage, 0);\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 83971174ac3..3ba19ba1c14 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -964,6 +964,16 @@ test_expect_success 'sparse index is not expanded: blame' '\n \tdone\n '\n \n+test_expect_success 'sparse index is not expanded: fetch/pull' '\n+\tinit_repos &&\n+\n+\tgit -C sparse-index remote add full \"file://$(pwd)/full-checkout\" &&\n+\tensure_not_expanded fetch full &&\n+\tgit -C full-checkout commit --allow-empty -m \"for pull merge\" &&\n+\tgit -C sparse-index commit --allow-empty -m \"for pull merge\" &&\n+\tensure_not_expanded pull full base\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"443810","messageId":"b81174ba54b94dac6bf82e783be12cdb325d644a.1639149192.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v3 2/5] ls-files: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:09Z","receivedAt":"2021-12-10T15:13:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExisting callers to 'git ls-files' are expecting file names, not\ndirectories. It is best to expand a sparse index to show all of the\ncontained files in this case.\n\nHowever, expert users may want to inspect the contents of the index\nitself including which directories are sparse. Add a --sparse option to\nallow users to request this information.\n\nDuring testing, I noticed that options such as --modified did not affect\nthe output when the files in question were outside the sparse-checkout\ndefinition. Tests are added to document this preexisting behavior and\nhow it remains unchanged with the sparse index and the --sparse option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ls-files.txt | 4 +\n builtin/ls-files.c | 12 ++-\n t/t1092-sparse-checkout-compatibility.sh | 96 ++++++++++++++++++++++++\n 3 files changed, 110 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex 6d11ab506b7..1c5d5f85ec5 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -187,6 +187,10 @@ Both the in the index (\"i/\")\n and in the working tree (\"w/\") are shown for regular files,\n followed by the (\"attr/\").\n \n+--sparse::\n+\tIf the index is sparse, show the sparse directories without expanding\n+\tto the contained files.\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 031fef1bcaa..c151eb1fb77 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -37,6 +37,7 @@ static int debug_mode;\n static int show_eol;\n static int recurse_submodules;\n static int skipping_duplicates;\n+static int show_sparse_dirs;\n \n static const char *prefix;\n static int max_prefix_len;\n@@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\n+\tif (!show_sparse_dirs)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tOPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n \t\tOPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n \t\t\t N_(\"suppress duplicate entries\")),\n+\t\tOPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n+\t\t\t N_(\"show sparse directories in the presence of a sparse index\")),\n \t\tOPT_END()\n \t};\n \tint ret = 0;\n@@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(ls_files_usage, builtin_ls_files_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tprefix = cmd_prefix;\n \tif (prefix)\n \t\tprefix_len = strlen(prefix);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 3ba19ba1c14..29b97667378 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -802,6 +802,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n \t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt &&\n+\n+\t# ls-files expands on read, but does not write.\n+\trm trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index ls-files &&\n \ttest_region index ensure_full_index trace2.txt\n '\n \n@@ -826,6 +832,7 @@ test_expect_success 'sparse-index is not expanded' '\n \tinit_repos &&\n \n \tensure_not_expanded status &&\n+\tensure_not_expanded ls-files --sparse &&\n \tensure_not_expanded commit --allow-empty -m empty &&\n \techo >>sparse-index/a &&\n \tensure_not_expanded commit -a -m a &&\n@@ -974,6 +981,95 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'ls-files' '\n+\tinit_repos &&\n+\n+\t# Behavior agrees by default. Sparse index is expanded.\n+\ttest_all_match git ls-files &&\n+\n+\t# With --sparse, the sparse index data changes behavior.\n+\tgit -C sparse-index ls-files >dense &&\n+\tgit -C sparse-index ls-files --sparse >sparse &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t@@ -13,13 +13,9 @@\n+\t e\n+\t folder1-\n+\t folder1.x\n+\t-folder1/0/0/0\n+\t-folder1/0/1\n+\t-folder1/a\n+\t+folder1/\n+\t folder10\n+\t-folder2/0/0/0\n+\t-folder2/0/1\n+\t-folder2/a\n+\t+folder2/\n+\t g\n+\t-x/a\n+\t+x/\n+\t z\n+\tEOF\n+\n+\tdiff -u dense sparse | tail -n +3 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# With --sparse and no sparse index, nothing changes.\n+\tgit -C sparse-checkout ls-files >dense &&\n+\tgit -C sparse-checkout ls-files --sparse >sparse &&\n+\ttest_cmp dense sparse &&\n+\n+\t# Set up a strange condition of having a file edit\n+\t# outside of the sparse-checkout cone. This is just\n+\t# to verify that sparse-checkout and sparse-index\n+\t# behave the same in this case.\n+\twrite_script edit-content <<-\\EOF &&\n+\tmkdir folder1 &&\n+\techo content >>folder1/a\n+\tEOF\n+\trun_on_sparse ../edit-content &&\n+\n+\t# ls-files does not currently notice modified files whose\n+\t# cache entries are marked SKIP_WORKTREE. This may change\n+\t# in the future, but here we test that sparse index does\n+\t# not accidentally create a change of behavior.\n+\ttest_sparse_match git ls-files --modified &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\t# Add folder1 to the sparse-checkout cone and\n+\t# check that ls-files shows the expanded files.\n+\ttest_sparse_match git sparse-checkout add folder1 &&\n+\ttest_sparse_match git ls-files --modified &&\n+\n+\tgit -C sparse-index ls-files >dense &&\n+\tgit -C sparse-index ls-files --sparse >sparse &&\n+\n+\tcat >expect <<-\\EOF &&\n+\t@@ -17,9 +17,7 @@\n+\t folder1/0/1\n+\t folder1/a\n+\t folder10\n+\t-folder2/0/0/0\n+\t-folder2/0/1\n+\t-folder2/a\n+\t+folder2/\n+\t g\n+\t-x/a\n+\t+x/\n+\t z\n+\tEOF\n+\n+\tdiff -u dense sparse | tail -n +3 >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Double-check index expansion is avoided\n+\tensure_not_expanded ls-files --sparse\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"443811","messageId":"2a6a1c5a39c9f07f4714d0d9916c3e1775836372.1639149192.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v3 3/5] t1092: replace 'read-cache --table' with 'ls-files --sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:10Z","receivedAt":"2021-12-10T15:13:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1092-sparse-checkout-compatibility.sh.\n\nThe important changes are due to the different output format. We need to\nuse the '--stage' output to get a file mode and OID, but it also\nincludes a stage value and drops the object type. This leads to some\ndifferences in how we handle looking for specific entries.\n\nSome places where we previously looked for no 'tree' entries, we can\ninstead directly compare the output across the repository with a sparse\nindex and the one without.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 31 +++++++++++-------------\n 1 file changed, 14 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 29b97667378..d4a0d8ce825 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -200,45 +200,42 @@ test_sparse_unstaged () {\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set folder1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set deep/deeper1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep/deeper2 folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n-\t# Disabling the sparse-index removes tree entries with full ones\n+\t# Disabling the sparse-index replaces tree entries with full ones\n \tgit -C sparse-index sparse-checkout init --no-sparse-index &&\n-\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\t! grep \"040000 tree\" cache &&\n-\ttest_sparse_match test-tool read-cache --table\n+\ttest_sparse_match git ls-files --stage --sparse\n '\n \n test_expect_success 'expanded in-memory index matches full index' '\n \tinit_repos &&\n-\ttest_sparse_match test-tool read-cache --expand --table\n+\ttest_sparse_match git ls-files --stage\n '\n \n test_expect_success 'status with options' '\n@@ -787,9 +784,9 @@ test_expect_success 'submodule handling' '\n \n \t# having a submodule prevents \"modules\" from collapse\n \ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\tgrep \"100644 blob .*\tmodules/a\" cache &&\n-\tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n+\tgrep \"100644 .*\tmodules/a\" cache &&\n+\tgrep \"160000 $(git -C initial-repo rev-parse HEAD) 0\tmodules/sub\" cache\n '\n \n # When working with a sparse index, some commands will need to expand the\n@@ -1085,13 +1082,13 @@ test_expect_success 'reset mixed and checkout orphan' '\n \t# the sparse checkouts skip \"adding\" the other side of\n \t# the conflict.\n \ttest_sparse_match git reset --mixed HEAD~1 &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2 &&\n \n \t# At this point, sparse-checkouts behave differently\n \t# from the full-checkout.\n \ttest_sparse_match git checkout --orphan new-branch &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2\n '\n \n-- \ngitgitgadget\n\n"},{"id":"443812","messageId":"f014368675495812dd45291a36902cecdfa00c63.1639149192.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v3 4/5] t1091/t3705: remove 'test-tool read-cache --table'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:11Z","receivedAt":"2021-12-10T15:13:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1091-sparse-checkout-builtin.sh and\nt3705-add-sparse-checkout.sh.\n\nThe important changes are due to the different output format. In t3705,\nwe need to use the '--stage' output to get a file mode and OID, but\nit also includes a stage value and drops the object type. This leads\nto some differences in how we handle looking for specific entries.\n\nIn t1091, the test focuses on enabling the sparse index, so we do not\nneed the --stage flag to demonstrate how the index changes, and instead\ncan use a diff.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 25 ++++++++++++++++++++-----\n t/t3705-add-sparse-checkout.sh | 8 ++++----\n 2 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 272ba1b566b..680e0043c36 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -212,12 +212,27 @@ test_expect_success 'sparse-index enabled and disabled' '\n \n \t\tgit -C repo sparse-checkout init --cone --sparse-index &&\n \t\ttest_cmp_config -C repo true index.sparse &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\tgrep \" tree \" cache &&\n-\n+\t\tgit -C repo ls-files --sparse >sparse &&\n \t\tgit -C repo sparse-checkout disable &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\t! grep \" tree \" cache &&\n+\t\tgit -C repo ls-files --sparse >full &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\t@@ -1,4 +1,7 @@\n+\t\t a\n+\t\t-deep/\n+\t\t-folder1/\n+\t\t-folder2/\n+\t\t+deep/a\n+\t\t+deep/deeper1/a\n+\t\t+deep/deeper1/deepest/a\n+\t\t+deep/deeper2/a\n+\t\t+folder1/a\n+\t\t+folder2/a\n+\t\tEOF\n+\n+\t\tdiff -u sparse full | tail -n +3 >actual &&\n+\t\ttest_cmp expect actual &&\n+\n \t\tgit -C repo config --list >config &&\n \t\t! grep index.sparse config\n \t)\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex f3143c92908..81f3384eeed 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -181,13 +181,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100644 .*sparse_entry\\$\" actual &&\n \n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100755 .*sparse_entry\\$\" actual &&\n \n \tgit reset &&\n \n-- \ngitgitgadget\n\n"},{"id":"443813","messageId":"9227dc54165e4d452d886905996e7164b3c5e525.1639149192.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v3 5/5] test-read-cache: remove --table, --expand options","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-10T15:13:12Z","receivedAt":"2021-12-10T15:13:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis commit effectively reverts 2782db3 (test-tool: don't force full\nindex, 2021-03-30) and e2df6c3 (test-read-cache: print cache entries\nwith --table, 2021-03-30) to remove the --table and --expand options\nfrom 'test-tool read-cache'. The previous changes already removed these\noptions from the test suite in favor of 'git ls-files --sparse'.\n\nThe initial thought of creating these options was to allow for tests to\nsee additional information with every cache entry. In particular, the\nobject type is still not mirrored in 'git ls-files'. Since sparse\ndirectory entries always end with a slash, the object type is not\ncritical to verify the sparse index is enabled. It was thought that it\nwould be helpful to have additional information, such as flags, but that\nwas not needed for the FS Monitor integration and hasn't been needed\nsince.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-cache.c | 64 ++++++--------------------------------\n 1 file changed, 10 insertions(+), 54 deletions(-)\n\ndiff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c\nindex 0d9f08931a1..b736ef16421 100644\n--- a/t/helper/test-read-cache.c\n+++ b/t/helper/test-read-cache.c\n@@ -1,83 +1,39 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n #include \"config.h\"\n-#include \"blob.h\"\n-#include \"commit.h\"\n-#include \"tree.h\"\n-#include \"sparse-index.h\"\n-\n-static void print_cache_entry(struct cache_entry *ce)\n-{\n-\tconst char *type;\n-\tprintf(\"%06o \", ce->ce_mode & 0177777);\n-\n-\tif (S_ISSPARSEDIR(ce->ce_mode))\n-\t\ttype = tree_type;\n-\telse if (S_ISGITLINK(ce->ce_mode))\n-\t\ttype = commit_type;\n-\telse\n-\t\ttype = blob_type;\n-\n-\tprintf(\"%s %s\\t%s\\n\",\n-\t type,\n-\t oid_to_hex(&ce->oid),\n-\t ce->name);\n-}\n-\n-static void print_cache(struct index_state *istate)\n-{\n-\tint i;\n-\tfor (i = 0; i < istate->cache_nr; i++)\n-\t\tprint_cache_entry(istate->cache[i]);\n-}\n \n int cmd__read_cache(int argc, const char **argv)\n {\n-\tstruct repository *r = the_repository;\n \tint i, cnt = 1;\n \tconst char *name = NULL;\n-\tint table = 0, expand = 0;\n \n \tinitialize_the_repository();\n \n-\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n-\t\tif (skip_prefix(*argv, \"--print-and-refresh=\", &name))\n-\t\t\tcontinue;\n-\t\tif (!strcmp(*argv, \"--table\"))\n-\t\t\ttable = 1;\n-\t\telse if (!strcmp(*argv, \"--expand\"))\n-\t\t\texpand = 1;\n+\tif (argc > 1 && skip_prefix(argv[1], \"--print-and-refresh=\", &name)) {\n+\t\targc--;\n+\t\targv++;\n \t}\n \n-\tif (argc == 1)\n-\t\tcnt = strtol(argv[0], NULL, 0);\n+\tif (argc == 2)\n+\t\tcnt = strtol(argv[1], NULL, 0);\n \tsetup_git_directory();\n \tgit_config(git_default_config, NULL);\n \n-\tprepare_repo_settings(r);\n-\tr->settings.command_requires_full_index = 0;\n-\n \tfor (i = 0; i < cnt; i++) {\n-\t\trepo_read_index(r);\n-\n-\t\tif (expand)\n-\t\t\tensure_full_index(r->index);\n-\n+\t\tread_cache();\n \t\tif (name) {\n \t\t\tint pos;\n \n-\t\t\trefresh_index(r->index, REFRESH_QUIET,\n+\t\t\trefresh_index(&the_index, REFRESH_QUIET,\n \t\t\t\t NULL, NULL, NULL);\n-\t\t\tpos = index_name_pos(r->index, name, strlen(name));\n+\t\t\tpos = index_name_pos(&the_index, name, strlen(name));\n \t\t\tif (pos < 0)\n \t\t\t\tdie(\"%s not in index\", name);\n \t\t\tprintf(\"%s is%s up to date\\n\", name,\n-\t\t\t ce_uptodate(r->index->cache[pos]) ? \"\" : \" not\");\n+\t\t\t ce_uptodate(the_index.cache[pos]) ? \"\" : \" not\");\n \t\t\twrite_file(name, \"%d\\n\", i);\n \t\t}\n-\t\tif (table)\n-\t\t\tprint_cache(r->index);\n-\t\tdiscard_index(r->index);\n+\t\tdiscard_cache();\n \t}\n \treturn 0;\n }\n-- \ngitgitgadget\n"},{"id":"443818","messageId":"211210.86zgp8bi48.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"745e79e3-521d-58bd-2ddf-1da4e0f6fe26@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-10T15:13:36Z","receivedAt":"2021-12-10T15:22:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 10 2021, Derrick Stolee wrote:\n\n> On 12/9/2021 7:50 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Dec 08 2021, Derrick Stolee wrote:\n>> \n>>> On 12/8/2021 1:36 PM, Elijah Newren wrote:\n>>>> On Wed, Dec 8, 2021 at 10:23 AM Derrick Stolee wrote:\n>>>>>\n>>>>> On 12/8/2021 12:04 PM, Elijah Newren wrote:\n>>>>>>\n>>>>>> This actually looks quite nice, though the magic '16' is kind of\n>>>>>> annoying. Could we get rid of that -- perhaps using something to rip\n>>>>>> out the diff header, or using comm instead?\n>>>>>\n>>>>> What I really want is \"remove the first two lines of this file\"\n>>>>\n>>>> Is `tail -n +3` portable? Looks like we have five uses of tail -n +N\n>>>> in the testsuite, so it should be okay to use.\n>>>\n>>> Ah, that's the magic incantation. Sounds good.\n>>>\n>>>>> but perhaps \"tail -n $(wc -l expect)\" would suffice to avoid a\n>>>>> magic number?\n>>>>\n>>>> That works too.\n>>>\n>>> If the \"-n +X\" syntax works, then I'll opt for that.\n>> \n>> I think it should be per\n>> https://pubs.opengroup.org/onlinepubs/009695299/utilities/tail.html\n>> \n>> But isn't that \"diff -u\" non-portable, per GIT_TEST_CMP. I.e. sometimes\n>> we'll fall back on \"cmp\" etc.\n>\n> You're talking about this hunk, right?\n>\n> if test -z \"$GIT_TEST_CMP\"\n> then\n> \tif test -n \"$GIT_TEST_CMP_USE_COPIED_CONTEXT\"\n> \tthen\n> \t\tGIT_TEST_CMP=\"$DIFF -c\"\n> \telse\n> \t\tGIT_TEST_CMP=\"$DIFF -u\"\n> \tfi\n> fi\n>\n> This only switches from \"diff -u\" to \"diff -c\" if the\n> GIT_TEST_CMP_USE_COPIED_CONTEXT variable is set, but it is not set\n> by default. Thus, we are using \"diff -u\" by default throughout.\n>\n> Please let me know if I'm misreading this.\n\nYes and no. Yes it's not on by default, but the reason (well, maybe not\n*the* reason, but definitely one reason) we have GIT_TEST_CMP is because\n-u isn't portable.\n\nIt's not in POSIX, and not just in the theoretical sense, e.g. when I\nrun tests on HP/UX it'll yell at me about not knowing about -u until I\nadjust GIT_TEST_CMP.\n\nThus you can't use \"diff -u\" as you did upthread without breaking the\ntest on those platforms, and you can't execute \"$GIT_TEST_CMP\" either\nbecause it might use -c, and then your \"skip the first three lines\"\nmight not hold.\n\nSo even if we avoid using \"git\" for the actual test helpers themselves\nusually using it in this case is probably fine, and a neat way out of\nthis particular hassle without worrying about portability.\n\nSo I don't know if we want a diff here, but if so I think it would be\nneat to wrap that in say a \"test_cmp_diff\" which we'd just use as:\n\n\ttest_cmp_diff a b <<-\\EOF\n \n\tEOF\n\nWhere under the hood we'd just run:\n\n\tgit diff --no-index a b >actual\n\tcat >expect &&\n\t[ munge diff header here? ] &&\n\ttest_cmp expect actual\n"},{"id":"443823","messageId":"211210.86v8zwbev9.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-10T16:16:35Z","receivedAt":"2021-12-10T16:32:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 10 2021, Derrick Stolee via GitGitGadget wrote:\n\n> Updates in v3\n> =============\n>\n> * Fixed typo in commit message.\n> * Added comments around doing strange things in an ls-files test.\n> * Fixed adjacent typo in a test comment.\n\nYay, I'm happy to see 5/5. Not because I didn't like the helper, but\nthat sparse is getting mature enough that we're getting ls-files to emit\ninformation about it. Thanks.\n\nThere's the small \"diff -u\" portability issue noted in my just-sent\n<211210.86zgp8bi48.gmgdl@evledraar.gmail.com>.\n\nOther than that 2/5 adds this documentation about ls-files --sparse:\n\n\tIf the index is sparse, show the sparse directories without expanding\n\tto the contained files.\n\nShouldn't we at least add:\n\n\tSparse directories will be shown with a trailing slash,\n\te.g. \"x/\" for a sparse directory \"x\".q\n\nIn addition to that I think this may have a buggy/unexpected interaction\nwith the --eol option:\n\n 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 i/ w/ attr/ x/\n\nI.e. should we be saying anything about the EOL state of these? OTOHO I\ntried adding a submodule and it says the same, which seems similarly\nodd, so maybe it's either correct, or this isn't updated for those\neither.\n\nIs the behavior of:\n\n $ git -C sparse-index ls-files --stage --sparse -- 'folder2/a'\n $ echo $?\n 0\n\nExpected? I.e. accepting /a when we'd just print \"folder2/\" and not\ne.g. erroring (probably, just asking)?\n\nHow about:\n\n $ ls -l sparse-index/x\n ls: cannot access 'sparse-index/x': No such file or directory\n $ git -C sparse-index ls-files --stage 'x/*'\n 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n $ git -C sparse-index ls-files --stage --no-empty-directory 'x/*' \n 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n $ git -C sparse-index ls-files --stage --no-empty-directory --sparse 'x/*' \n 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n\nThe answer is probably \"yes that's fine\" because I've got no idea how\nsparse really works, but just checking..\n\nSo it's very nice to have the new diff test in 2/5, but would be much\nnicer/assuring to have that split into a trivial function followed by\nseeing how the diff looked in combination with each of the other option\nthat \"ls-files\" accepts.\n"},{"id":"443829","messageId":"CABPp-BHLRUQmjEoO_eEvZWTrVFK_xdMoQH5Q9DAj7y=JrKDzig@mail.gmail.com","threadId":"56917","inReplyTo":"211210.86v8zwbev9.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-10T18:45:10Z","receivedAt":"2021-12-10T18:45:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Dec 10, 2021 at 8:31 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Fri, Dec 10 2021, Derrick Stolee via GitGitGadget wrote:\n>\n> > Updates in v3\n> > =============\n> >\n> > * Fixed typo in commit message.\n> > * Added comments around doing strange things in an ls-files test.\n> > * Fixed adjacent typo in a test comment.\n>\n> Yay, I'm happy to see 5/5. Not because I didn't like the helper, but\n> that sparse is getting mature enough that we're getting ls-files to emit\n> information about it. Thanks.\n>\n> There's the small \"diff -u\" portability issue noted in my just-sent\n> <211210.86zgp8bi48.gmgdl@evledraar.gmail.com>.\n\nYeah, that one is an important point.\n\n> Other than that 2/5 adds this documentation about ls-files --sparse:\n>\n> If the index is sparse, show the sparse directories without expanding\n> to the contained files.\n>\n> Shouldn't we at least add:\n>\n> Sparse directories will be shown with a trailing slash,\n> e.g. \"x/\" for a sparse directory \"x\".q\n\nMakes sense. Except I don't understand the trailing 'q' -- typo?\n\n>\n> In addition to that I think this may have a buggy/unexpected interaction\n> with the --eol option:\n>\n> 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 i/ w/ attr/ x/\n>\n> I.e. should we be saying anything about the EOL state of these? OTOHO I\n> tried adding a submodule and it says the same, which seems similarly\n> odd, so maybe it's either correct, or this isn't updated for those\n> either.\n\nIf it matches what we do for submodules, for which eol values are also\nnon-sensical, then I think we're good enough for this series. Perhaps\nwe just shouldn't print anything eol related for directories with\n--eol, but that sounds like an orthogonal series rather than something\nthat should go in this one.\n\n> Is the behavior of:\n>\n> $ git -C sparse-index ls-files --stage --sparse -- 'folder2/a'\n> $ echo $?\n> 0\n>\n> Expected? I.e. accepting /a when we'd just print \"folder2/\" and not\n> e.g. erroring (probably, just asking)?\n\nFair question. I think it's fine; by way of comparison:\n\n$ git rm --cached removed-and-no-longer-tracked-file\n$ git ls-files --stage -- non-existent-file\nremoved-and-no-longer-tracked-file untracked-file\n$ echo $?\n0\n\nSo it also shows nothing and displays nothing when asked for file(s)\nthat are not in the index.\n\nYes, there is a slight semantic difference in that in your example we\nhave a \"folder2/\" entry which *could be* expanded, but I am quite\nhappy with the literal interpretation of the command that there is no\n\"folder2/a\" in the index. Said another way, I'm happy with ls-files\nshowing what is in the index right now, rather than what could be in\nit, or listing things that HEAD contains that we don't for whatever\nreason.\n\n> How about:\n>\n> $ ls -l sparse-index/x\n> ls: cannot access 'sparse-index/x': No such file or directory\n> $ git -C sparse-index ls-files --stage 'x/*'\n> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n> $ git -C sparse-index ls-files --stage --no-empty-directory 'x/*'\n> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n> $ git -C sparse-index ls-files --stage --no-empty-directory --sparse 'x/*'\n> 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n>\n> The answer is probably \"yes that's fine\" because I've got no idea how\n> sparse really works, but just checking..\n\nYou should read the docs for this option you are trying: \"Do not list\nempty directories. Has no effect without --directory.\" (Also,\n--directory only takes effect with --other, which you are also\nmissing.)\n\nSo yeah, that flag is irrelevant. Perhaps ls-files should print a\nwarning when flags are passed but ignored due to other flags not being\npassed, but that would belong in an orthogonal series rather than this\none.\n\n> So it's very nice to have the new diff test in 2/5, but would be much\n> nicer/assuring to have that split into a trivial function followed by\n> seeing how the diff looked in combination with each of the other option\n> that \"ls-files\" accepts.\n\nThere's no point testing in combination with flags that only affect\nuntracked files. And I'm very dubious of adding testing for a case\nwhere we would need to add an explicit disclaimer that \"We have no\nidea what the output should be but we are testing it anyway\". So the\noptions you suggest at least are things I'd rather not see us trying\nto add to the testing here.\n"},{"id":"443830","messageId":"CABPp-BEs5gnX+HfPNMRBqtMdhVfkk5oKDT8sN4fJiPG87JDjcw@mail.gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-10T18:53:55Z","receivedAt":"2021-12-10T18:54:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Dec 10, 2021 at 7:13 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is based on ld/sparse-index-blame (merged with 'master' due to an\n> unrelated build issue).\n>\n> Here are two relatively-simple patches that further the sparse index\n> integrations.\n>\n> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n> have been an integration much earlier in the cycle. They read the index to\n> look for the .gitmodules file in case there are submodules that need to be\n> fetched. Since looking for a file by name is already protected, we only need\n> to disable 'command_requires_full_index' and we are done.\n>\n> The 'ls-files' builtin is useful when debugging the index, and some scripts\n> use it, too. We are not changing the default behavior which expands a sparse\n> index in order to show all of the cached blobs. Instead, we add a '--sparse'\n> option that allows us to see the sparse directory entries upon request.\n> Combined with --debug, we can see a lot of index details, such as:\n>\n> $ git ls-files --debug --sparse\n> LICENSE\n> ctime: 1634910503:287405820\n> mtime: 1634910503:287405820\n> dev: 16777220 ino: 119325319\n> uid: 501 gid: 20\n> size: 1098 flags: 200000\n> README.md\n> ctime: 1634910503:288090279\n> mtime: 1634910503:288090279\n> dev: 16777220 ino: 119325320\n> uid: 501 gid: 20\n> size: 934 flags: 200000\n> bin/index.js\n> ctime: 1634910767:828434033\n> mtime: 1634910767:828434033\n> dev: 16777220 ino: 119325520\n> uid: 501 gid: 20\n> size: 7292 flags: 200000\n> examples/\n> ctime: 0:0\n> mtime: 0:0\n> dev: 0 ino: 0\n> uid: 0 gid: 0\n> size: 0 flags: 40004000\n> package.json\n> ctime: 1634910503:288676330\n> mtime: 1634910503:288676330\n> dev: 16777220 ino: 119325321\n> uid: 501 gid: 20\n> size: 680 flags: 200000\n>\n>\n> (In this example, the 'examples/' directory is sparse.)\n>\n> Thanks!\n>\n>\n> Updates in v2\n> =============\n>\n> * Rebased onto latest ld/sparse-index-blame without issue.\n> * Updated the test to use diff-of-diffs instead of a sequence of greps.\n> * Added patches that remove the use of 'test-tool read-cache --table' and\n> its implementation.\n>\n>\n> Updates in v3\n> =============\n>\n> * Fixed typo in commit message.\n> * Added comments around doing strange things in an ls-files test.\n> * Fixed adjacent typo in a test comment.\n\nThanks, this round addresses all my previous feedback. However, there\nare two things Ævar has brought up that I think are important:\n * cannot rely on `diff -u` for portability reasons[1] (his\nsuggestion of git diff --no-index sounds good, or you can use comm(1))\n * have documentation mention the trailing slash that sparse\ndirectory entries are mentioned with[2]\n\n[1] https://lore.kernel.org/git/211210.86zgp8bi48.gmgdl@evledraar.gmail.com/\n[2] https://lore.kernel.org/git/211210.86v8zwbev9.gmgdl@evledraar.gmail.com/\n"},{"id":"443859","messageId":"211211.86ee6jc1t2.gmgdl@evledraar.gmail.com","threadId":"56917","inReplyTo":"CABPp-BHLRUQmjEoO_eEvZWTrVFK_xdMoQH5Q9DAj7y=JrKDzig@mail.gmail.com","subject":"Re: [PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-12-11T02:24:57Z","receivedAt":"2021-12-11T02:28:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 10 2021, Elijah Newren wrote:\n\n> On Fri, Dec 10, 2021 at 8:31 AM Ævar Arnfjörð Bjarmason\n> wrote:\n> [...]\n>> Other than that 2/5 adds this documentation about ls-files --sparse:\n>>\n>> If the index is sparse, show the sparse directories without expanding\n>> to the contained files.\n>>\n>> Shouldn't we at least add:\n>>\n>> Sparse directories will be shown with a trailing slash,\n>> e.g. \"x/\" for a sparse directory \"x\".q\n>\n> Makes sense. Except I don't understand the trailing 'q' -- typo?\n\nYes, sorry.\n\n>>\n>> In addition to that I think this may have a buggy/unexpected interaction\n>> with the --eol option:\n>>\n>> 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 i/ w/ attr/ x/\n>>\n>> I.e. should we be saying anything about the EOL state of these? OTOHO I\n>> tried adding a submodule and it says the same, which seems similarly\n>> odd, so maybe it's either correct, or this isn't updated for those\n>> either.\n>\n> If it matches what we do for submodules, for which eol values are also\n> non-sensical, then I think we're good enough for this series. Perhaps\n> we just shouldn't print anything eol related for directories with\n> --eol, but that sounds like an orthogonal series rather than something\n> that should go in this one.\n\n*nod*, probably. \n\n>> Is the behavior of:\n>>\n>> $ git -C sparse-index ls-files --stage --sparse -- 'folder2/a'\n>> $ echo $?\n>> 0\n>>\n>> Expected? I.e. accepting /a when we'd just print \"folder2/\" and not\n>> e.g. erroring (probably, just asking)?\n>\n> Fair question. I think it's fine; by way of comparison:\n>\n> $ git rm --cached removed-and-no-longer-tracked-file\n> $ git ls-files --stage -- non-existent-file\n> removed-and-no-longer-tracked-file untracked-file\n> $ echo $?\n> 0\n>\n> So it also shows nothing and displays nothing when asked for file(s)\n> that are not in the index.\n>\n> Yes, there is a slight semantic difference in that in your example we\n> have a \"folder2/\" entry which *could be* expanded, but I am quite\n> happy with the literal interpretation of the command that there is no\n> \"folder2/a\" in the index. Said another way, I'm happy with ls-files\n> showing what is in the index right now, rather than what could be in\n> it, or listing things that HEAD contains that we don't for whatever\n> reason.\n\nSounds good.\n\n>> How about:\n>>\n>> $ ls -l sparse-index/x\n>> ls: cannot access 'sparse-index/x': No such file or directory\n>> $ git -C sparse-index ls-files --stage 'x/*'\n>> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n>> $ git -C sparse-index ls-files --stage --no-empty-directory 'x/*'\n>> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n>> $ git -C sparse-index ls-files --stage --no-empty-directory --sparse 'x/*'\n>> 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n>>\n>> The answer is probably \"yes that's fine\" because I've got no idea how\n>> sparse really works, but just checking..\n>\n> You should read the docs for this option you are trying: \"Do not list\n> empty directories. Has no effect without --directory.\" (Also,\n> --directory only takes effect with --other, which you are also\n> missing.)\n>\n> So yeah, that flag is irrelevant. Perhaps ls-files should print a\n> warning when flags are passed but ignored due to other flags not being\n> passed, but that would belong in an orthogonal series rather than this\n> one.\n\n...\n\n>> So it's very nice to have the new diff test in 2/5, but would be much\n>> nicer/assuring to have that split into a trivial function followed by\n>> seeing how the diff looked in combination with each of the other option\n>> that \"ls-files\" accepts.\n>\n> There's no point testing in combination with flags that only affect\n> untracked files. And I'm very dubious of adding testing for a case\n> where we would need to add an explicit disclaimer that \"We have no\n> idea what the output should be but we are testing it anyway\". So the\n> options you suggest at least are things I'd rather not see us trying\n> to add to the testing here.\n\nThis series is adding a new flag to ls-files, it doesn't error out when\ncombined with other existing flags, and observably changes their output.\n\nI think erroring out would be fine, or doing whatever it's doing now,\nbut either way the gap in test coverage should be closed, shouldn't it?\n\nI'd think the easiest and probably most prudent fix would just be to say\nthat we don't think some of these make sense with --sparse and have them\nerror out if they're combined, no?\n"},{"id":"443864","messageId":"CABPp-BFEayJvGytbsi76Nqe=tWj6zX5Aa+ErAxG=akC-yp1JFw@mail.gmail.com","threadId":"56917","inReplyTo":"211211.86ee6jc1t2.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 0/5] Sparse index: fetch, pull, ls-files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-11T04:45:42Z","receivedAt":"2021-12-11T04:45:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Dec 10, 2021 at 6:28 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Fri, Dec 10 2021, Elijah Newren wrote:\n>\n> > On Fri, Dec 10, 2021 at 8:31 AM Ævar Arnfjörð Bjarmason\n> > wrote:\n> > [...]\n> >> So it's very nice to have the new diff test in 2/5, but would be much\n> >> nicer/assuring to have that split into a trivial function followed by\n> >> seeing how the diff looked in combination with each of the other option\n> >> that \"ls-files\" accepts.\n> >\n> > There's no point testing in combination with flags that only affect\n> > untracked files. And I'm very dubious of adding testing for a case\n> > where we would need to add an explicit disclaimer that \"We have no\n> > idea what the output should be but we are testing it anyway\". So the\n> > options you suggest at least are things I'd rather not see us trying\n> > to add to the testing here.\n>\n> This series is adding a new flag to ls-files, it doesn't error out when\n> combined with other existing flags, and observably changes their output.\n\nAh, I think you had a misunderstanding here. If what you say here\nwere true, then indeed we would need some testing and it'd suggest\nsome kind of bug. But the combination here does not observably change\nthe output. You were missing an important testcase for comparison.\nLet me repeat your testing and sprinkle in some commentary:\n\n> >> $ ls -l sparse-index/x\n> >> ls: cannot access 'sparse-index/x': No such file or directory\n\nRight, this is a sparse directory; good to double check.\n\n> >> $ git -C sparse-index ls-files --stage 'x/*'\n> >> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n> >> $ git -C sparse-index ls-files --stage --no-empty-directory 'x/*'\n> >> 100644 78981922613b2afb6025042ff6bd878ac1994e85 0 x/a\n\nRight, --no-empty-directory by itself is a useless option that won't\naffect the output of ls-files; it only takes affect with --directory,\nwhich in turn only takes affect with other options. Since that\noptions is useless, the output is the same for both of these.\n\n> >> $ git -C sparse-index ls-files --stage --no-empty-directory --sparse 'x/*'\n> >> 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n\nHere you added --sparse, but you neglected what it would show without\n--no-empty-directory, which is a critical comparison point. So let me\nfill it in:\n\n $ git -C sparse-index ls-files --stage --sparse 'x/*'\n 040000 aaff74984cccd156a469afa7d9ab10e4777beb24 0 x/\n\nNow, this case I added in comparison to the one three above it shows\nthat, yes, --sparse does indeed change the output relative to --stage.\nAnd it does so by design. Now if you compare my added case to the\nlast one you showed, you can verify that adding --no-empty-directory\nto that mix does not change the output further; --no-empty-directory\nis a useless/ignored option unless you also include other flags that\nwere not involved here.\n\n> I think erroring out would be fine, or doing whatever it's doing now,\n> but either way the gap in test coverage should be closed, shouldn't it?\n>\n> I'd think the easiest and probably most prudent fix would just be to say\n> that we don't think some of these make sense with --sparse and have them\n> error out if they're combined, no?\n\nls-files offers several options that allow you to either slice and\ndice or tweak the output, and function on two kinds of files: tracked,\nand not tracked. Several examples of such flags:\n * tracked: --cached, --stage, --unmerged, --modified, --sparse (and\nI think --error-unmatch)\n * not tracked: --others, --ignored, --exclude, --exclude-from,\n--exclude-standard, --directory, --no-empty-directory\n\nNow, in particular, specifying any of --exclude, --exclude-from,\n--exclude-standard, --directory, or --no-empty-directory is a complete\nwaste of breath and will do nothing unless you also specify --others\nor --ignored. None of these options interact in any way with any of\nthe flags from the --tracked category.\n\nI don't think we want an n! permutation of all combinations tested. I\ndon't even think an n^2 pair-wise combination makes sense when we know\nthat some flags have no effect on their own. What would make sense is\nperhaps adding a warning to ls-files when specified flags will have no\nutility due to depending on other flags that have not been specified.\nBut that's in no way specific to --sparse and does not make sense to\nme to make part of this topic.\n"},{"id":"444019","messageId":"xmqq7dc8co3d.fsf@gitster.g","threadId":"56917","inReplyTo":"745e79e3-521d-58bd-2ddf-1da4e0f6fe26@gmail.com","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-13T19:16:22Z","receivedAt":"2021-12-13T19:16:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> You're talking about this hunk, right?\n>\n> if test -z \"$GIT_TEST_CMP\"\n> then\n> \tif test -n \"$GIT_TEST_CMP_USE_COPIED_CONTEXT\"\n> \tthen\n> \t\tGIT_TEST_CMP=\"$DIFF -c\"\n> \telse\n> \t\tGIT_TEST_CMP=\"$DIFF -u\"\n> \tfi\n> fi\n>\n> This only switches from \"diff -u\" to \"diff -c\" if the\n> GIT_TEST_CMP_USE_COPIED_CONTEXT variable is set, but it is not set\n> by default. Thus, we are using \"diff -u\" by default throughout.\n\nThat it can be set merely means that somebody needed to work around\nthe lack of \"-u\" format in their implementation of \"diff\". It came\nto POSIX only at the Issue 7 of the standard (cf. [*1*]).\n\nUnconditional use of \"diff -u\" is a breaking change for them.\n\nIf these people still exist, that is ;-)\n\n\n[Reference]\n\n*1* https://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html\nThe Open Group Base Specifications Issue 7, 2018 edition\n"},{"id":"444225","messageId":"3910553a-019c-24bf-a058-ef47dfe28393@gmail.com","threadId":"56917","inReplyTo":"xmqq7dc8co3d.fsf@gitster.g","subject":"Re: [PATCH 2/2] ls-files: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-12-16T14:11:49Z","receivedAt":"2021-12-16T14:11:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/13/2021 2:16 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> You're talking about this hunk, right?\n>>\n>> if test -z \"$GIT_TEST_CMP\"\n>> then\n>> \tif test -n \"$GIT_TEST_CMP_USE_COPIED_CONTEXT\"\n>> \tthen\n>> \t\tGIT_TEST_CMP=\"$DIFF -c\"\n>> \telse\n>> \t\tGIT_TEST_CMP=\"$DIFF -u\"\n>> \tfi\n>> fi\n>>\n>> This only switches from \"diff -u\" to \"diff -c\" if the\n>> GIT_TEST_CMP_USE_COPIED_CONTEXT variable is set, but it is not set\n>> by default. Thus, we are using \"diff -u\" by default throughout.\n> \n> That it can be set merely means that somebody needed to work around\n> the lack of \"-u\" format in their implementation of \"diff\". It came\n> to POSIX only at the Issue 7 of the standard (cf. [*1*]).\n> \n> Unconditional use of \"diff -u\" is a breaking change for them.\n\nYes, noted. I'm currently waiting for ld/sparse-diff-blame to\nmerge to 'master' before sending a new version, since there are\nunrelated build failures without taking newer changes in 'master'.\n\nThanks,\n-Stolee\n"},{"id":"444770","messageId":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v3.git.1639149192.gitgitgadget@gmail.com","subject":"[PATCH v4 0/5] Sparse index: fetch, pull, ls-files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:51Z","receivedAt":"2021-12-22T14:21:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is now based on 'master'.\n\nDid you know that 'fetch' and 'pull' read the index? I didn't, or this would\nhave been an integration much earlier in the cycle. They read the index to\nlook for the .gitmodules file in case there are submodules that need to be\nfetched. Since looking for a file by name is already protected, we only need\nto disable 'command_requires_full_index' and we are done.\n\nThe 'ls-files' builtin is useful when debugging the index, and some scripts\nuse it, too. We are not changing the default behavior which expands a sparse\nindex in order to show all of the cached blobs. Instead, we add a '--sparse'\noption that allows us to see the sparse directory entries upon request.\nCombined with --debug, we can see a lot of index details, such as:\n\n$ git ls-files --debug --sparse\nLICENSE\n ctime: 1634910503:287405820\n mtime: 1634910503:287405820\n dev: 16777220 ino: 119325319\n uid: 501 gid: 20\n size: 1098 flags: 200000\nREADME.md\n ctime: 1634910503:288090279\n mtime: 1634910503:288090279\n dev: 16777220 ino: 119325320\n uid: 501 gid: 20\n size: 934 flags: 200000\nbin/index.js\n ctime: 1634910767:828434033\n mtime: 1634910767:828434033\n dev: 16777220 ino: 119325520\n uid: 501 gid: 20\n size: 7292 flags: 200000\nexamples/\n ctime: 0:0\n mtime: 0:0\n dev: 0 ino: 0\n uid: 0 gid: 0\n size: 0 flags: 40004000\npackage.json\n ctime: 1634910503:288676330\n mtime: 1634910503:288676330\n dev: 16777220 ino: 119325321\n uid: 501 gid: 20\n size: 680 flags: 200000\n\n\n(In this example, the 'examples/' directory is sparse.)\n\nThanks!\n\n\nUpdates in v2\n=============\n\n * Rebased onto latest ld/sparse-index-blame without issue.\n * Updated the test to use diff-of-diffs instead of a sequence of greps.\n * Added patches that remove the use of 'test-tool read-cache --table' and\n its implementation.\n\n\nUpdates in v3\n=============\n\n * Fixed typo in commit message.\n * Added comments around doing strange things in an ls-files test.\n * Fixed adjacent typo in a test comment.\n\n\nUpdates in v4\n=============\n\n * Rebased on to 'master' now that ld/sparse-index-blame is merged.\n * Change testing strategy to check exact output instead of using 'diff -u'.\n * Updated documentation to state that directories have a trailing slash.\n\nDerrick Stolee (5):\n fetch/pull: use the sparse index\n ls-files: add --sparse option\n t1092: replace 'read-cache --table' with 'ls-files --sparse'\n t1091/t3705: remove 'test-tool read-cache --table'\n test-read-cache: remove --table, --expand options\n\n Documentation/git-ls-files.txt | 5 +\n builtin/fetch.c | 2 +\n builtin/ls-files.c | 12 ++-\n builtin/pull.c | 2 +\n t/helper/test-read-cache.c | 64 ++---------\n t/t1091-sparse-checkout-builtin.sh | 25 ++++-\n t/t1092-sparse-checkout-compatibility.sh | 132 ++++++++++++++++++++---\n t/t3705-add-sparse-checkout.sh | 8 +-\n 8 files changed, 168 insertions(+), 82 deletions(-)\n\n\nbase-commit: 597af311a2899bfd6640b9b107622c5795d5f998\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1080%2Fderrickstolee%2Fsparse-index%2Ffetch-pull-ls-files-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1080/derrickstolee/sparse-index/fetch-pull-ls-files-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1080\n\nRange-diff vs v3:\n\n 1: f72001638d1 = 1: 1043a1927d2 fetch/pull: use the sparse index\n 2: b81174ba54b ! 2: e1ec52881d9 ls-files: add --sparse option\n @@ Documentation/git-ls-files.txt: Both the in the index (\"i/\")\n \n +--sparse::\n +\tIf the index is sparse, show the sparse directories without expanding\n -+\tto the contained files.\n ++\tto the contained files. Sparse directories will be shown with a\n ++\ttrailing slash, such as \"x/\" for a sparse directory \"x\".\n +\n \\--::\n \tDo not interpret any more arguments as options.\n @@ builtin/ls-files.c: int cmd_ls_files(int argc, const char **argv, const char *cm\n \n ## t/t1092-sparse-checkout-compatibility.sh ##\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse-index is expanded and converted back' '\n - \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n + \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n +\ttest_region index ensure_full_index trace2.txt &&\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +test_expect_success 'ls-files' '\n +\tinit_repos &&\n +\n ++\t# Use a smaller sparse-checkout for reduced output\n ++\ttest_sparse_match git sparse-checkout set &&\n ++\n +\t# Behavior agrees by default. Sparse index is expanded.\n +\ttest_all_match git ls-files &&\n +\n +\t# With --sparse, the sparse index data changes behavior.\n -+\tgit -C sparse-index ls-files >dense &&\n -+\tgit -C sparse-index ls-files --sparse >sparse &&\n ++\tgit -C sparse-index ls-files --sparse >actual &&\n +\n +\tcat >expect <<-\\EOF &&\n -+\t@@ -13,13 +13,9 @@\n -+\t e\n -+\t folder1-\n -+\t folder1.x\n -+\t-folder1/0/0/0\n -+\t-folder1/0/1\n -+\t-folder1/a\n -+\t+folder1/\n -+\t folder10\n -+\t-folder2/0/0/0\n -+\t-folder2/0/1\n -+\t-folder2/a\n -+\t+folder2/\n -+\t g\n -+\t-x/a\n -+\t+x/\n -+\t z\n ++\ta\n ++\tdeep/\n ++\te\n ++\tfolder1-\n ++\tfolder1.x\n ++\tfolder1/\n ++\tfolder10\n ++\tfolder2/\n ++\tg\n ++\tx/\n ++\tz\n +\tEOF\n +\n -+\tdiff -u dense sparse | tail -n +3 >actual &&\n +\ttest_cmp expect actual &&\n +\n +\t# With --sparse and no sparse index, nothing changes.\n @@ t/t1092-sparse-checkout-compatibility.sh: test_expect_success 'sparse index is n\n +\ttest_sparse_match git sparse-checkout add folder1 &&\n +\ttest_sparse_match git ls-files --modified &&\n +\n -+\tgit -C sparse-index ls-files >dense &&\n -+\tgit -C sparse-index ls-files --sparse >sparse &&\n ++\ttest_all_match git ls-files &&\n ++\tgit -C sparse-index ls-files --sparse >actual &&\n +\n +\tcat >expect <<-\\EOF &&\n -+\t@@ -17,9 +17,7 @@\n -+\t folder1/0/1\n -+\t folder1/a\n -+\t folder10\n -+\t-folder2/0/0/0\n -+\t-folder2/0/1\n -+\t-folder2/a\n -+\t+folder2/\n -+\t g\n -+\t-x/a\n -+\t+x/\n -+\t z\n ++\ta\n ++\tdeep/\n ++\te\n ++\tfolder1-\n ++\tfolder1.x\n ++\tfolder1/0/0/0\n ++\tfolder1/0/1\n ++\tfolder1/a\n ++\tfolder10\n ++\tfolder2/\n ++\tg\n ++\tx/\n ++\tz\n +\tEOF\n +\n -+\tdiff -u dense sparse | tail -n +3 >actual &&\n +\ttest_cmp expect actual &&\n +\n +\t# Double-check index expansion is avoided\n 3: 2a6a1c5a39c = 3: 0c53bd09ba4 t1092: replace 'read-cache --table' with 'ls-files --sparse'\n 4: f0143686754 = 4: 4952c9e724b t1091/t3705: remove 'test-tool read-cache --table'\n 5: 9227dc54165 = 5: 3efffad814c test-read-cache: remove --table, --expand options\n\n-- \ngitgitgadget\n"},{"id":"444771","messageId":"1043a1927d220345c3a2fdba518708869f71b542.1640182856.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"[PATCH v4 1/5] fetch/pull: use the sparse index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:52Z","receivedAt":"2021-12-22T14:21:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git fetch' and 'git pull' commands parse the index in order to\ndetermine if submodules exist. Without command_requires_full_index=0,\nthis will expand a sparse index, causing slow performance even when\nthere is no new data to fetch.\n\nThe .gitmodules file will never be inside a sparse directory entry, and\neven if it was, the index_name_pos() method would expand the sparse\nindex if needed as we search for the path by name. These commands do not\niterate over the index, which is the typical thing we are careful about\nwhen integrating with the sparse index.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 2 ++\n builtin/pull.c | 2 ++\n t/t1092-sparse-checkout-compatibility.sh | 10 ++++++++++\n 3 files changed, 14 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e45185cf9cf..a0757dd0158 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1999,6 +1999,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tgit_config(git_fetch_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_fetch_options, builtin_fetch_usage, 0);\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex c8457619d86..100cbf9fb85 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -994,6 +994,8 @@ int cmd_pull(int argc, const char **argv, const char *prefix)\n \t\tset_reflog_message(argc, argv);\n \n \tgit_config(git_pull_config, NULL);\n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n \n \targc = parse_options(argc, argv, prefix, pull_options, pull_usage, 0);\n \ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 49f70a65692..8d3c21fc84c 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -1009,6 +1009,16 @@ test_expect_success 'sparse index is not expanded: blame' '\n \tdone\n '\n \n+test_expect_success 'sparse index is not expanded: fetch/pull' '\n+\tinit_repos &&\n+\n+\tgit -C sparse-index remote add full \"file://$(pwd)/full-checkout\" &&\n+\tensure_not_expanded fetch full &&\n+\tgit -C full-checkout commit --allow-empty -m \"for pull merge\" &&\n+\tgit -C sparse-index commit --allow-empty -m \"for pull merge\" &&\n+\tensure_not_expanded pull full base\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"444772","messageId":"e1ec52881d9ad18097254040824902b799667113.1640182856.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"[PATCH v4 2/5] ls-files: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:53Z","receivedAt":"2021-12-22T14:21:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExisting callers to 'git ls-files' are expecting file names, not\ndirectories. It is best to expand a sparse index to show all of the\ncontained files in this case.\n\nHowever, expert users may want to inspect the contents of the index\nitself including which directories are sparse. Add a --sparse option to\nallow users to request this information.\n\nDuring testing, I noticed that options such as --modified did not affect\nthe output when the files in question were outside the sparse-checkout\ndefinition. Tests are added to document this preexisting behavior and\nhow it remains unchanged with the sparse index and the --sparse option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-ls-files.txt | 5 ++\n builtin/ls-files.c | 12 +++-\n t/t1092-sparse-checkout-compatibility.sh | 91 ++++++++++++++++++++++++\n 3 files changed, 106 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.txt b/Documentation/git-ls-files.txt\nindex 2e3d695fa21..48cc7c0b6f4 100644\n--- a/Documentation/git-ls-files.txt\n+++ b/Documentation/git-ls-files.txt\n@@ -187,6 +187,11 @@ Both the in the index (\"i/\")\n and in the working tree (\"w/\") are shown for regular files,\n followed by the (\"attr/\").\n \n+--sparse::\n+\tIf the index is sparse, show the sparse directories without expanding\n+\tto the contained files. Sparse directories will be shown with a\n+\ttrailing slash, such as \"x/\" for a sparse directory \"x\".\n+\n \\--::\n \tDo not interpret any more arguments as options.\n \ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 031fef1bcaa..c151eb1fb77 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -37,6 +37,7 @@ static int debug_mode;\n static int show_eol;\n static int recurse_submodules;\n static int skipping_duplicates;\n+static int show_sparse_dirs;\n \n static const char *prefix;\n static int max_prefix_len;\n@@ -315,8 +316,10 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n-\t/* TODO: audit for interaction with sparse-index. */\n-\tensure_full_index(repo->index);\n+\n+\tif (!show_sparse_dirs)\n+\t\tensure_full_index(repo->index);\n+\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -670,6 +673,8 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tOPT_BOOL(0, \"debug\", &debug_mode, N_(\"show debugging data\")),\n \t\tOPT_BOOL(0, \"deduplicate\", &skipping_duplicates,\n \t\t\t N_(\"suppress duplicate entries\")),\n+\t\tOPT_BOOL(0, \"sparse\", &show_sparse_dirs,\n+\t\t\t N_(\"show sparse directories in the presence of a sparse index\")),\n \t\tOPT_END()\n \t};\n \tint ret = 0;\n@@ -677,6 +682,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage_with_options(ls_files_usage, builtin_ls_files_options);\n \n+\tprepare_repo_settings(the_repository);\n+\tthe_repository->settings.command_requires_full_index = 0;\n+\n \tprefix = cmd_prefix;\n \tif (prefix)\n \t\tprefix_len = strlen(prefix);\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 8d3c21fc84c..0c51e9bd3b7 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -816,6 +816,12 @@ test_expect_success 'sparse-index is expanded and converted back' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tgit -C sparse-index reset -- folder1/a &&\n \ttest_region index convert_to_sparse trace2.txt &&\n+\ttest_region index ensure_full_index trace2.txt &&\n+\n+\t# ls-files expands on read, but does not write.\n+\trm trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" GIT_TRACE2_EVENT_NESTING=10 \\\n+\t\tgit -C sparse-index ls-files &&\n \ttest_region index ensure_full_index trace2.txt\n '\n \n@@ -871,6 +877,7 @@ test_expect_success 'sparse-index is not expanded' '\n \tinit_repos &&\n \n \tensure_not_expanded status &&\n+\tensure_not_expanded ls-files --sparse &&\n \tensure_not_expanded commit --allow-empty -m empty &&\n \techo >>sparse-index/a &&\n \tensure_not_expanded commit -a -m a &&\n@@ -1019,6 +1026,90 @@ test_expect_success 'sparse index is not expanded: fetch/pull' '\n \tensure_not_expanded pull full base\n '\n \n+test_expect_success 'ls-files' '\n+\tinit_repos &&\n+\n+\t# Use a smaller sparse-checkout for reduced output\n+\ttest_sparse_match git sparse-checkout set &&\n+\n+\t# Behavior agrees by default. Sparse index is expanded.\n+\ttest_all_match git ls-files &&\n+\n+\t# With --sparse, the sparse index data changes behavior.\n+\tgit -C sparse-index ls-files --sparse >actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta\n+\tdeep/\n+\te\n+\tfolder1-\n+\tfolder1.x\n+\tfolder1/\n+\tfolder10\n+\tfolder2/\n+\tg\n+\tx/\n+\tz\n+\tEOF\n+\n+\ttest_cmp expect actual &&\n+\n+\t# With --sparse and no sparse index, nothing changes.\n+\tgit -C sparse-checkout ls-files >dense &&\n+\tgit -C sparse-checkout ls-files --sparse >sparse &&\n+\ttest_cmp dense sparse &&\n+\n+\t# Set up a strange condition of having a file edit\n+\t# outside of the sparse-checkout cone. This is just\n+\t# to verify that sparse-checkout and sparse-index\n+\t# behave the same in this case.\n+\twrite_script edit-content <<-\\EOF &&\n+\tmkdir folder1 &&\n+\techo content >>folder1/a\n+\tEOF\n+\trun_on_sparse ../edit-content &&\n+\n+\t# ls-files does not currently notice modified files whose\n+\t# cache entries are marked SKIP_WORKTREE. This may change\n+\t# in the future, but here we test that sparse index does\n+\t# not accidentally create a change of behavior.\n+\ttest_sparse_match git ls-files --modified &&\n+\ttest_must_be_empty sparse-checkout-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\tgit -C sparse-index ls-files --sparse --modified >sparse-index-out &&\n+\ttest_must_be_empty sparse-index-out &&\n+\n+\t# Add folder1 to the sparse-checkout cone and\n+\t# check that ls-files shows the expanded files.\n+\ttest_sparse_match git sparse-checkout add folder1 &&\n+\ttest_sparse_match git ls-files --modified &&\n+\n+\ttest_all_match git ls-files &&\n+\tgit -C sparse-index ls-files --sparse >actual &&\n+\n+\tcat >expect <<-\\EOF &&\n+\ta\n+\tdeep/\n+\te\n+\tfolder1-\n+\tfolder1.x\n+\tfolder1/0/0/0\n+\tfolder1/0/1\n+\tfolder1/a\n+\tfolder10\n+\tfolder2/\n+\tg\n+\tx/\n+\tz\n+\tEOF\n+\n+\ttest_cmp expect actual &&\n+\n+\t# Double-check index expansion is avoided\n+\tensure_not_expanded ls-files --sparse\n+'\n+\n # NEEDSWORK: a sparse-checkout behaves differently from a full checkout\n # in this scenario, but it shouldn't.\n test_expect_success 'reset mixed and checkout orphan' '\n-- \ngitgitgadget\n\n"},{"id":"444773","messageId":"0c53bd09ba4a4c16e2be4b92c4386e22dbe98b50.1640182856.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"[PATCH v4 3/5] t1092: replace 'read-cache --table' with 'ls-files --sparse'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:54Z","receivedAt":"2021-12-22T14:21:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1092-sparse-checkout-compatibility.sh.\n\nThe important changes are due to the different output format. We need to\nuse the '--stage' output to get a file mode and OID, but it also\nincludes a stage value and drops the object type. This leads to some\ndifferences in how we handle looking for specific entries.\n\nSome places where we previously looked for no 'tree' entries, we can\ninstead directly compare the output across the repository with a sparse\nindex and the one without.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1092-sparse-checkout-compatibility.sh | 31 +++++++++++-------------\n 1 file changed, 14 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh\nindex 0c51e9bd3b7..4ba16177528 100755\n--- a/t/t1092-sparse-checkout-compatibility.sh\n+++ b/t/t1092-sparse-checkout-compatibility.sh\n@@ -206,45 +206,42 @@ test_sparse_unstaged () {\n test_expect_success 'sparse-index contents' '\n \tinit_repos &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set folder1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n \tgit -C sparse-index sparse-checkout set deep/deeper1 &&\n \n-\ttest-tool -C sparse-index read-cache --table >cache &&\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n \tfor dir in deep/deeper2 folder1 folder2 x\n \tdo\n \t\tTREE=$(git -C sparse-index rev-parse HEAD:$dir) &&\n-\t\tgrep \"040000 tree $TREE\t$dir/\" cache \\\n+\t\tgrep \"040000 $TREE 0\t$dir/\" cache \\\n \t\t\t|| return 1\n \tdone &&\n \n-\t# Disabling the sparse-index removes tree entries with full ones\n+\t# Disabling the sparse-index replaces tree entries with full ones\n \tgit -C sparse-index sparse-checkout init --no-sparse-index &&\n-\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\t! grep \"040000 tree\" cache &&\n-\ttest_sparse_match test-tool read-cache --table\n+\ttest_sparse_match git ls-files --stage --sparse\n '\n \n test_expect_success 'expanded in-memory index matches full index' '\n \tinit_repos &&\n-\ttest_sparse_match test-tool read-cache --expand --table\n+\ttest_sparse_match git ls-files --stage\n '\n \n test_expect_success 'status with options' '\n@@ -801,9 +798,9 @@ test_expect_success 'submodule handling' '\n \n \t# having a submodule prevents \"modules\" from collapse\n \ttest_sparse_match git sparse-checkout set deep/deeper1 &&\n-\ttest-tool -C sparse-index read-cache --table >cache &&\n-\tgrep \"100644 blob .*\tmodules/a\" cache &&\n-\tgrep \"160000 commit $(git -C initial-repo rev-parse HEAD)\tmodules/sub\" cache\n+\tgit -C sparse-index ls-files --sparse --stage >cache &&\n+\tgrep \"100644 .*\tmodules/a\" cache &&\n+\tgrep \"160000 $(git -C initial-repo rev-parse HEAD) 0\tmodules/sub\" cache\n '\n \n # When working with a sparse index, some commands will need to expand the\n@@ -1125,13 +1122,13 @@ test_expect_success 'reset mixed and checkout orphan' '\n \t# the sparse checkouts skip \"adding\" the other side of\n \t# the conflict.\n \ttest_sparse_match git reset --mixed HEAD~1 &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2 &&\n \n \t# At this point, sparse-checkouts behave differently\n \t# from the full-checkout.\n \ttest_sparse_match git checkout --orphan new-branch &&\n-\ttest_sparse_match test-tool read-cache --table --expand &&\n+\ttest_sparse_match git ls-files --stage &&\n \ttest_sparse_match git status --porcelain=v2\n '\n \n-- \ngitgitgadget\n\n"},{"id":"444774","messageId":"4952c9e724b82356ca2debfe4fd115b63198bc74.1640182856.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"[PATCH v4 4/5] t1091/t3705: remove 'test-tool read-cache --table'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:55Z","receivedAt":"2021-12-22T14:21:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'git ls-files --sparse' exists, we can use it to verify the\nstate of a sparse index instead of 'test-tool read-cache table'. Replace\nthese usages within t1091-sparse-checkout-builtin.sh and\nt3705-add-sparse-checkout.sh.\n\nThe important changes are due to the different output format. In t3705,\nwe need to use the '--stage' output to get a file mode and OID, but\nit also includes a stage value and drops the object type. This leads\nto some differences in how we handle looking for specific entries.\n\nIn t1091, the test focuses on enabling the sparse index, so we do not\nneed the --stage flag to demonstrate how the index changes, and instead\ncan use a diff.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 25 ++++++++++++++++++++-----\n t/t3705-add-sparse-checkout.sh | 8 ++++----\n 2 files changed, 24 insertions(+), 9 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 272ba1b566b..680e0043c36 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -212,12 +212,27 @@ test_expect_success 'sparse-index enabled and disabled' '\n \n \t\tgit -C repo sparse-checkout init --cone --sparse-index &&\n \t\ttest_cmp_config -C repo true index.sparse &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\tgrep \" tree \" cache &&\n-\n+\t\tgit -C repo ls-files --sparse >sparse &&\n \t\tgit -C repo sparse-checkout disable &&\n-\t\ttest-tool -C repo read-cache --table >cache &&\n-\t\t! grep \" tree \" cache &&\n+\t\tgit -C repo ls-files --sparse >full &&\n+\n+\t\tcat >expect <<-\\EOF &&\n+\t\t@@ -1,4 +1,7 @@\n+\t\t a\n+\t\t-deep/\n+\t\t-folder1/\n+\t\t-folder2/\n+\t\t+deep/a\n+\t\t+deep/deeper1/a\n+\t\t+deep/deeper1/deepest/a\n+\t\t+deep/deeper2/a\n+\t\t+folder1/a\n+\t\t+folder2/a\n+\t\tEOF\n+\n+\t\tdiff -u sparse full | tail -n +3 >actual &&\n+\t\ttest_cmp expect actual &&\n+\n \t\tgit -C repo config --list >config &&\n \t\t! grep index.sparse config\n \t)\ndiff --git a/t/t3705-add-sparse-checkout.sh b/t/t3705-add-sparse-checkout.sh\nindex f3143c92908..81f3384eeed 100755\n--- a/t/t3705-add-sparse-checkout.sh\n+++ b/t/t3705-add-sparse-checkout.sh\n@@ -181,13 +181,13 @@ test_expect_success 'git add fails outside of sparse-checkout definition' '\n \t# Avoid munging CRLFs to avoid an error message\n \tgit -c core.autocrlf=input add --sparse sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100644 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100644 .*sparse_entry\\$\" actual &&\n \n \tgit add --sparse --chmod=+x sparse_entry 2>stderr &&\n \ttest_must_be_empty stderr &&\n-\ttest-tool read-cache --table >actual &&\n-\tgrep \"^100755 blob.*sparse_entry\\$\" actual &&\n+\tgit ls-files --stage >actual &&\n+\tgrep \"^100755 .*sparse_entry\\$\" actual &&\n \n \tgit reset &&\n \n-- \ngitgitgadget\n\n"},{"id":"444775","messageId":"3efffad814c7183ebe5538c145c4fd263e5a9bca.1640182856.git.gitgitgadget@gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"[PATCH v4 5/5] test-read-cache: remove --table, --expand options","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-12-22T14:20:56Z","receivedAt":"2021-12-22T14:21:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis commit effectively reverts 2782db3 (test-tool: don't force full\nindex, 2021-03-30) and e2df6c3 (test-read-cache: print cache entries\nwith --table, 2021-03-30) to remove the --table and --expand options\nfrom 'test-tool read-cache'. The previous changes already removed these\noptions from the test suite in favor of 'git ls-files --sparse'.\n\nThe initial thought of creating these options was to allow for tests to\nsee additional information with every cache entry. In particular, the\nobject type is still not mirrored in 'git ls-files'. Since sparse\ndirectory entries always end with a slash, the object type is not\ncritical to verify the sparse index is enabled. It was thought that it\nwould be helpful to have additional information, such as flags, but that\nwas not needed for the FS Monitor integration and hasn't been needed\nsince.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-cache.c | 64 ++++++--------------------------------\n 1 file changed, 10 insertions(+), 54 deletions(-)\n\ndiff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c\nindex 0d9f08931a1..b736ef16421 100644\n--- a/t/helper/test-read-cache.c\n+++ b/t/helper/test-read-cache.c\n@@ -1,83 +1,39 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n #include \"config.h\"\n-#include \"blob.h\"\n-#include \"commit.h\"\n-#include \"tree.h\"\n-#include \"sparse-index.h\"\n-\n-static void print_cache_entry(struct cache_entry *ce)\n-{\n-\tconst char *type;\n-\tprintf(\"%06o \", ce->ce_mode & 0177777);\n-\n-\tif (S_ISSPARSEDIR(ce->ce_mode))\n-\t\ttype = tree_type;\n-\telse if (S_ISGITLINK(ce->ce_mode))\n-\t\ttype = commit_type;\n-\telse\n-\t\ttype = blob_type;\n-\n-\tprintf(\"%s %s\\t%s\\n\",\n-\t type,\n-\t oid_to_hex(&ce->oid),\n-\t ce->name);\n-}\n-\n-static void print_cache(struct index_state *istate)\n-{\n-\tint i;\n-\tfor (i = 0; i < istate->cache_nr; i++)\n-\t\tprint_cache_entry(istate->cache[i]);\n-}\n \n int cmd__read_cache(int argc, const char **argv)\n {\n-\tstruct repository *r = the_repository;\n \tint i, cnt = 1;\n \tconst char *name = NULL;\n-\tint table = 0, expand = 0;\n \n \tinitialize_the_repository();\n \n-\tfor (++argv, --argc; *argv && starts_with(*argv, \"--\"); ++argv, --argc) {\n-\t\tif (skip_prefix(*argv, \"--print-and-refresh=\", &name))\n-\t\t\tcontinue;\n-\t\tif (!strcmp(*argv, \"--table\"))\n-\t\t\ttable = 1;\n-\t\telse if (!strcmp(*argv, \"--expand\"))\n-\t\t\texpand = 1;\n+\tif (argc > 1 && skip_prefix(argv[1], \"--print-and-refresh=\", &name)) {\n+\t\targc--;\n+\t\targv++;\n \t}\n \n-\tif (argc == 1)\n-\t\tcnt = strtol(argv[0], NULL, 0);\n+\tif (argc == 2)\n+\t\tcnt = strtol(argv[1], NULL, 0);\n \tsetup_git_directory();\n \tgit_config(git_default_config, NULL);\n \n-\tprepare_repo_settings(r);\n-\tr->settings.command_requires_full_index = 0;\n-\n \tfor (i = 0; i < cnt; i++) {\n-\t\trepo_read_index(r);\n-\n-\t\tif (expand)\n-\t\t\tensure_full_index(r->index);\n-\n+\t\tread_cache();\n \t\tif (name) {\n \t\t\tint pos;\n \n-\t\t\trefresh_index(r->index, REFRESH_QUIET,\n+\t\t\trefresh_index(&the_index, REFRESH_QUIET,\n \t\t\t\t NULL, NULL, NULL);\n-\t\t\tpos = index_name_pos(r->index, name, strlen(name));\n+\t\t\tpos = index_name_pos(&the_index, name, strlen(name));\n \t\t\tif (pos < 0)\n \t\t\t\tdie(\"%s not in index\", name);\n \t\t\tprintf(\"%s is%s up to date\\n\", name,\n-\t\t\t ce_uptodate(r->index->cache[pos]) ? \"\" : \" not\");\n+\t\t\t ce_uptodate(the_index.cache[pos]) ? \"\" : \" not\");\n \t\t\twrite_file(name, \"%d\\n\", i);\n \t\t}\n-\t\tif (table)\n-\t\t\tprint_cache(r->index);\n-\t\tdiscard_index(r->index);\n+\t\tdiscard_cache();\n \t}\n \treturn 0;\n }\n-- \ngitgitgadget\n"},{"id":"444814","messageId":"CABPp-BGuoY_R8Cfoa7OfoqqFt4ZuWn9dNyN94vcJb8XJMXOvRA@mail.gmail.com","threadId":"56917","inReplyTo":"pull.1080.v4.git.1640182856.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/5] Sparse index: fetch, pull, ls-files","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-12-22T19:17:27Z","receivedAt":"2021-12-22T19:17:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 22, 2021 at 6:20 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This is now based on 'master'.\n>\n> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n> have been an integration much earlier in the cycle. They read the index to\n> look for the .gitmodules file in case there are submodules that need to be\n> fetched. Since looking for a file by name is already protected, we only need\n> to disable 'command_requires_full_index' and we are done.\n>\n> The 'ls-files' builtin is useful when debugging the index, and some scripts\n> use it, too. We are not changing the default behavior which expands a sparse\n> index in order to show all of the cached blobs. Instead, we add a '--sparse'\n> option that allows us to see the sparse directory entries upon request.\n...\n> Updates in v2\n> =============\n>\n> * Rebased onto latest ld/sparse-index-blame without issue.\n> * Updated the test to use diff-of-diffs instead of a sequence of greps.\n> * Added patches that remove the use of 'test-tool read-cache --table' and\n> its implementation.\n>\n>\n> Updates in v3\n> =============\n>\n> * Fixed typo in commit message.\n> * Added comments around doing strange things in an ls-files test.\n> * Fixed adjacent typo in a test comment.\n>\n>\n> Updates in v4\n> =============\n>\n> * Rebased on to 'master' now that ld/sparse-index-blame is merged.\n> * Change testing strategy to check exact output instead of using 'diff -u'.\n> * Updated documentation to state that directories have a trailing slash.\n\nThis version looks good to me:\n\nReviewed-by: Elijah Newren \n"},{"id":"444845","messageId":"xmqqa6gsdwi0.fsf@gitster.g","threadId":"56917","inReplyTo":"CABPp-BGuoY_R8Cfoa7OfoqqFt4ZuWn9dNyN94vcJb8XJMXOvRA@mail.gmail.com","subject":"Re: [PATCH v4 0/5] Sparse index: fetch, pull, ls-files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-12-22T23:56:39Z","receivedAt":"2021-12-22T23:56:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Wed, Dec 22, 2021 at 6:20 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> This is now based on 'master'.\n>>\n>> Did you know that 'fetch' and 'pull' read the index? I didn't, or this would\n>> have been an integration much earlier in the cycle. They read the index to\n>> look for the .gitmodules file in case there are submodules that need to be\n>> fetched. Since looking for a file by name is already protected, we only need\n>> to disable 'command_requires_full_index' and we are done.\n>>\n>> The 'ls-files' builtin is useful when debugging the index, and some scripts\n>> use it, too. We are not changing the default behavior which expands a sparse\n>> index in order to show all of the cached blobs. Instead, we add a '--sparse'\n>> option that allows us to see the sparse directory entries upon request.\n> ...\n>> Updates in v2\n>> =============\n>>\n>> * Rebased onto latest ld/sparse-index-blame without issue.\n>> * Updated the test to use diff-of-diffs instead of a sequence of greps.\n>> * Added patches that remove the use of 'test-tool read-cache --table' and\n>> its implementation.\n>>\n>>\n>> Updates in v3\n>> =============\n>>\n>> * Fixed typo in commit message.\n>> * Added comments around doing strange things in an ls-files test.\n>> * Fixed adjacent typo in a test comment.\n>>\n>>\n>> Updates in v4\n>> =============\n>>\n>> * Rebased on to 'master' now that ld/sparse-index-blame is merged.\n>> * Change testing strategy to check exact output instead of using 'diff -u'.\n>> * Updated documentation to state that directories have a trailing slash.\n>\n> This version looks good to me:\n>\n> Reviewed-by: Elijah Newren \n\nYup, they looked good to me too.\n\nThanks.\n"}]}