{"thread":{"id":"57444","subject":"[PATCH] Provide config option to expect files outside sparse patterns","startedAt":"2022-02-20T05:10:00Z","lastAt":"2022-03-02T13:37:17Z","messageCount":29,"participants":["Elijah Newren via GitGitGadget","Derrick Stolee","Junio C Hamano","Johannes Schindelin","Ævar Arnfjörð Bjarmason","Elijah Newren","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"448878","messageId":"pull.1153.git.1645333542011.gitgitgadget@gmail.com","threadId":"57444","inReplyTo":null,"subject":"[PATCH] Provide config option to expect files outside sparse patterns","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-20T05:05:41Z","receivedAt":"2022-02-20T05:10:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nTypically with sparse checkouts, we expect files outside the sparsity\npatterns to be marked as SKIP_WORKTREE and be missing from the working\ntree. VFS for Git can be used to turn this expectation on its head:\nall files are considered present in the working copy, though they are\nnot vivified until first access access. With VFS for Git, most of the\nfiles do not match the sparsity patterns at first, and the VFS layer\nautomatically updates the sparsity patterns to add more files whenever\nfiles are written.\n\nWith this background, this special usecase does not play well with the\nsafety check we added in commit 11d46a399d (\"repo_read_index: clear\nSKIP_WORKTREE bit from files present in worktree\", 2022-01-06).\nChecking SKIP_WORKTREE files to see if they are present in the working\ntree causes them all to be immediately vivified. Further, the special\nVFS layer, by virtue of automatically updating the sparsity patterns and\ncatching all accesses, isn't in need of that safety check either.\nProvide a configuration option, core.expectFilesOutsideSparsePatterns\nso that those with this special usecase can turn off the safety check.\n\nSigned-off-by: Elijah Newren \n---\n Provide config option to expect files outside sparse patterns\n \n Builds on en/present-despite-skipped, and addresses issue reported at\n https://lore.kernel.org/git/YhBCsg2DCEd9FXjE@google.com/\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1153%2Fnewren%2Fpresent-despite-skip-worktree-toggle-for-vfs-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1153/newren/present-despite-skip-worktree-toggle-for-vfs-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1153\n\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n sparse-index.c | 3 ++-\n t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n 5 files changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b1..1f35d8a59ea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\n+extern int core_expect_files_outside_sparse_patterns;\n \n /*\n * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a01..68e877a1d80 100644\n--- a/config.c\n+++ b/config.c\n@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n+\t\tcore_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..c86ebe9ce74 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -70,6 +70,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int core_sparse_checkout_cone;\n+int core_expect_files_outside_sparse_patterns;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/sparse-index.c b/sparse-index.c\nindex eed170cd8f7..8e2a3973856 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n \n \tint i;\n \n-\tif (!core_apply_sparse_checkout)\n+\tif (!core_apply_sparse_checkout ||\n+\t core_expect_files_outside_sparse_patterns)\n \t\treturn;\n \n restart:\ndiff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\nindex 3deb4901874..feef883523b 100755\n--- a/t/t1090-sparse-checkout-scope.sh\n+++ b/t/t1090-sparse-checkout-scope.sh\n@@ -84,4 +84,23 @@ test_expect_success 'in partial clone, sparse checkout only fetches needed blobs\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'skip-worktree on files outside sparse patterns' '\n+\tgit sparse-checkout disable &&\n+\tgit sparse-checkout set --no-cone \"a*\" &&\n+\tgit checkout-index --all --ignore-skip-worktree-bits &&\n+\n+\tgit ls-files -t >output &&\n+\t! grep ^S output >actual &&\n+\ttest_must_be_empty actual &&\n+\n+\ttest_config core.expectFilesOutsideSparsePatterns true &&\n+\tcat <<-\\EOF >expect &&\n+\tS b\n+\tS c\n+\tEOF\n+\tgit ls-files -t >output &&\n+\tgrep ^S output >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n\nbase-commit: d79d29935262f8422a6e620bd33cc2fce28e669d\n-- \ngitgitgadget\n"},{"id":"448915","messageId":"54a0aa74-57c2-ee65-ae07-cb1b0daf947f@github.com","threadId":"57444","inReplyTo":"pull.1153.git.1645333542011.gitgitgadget@gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-20T19:41:55Z","receivedAt":"2022-02-20T19:41:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/20/2022 12:05 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Typically with sparse checkouts, we expect files outside the sparsity\n> patterns to be marked as SKIP_WORKTREE and be missing from the working\n> tree. VFS for Git can be used to turn this expectation on its head:\n> all files are considered present in the working copy, though they are\n> not vivified until first access access. With VFS for Git, most of the\n> files do not match the sparsity patterns at first, and the VFS layer\n> automatically updates the sparsity patterns to add more files whenever\n> files are written.\n> \n> With this background, this special usecase does not play well with the\n> safety check we added in commit 11d46a399d (\"repo_read_index: clear\n> SKIP_WORKTREE bit from files present in worktree\", 2022-01-06).\n> Checking SKIP_WORKTREE files to see if they are present in the working\n> tree causes them all to be immediately vivified. Further, the special\n> VFS layer, by virtue of automatically updating the sparsity patterns and\n> catching all accesses, isn't in need of that safety check either.\n> Provide a configuration option, core.expectFilesOutsideSparsePatterns\n> so that those with this special usecase can turn off the safety check.\n\nThis patch looks like a good solution to the concerns brought up by\nJonathan N. around vfsd. VFS for Git uses the microsoft/git fork with\nits own custom config to protect things like this. I imagine that we\nwill start setting your core_expect_files_outside_sparse_patterns\nvariable when reading the virtual filesystem info. We might even modify\nsome of our custom checks to use this variable instead. That would make\nthem appropriate to send upstream.\n\nShould we update Documentation/config/core.txt describing this config\nkey? Or is this intended to be an internal detail only for something\nlike vfsd?\n\nThe only concern here really is if we want to be picky about the \"VFS\nfor Git\" references instead of \"vfsd\" references in the commit message.\n\nThanks,\n-Stolee\n"},{"id":"448923","messageId":"xmqqsfsduwdp.fsf@gitster.g","threadId":"57444","inReplyTo":"54a0aa74-57c2-ee65-ae07-cb1b0daf947f@github.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-20T20:16:18Z","receivedAt":"2022-02-20T20:16:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> This patch looks like a good solution to the concerns brought up by\n> Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with\n> its own custom config to protect things like this. I imagine that we\n> will start setting your core_expect_files_outside_sparse_patterns\n> variable when reading the virtual filesystem info. We might even modify\n> some of our custom checks to use this variable instead. That would make\n> them appropriate to send upstream.\n>\n> Should we update Documentation/config/core.txt describing this config\n> key? Or is this intended to be an internal detail only for something\n> like vfsd?\n>\n> The only concern here really is if we want to be picky about the \"VFS\n> for Git\" references instead of \"vfsd\" references in the commit message.\n\nThanks for a quick response to make sure what our left hand is\ntrying to do will not interfere with what our right hand has already\nbeen doing ;-)\n\n"},{"id":"449057","messageId":"nycvar.QRO.7.76.6.2202212100080.4418@tvgsbejvaqbjf.bet","threadId":"57444","inReplyTo":"pull.1153.git.1645333542011.gitgitgadget@gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-21T20:34:35Z","receivedAt":"2022-02-21T20:34:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nIn addition to Stolee's feedback...\n\nOn Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n\n> diff --git a/config.c b/config.c\n> index 2bffa8d4a01..68e877a1d80 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> \t\treturn 0;\n> \t}\n>\n> +\tif (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n> +\t\tcore_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n\nThe `core` section is already quite crowded (for which I am partially\nresponsible, of course).\n\nMaybe it would be a good idea to introduce the `sparse` section, using\n`sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\nfalse`?\n\nCiao,\nDscho\n"},{"id":"449063","messageId":"220221.86a6ejakun.gmgdl@evledraar.gmail.com","threadId":"57444","inReplyTo":"nycvar.QRO.7.76.6.2202212100080.4418@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-21T22:53:08Z","receivedAt":"2022-02-21T22:57:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 21 2022, Johannes Schindelin wrote:\n\n> Hi Elijah,\n>\n> In addition to Stolee's feedback...\n>\n> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n>\n>> diff --git a/config.c b/config.c\n>> index 2bffa8d4a01..68e877a1d80 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>> \t\treturn 0;\n>> \t}\n>>\n>> +\tif (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n>> +\t\tcore_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n>> +\t\treturn 0;\n>> +\t}\n>\n> The `core` section is already quite crowded (for which I am partially\n> responsible, of course).\n>\n> Maybe it would be a good idea to introduce the `sparse` section, using\n> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n> false`?\n\nThere's a large list of exceptions to this, but generally we have core.*\nfor cross-command configuration, and .* for command-specific\nconfiguration.\n\nExcetions include http.*, mailmap.*, pack.* (arguably), trace2.*.\n\nIn this case though we have 2x /^core\\.sparse[A-Z].*/ variables in\ngit-config(1) already.\n\nSo perhaps we could name it like that, then at least you'd find all of\nthese in the same place when looking through that documentation,\nalthough a name starting with \"core.sparse\" might not be otherwise\nideal.\n\nThere's also no law that says we can't have a three-level variable as\ncore.sparse.*, perhaps that would be the start of a good trend, and it\nwould sort adjacent to core.sparse[A-Z].* ...\n"},{"id":"449083","messageId":"CABPp-BGz0Y+Gk0uLy7K-pqBY955=rN-E2SJFofDaje60b-XZuw@mail.gmail.com","threadId":"57444","inReplyTo":"54a0aa74-57c2-ee65-ae07-cb1b0daf947f@github.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-22T02:17:21Z","receivedAt":"2022-02-22T02:17:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee\n wrote:\n>\n> On 2/20/2022 12:05 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Typically with sparse checkouts, we expect files outside the sparsity\n> > patterns to be marked as SKIP_WORKTREE and be missing from the working\n> > tree. VFS for Git can be used to turn this expectation on its head:\n> > all files are considered present in the working copy, though they are\n> > not vivified until first access access. With VFS for Git, most of the\n> > files do not match the sparsity patterns at first, and the VFS layer\n> > automatically updates the sparsity patterns to add more files whenever\n> > files are written.\n> >\n> > With this background, this special usecase does not play well with the\n> > safety check we added in commit 11d46a399d (\"repo_read_index: clear\n> > SKIP_WORKTREE bit from files present in worktree\", 2022-01-06).\n> > Checking SKIP_WORKTREE files to see if they are present in the working\n> > tree causes them all to be immediately vivified. Further, the special\n> > VFS layer, by virtue of automatically updating the sparsity patterns and\n> > catching all accesses, isn't in need of that safety check either.\n> > Provide a configuration option, core.expectFilesOutsideSparsePatterns\n> > so that those with this special usecase can turn off the safety check.\n>\n> This patch looks like a good solution to the concerns brought up by\n> Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with\n> its own custom config to protect things like this. I imagine that we\n> will start setting your core_expect_files_outside_sparse_patterns\n> variable when reading the virtual filesystem info. We might even modify\n> some of our custom checks to use this variable instead. That would make\n> them appropriate to send upstream.\n>\n> Should we update Documentation/config/core.txt describing this config\n> key? Or is this intended to be an internal detail only for something\n> like vfsd?\n\nI think it's probably only for vfsd, but vfsd users probably deserve\nto be able to look it up in the documentation, so it's probably worth\ndocumenting. Once we figure out it's name and section, that is (as\nper Dscho's email -- comments from you on that would be appreciated).\n\n> The only concern here really is if we want to be picky about the \"VFS\n> for Git\" references instead of \"vfsd\" references in the commit message.\n\nI'm not sure I'm understanding the distinction. Was \"VFS for Git\"\nrenamed to \"vfsd\"? Is \"vfsd\" Google's modified version? Something\nelse?\n\nAlso, vfsd doesn't seem to be Google-able whereas \"VFS for Git\" is,\nand since it's not an internal git project, it might be nice to use a\nname that users can find.\n\nBut anyway, I have no idea what the \"correct\" text here is (as\nprobably evidenced by my questions), so if anyone wants to provide\nsuggestions or corrections to the commit message, I'm happy to take\nthem.\n"},{"id":"449084","messageId":"CABPp-BH_fJYWSySh_-Pk5w2j7U2q4CAXi_mEextmmd4YBXOiGg@mail.gmail.com","threadId":"57444","inReplyTo":"nycvar.QRO.7.76.6.2202212100080.4418@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-22T02:23:02Z","receivedAt":"2022-02-22T02:23:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n> In addition to Stolee's feedback...\n>\n> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n>\n> > diff --git a/config.c b/config.c\n> > index 2bffa8d4a01..68e877a1d80 100644\n> > --- a/config.c\n> > +++ b/config.c\n> > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> > return 0;\n> > }\n> >\n> > + if (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n> > + core_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n> > + return 0;\n> > + }\n>\n> The `core` section is already quite crowded (for which I am partially\n> responsible, of course).\n>\n> Maybe it would be a good idea to introduce the `sparse` section, using\n> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n> false`?\n\nThat's a fair point. At one point Stolee wanted to change from\ncore.sparse* to sparse.* -- but by that point we already had users and\nwould have had to deal with a bit of a migration story (and wondering\nwhat to do if people had both old and new config variables set\ninconsistently).\n\nI'm not sure if it's optimal to try to keep the sparse settings\ntogether (thus put new ones under core), or try to avoid filling core.\nI guess if we moved towards sparse.* now, it might be an easier\nmigration story if we only have two options to move. And besides,\nwe're already split between multiple sections with\nextensions.worktreeConfig, core.sparseCheckout{,Cone}, and\nindex.sparse already...so maybe adding one more section would be par\nfor the course. ;-)\n\nSo, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd\nlike to hear Stolee's thoughts too.\n"},{"id":"449085","messageId":"CABPp-BHmU8-a+McANE2bdAndGEtVudr74FHEEj6K6NwYECEZ6Q@mail.gmail.com","threadId":"57444","inReplyTo":"220221.86a6ejakun.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-22T02:25:50Z","receivedAt":"2022-02-22T02:26:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> On Mon, Feb 21 2022, Johannes Schindelin wrote:\n>\n> > Hi Elijah,\n> >\n> > In addition to Stolee's feedback...\n> >\n> > On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n> >\n> >> diff --git a/config.c b/config.c\n> >> index 2bffa8d4a01..68e877a1d80 100644\n> >> --- a/config.c\n> >> +++ b/config.c\n> >> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> >> return 0;\n> >> }\n> >>\n> >> + if (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n> >> + core_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n> >> + return 0;\n> >> + }\n> >\n> > The `core` section is already quite crowded (for which I am partially\n> > responsible, of course).\n> >\n> > Maybe it would be a good idea to introduce the `sparse` section, using\n> > `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n> > false`?\n>\n> There's a large list of exceptions to this, but generally we have core.*\n> for cross-command configuration, and .* for command-specific\n> configuration.\n>\n> Excetions include http.*, mailmap.*, pack.* (arguably), trace2.*.\n>\n> In this case though we have 2x /^core\\.sparse[A-Z].*/ variables in\n> git-config(1) already.\n\nThose wouldn't be an exception as worded. They are very much\ncross-command configuration variables, affecting basically every\ncommand that touches the working tree.\n\n> So perhaps we could name it like that, then at least you'd find all of\n> these in the same place when looking through that documentation,\n> although a name starting with \"core.sparse\" might not be otherwise\n> ideal.\n>\n> There's also no law that says we can't have a three-level variable as\n> core.sparse.*, perhaps that would be the start of a good trend, and it\n> would sort adjacent to core.sparse[A-Z].* ...\n\nInteresting thought. I'm curious what others think of this.\n"},{"id":"449095","messageId":"220222.86wnhn8bbt.gmgdl@evledraar.gmail.com","threadId":"57444","inReplyTo":"CABPp-BH_fJYWSySh_-Pk5w2j7U2q4CAXi_mEextmmd4YBXOiGg@mail.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-22T10:05:20Z","receivedAt":"2022-02-22T10:06:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 21 2022, Elijah Newren wrote:\n\n> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin\n> wrote:\n>>\n>> Hi Elijah,\n>>\n>> In addition to Stolee's feedback...\n>>\n>> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n>>\n>> > diff --git a/config.c b/config.c\n>> > index 2bffa8d4a01..68e877a1d80 100644\n>> > --- a/config.c\n>> > +++ b/config.c\n>> > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>> > return 0;\n>> > }\n>> >\n>> > + if (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n>> > + core_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n>> > + return 0;\n>> > + }\n>>\n>> The `core` section is already quite crowded (for which I am partially\n>> responsible, of course).\n>>\n>> Maybe it would be a good idea to introduce the `sparse` section, using\n>> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n>> false`?\n>\n> That's a fair point. At one point Stolee wanted to change from\n> core.sparse* to sparse.* -- but by that point we already had users and\n> would have had to deal with a bit of a migration story (and wondering\n> what to do if people had both old and new config variables set\n> inconsistently).\n>\n> I'm not sure if it's optimal to try to keep the sparse settings\n> together (thus put new ones under core), or try to avoid filling core.\n> I guess if we moved towards sparse.* now, it might be an easier\n> migration story if we only have two options to move. And besides,\n> we're already split between multiple sections with\n> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and\n> index.sparse already...so maybe adding one more section would be par\n> for the course. ;-)\n>\n> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd\n> like to hear Stolee's thoughts too.\n\nYes, maybe sparse.* etc. is best. I'm fine with whatever you two come up\nwith.\n\nI just wanted to raise the point that it's less important to consider\nnames/namespaces for variables one-by-one, and more important that all\nthe variables in a \"family\" fit well together, the users can find them\nall in one place etc. (ideally, not always possible).\n\n\n"},{"id":"449116","messageId":"nycvar.QRO.7.76.6.2202221309220.11118@tvgsbejvaqbjf.bet","threadId":"57444","inReplyTo":"CABPp-BH_fJYWSySh_-Pk5w2j7U2q4CAXi_mEextmmd4YBXOiGg@mail.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-22T12:11:27Z","receivedAt":"2022-02-22T12:11:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Mon, 21 Feb 2022, Elijah Newren wrote:\n\n> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin\n> wrote:\n> >\n> > In addition to Stolee's feedback...\n> >\n> > On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n> >\n> > > diff --git a/config.c b/config.c\n> > > index 2bffa8d4a01..68e877a1d80 100644\n> > > --- a/config.c\n> > > +++ b/config.c\n> > > @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> > > return 0;\n> > > }\n> > >\n> > > + if (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n> > > + core_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n> > > + return 0;\n> > > + }\n> >\n> > The `core` section is already quite crowded (for which I am partially\n> > responsible, of course).\n> >\n> > Maybe it would be a good idea to introduce the `sparse` section, using\n> > `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n> > false`?\n>\n> That's a fair point. At one point Stolee wanted to change from\n> core.sparse* to sparse.* -- but by that point we already had users and\n> would have had to deal with a bit of a migration story (and wondering\n> what to do if people had both old and new config variables set\n> inconsistently).\n\nRight, migration is always hard.\n\nAnd it's outside of the scope of this here patch series, of course.\n\n> I'm not sure if it's optimal to try to keep the sparse settings\n> together (thus put new ones under core), or try to avoid filling core.\n> I guess if we moved towards sparse.* now, it might be an easier\n> migration story if we only have two options to move. And besides,\n> we're already split between multiple sections with\n> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and\n> index.sparse already...so maybe adding one more section would be par\n> for the course. ;-)\n\nFWIW as a potential #leftoverbits, we could migrate those to `sparse.*`\nwhere `sparse.*` would take precendence over `core.sparse*` and the usual\ndeprecation notice would be shown via the `advice` mechanism.\n\n> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd\n> like to hear Stolee's thoughts too.\n\nIndeed, his opinion weighs more than mine on this matter.\n\nCiao,\nDscho\n"},{"id":"449117","messageId":"nycvar.QRO.7.76.6.2202221311480.11118@tvgsbejvaqbjf.bet","threadId":"57444","inReplyTo":"CABPp-BHmU8-a+McANE2bdAndGEtVudr74FHEEj6K6NwYECEZ6Q@mail.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-22T12:13:06Z","receivedAt":"2022-02-22T12:13:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Mon, 21 Feb 2022, Elijah Newren wrote:\n\n> On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>\n> > There's also no law that says we can't have a three-level variable as\n> > core.sparse.*, perhaps that would be the start of a good trend, and it\n> > would sort adjacent to core.sparse[A-Z].* ...\n>\n> Interesting thought. I'm curious what others think of this.\n\nWhat would make it `core`?\n\nIn other words, while it is valuable to consider this option, I don't\nthink that it solves anything, for the price of additional complexity.\n\nLet's not go there.\n\nCiao,\nDscho\n"},{"id":"449118","messageId":"nycvar.QRO.7.76.6.2202221313400.11118@tvgsbejvaqbjf.bet","threadId":"57444","inReplyTo":"CABPp-BGz0Y+Gk0uLy7K-pqBY955=rN-E2SJFofDaje60b-XZuw@mail.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-02-22T12:28:30Z","receivedAt":"2022-02-22T12:28:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Mon, 21 Feb 2022, Elijah Newren wrote:\n\n> On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee\n> wrote:\n>\n> > The only concern here really is if we want to be picky about the \"VFS\n> > for Git\" references instead of \"vfsd\" references in the commit\n> > message.\n>\n> I'm not sure I'm understanding the distinction. Was \"VFS for Git\"\n> renamed to \"vfsd\"? Is \"vfsd\" Google's modified version? Something\n> else?\n\nVFS for Git is the existing project at\nhttps://github.com/microsoft/VFSforGit which is pretty much in maintenance\nmode.\n\nvfsd was mentioned by Jonathan Tan in\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/.\nFrom what I gather, it is a completely separate implementation of the same\nideas of VFS for Git, but from what I see it does not share any code with\nit (and it is unclear how much vfsd tries/tried to learn from VFS for Git,\nit looks like it's being done from scratch but that impression could be\nincorrect).\n\n> Also, vfsd doesn't seem to be Google-able whereas \"VFS for Git\" is, and\n> since it's not an internal git project, it might be nice to use a name\n> that users can find.\n>\n> But anyway, I have no idea what the \"correct\" text here is (as\n> probably evidenced by my questions), so if anyone wants to provide\n> suggestions or corrections to the commit message, I'm happy to take\n> them.\n\nHow about this?\n\n\tTypically with sparse checkouts, we expect files outside the\n\tsparsity patterns to be marked as SKIP_WORKTREE and be missing\n\tfrom the working tree.\n\n\tThere is currently an effort code-named `vfsd` (see\n\thttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\tfor details) to implement a Git-aware virtual file system layer\n\t(similar in spirit to the VFS for Git project) that can be used to\n\tturn this expectation on its head: all files are considered\n\tpresent in the working copy, though they are not vivified until\n\tactually accessed.\n\n\tWith such a virtual file system (VFS) layer, most of the files do\n\tnot match the sparsity patterns at first, and the VFS layer\n\tautomatically updates the sparsity patterns to add more files whenever\n\tfiles are written.\n\n\tSide note: VFS for Git itself requires the Microsoft fork of Git\n\tto work, therefore this patch would only be needed in that fork,\n\tas far as VFS for Git is concerned.\n\nCiao,\nDscho\n"},{"id":"449122","messageId":"220222.8635kb832w.gmgdl@evledraar.gmail.com","threadId":"57444","inReplyTo":"nycvar.QRO.7.76.6.2202221311480.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-22T12:57:00Z","receivedAt":"2022-02-22T13:04:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 22 2022, Johannes Schindelin wrote:\n\n> Hi Elijah,\n>\n> On Mon, 21 Feb 2022, Elijah Newren wrote:\n>\n>> On Mon, Feb 21, 2022 at 2:57 PM Ævar Arnfjörð Bjarmason\n>> wrote:\n>>\n>> > There's also no law that says we can't have a three-level variable as\n>> > core.sparse.*, perhaps that would be the start of a good trend, and it\n>> > would sort adjacent to core.sparse[A-Z].* ...\n>>\n>> Interesting thought. I'm curious what others think of this.\n>\n> What would make it `core`?\n>\n> In other words, while it is valuable to consider this option, I don't\n> think that it solves anything, for the price of additional complexity.\n\nTo a first approximation what makes it core.* is that we have a\ncore.sparseCheckout.\n\nIt's hard come up with some definition of config namespacing where it\nbelongs there, but not a setting that tweaks it further, no?\n\nI.e. we could quibble with the benefit of hindsight whether\ncore.sshCommand was the best idea at the time, or if it should be\nremote.sshCommand or whatever.\n\nBut if we introduced a more specific sshCommandSomething having it there\ninstead of ... well, damn, I forgot that we have ssh.variant :)\n\nAnyway, as noted (and I feel I have to comment as the originator of this\nthread) I think the bikeshedding about namespacing is less important\nthan making sure from a user perspective that these are discoverable.\n\nE.g. core.sshCommand doesn't reference ssh.variant or the other way\naround, and in my \"man git-config\" they're at ~25% and ~90% in to what's\nnow a *huge* document. You need to read the whole thing or have the\nforesight to search through it to discover both.\n\nSo for core.sshCommand and ssh.variant, and core.sparse* and sparse.*\ncross-linking etc. would go a long way...\n\nAside: I do have some local patches (waiting on a couple of other things\nI have outstanding) to add a \"CONFIGURATION\" section to all of the\nbuilt-in commands with some including/splitting up of\nDocumentation/config**.txt, so \"man git-sparse-checkout\" would have a\nsection listing the subset of \"git-config\" applicable to the command.\n"},{"id":"449127","messageId":"f65462bf-eed7-f8b9-6255-249e5128172f@github.com","threadId":"57444","inReplyTo":"nycvar.QRO.7.76.6.2202221313400.11118@tvgsbejvaqbjf.bet","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-22T13:43:19Z","receivedAt":"2022-02-22T13:43:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/22/2022 7:28 AM, Johannes Schindelin wrote:\n> Hi Elijah,\n> \n> On Mon, 21 Feb 2022, Elijah Newren wrote:\n> \n>> On Sun, Feb 20, 2022 at 11:41 AM Derrick Stolee\n>> wrote:\n>>\n>>> The only concern here really is if we want to be picky about the \"VFS\n>>> for Git\" references instead of \"vfsd\" references in the commit\n>>> message.\n>>\n>> I'm not sure I'm understanding the distinction. Was \"VFS for Git\"\n>> renamed to \"vfsd\"? Is \"vfsd\" Google's modified version? Something\n>> else?\n> \n> VFS for Git is the existing project at\n> https://github.com/microsoft/VFSforGit which is pretty much in maintenance\n> mode.\n> \n> vfsd was mentioned by Jonathan Tan in\n> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/.\n> From what I gather, it is a completely separate implementation of the same\n> ideas of VFS for Git, but from what I see it does not share any code with\n> it (and it is unclear how much vfsd tries/tried to learn from VFS for Git,\n> it looks like it's being done from scratch but that impression could be\n> incorrect).\n\nElijah is correct that \"vfsd\" is not publicly visible outside of this\nmessage, so that makes it difficult for us to verify that these patches\nbeing made for it actually work as we intend.\n\n> \tSide note: VFS for Git itself requires the Microsoft fork of Git\n> \tto work, therefore this patch would only be needed in that fork,\n> \tas far as VFS for Git is concerned.\n\nI would drop this side note, since a big reason for the microsoft/git\nfork is so we can create custom patches that handle these issues. We\nwould probably have created a similar patch during release integration,\nbut we are grateful to use this patch as a base for making these\nintegrations simpler in the future.\n\nThanks,\n-Stolee\n"},{"id":"449129","messageId":"d33e3253-dc2f-18a7-04b7-2c8a44405f6b@github.com","threadId":"57444","inReplyTo":"CABPp-BH_fJYWSySh_-Pk5w2j7U2q4CAXi_mEextmmd4YBXOiGg@mail.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-22T13:47:36Z","receivedAt":"2022-02-22T13:47:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/21/2022 9:23 PM, Elijah Newren wrote:\n> On Mon, Feb 21, 2022 at 12:34 PM Johannes Schindelin\n> wrote:\n>>\n>> Hi Elijah,\n>>\n>> In addition to Stolee's feedback...\n>>\n>> On Sun, 20 Feb 2022, Elijah Newren via GitGitGadget wrote:\n>>\n>>> diff --git a/config.c b/config.c\n>>> index 2bffa8d4a01..68e877a1d80 100644\n>>> --- a/config.c\n>>> +++ b/config.c\n>>> @@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>>> return 0;\n>>> }\n>>>\n>>> + if (!strcmp(var, \"core.expectfilesoutsidesparsepatterns\")) {\n>>> + core_expect_files_outside_sparse_patterns = git_config_bool(var, value);\n>>> + return 0;\n>>> + }\n>>\n>> The `core` section is already quite crowded (for which I am partially\n>> responsible, of course).\n>>\n>> Maybe it would be a good idea to introduce the `sparse` section, using\n>> `sparse.allowFilesMatchingPatterns` or `sparse.applyPatternsToWorktree =\n>> false`?\n> \n> That's a fair point. At one point Stolee wanted to change from\n> core.sparse* to sparse.* -- but by that point we already had users and\n> would have had to deal with a bit of a migration story (and wondering\n> what to do if people had both old and new config variables set\n> inconsistently).\n> \n> I'm not sure if it's optimal to try to keep the sparse settings\n> together (thus put new ones under core), or try to avoid filling core.\n> I guess if we moved towards sparse.* now, it might be an easier\n> migration story if we only have two options to move. And besides,\n> we're already split between multiple sections with\n> extensions.worktreeConfig, core.sparseCheckout{,Cone}, and\n> index.sparse already...so maybe adding one more section would be par\n> for the course. ;-)\n> \n> So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd\n> like to hear Stolee's thoughts too.\n\nThis has been an interesting discussion.\n\nI think that the existence of core.sparseCheckout[Cone] isn't a good\nreason to keep adding to the core.* namespace. Creating a sparse.*\nnamespace is a good idea. I doubt this will be the last time we want\na config option for some behavior custom to sparse-checkouts (or\nvirtual environments).\n\nI agree that migrating core.sparseCheckout[Cone] to the sparse.*\nnamespace would need to be done carefully. In particular, we would\nprobably need to _always_ understand the core.* versions, but start\nto prefer the sparse.* versions if there is a conflict. This work\ncould be delayed indefinitely, or could be offered as a project for\na new contributor (if we experienced contributors could agree on a\nclear path to make this transition).\n\nThanks,\n-Stolee\n"},{"id":"449200","messageId":"YhVuCfDgFF/K5Gtl@google.com","threadId":"57444","inReplyTo":"220222.8635kb832w.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-22T23:13:13Z","receivedAt":"2022-02-22T23:13:20Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n\n> To a first approximation what makes it core.* is that we have a\n> core.sparseCheckout.\n[...]\n> Anyway, as noted (and I feel I have to comment as the originator of this\n> thread) I think the bikeshedding about namespacing is less important\n> than making sure from a user perspective that these are discoverable.\n\nAgreed. Though there's already also advice.updateSparsePath; even\nthough I suspect that ctrl+F for \"sparse\" is what people may already\nbe using in practice, a pointer from the git-sparse-checkout(1) page\nwould be welcome.\n\n> E.g. core.sshCommand doesn't reference ssh.variant or the other way\n> around, and in my \"man git-config\" they're at ~25% and ~90% in to what's\n> now a *huge* document. You need to read the whole thing or have the\n> foresight to search through it to discover both.\n>\n> So for core.sshCommand and ssh.variant, and core.sparse* and sparse.*\n> cross-linking etc. would go a long way...\n\nAgreed as well.\n\n> Aside: I do have some local patches (waiting on a couple of other things\n> I have outstanding) to add a \"CONFIGURATION\" section to all of the\n> built-in commands with some including/splitting up of\n> Documentation/config**.txt, so \"man git-sparse-checkout\" would have a\n> section listing the subset of \"git-config\" applicable to the command.\n\nOh! I was about to do a quick patch for git-sparse-checkout.txt, but\nthis is even better, so I think I'll wait for you to send those. :)\n\nThanks,\nJonathan\n"},{"id":"449206","messageId":"YhWbWOd6PeF1RZw1@google.com","threadId":"57444","inReplyTo":"pull.1153.git.1645333542011.gitgitgadget@gmail.com","subject":"[PATCH v2] repo_read_index: add config to expect files outside sparse patterns","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-23T02:26:32Z","receivedAt":"2022-02-23T02:26:40Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"From: Elijah Newren \n\nTypically with sparse checkouts, we expect files outside the sparsity\npatterns to be marked as SKIP_WORKTREE and be missing from the working\ntree. In edge cases, this can be violated and cause confusion, so in\na sparse checkout, since 11d46a399d (\"repo_read_index: clear\nSKIP_WORKTREE bit from files present in worktree\", 2022-01-06), Git\nautomatically clears the SKIP_WORKTREE bit at read time for entries\ncorresponding to files that are present in the working tree.\n\nHowever, there is a more atypical situation where this situation would\nbe expected. A Git-aware virtual file system[1] takes advantage of\nits position as a file system driver to expose all files in the\nworking tree, fetch them on demand using partial clone on access, and\ntell Git to pay attention to them on demand by updating the sparse\ncheckout pattern on writes. This means that commands like \"git\nstatus\" only has to examine files that have potentially been modified,\nwhereas commands like \"ls\" are able to show the entire codebase\nwithout requiring manual updates to the sparse checkout pattern.\n\nThus since 11d46a399d, Git with such Git-aware virtual file systems\nunsets the SKIP_WORKTREE bit for all files and commands like \"git\nstatus\" have to fetch and examine them all.\n\nIntroduce a configuration setting sparse.expectFilesOutsideOfPatterns\nto allow limiting the tracked set of files to a small set once again.\nA Git-aware virtual file system or other application that wants to\nmaintain files outside of the sparse checkout can set this in a\nrepository to instruct Git not to check for the presence of\nSKIP_WORKTREE files. The setting defaults to false, so most users of\nsparse checkout will still get the benefit of an automatically\nupdating index to recover from interrupted updates that forget to\ndelete some files or unset SKIP_WORKTREE for them.\n\n[1] such as the vfsd described in\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\n[jn: fleshed out commit message and documentation, added missing\n include to config.txt, moved to a separate config callback]\n\nSigned-off-by: Elijah Newren \nSigned-off-by: Jonathan Nieder \n---\nSome minor updates, but this is basically the same as the patch you\nsent. Thoughts?\n\nThanks,\nJonathan\n\n Documentation/config.txt | 2 ++\n Documentation/config/sparse.txt | 24 ++++++++++++++++++++++++\n cache.h | 1 +\n config.c | 14 ++++++++++++++\n environment.c | 1 +\n sparse-index.c | 3 ++-\n t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n 7 files changed, 63 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/sparse.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b168f02dc3d..8628ae2634d 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -468,6 +468,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 00000000000..c790c728276\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,24 @@\n+sparse.expectFilesOutsideOfPatterns::\n+\tTypically with sparse checkouts, files not matching any\n+\tsparsity patterns are marked as such in the index file and\n+\tmissing from the working tree. Accordingly, Git will\n+\tordinarily check whether files that the index indicates are\n+\toutside of the sparse area are present in the working tree and\n+\tmark them as present in the index if so. This option can be\n+\tused to tell Git that such present-but-unmatching files are\n+\texpected and to stop checking for them.\n++\n+The default is `false`. Leaving this set to `false` is recommended in\n+most situations because it allows Git to recover from an interrupted\n+operation that updated the working tree without updating the index or\n+vice versa.\n++\n+A Git-based virtual file system (VFS) can turn the usual expectation\n+on its head: files are present in the working copy but do not take\n+up much disk space because their contents are not downloaded until\n+they are accessed. With such a virtual file system layer, most files\n+do not match the sparsity patterns at first, and the VFS layer\n+updates the sparsity patterns to add more files whenever files are\n+written. Setting this to `true` supports such a setup where files are\n+expected to be present outside the sparse area and a separate, robust\n+mechanism is responsible for keeping the sparsity patterns up to date.\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b1..b6b8e83ae35 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\n+extern int sparse_expect_files_outside_of_patterns;\n \n /*\n * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a01..9b9ad1500aa 100644\n--- a/config.c\n+++ b/config.c\n@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \treturn platform_core_config(var, value, cb);\n }\n \n+static int git_default_sparse_config(const char *var, const char *value)\n+{\n+\tif (!strcmp(var, \"sparse.expectfilesoutsideofpatterns\")) {\n+\t\tsparse_expect_files_outside_of_patterns = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* Add other config variables here and to Documentation/config/sparse.txt. */\n+\treturn 0;\n+}\n+\n static int git_default_i18n_config(const char *var, const char *value)\n {\n \tif (!strcmp(var, \"i18n.commitencoding\"))\n@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (starts_with(var, \"sparse.\"))\n+\t\treturn git_default_sparse_config(var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a5..fb55bf61290 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -70,6 +70,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int core_sparse_checkout_cone;\n+int sparse_expect_files_outside_of_patterns;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/sparse-index.c b/sparse-index.c\nindex eed170cd8f7..daeb5112a18 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n \n \tint i;\n \n-\tif (!core_apply_sparse_checkout)\n+\tif (!core_apply_sparse_checkout ||\n+\t sparse_expect_files_outside_of_patterns)\n \t\treturn;\n \n restart:\ndiff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\nindex 3deb4901874..d1833c0f31b 100755\n--- a/t/t1090-sparse-checkout-scope.sh\n+++ b/t/t1090-sparse-checkout-scope.sh\n@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '\n \ttest \"$(cat b)\" = \"modified\"\n '\n \n+test_expect_success 'skip-worktree on files outside sparse patterns' '\n+\tgit sparse-checkout disable &&\n+\tgit sparse-checkout set --no-cone \"a*\" &&\n+\tgit checkout-index --all --ignore-skip-worktree-bits &&\n+\n+\tgit ls-files -t >output &&\n+\t! grep ^S output >actual &&\n+\ttest_must_be_empty actual &&\n+\n+\ttest_config sparse.expectFilesOutsideOfPatterns true &&\n+\tcat <<-\\EOF >expect &&\n+\tS b\n+\tS c\n+\tEOF\n+\tgit ls-files -t >output &&\n+\tgrep ^S output >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '\n \ttest_create_repo server &&\n \tgit clone \"file://$(pwd)/server\" client &&\n-- \n2.35.1.574.g5d30c73bfb\n\n"},{"id":"449207","messageId":"CABPp-BHv4VVyPv1v-pgPXf_u=6EpRhcn7Cm551o2HEbdEthAaw@mail.gmail.com","threadId":"57444","inReplyTo":"YhWbWOd6PeF1RZw1@google.com","subject":"Re: [PATCH v2] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-23T03:10:46Z","receivedAt":"2022-02-23T03:11:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Feb 22, 2022 at 6:26 PM Jonathan Nieder wrote:\n>\n> From: Elijah Newren \n>\n> Typically with sparse checkouts, we expect files outside the sparsity\n> patterns to be marked as SKIP_WORKTREE and be missing from the working\n> tree. In edge cases, this can be violated and cause confusion, so in\n> a sparse checkout, since 11d46a399d (\"repo_read_index: clear\n> SKIP_WORKTREE bit from files present in worktree\", 2022-01-06), Git\n> automatically clears the SKIP_WORKTREE bit at read time for entries\n> corresponding to files that are present in the working tree.\n>\n> However, there is a more atypical situation where this situation would\n> be expected. A Git-aware virtual file system[1] takes advantage of\n> its position as a file system driver to expose all files in the\n> working tree, fetch them on demand using partial clone on access, and\n> tell Git to pay attention to them on demand by updating the sparse\n> checkout pattern on writes. This means that commands like \"git\n> status\" only has to examine files that have potentially been modified,\n> whereas commands like \"ls\" are able to show the entire codebase\n> without requiring manual updates to the sparse checkout pattern.\n\nShould that be s/commands/a command/ or else s/only has/only have/?\n\n> Thus since 11d46a399d, Git with such Git-aware virtual file systems\n> unsets the SKIP_WORKTREE bit for all files and commands like \"git\n> status\" have to fetch and examine them all.\n>\n> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns\n> to allow limiting the tracked set of files to a small set once again.\n> A Git-aware virtual file system or other application that wants to\n> maintain files outside of the sparse checkout can set this in a\n> repository to instruct Git not to check for the presence of\n> SKIP_WORKTREE files. The setting defaults to false, so most users of\n> sparse checkout will still get the benefit of an automatically\n> updating index to recover from interrupted updates that forget to\n\nPlease don't presume that these only come from interrupted updates.\nAs per the referenced 11d46a399d that started all this:\n\n\"\"\"\n There are various ways for users to get files to be present in the\n working copy despite having the SKIP_WORKTREE bit set for that file in\n the index. This may come from:\n * various git commands not really supporting the SKIP_WORKTREE bit[1,2]\n * users grabbing files from elsewhere and writing them to the worktree\n (perhaps even cached in their editor)\n * users attempting to \"abort\" a sparse-checkout operation with a\n not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n working tree is not atomic)[3].\n\"\"\"\n\n> delete some files or unset SKIP_WORKTREE for them.\n\nThe only problem 11d46a399d corrects is having the SKIP_WORKTREE being\n*set* despite the file being present. So the \"recover from...updates\nthat...unset SKIP_WORKTREE\" doesn't make any sense.\n\n> [1] such as the vfsd described in\n> https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n>\n> [jn: fleshed out commit message and documentation, added missing\n> include to config.txt, moved to a separate config callback]\n>\n> Signed-off-by: Elijah Newren \n> Signed-off-by: Jonathan Nieder \n> ---\n> Some minor updates, but this is basically the same as the patch you\n> sent. Thoughts?\n\n...and it looks even more like the v2 I was about to send, so it seems\nlike we're pretty much on the same page. :-)\n\nOverall, it looks good, and your version has some things that are\nnicer than mine, but I did have a couple small notes on the commit\nmessage above and a similar one on the config description below.\n\n> Thanks,\n> Jonathan\n>\n> Documentation/config.txt | 2 ++\n> Documentation/config/sparse.txt | 24 ++++++++++++++++++++++++\n> cache.h | 1 +\n> config.c | 14 ++++++++++++++\n> environment.c | 1 +\n> sparse-index.c | 3 ++-\n> t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n> 7 files changed, 63 insertions(+), 1 deletion(-)\n> create mode 100644 Documentation/config/sparse.txt\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index b168f02dc3d..8628ae2634d 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -468,6 +468,8 @@ include::config/sequencer.txt[]\n>\n> include::config/showbranch.txt[]\n>\n> +include::config/sparse.txt[]\n> +\n> include::config/splitindex.txt[]\n>\n> include::config/ssh.txt[]\n> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> new file mode 100644\n> index 00000000000..c790c728276\n> --- /dev/null\n> +++ b/Documentation/config/sparse.txt\n> @@ -0,0 +1,24 @@\n> +sparse.expectFilesOutsideOfPatterns::\n> + Typically with sparse checkouts, files not matching any\n> + sparsity patterns are marked as such in the index file and\n> + missing from the working tree. Accordingly, Git will\n> + ordinarily check whether files that the index indicates are\n> + outside of the sparse area are present in the working tree and\n> + mark them as present in the index if so. This option can be\n> + used to tell Git that such present-but-unmatching files are\n> + expected and to stop checking for them.\n> ++\n> +The default is `false`. Leaving this set to `false` is recommended in\n> +most situations because it allows Git to recover from an interrupted\n> +operation that updated the working tree without updating the index or\n> +vice versa.\n\nAgain, please don't claim this is only for recovering from interrupted\noperations; there are other cases -- Git commands that write the\nworking tree but not the index (checkout-index, git apply), and users\nmucking around with files, for example.\n\n> ++\n> +A Git-based virtual file system (VFS) can turn the usual expectation\n> +on its head: files are present in the working copy but do not take\n> +up much disk space because their contents are not downloaded until\n> +they are accessed. With such a virtual file system layer, most files\n> +do not match the sparsity patterns at first, and the VFS layer\n> +updates the sparsity patterns to add more files whenever files are\n> +written. Setting this to `true` supports such a setup where files are\n> +expected to be present outside the sparse area and a separate, robust\n> +mechanism is responsible for keeping the sparsity patterns up to date.\n> diff --git a/cache.h b/cache.h\n> index 281f00ab1b1..b6b8e83ae35 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n>\n> extern int core_apply_sparse_checkout;\n> extern int core_sparse_checkout_cone;\n> +extern int sparse_expect_files_outside_of_patterns;\n>\n> /*\n> * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\n> diff --git a/config.c b/config.c\n> index 2bffa8d4a01..9b9ad1500aa 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> return platform_core_config(var, value, cb);\n> }\n>\n> +static int git_default_sparse_config(const char *var, const char *value)\n> +{\n> + if (!strcmp(var, \"sparse.expectfilesoutsideofpatterns\")) {\n> + sparse_expect_files_outside_of_patterns = git_config_bool(var, value);\n> + return 0;\n> + }\n> +\n> + /* Add other config variables here and to Documentation/config/sparse.txt. */\n> + return 0;\n> +}\n> +\n> static int git_default_i18n_config(const char *var, const char *value)\n> {\n> if (!strcmp(var, \"i18n.commitencoding\"))\n> @@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)\n> return 0;\n> }\n>\n> + if (starts_with(var, \"sparse.\"))\n> + return git_default_sparse_config(var, value);\n> +\n> /* Add other config variables here and to Documentation/config.txt. */\n> return 0;\n> }\n> diff --git a/environment.c b/environment.c\n> index fd0501e77a5..fb55bf61290 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -70,6 +70,7 @@ char *notes_ref_name;\n> int grafts_replace_parents = 1;\n> int core_apply_sparse_checkout;\n> int core_sparse_checkout_cone;\n> +int sparse_expect_files_outside_of_patterns;\n> int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> unsigned long pack_size_limit_cfg;\n> diff --git a/sparse-index.c b/sparse-index.c\n> index eed170cd8f7..daeb5112a18 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n>\n> int i;\n>\n> - if (!core_apply_sparse_checkout)\n> + if (!core_apply_sparse_checkout ||\n> + sparse_expect_files_outside_of_patterns)\n> return;\n>\n> restart:\n> diff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\n> index 3deb4901874..d1833c0f31b 100755\n> --- a/t/t1090-sparse-checkout-scope.sh\n> +++ b/t/t1090-sparse-checkout-scope.sh\n> @@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '\n> test \"$(cat b)\" = \"modified\"\n> '\n>\n> +test_expect_success 'skip-worktree on files outside sparse patterns' '\n> + git sparse-checkout disable &&\n> + git sparse-checkout set --no-cone \"a*\" &&\n> + git checkout-index --all --ignore-skip-worktree-bits &&\n> +\n> + git ls-files -t >output &&\n> + ! grep ^S output >actual &&\n> + test_must_be_empty actual &&\n> +\n> + test_config sparse.expectFilesOutsideOfPatterns true &&\n> + cat <<-\\EOF >expect &&\n> + S b\n> + S c\n> + EOF\n> + git ls-files -t >output &&\n> + grep ^S output >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '\n> test_create_repo server &&\n> git clone \"file://$(pwd)/server\" client &&\n> --\n> 2.35.1.574.g5d30c73bfb\n"},{"id":"449388","messageId":"20220224052259.30498-1-newren@gmail.com","threadId":"57444","inReplyTo":"YhWbWOd6PeF1RZw1@google.com","subject":"[PATCH v3] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-24T05:22:59Z","receivedAt":"2022-02-24T05:23:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Typically with sparse checkouts, we expect files outside the sparsity\npatterns to be marked as SKIP_WORKTREE and be missing from the working\ntree. In edge cases, this can be violated and cause confusion, so in a\nsparse checkout, since 11d46a399d (\"repo_read_index: clear SKIP_WORKTREE\nbit from files present in worktree\", 2022-01-06), Git automatically\nclears the SKIP_WORKTREE bit at read time for entries corresponding to\nfiles that are present in the working tree.\n\nHowever, there is a more atypical situation where this situation would\nbe expected. A Git-aware virtual file system[1] takes advantage of its\nposition as a file system driver to expose all files in the working\ntree, fetch them on demand using partial clone on access, and tell Git\nto pay attention to them on demand by updating the sparse checkout\npattern on writes. This means that commands like \"git status\" only have\nto examine files that have potentially been modified, whereas commands\nlike \"ls\" are able to show the entire codebase without requiring manual\nupdates to the sparse checkout pattern.\n\nThus since 11d46a399d, Git with such Git-aware virtual file systems\nunsets the SKIP_WORKTREE bit for all files and commands like \"git\nstatus\" have to fetch and examine them all.\n\nIntroduce a configuration setting sparse.expectFilesOutsideOfPatterns to\nallow limiting the tracked set of files to a small set once again. A\nGit-aware virtual file system or other application that wants to\nmaintain files outside of the sparse checkout can set this in a\nrepository to instruct Git not to check for the presence of\nSKIP_WORKTREE files. The setting defaults to false, so most users of\nsparse checkout will still get the benefit of an automatically updating\nindex to recover from the variety of difficult issues detailed in\n11d46a399d for paths with SKIP_WORKTREE set despite the path being\npresent.\n\n[1] such as the vfsd described in\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\nHelped-by: Jonathan Nieder \nSigned-off-by: Jonathan Nieder \nSigned-off-by: Elijah Newren \n---\nChanges since v2:\n * Made tweaks to the commit message and the text of the config option as\n highlighted in my response to Jonathan's v2.\n\nI'm guessing that since there are no code (only documentation) changes since\nJonathan's v2 submission, that this patch satisfies vfsd/Google's needs.\nI'm also guessing it matches what Stolee and Dscho stated in their comments\non v1. But it'd be nice to have an ack from each side just to make sure.\n \n Documentation/config.txt | 2 ++\n Documentation/config/sparse.txt | 28 ++++++++++++++++++++++++++++\n cache.h | 1 +\n config.c | 14 ++++++++++++++\n environment.c | 1 +\n sparse-index.c | 3 ++-\n t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n 7 files changed, 67 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/sparse.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b168f02dc3..8628ae2634 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -468,6 +468,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..fba504173c\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,28 @@\n+sparse.expectFilesOutsideOfPatterns::\n+\tTypically with sparse checkouts, files not matching any\n+\tsparsity patterns are marked as such in the index file and\n+\tmissing from the working tree. Accordingly, Git will\n+\tordinarily check whether files that the index indicates are\n+\toutside of the sparse area are present in the working tree and\n+\tmark them as present in the index if so. This option can be\n+\tused to tell Git that such present-but-unmatching files are\n+\texpected and to stop checking for them.\n++\n+The default is `false`. Paths which are marked as SKIP_WORKTREE\n+despite being present (which can occur for a few different reasons)\n+typically present a range of problems which are difficult for users to\n+discover and recover from. The default setting avoids such issues.\n++\n+A Git-based virtual file system (VFS) can turn the usual expectation\n+on its head: files are present in the working copy but do not take\n+up much disk space because their contents are not downloaded until\n+they are accessed. With such a virtual file system layer, most files\n+do not match the sparsity patterns at first, and the VFS layer\n+updates the sparsity patterns to add more files whenever files are\n+written. Setting this to `true` supports such a setup where files are\n+expected to be present outside the sparse area and a separate, robust\n+mechanism is responsible for keeping the sparsity patterns up to date.\n++\n+Note that the checking and clearing of the SKIP_WORKTREE bit only\n+happens when core.sparseCheckout is true, so this config option has no\n+effect unless core.sparseCheckout is true.\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b..b6b8e83ae3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\n+extern int sparse_expect_files_outside_of_patterns;\n \n /*\n * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a0..9b9ad1500a 100644\n--- a/config.c\n+++ b/config.c\n@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \treturn platform_core_config(var, value, cb);\n }\n \n+static int git_default_sparse_config(const char *var, const char *value)\n+{\n+\tif (!strcmp(var, \"sparse.expectfilesoutsideofpatterns\")) {\n+\t\tsparse_expect_files_outside_of_patterns = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* Add other config variables here and to Documentation/config/sparse.txt. */\n+\treturn 0;\n+}\n+\n static int git_default_i18n_config(const char *var, const char *value)\n {\n \tif (!strcmp(var, \"i18n.commitencoding\"))\n@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (starts_with(var, \"sparse.\"))\n+\t\treturn git_default_sparse_config(var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a..fb55bf6129 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -70,6 +70,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int core_sparse_checkout_cone;\n+int sparse_expect_files_outside_of_patterns;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/sparse-index.c b/sparse-index.c\nindex eed170cd8f..daeb5112a1 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n \n \tint i;\n \n-\tif (!core_apply_sparse_checkout)\n+\tif (!core_apply_sparse_checkout ||\n+\t sparse_expect_files_outside_of_patterns)\n \t\treturn;\n \n restart:\ndiff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\nindex 3deb490187..d1833c0f31 100755\n--- a/t/t1090-sparse-checkout-scope.sh\n+++ b/t/t1090-sparse-checkout-scope.sh\n@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '\n \ttest \"$(cat b)\" = \"modified\"\n '\n \n+test_expect_success 'skip-worktree on files outside sparse patterns' '\n+\tgit sparse-checkout disable &&\n+\tgit sparse-checkout set --no-cone \"a*\" &&\n+\tgit checkout-index --all --ignore-skip-worktree-bits &&\n+\n+\tgit ls-files -t >output &&\n+\t! grep ^S output >actual &&\n+\ttest_must_be_empty actual &&\n+\n+\ttest_config sparse.expectFilesOutsideOfPatterns true &&\n+\tcat <<-\\EOF >expect &&\n+\tS b\n+\tS c\n+\tEOF\n+\tgit ls-files -t >output &&\n+\tgrep ^S output >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '\n \ttest_create_repo server &&\n \tgit clone \"file://$(pwd)/server\" client &&\n-- \n2.34.0.22.g587c05af5b.dirty\n\n"},{"id":"449491","messageId":"xmqq35k8nmwj.fsf@gitster.g","threadId":"57444","inReplyTo":"20220224052259.30498-1-newren@gmail.com","subject":"Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T18:24:12Z","receivedAt":"2022-02-24T18:24:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Typically with sparse checkouts, we expect files outside the sparsity\n> patterns to be marked as SKIP_WORKTREE and be missing from the working\n> tree. In edge cases, this can be violated and cause confusion, so in a\n> sparse checkout, since 11d46a399d (\"repo_read_index: clear SKIP_WORKTREE\n\nI think this refers to af6a5187 (repo_read_index: clear\nSKIP_WORKTREE bit from files present in worktree, 2022-01-14).\n\n> bit from files present in worktree\", 2022-01-06), Git automatically\n> clears the SKIP_WORKTREE bit at read time for entries corresponding to\n> files that are present in the working tree.\n\nSo, this is a workflow where the user deliberately \"creates\" these\nfiles outside the sparsity cone or pattern (by various non-automated\nmeans like editing, copying/renaming, or untarring). If they did so\non purpose, they may be interested in comparing them with existing\ncommits, or even including them as a newer version in the next\ncommit they create. To help that workflow, clearing the bit makes\nsense.\n\nAm I on the right path? I am wondering if mentioning some of that\nwould help understanding by the reader when it is contrasted with\nthe (competing) goal of supporting VFS use case mentioned next.\n\n> However, there is a more atypical situation where this situation would\n\nI wonder if that is \"more atypical\" (read: makes me wonder if it\ndepends on who the reader is what is typoical), and more importantly,\nif it helps understanding of the reader (read: whether which one is\nmore common, we'd want to support both camps anyway). \n\n There is another workflow, however, that it is expected that\n paths outside the sparsity patterns appear to exist in the\n working tree and that they do not lose the SKIP_WORKTREE bit, at\n least until they get modified.\n\nor something?\n\n> be expected. A Git-aware virtual file system[1] takes advantage of its\n> position as a file system driver to expose all files in the working\n> tree, fetch them on demand using partial clone on access, and tell Git\n> to pay attention to them on demand by updating the sparse checkout\n> pattern on writes. This means that commands like \"git status\" only have\n> to examine files that have potentially been modified, whereas commands\n> like \"ls\" are able to show the entire codebase without requiring manual\n> updates to the sparse checkout pattern.\n\nWell explained.\n\n> Thus since 11d46a399d, Git with such Git-aware virtual file systems\n\nThe same stale reference.\n\n> unsets the SKIP_WORKTREE bit for all files and commands like \"git\n> status\" have to fetch and examine them all.\n>\n> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to\n> allow limiting the tracked set of files to a small set once again. A\n> Git-aware virtual file system or other application that wants to\n> maintain files outside of the sparse checkout can set this in a\n> repository to instruct Git not to check for the presence of\n> SKIP_WORKTREE files. The setting defaults to false, so most users of\n> sparse checkout will still get the benefit of an automatically updating\n> index to recover from the variety of difficult issues detailed in\n> 11d46a399d for paths with SKIP_WORKTREE set despite the path being\n\nDitto.\n\n> I'm guessing that since there are no code (only documentation) changes since\n> Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs.\n> I'm also guessing it matches what Stolee and Dscho stated in their comments\n> on v1. But it'd be nice to have an ack from each side just to make sure.\n\nTrue. Let me queue but leave it just outside 'next' until that\nhappens.\n\nI think the name of the knob is what Jonathan suggested, so I\npresume that their side would be fine with it, but I am curious (I\ndo not wonder, though) what the plan on the Microsoft's side going\nforward. When they update the version of Git bundled in their vfsd,\nwould this be reverted and an equivalent they have (and they may\nhave more such \"workaround\" in other areas as well?) will be kept,\nso whatever we do here will add a minor inconvenience to them but\nwill not hurt them otherwise?\n\n> diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> new file mode 100644\n> index 0000000000..fba504173c\n> --- /dev/null\n> +++ b/Documentation/config/sparse.txt\n> @@ -0,0 +1,28 @@\n> +sparse.expectFilesOutsideOfPatterns::\n> +\tTypically with sparse checkouts, files not matching any\n> +\tsparsity patterns are marked as such in the index file and\n\ns/index file/index/ perhaps.\n\n> +\tmissing from the working tree. Accordingly, Git will\n> +\tordinarily check whether files that the index indicates are\n> +\toutside of the sparse area are present in the working tree and\n> +\tmark them as present in the index if so. This option can be\n\nJust an observation. According to this sentence, \"sparse area\" is\n\"paths that ought to be present in the working tree\", so paths\n\"outside of the sparse area\" that are present need to be corrected\nto be \"in\" the sparse area by futzing bits. I always get confused\nwhen I hear \"sparse area\" if the author meant \"paths that ought to\nbe missing\" or \"present\", but maybe it is just me.\n\n> +\tused to tell Git that such present-but-unmatching files are\n> +\texpected and to stop checking for them.\n\nOK.\n\n> ++\n> +The default is `false`. Paths which are marked as SKIP_WORKTREE\n> +despite being present (which can occur for a few different reasons)\n> +typically present a range of problems which are difficult for users to\n\ns/typically // perhaps.\n\n> +discover and recover from. The default setting avoids such issues.\n> ++\n> +A Git-based virtual file system (VFS) can turn the usual expectation\n> +on its head: files are present in the working copy but do not take\n> +up much disk space because their contents are not downloaded until\n> +they are accessed. With such a virtual file system layer, most files\n> +do not match the sparsity patterns at first, and the VFS layer\n> +updates the sparsity patterns to add more files whenever files are\n> +written. Setting this to `true` supports such a setup where files are\n> +expected to be present outside the sparse area and a separate, robust\n> +mechanism is responsible for keeping the sparsity patterns up to date.\n\ns/separate, robust/separate/ I would think.\n\nWe make the outside mechanism that makes these files appear to be\npresent to also be responsible for maintaining the sparse bit and\npatterns.\n\nWhen the user (or IDE) sets this knob to 'true', do we even have to\nexpect that files appear to be present? In the use case we intend\nto support with this feature, i.e. some VFS, we might expect all\npaths to appear to be present, but if that VFS also allows users to\nconfigure to expose only a subset of paths, not all paths may appear\nto be present. And we are perfectly OK with that, becuase we do not\nexpect anything about the working tree paths outside the sparsity\npattern. Am I mistaken?\n\nSo, \"... supports such a setup where some external system releaves\nus of the responsibility of maintaining the consistency between the\npresence of working tree files and sparsity patterns, so we stop\nexpecting whether files are present or missing outside the sparse\narea\", might be closer to the truth?\n\n> +Note that the checking and clearing of the SKIP_WORKTREE bit only\n> +happens when core.sparseCheckout is true, so this config option has no\n> +effect unless core.sparseCheckout is true.\n\nGood note to have. There is no mention of \"cone\" mode in the entire\ndescription; it is unclear if this only applies to \"pattern\" mode or\nto both \"pattern\" and \"cone\" modes, which may want to be clarified.\n\nThanks.\n"},{"id":"449597","messageId":"YhkE2vxI4nM3ut0K@google.com","threadId":"57444","inReplyTo":"20220224052259.30498-1-newren@gmail.com","subject":"Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2022-02-25T16:33:30Z","receivedAt":"2022-02-25T16:33:38Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nElijah Newren wrote:\n\n> Signed-off-by: Elijah Newren \n\nReviewed-by: Jonathan Nieder \n\nThanks, and sorry for the slow review. My one remaining area for nits\nis the documentation, but that can be improved iteratively via patches\non top.\n\n[...]\n> --- /dev/null\n> +++ b/Documentation/config/sparse.txt\n> @@ -0,0 +1,28 @@\n> +sparse.expectFilesOutsideOfPatterns::\n> +\tTypically with sparse checkouts, files not matching any\n> +\tsparsity patterns are marked as such in the index file and\n> +\tmissing from the working tree. Accordingly, Git will\n> +\tordinarily check whether files that the index indicates are\n> +\toutside of the sparse area are present in the working tree and\n\nJunio mentioned the \"sparse area\" could suggest that the area is\nitself sparse and devoid of files, so it might not have been the best\nchoice of words on my part. Perhaps \"whether files that the index\nindicates are not checked out are present in the working tree\" would\nwork here?\n\n> +\tmark them as present in the index if so. This option can be\n> +\tused to tell Git that such present-but-unmatching files are\n> +\texpected and to stop checking for them.\n> ++\n> +The default is `false`. Paths which are marked as SKIP_WORKTREE\n> +despite being present (which can occur for a few different reasons)\n> +typically present a range of problems which are difficult for users to\n> +discover and recover from. The default setting avoids such issues.\n\nThe git-sparse-checkout(1) page never describes what SKIP_WORKTREE\nmeans, so it might not be obvious to them what this means. Also, the\n\"can occur for a few different reasons\" may leave the user wondering\nwhether they are subject to those reasons. What the reader wants to\nknow is \"I should keep using the default because it makes Git work\nbetter\", so how about something like\n\n The default is `false`, which allows Git to automatically recover\n from the list of files in the index and working tree falling out of\n sync.\n +\n\n?\n\n> ++\n> +A Git-based virtual file system (VFS) can turn the usual expectation\n> +on its head: files are present in the working copy but do not take\n> +up much disk space because their contents are not downloaded until\n> +they are accessed. With such a virtual file system layer, most files\n> +do not match the sparsity patterns at first, and the VFS layer\n> +updates the sparsity patterns to add more files whenever files are\n> +written. Setting this to `true` supports such a setup where files are\n> +expected to be present outside the sparse area and a separate, robust\n> +mechanism is responsible for keeping the sparsity patterns up to date.\n\nHere I spent most of the words explaining what a Git-based VFS layer\nis, which is also not too relevant to most users (who are just\ninterested in \"is `true` the right value for me?\"). How about\nreducing it to the following?\n\n Set this to `true` if you are in a setup where extra files are expected\n to be present and a separate, robust mechanism is responsible for\n keeping the sparsity patterns up to date, such as a Git-aware virtual\n file system.\n\n?\n\n> ++\n> +Note that the checking and clearing of the SKIP_WORKTREE bit only\n> +happens when core.sparseCheckout is true, so this config option has no\n> +effect unless core.sparseCheckout is true.\n\nGood note. Same nit about the user not necessarily knowing what\nSKIP_WORKTREE means applies. Also, we can remove the extra words\n\"Note that\" since the dutiful reader should be noting everything we\nsay. :) I think that would make\n\n +\n Regardless of this setting, Git does not check for\n present-but-unmatching files unless sparse checkout is enabled, so\n this config option has no effect unless `core.sparseCheckout` is\n `true`.\n\nThanks,\nJonathan\n"},{"id":"449599","messageId":"220225.86r17q7v9g.gmgdl@evledraar.gmail.com","threadId":"57444","inReplyTo":"YhVuCfDgFF/K5Gtl@google.com","subject":"Re: [PATCH] Provide config option to expect files outside sparse patterns","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-25T16:39:43Z","receivedAt":"2022-02-25T16:42:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 22 2022, Jonathan Nieder wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>\n>> To a first approximation what makes it core.* is that we have a\n>> core.sparseCheckout.\n> [...]\n>> Anyway, as noted (and I feel I have to comment as the originator of this\n>> thread) I think the bikeshedding about namespacing is less important\n>> than making sure from a user perspective that these are discoverable.\n>\n> Agreed. Though there's already also advice.updateSparsePath; even\n> though I suspect that ctrl+F for \"sparse\" is what people may already\n> be using in practice, a pointer from the git-sparse-checkout(1) page\n> would be welcome.\n>\n>> E.g. core.sshCommand doesn't reference ssh.variant or the other way\n>> around, and in my \"man git-config\" they're at ~25% and ~90% in to what's\n>> now a *huge* document. You need to read the whole thing or have the\n>> foresight to search through it to discover both.\n>>\n>> So for core.sshCommand and ssh.variant, and core.sparse* and sparse.*\n>> cross-linking etc. would go a long way...\n>\n> Agreed as well.\n>\n>> Aside: I do have some local patches (waiting on a couple of other things\n>> I have outstanding) to add a \"CONFIGURATION\" section to all of the\n>> built-in commands with some including/splitting up of\n>> Documentation/config**.txt, so \"man git-sparse-checkout\" would have a\n>> section listing the subset of \"git-config\" applicable to the command.\n>\n> Oh! I was about to do a quick patch for git-sparse-checkout.txt, but\n> this is even better, so I think I'll wait for you to send those. :)\n\nI think if you're interested in fixing just that case you shouldn't let\nme stop you.\n\nThe thing I have is currently dependant on [1], so a review would speed\nit up. I.e. needing to create new sub-dirs in Documentation/config that\nwe lint etc. needs some of those Makefile bits (or rather, is a lot\neasier with them).\n\n1. https://lore.kernel.org/git/cover-v2-0.8-00000000000-20211224T173558Z-avarab@gmail.com/\n\nThe diff for the ad-hoc series I have following that locally follows, to\ngive you some idea.\n\nI hacked it up before there was git-sparse*, so if I submit it I'd\nrevisit that, and I think even for the below I stopped at some point and\nwould need to finish it.\n\n Documentation/Makefile | 2 +\n Documentation/config/color.txt | 155 ++-------------------\n Documentation/config/color/blame.txt | 22 +++\n Documentation/config/color/branch.txt | 13 ++\n Documentation/config/color/diff.txt | 27 ++++\n Documentation/config/color/grep.txt | 70 ++++++++++\n Documentation/config/color/log.txt | 9 ++\n Documentation/config/color/push.txt | 8 ++\n Documentation/config/color/showbranch.txt | 6 +\n Documentation/config/color/status.txt | 20 +++\n Documentation/config/core.txt | 13 +-\n Documentation/config/core/bigFileThreshold.txt | 12 ++\n Documentation/config/diff.txt | 15 --\n Documentation/config/diff/difftool.txt | 14 ++\n Documentation/config/difftool.txt | 7 +\n Documentation/config/gc.txt | 12 +-\n Documentation/config/gc/rerere.txt | 11 ++\n Documentation/config/grep.txt | 7 +-\n Documentation/config/log.txt | 18 ++-\n Documentation/config/notes.txt | 43 +++---\n Documentation/config/sendemail.txt | 40 +++++-\n Documentation/config/uploadpack.txt | 26 +---\n Documentation/config/uploadpack/pack-objects.txt | 25 ++++\n Documentation/git-add.txt | 7 +\n Documentation/git-am.txt | 7 +\n Documentation/git-apply.txt | 10 +-\n Documentation/git-blame.txt | 10 ++\n Documentation/git-branch.txt | 6 +\n Documentation/git-checkout.txt | 7 +\n Documentation/git-clean.txt | 7 +\n Documentation/git-clone.txt | 9 ++\n Documentation/git-commit-graph.txt | 8 ++\n Documentation/git-commit.txt | 4 +\n Documentation/git-credential.txt | 7 +\n Documentation/git-diff.txt | 9 ++\n Documentation/git-difftool.txt | 28 +---\n Documentation/git-fast-import.txt | 7 +\n Documentation/git-fetch.txt | 9 ++\n Documentation/git-gc.txt | 14 +-\n Documentation/git-grep.txt | 29 +---\n Documentation/git-log.txt | 50 ++-----\n Documentation/git-mergetool.txt | 3 +\n Documentation/git-notes.txt | 54 +------\n Documentation/git-pack-objects.txt | 8 +-\n Documentation/git-remote.txt | 15 +-\n Documentation/git-repack.txt | 19 +--\n Documentation/git-rerere.txt | 9 ++\n Documentation/git-reset.txt | 7 +\n Documentation/git-send-email.txt | 36 +----\n Documentation/git-show-branch.txt | 7 +\n Documentation/githooks.txt | 4 +\n Documentation/includes/cmd-config-section-all.txt | 3 +\n Documentation/includes/cmd-config-section-rest.txt | 3 +\n .../templates/configuration-include-pack.txt | 10 ++\n builtin/help.c | 2 +-\n builtin/log.c | 3 +-\n generate-configlist.sh | 2 +-\n t/t4207-log-decoration-colors.sh | 12 +-\n 58 files changed, 544 insertions(+), 456 deletions(-)\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex 1eb9192dae8..15b71f24735 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -50,7 +50,9 @@ MAN7_TXT += gitworkflows.txt\n HOWTO_TXT += $(wildcard howto/*.txt)\n \n DOC_DEP_TXT += $(wildcard *.txt)\n+DOC_DEP_TXT += $(wildcard includes/*.txt)\n DOC_DEP_TXT += $(wildcard config/*.txt)\n+DOC_DEP_TXT += $(wildcard config/*/*.txt)\n \n ifdef MAN_FILTER\n MAN_TXT = $(filter $(MAN_FILTER),$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT))\ndiff --git a/Documentation/config/color.txt b/Documentation/config/color.txt\nindex 1795b2d16be..4ca476b99ac 100644\n--- a/Documentation/config/color.txt\n+++ b/Documentation/config/color.txt\n@@ -8,113 +8,15 @@ color.advice::\n color.advice.hint::\n \tUse customized color for hints.\n \n-color.blame.highlightRecent::\n-\tSpecify the line annotation color for `git blame --color-by-age`\n-\tdepending upon the age of the line.\n-+\n-This setting should be set to a comma-separated list of color and\n-date settings, starting and ending with a color, the dates should be\n-set from oldest to newest. The metadata will be colored with the\n-specified colors if the line was introduced before the given\n-timestamp, overwriting older timestamped colors.\n-+\n-Instead of an absolute timestamp relative timestamps work as well,\n-e.g. `2.weeks.ago` is valid to address anything older than 2 weeks.\n-+\n-It defaults to `blue,12 month ago,white,1 month ago,red`, which\n-colors everything older than one year blue, recent changes between\n-one month and one year old are kept white, and lines introduced\n-within the last month are colored red.\n-\n-color.blame.repeatedLines::\n-\tUse the specified color to colorize line annotations for\n-\t`git blame --color-lines`, if they come from the same commit as the\n-\tpreceding line. Defaults to cyan.\n-\n-color.branch::\n-\tA boolean to enable/disable color in the output of\n-\tlinkgit:git-branch[1]. May be set to `always`,\n-\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n-\tonly when the output is to a terminal. If unset, then the\n-\tvalue of `color.ui` is used (`auto` by default).\n-\n-color.branch.::\n-\tUse customized color for branch coloration. `` is one of\n-\t`current` (the current branch), `local` (a local branch),\n-\t`remote` (a remote-tracking branch in refs/remotes/),\n-\t`upstream` (upstream tracking branch), `plain` (other\n-\trefs).\n-\n-color.diff::\n-\tWhether to use ANSI escape sequences to add color to patches.\n-\tIf this is set to `always`, linkgit:git-diff[1],\n-\tlinkgit:git-log[1], and linkgit:git-show[1] will use color\n-\tfor all patches. If it is set to `true` or `auto`, those\n-\tcommands will only use color when output is to the terminal.\n-\tIf unset, then the value of `color.ui` is used (`auto` by\n-\tdefault).\n-+\n-This does not affect linkgit:git-format-patch[1] or the\n-'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n-command line with the `--color[=]` option.\n-\n-color.diff.::\n-\tUse customized color for diff colorization. `` specifies\n-\twhich part of the patch to use the specified color, and is one\n-\tof `context` (context text - `plain` is a historical synonym),\n-\t`meta` (metainformation), `frag`\n-\t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n-\t`new` (added lines), `commit` (commit headers), `whitespace`\n-\t(highlighting whitespace errors), `oldMoved` (deleted lines),\n-\t`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,\n-\t`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`\n-\t`newMovedAlternativeDimmed` (See the ''\n-\tsetting of '--color-moved' in linkgit:git-diff[1] for details),\n-\t`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,\n-\t`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).\n-\n-color.decorate.::\n-\tUse customized color for 'git log --decorate' output. `` is one\n-\tof `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local\n-\tbranches, remote-tracking branches, tags, stash and HEAD, respectively\n-\tand `grafted` for grafted commits.\n-\n-color.grep::\n-\tWhen set to `always`, always highlight matches. When `false` (or\n-\t`never`), never. When set to `true` or `auto`, use color only\n-\twhen the output is written to the terminal. If unset, then the\n-\tvalue of `color.ui` is used (`auto` by default).\n-\n-color.grep.::\n-\tUse customized color for grep colorization. `` specifies which\n-\tpart of the line to use the specified color, and is one of\n-+\n---\n-`context`;;\n-\tnon-matching text in context lines (when using `-A`, `-B`, or `-C`)\n-`filename`;;\n-\tfilename prefix (when not using `-h`)\n-`function`;;\n-\tfunction name lines (when using `-p`)\n-`lineNumber`;;\n-\tline number prefix (when using `-n`)\n-`column`;;\n-\tcolumn number prefix (when using `--column`)\n-`match`;;\n-\tmatching text (same as setting `matchContext` and `matchSelected`)\n-`matchContext`;;\n-\tmatching text in context lines\n-`matchSelected`;;\n-\tmatching text in selected lines. Also, used to customize the following\n-\tlinkgit:git-log[1] subcommands: `--grep`, `--author` and `--committer`.\n-`selected`;;\n-\tnon-matching text in selected lines. Also, used to customize the\n-\tfollowing linkgit:git-log[1] subcommands: `--grep`, `--author` and\n-\t`--committer`.\n-`separator`;;\n-\tseparators between fields on a line (`:`, `-`, and `=`)\n-\tand between hunks (`--`)\n---\n+include::color/blame.txt[]\n+\n+include::color/branch.txt[]\n+\n+include::color/diff.txt[]\n+\n+include::color/log.txt[]\n+\n+include::color/grep.txt[]\n \n color.interactive::\n \tWhen set to `always`, always use colors for interactive prompts\n@@ -135,14 +37,7 @@ color.pager::\n \toutput going to the pager. Defaults to true; set this to false\n \tif your pager does not understand ANSI color codes.\n \n-color.push::\n-\tA boolean to enable/disable color in push errors. May be set to\n-\t`always`, `false` (or `never`) or `auto` (or `true`), in which\n-\tcase colors are used only when the error output goes to a terminal.\n-\tIf unset, then the value of `color.ui` is used (`auto` by default).\n-\n-color.push.error::\n-\tUse customized color for push errors.\n+include::color/push.txt[]\n \n color.remote::\n \tIf set, keywords at the start of the line are highlighted. The\n@@ -156,33 +51,9 @@ color.remote.::\n \t`hint`, `warning`, `success` or `error` which match the\n \tcorresponding keyword.\n \n-color.showBranch::\n-\tA boolean to enable/disable color in the output of\n-\tlinkgit:git-show-branch[1]. May be set to `always`,\n-\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n-\tonly when the output is to a terminal. If unset, then the\n-\tvalue of `color.ui` is used (`auto` by default).\n-\n-color.status::\n-\tA boolean to enable/disable color in the output of\n-\tlinkgit:git-status[1]. May be set to `always`,\n-\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n-\tonly when the output is to a terminal. If unset, then the\n-\tvalue of `color.ui` is used (`auto` by default).\n-\n-color.status.::\n-\tUse customized color for status colorization. `` is\n-\tone of `header` (the header text of the status message),\n-\t`added` or `updated` (files which are added but not committed),\n-\t`changed` (files which are changed but not added in the index),\n-\t`untracked` (files which are not tracked by Git),\n-\t`branch` (the current branch),\n-\t`nobranch` (the color the 'no branch' warning is shown in, defaulting\n-\tto red),\n-\t`localBranch` or `remoteBranch` (the local and remote branch names,\n-\trespectively, when branch and tracking information is displayed in the\n-\tstatus short-format), or\n-\t`unmerged` (files which have unmerged changes).\n+include::color/showbranch.txt[]\n+\n+include::color/status.txt[]\n \n color.transport::\n \tA boolean to enable/disable color when pushes are rejected. May be\ndiff --git a/Documentation/config/color/blame.txt b/Documentation/config/color/blame.txt\nnew file mode 100644\nindex 00000000000..423290c597b\n--- /dev/null\n+++ b/Documentation/config/color/blame.txt\n@@ -0,0 +1,22 @@\n+color.blame.highlightRecent::\n+\tSpecify the line annotation color for `git blame --color-by-age`\n+\tdepending upon the age of the line.\n++\n+This setting should be set to a comma-separated list of color and\n+date settings, starting and ending with a color, the dates should be\n+set from oldest to newest. The metadata will be colored with the\n+specified colors if the line was introduced before the given\n+timestamp, overwriting older timestamped colors.\n++\n+Instead of an absolute timestamp relative timestamps work as well,\n+e.g. `2.weeks.ago` is valid to address anything older than 2 weeks.\n++\n+It defaults to `blue,12 month ago,white,1 month ago,red`, which\n+colors everything older than one year blue, recent changes between\n+one month and one year old are kept white, and lines introduced\n+within the last month are colored red.\n+\n+color.blame.repeatedLines::\n+\tUse the specified color to colorize line annotations for\n+\t`git blame --color-lines`, if they come from the same commit as the\n+\tpreceding line. Defaults to cyan.\ndiff --git a/Documentation/config/color/branch.txt b/Documentation/config/color/branch.txt\nnew file mode 100644\nindex 00000000000..30bf0c3d108\n--- /dev/null\n+++ b/Documentation/config/color/branch.txt\n@@ -0,0 +1,13 @@\n+color.branch::\n+\tA boolean to enable/disable color in the output of\n+\tlinkgit:git-branch[1]. May be set to `always`,\n+\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n+\tonly when the output is to a terminal. If unset, then the\n+\tvalue of `color.ui` is used (`auto` by default).\n+\n+color.branch.::\n+\tUse customized color for branch coloration. `` is one of\n+\t`current` (the current branch), `local` (a local branch),\n+\t`remote` (a remote-tracking branch in refs/remotes/),\n+\t`upstream` (upstream tracking branch), `plain` (other\n+\trefs).\ndiff --git a/Documentation/config/color/diff.txt b/Documentation/config/color/diff.txt\nnew file mode 100644\nindex 00000000000..609c7a5ca91\n--- /dev/null\n+++ b/Documentation/config/color/diff.txt\n@@ -0,0 +1,27 @@\n+color.diff::\n+\tWhether to use ANSI escape sequences to add color to patches.\n+\tIf this is set to `always`, linkgit:git-diff[1],\n+\tlinkgit:git-log[1], and linkgit:git-show[1] will use color\n+\tfor all patches. If it is set to `true` or `auto`, those\n+\tcommands will only use color when output is to the terminal.\n+\tIf unset, then the value of `color.ui` is used (`auto` by\n+\tdefault).\n++\n+This does not affect linkgit:git-format-patch[1] or the\n+'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n+command line with the `--color[=]` option.\n+\n+color.diff.::\n+\tUse customized color for diff colorization. `` specifies\n+\twhich part of the patch to use the specified color, and is one\n+\tof `context` (context text - `plain` is a historical synonym),\n+\t`meta` (metainformation), `frag`\n+\t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `oldMoved` (deleted lines),\n+\t`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,\n+\t`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`\n+\t`newMovedAlternativeDimmed` (See the ''\n+\tsetting of '--color-moved' in linkgit:git-diff[1] for details),\n+\t`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,\n+\t`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).\ndiff --git a/Documentation/config/color/grep.txt b/Documentation/config/color/grep.txt\nnew file mode 100644\nindex 00000000000..390838450b3\n--- /dev/null\n+++ b/Documentation/config/color/grep.txt\n@@ -0,0 +1,70 @@\n+color.diff::\n+\tWhether to use ANSI escape sequences to add color to patches.\n+\tIf this is set to `always`, linkgit:git-diff[1],\n+\tlinkgit:git-log[1], and linkgit:git-show[1] will use color\n+\tfor all patches. If it is set to `true` or `auto`, those\n+\tcommands will only use color when output is to the terminal.\n+\tIf unset, then the value of `color.ui` is used (`auto` by\n+\tdefault).\n++\n+This does not affect linkgit:git-format-patch[1] or the\n+'git-diff-{asterisk}' plumbing commands. Can be overridden on the\n+command line with the `--color[=]` option.\n+\n+color.diff.::\n+\tUse customized color for diff colorization. `` specifies\n+\twhich part of the patch to use the specified color, and is one\n+\tof `context` (context text - `plain` is a historical synonym),\n+\t`meta` (metainformation), `frag`\n+\t(hunk header), 'func' (function in hunk header), `old` (removed lines),\n+\t`new` (added lines), `commit` (commit headers), `whitespace`\n+\t(highlighting whitespace errors), `oldMoved` (deleted lines),\n+\t`newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`,\n+\t`oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative`\n+\t`newMovedAlternativeDimmed` (See the ''\n+\tsetting of '--color-moved' in linkgit:git-diff[1] for details),\n+\t`contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`,\n+\t`oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).\n+\n+color.decorate.::\n+\tUse customized color for 'git log --decorate' output. `` is one\n+\tof `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local\n+\tbranches, remote-tracking branches, tags, stash and HEAD, respectively\n+\tand `grafted` for grafted commits.\n+\n+color.grep::\n+\tWhen set to `always`, always highlight matches. When `false` (or\n+\t`never`), never. When set to `true` or `auto`, use color only\n+\twhen the output is written to the terminal. If unset, then the\n+\tvalue of `color.ui` is used (`auto` by default).\n+\n+color.grep.::\n+\tUse customized color for grep colorization. `` specifies which\n+\tpart of the line to use the specified color, and is one of\n++\n+--\n+`context`;;\n+\tnon-matching text in context lines (when using `-A`, `-B`, or `-C`)\n+`filename`;;\n+\tfilename prefix (when not using `-h`)\n+`function`;;\n+\tfunction name lines (when using `-p`)\n+`lineNumber`;;\n+\tline number prefix (when using `-n`)\n+`column`;;\n+\tcolumn number prefix (when using `--column`)\n+`match`;;\n+\tmatching text (same as setting `matchContext` and `matchSelected`)\n+`matchContext`;;\n+\tmatching text in context lines\n+`matchSelected`;;\n+\tmatching text in selected lines. Also, used to customize the following\n+\tlinkgit:git-log[1] subcommands: `--grep`, `--author` and `--committer`.\n+`selected`;;\n+\tnon-matching text in selected lines. Also, used to customize the\n+\tfollowing linkgit:git-log[1] subcommands: `--grep`, `--author` and\n+\t`--committer`.\n+`separator`;;\n+\tseparators between fields on a line (`:`, `-`, and `=`)\n+\tand between hunks (`--`)\n+--\ndiff --git a/Documentation/config/color/log.txt b/Documentation/config/color/log.txt\nnew file mode 100644\nindex 00000000000..056a027804c\n--- /dev/null\n+++ b/Documentation/config/color/log.txt\n@@ -0,0 +1,9 @@\n+color.decorate.::\n+color.logDecorate.::\n+\tUse customized color for 'git log --decorate' output. `` is one\n+\tof `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local\n+\tbranches, remote-tracking branches, tags, stash and HEAD, respectively\n+\tand `grafted` for grafted commits.\n++\n+The `color.logDecorate` name is preferred to the historical\n+`color.decorate.\ndiff --git a/Documentation/config/color/push.txt b/Documentation/config/color/push.txt\nnew file mode 100644\nindex 00000000000..8c5f19d12df\n--- /dev/null\n+++ b/Documentation/config/color/push.txt\n@@ -0,0 +1,8 @@\n+color.push::\n+\tA boolean to enable/disable color in push errors. May be set to\n+\t`always`, `false` (or `never`) or `auto` (or `true`), in which\n+\tcase colors are used only when the error output goes to a terminal.\n+\tIf unset, then the value of `color.ui` is used (`auto` by default).\n+\n+color.push.error::\n+\tUse customized color for push errors.\ndiff --git a/Documentation/config/color/showbranch.txt b/Documentation/config/color/showbranch.txt\nnew file mode 100644\nindex 00000000000..af17e085213\n--- /dev/null\n+++ b/Documentation/config/color/showbranch.txt\n@@ -0,0 +1,6 @@\n+color.showBranch::\n+\tA boolean to enable/disable color in the output of\n+\tlinkgit:git-show-branch[1]. May be set to `always`,\n+\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n+\tonly when the output is to a terminal. If unset, then the\n+\tvalue of `color.ui` is used (`auto` by default).\ndiff --git a/Documentation/config/color/status.txt b/Documentation/config/color/status.txt\nnew file mode 100644\nindex 00000000000..d37bec5f282\n--- /dev/null\n+++ b/Documentation/config/color/status.txt\n@@ -0,0 +1,20 @@\n+color.status::\n+\tA boolean to enable/disable color in the output of\n+\tlinkgit:git-status[1]. May be set to `always`,\n+\t`false` (or `never`) or `auto` (or `true`), in which case colors are used\n+\tonly when the output is to a terminal. If unset, then the\n+\tvalue of `color.ui` is used (`auto` by default).\n+\n+color.status.::\n+\tUse customized color for status colorization. `` is\n+\tone of `header` (the header text of the status message),\n+\t`added` or `updated` (files which are added but not committed),\n+\t`changed` (files which are changed but not added in the index),\n+\t`untracked` (files which are not tracked by Git),\n+\t`branch` (the current branch),\n+\t`nobranch` (the color the 'no branch' warning is shown in, defaulting\n+\tto red),\n+\t`localBranch` or `remoteBranch` (the local and remote branch names,\n+\trespectively, when branch and tracking information is displayed in the\n+\tstatus short-format), or\n+\t`unmerged` (files which have unmerged changes).\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex c04f62a54a1..6540008f319 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -411,18 +411,7 @@ You probably do not need to adjust this value.\n +\n Common unit suffixes of 'k', 'm', or 'g' are supported.\n \n-core.bigFileThreshold::\n-\tFiles larger than this size are stored deflated, without\n-\tattempting delta compression. Storing large files without\n-\tdelta compression avoids excessive memory usage, at the\n-\tslight expense of increased disk usage. Additionally files\n-\tlarger than this size are always treated as binary.\n-+\n-Default is 512 MiB on all platforms. This should be reasonable\n-for most projects as source code and other text files can still\n-be delta compressed, but larger binary media files won't be.\n-+\n-Common unit suffixes of 'k', 'm', or 'g' are supported.\n+include::core/bigFileThreshold.txt[]\n \n core.excludesFile::\n \tSpecifies the pathname to the file that contains patterns to\ndiff --git a/Documentation/config/core/bigFileThreshold.txt b/Documentation/config/core/bigFileThreshold.txt\nnew file mode 100644\nindex 00000000000..4eafd44013b\n--- /dev/null\n+++ b/Documentation/config/core/bigFileThreshold.txt\n@@ -0,0 +1,12 @@\n+core.bigFileThreshold::\n+\tFiles larger than this size are stored deflated, without\n+\tattempting delta compression. Storing large files without\n+\tdelta compression avoids excessive memory usage, at the\n+\tslight expense of increased disk usage. Additionally files\n+\tlarger than this size are always treated as binary.\n++\n+Default is 512 MiB on all platforms. This should be reasonable\n+for most projects as source code and other text files can still\n+be delta compressed, but larger binary media files won't be.\n++\n+Common unit suffixes of 'k', 'm', or 'g' are supported.\ndiff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt\nindex 32f84838ac1..35a7bf86d77 100644\n--- a/Documentation/config/diff.txt\n+++ b/Documentation/config/diff.txt\n@@ -178,21 +178,6 @@ diff..cachetextconv::\n \tSet this option to true to make the diff driver cache the text\n \tconversion outputs. See linkgit:gitattributes[5] for details.\n \n-diff.tool::\n-\tControls which diff tool is used by linkgit:git-difftool[1].\n-\tThis variable overrides the value configured in `merge.tool`.\n-\tThe list below shows the valid built-in values.\n-\tAny other value is treated as a custom diff tool and requires\n-\tthat a corresponding difftool..cmd variable is defined.\n-\n-diff.guitool::\n-\tControls which diff tool is used by linkgit:git-difftool[1] when\n-\tthe -g/--gui flag is specified. This variable overrides the value\n-\tconfigured in `merge.guitool`. The list below shows the valid\n-\tbuilt-in values. Any other value is treated as a custom diff tool\n-\tand requires that a corresponding difftool..cmd variable\n-\tis defined.\n-\n include::../mergetools-diff.txt[]\n \n diff.indentHeuristic::\ndiff --git a/Documentation/config/diff/difftool.txt b/Documentation/config/diff/difftool.txt\nnew file mode 100644\nindex 00000000000..652bc2bd8e1\n--- /dev/null\n+++ b/Documentation/config/diff/difftool.txt\n@@ -0,0 +1,14 @@\n+diff.tool::\n+\tControls which diff tool is used by linkgit:git-difftool[1].\n+\tThis variable overrides the value configured in `merge.tool`.\n+\tThe list below shows the valid built-in values.\n+\tAny other value is treated as a custom diff tool and requires\n+\tthat a corresponding difftool..cmd variable is defined.\n+\n+diff.guitool::\n+\tControls which diff tool is used by linkgit:git-difftool[1] when\n+\tthe -g/--gui flag is specified. This variable overrides the value\n+\tconfigured in `merge.guitool`. The list below shows the valid\n+\tbuilt-in values. Any other value is treated as a custom diff tool\n+\tand requires that a corresponding difftool..cmd variable\n+\tis defined.\ndiff --git a/Documentation/config/difftool.txt b/Documentation/config/difftool.txt\nindex 67625944804..dcf6213d31d 100644\n--- a/Documentation/config/difftool.txt\n+++ b/Documentation/config/difftool.txt\n@@ -9,6 +9,13 @@ difftool..cmd::\n \tfile containing the contents of the diff pre-image and 'REMOTE'\n \tis set to the name of the temporary file containing the contents\n \tof the diff post-image.\n++\n+See the `--tool=` option linkgit:git-difftool[1] for more details.\n \n difftool.prompt::\n \tPrompt before each invocation of the diff tool.\n+\n+difftool.trustExitCode::\n+\tExit difftool if the invoked diff tool returns a non-zero exit status.\n++\n+See the `--trust-exit-code` option in linkgit:git-difftool[1] for more details.\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex c834e07991f..a8873dca580 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -123,14 +123,4 @@ or rebase occurring. Since these changes are not part of the current\n project most users will want to expire them sooner, which is why the\n default is more aggressive than `gc.reflogExpire`.\n \n-gc.rerereResolved::\n-\tRecords of conflicted merge you resolved earlier are\n-\tkept for this many days when 'git rerere gc' is run.\n-\tYou can also use more human-readable \"1.month.ago\", etc.\n-\tThe default is 60 days. See linkgit:git-rerere[1].\n-\n-gc.rerereUnresolved::\n-\tRecords of conflicted merge you have not resolved are\n-\tkept for this many days when 'git rerere gc' is run.\n-\tYou can also use more human-readable \"1.month.ago\", etc.\n-\tThe default is 15 days. See linkgit:git-rerere[1].\n+include::gc/rerere.txt[]\ndiff --git a/Documentation/config/gc/rerere.txt b/Documentation/config/gc/rerere.txt\nnew file mode 100644\nindex 00000000000..6d0544771d8\n--- /dev/null\n+++ b/Documentation/config/gc/rerere.txt\n@@ -0,0 +1,11 @@\n+gc.rerereResolved::\n+\tRecords of conflicted merge you resolved earlier are\n+\tkept for this many days when 'git rerere gc' is run.\n+\tYou can also use more human-readable \"1.month.ago\", etc.\n+\tThe default is 60 days. See linkgit:git-rerere[1].\n+\n+gc.rerereUnresolved::\n+\tRecords of conflicted merge you have not resolved are\n+\tkept for this many days when 'git rerere gc' is run.\n+\tYou can also use more human-readable \"1.month.ago\", etc.\n+\tThe default is 15 days. See linkgit:git-rerere[1].\ndiff --git a/Documentation/config/grep.txt b/Documentation/config/grep.txt\nindex 182edd813a5..e521f20390c 100644\n--- a/Documentation/config/grep.txt\n+++ b/Documentation/config/grep.txt\n@@ -17,8 +17,11 @@ grep.extendedRegexp::\n \tother than 'default'.\n \n grep.threads::\n-\tNumber of grep worker threads to use.\n-\tSee `grep.threads` in linkgit:git-grep[1] for more information.\n+\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n+\tuse as many threads as the number of logical cores available.\n+\n+grep.fullName::\n+\tIf set to true, enable `--full-name` option by default.\n \n grep.fallbackToNoIndex::\n \tIf set to true, fall back to git grep --no-index if git grep\ndiff --git a/Documentation/config/log.txt b/Documentation/config/log.txt\nindex 456eb07800c..12ede8e31b8 100644\n--- a/Documentation/config/log.txt\n+++ b/Documentation/config/log.txt\n@@ -1,12 +1,11 @@\n-log.abbrevCommit::\n-\tIf true, makes linkgit:git-log[1], linkgit:git-show[1], and\n-\tlinkgit:git-whatchanged[1] assume `--abbrev-commit`. You may\n-\toverride this option with `--no-abbrev-commit`.\n-\n log.date::\n \tSet the default date-time mode for the 'log' command.\n \tSetting a value for log.date is similar to using 'git log''s\n \t`--date` option. See linkgit:git-log[1] for details.\n++\n+If the format is set to \"auto:foo\" and the pager is in use, format\n+\"foo\" will be the used for the date format. Otherwise \"default\" will\n+be used.\n \n log.decorate::\n \tPrint out the ref names of any commits that are shown by the log\n@@ -45,6 +44,11 @@ log.showRoot::\n \tTools like linkgit:git-log[1] or linkgit:git-whatchanged[1], which\n \tnormally hide the root commit will now show it. True by default.\n \n+log.abbrevCommit::\n+\tIf true, makes linkgit:git-log[1], linkgit:git-show[1], and\n+\tlinkgit:git-whatchanged[1] assume `--abbrev-commit`. You may\n+\toverride this option with `--no-abbrev-commit`.\n+\n log.showSignature::\n \tIf true, makes linkgit:git-log[1], linkgit:git-show[1], and\n \tlinkgit:git-whatchanged[1] assume `--show-signature`.\n@@ -53,3 +57,7 @@ log.mailmap::\n \tIf true, makes linkgit:git-log[1], linkgit:git-show[1], and\n \tlinkgit:git-whatchanged[1] assume `--use-mailmap`, otherwise\n \tassume `--no-use-mailmap`. True by default.\n++\n+The behavior of `log.mailmap` is impacted by the setting of the more\n+general 'mailmap.*' configuration variables, i.e. `mailmap.file` and\n+`mailmap.blob`.\ndiff --git a/Documentation/config/notes.txt b/Documentation/config/notes.txt\nindex aeef56d49ae..c7c4811734b 100644\n--- a/Documentation/config/notes.txt\n+++ b/Documentation/config/notes.txt\n@@ -3,6 +3,9 @@ notes.mergeStrategy::\n \tconflicts. Must be one of `manual`, `ours`, `theirs`, `union`, or\n \t`cat_sort_uniq`. Defaults to `manual`. See \"NOTES MERGE STRATEGIES\"\n \tsection of linkgit:git-notes[1] for more information on each strategy.\n++\n+This setting can be overridden by passing the `--strategy` option to\n+linkgit:git-notes[1].\n \n notes..mergeStrategy::\n \tWhich merge strategy to choose when doing a notes merge into\n@@ -11,28 +14,35 @@ notes..mergeStrategy::\n \tlinkgit:git-notes[1] for more information on the available strategies.\n \n notes.displayRef::\n-\tThe (fully qualified) refname from which to show notes when\n-\tshowing commit messages. The value of this variable can be set\n-\tto a glob, in which case notes from all matching refs will be\n-\tshown. You may also specify this configuration variable\n-\tseveral times. A warning will be issued for refs that do not\n-\texist, but a glob that does not match any refs is silently\n-\tignored.\n+\tWhich ref (or refs, if a glob or specified more than once), in\n+\taddition to the default set by `core.notesRef` or\n+\t`GIT_NOTES_REF`, to read notes from when showing commit\n+\tmessages with the 'git log' family of commands.\n +\n This setting can be overridden with the `GIT_NOTES_DISPLAY_REF`\n environment variable, which must be a colon separated list of refs or\n globs.\n +\n+A warning will be issued for refs that do not exist,\n+but a glob that does not match any refs is silently ignored.\n++\n+This setting can be disabled by the `--no-notes` option to the 'git\n+log' family of commands, or by the `--notes=` option accepted by\n+those commands.\n++\n The effective value of \"core.notesRef\" (possibly overridden by\n GIT_NOTES_REF) is also implicitly added to the list of refs to be\n displayed.\n \n notes.rewrite.::\n \tWhen rewriting commits with (currently `amend` or\n-\t`rebase`) and this variable is set to `true`, Git\n-\tautomatically copies your notes from the original to the\n-\trewritten commit. Defaults to `true`, but see\n-\t\"notes.rewriteRef\" below.\n+\t`rebase`), if this variable is `false`, git will not copy\n+\tnotes from the original to the rewritten commit. Defaults to\n+\t`true`. See also \"`notes.rewriteRef`\" below.\n++\n+This setting can be overridden with the `GIT_NOTES_REWRITE_REF`\n+environment variable, which must be a colon separated list of refs or\n+globs.\n \n notes.rewriteMode::\n \tWhen copying notes during a rewrite (see the\n@@ -46,14 +56,13 @@ environment variable.\n \n notes.rewriteRef::\n \tWhen copying notes during a rewrite, specifies the (fully\n-\tqualified) ref whose notes should be copied. The ref may be a\n-\tglob, in which case notes in all matching refs will be copied.\n-\tYou may also specify this configuration several times.\n+\tqualified) ref whose notes should be copied. May be a glob,\n+\tin which case notes in all matching refs will be copied. You\n+\tmay also specify this configuration several times.\n +\n Does not have a default value; you must configure this variable to\n enable note rewriting. Set it to `refs/notes/commits` to enable\n rewriting for the default commit notes.\n +\n-This setting can be overridden with the `GIT_NOTES_REWRITE_REF`\n-environment variable, which must be a colon separated list of refs or\n-globs.\n+Can be overridden with the `GIT_NOTES_REWRITE_REF` environment variable.\n+See `notes.rewrite.` above for a further description of its format.\ndiff --git a/Documentation/config/sendemail.txt b/Documentation/config/sendemail.txt\nindex 50baa5d6bfb..51da7088a84 100644\n--- a/Documentation/config/sendemail.txt\n+++ b/Documentation/config/sendemail.txt\n@@ -18,17 +18,49 @@ sendemail..*::\n \tidentity is selected, through either the command-line or\n \t`sendemail.identity`.\n \n+sendemail.multiEdit::\n+\tIf true (default), a single editor instance will be spawned to edit\n+\tfiles you have to edit (patches when `--annotate` is used, and the\n+\tsummary when `--compose` is used). If false, files will be edited one\n+\tafter the other, spawning a new editor each time.\n+\n+sendemail.confirm::\n+\tSets the default for whether to confirm before sending. Must be\n+\tone of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm`\n+\tin the linkgit:git-send-email[1] documentation for the meaning of these\n+\tvalues.\n+\n sendemail.aliasesFile::\n+\tTo avoid typing long email addresses, point this to one or more\n+\temail aliases files. You must also supply `sendemail.aliasFileType`.\n+\n sendemail.aliasFileType::\n+\tFormat of the file(s) specified in sendemail.aliasesFile. Must be\n+\tone of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'.\n++\n+What an alias file in each format looks like can be found in\n+the documentation of the email program of the same name. The\n+differences and limitations from the standard formats are\n+described below:\n++\n+--\n+sendmail;;\n+*\tQuoted aliases and quoted addresses are not supported: lines that\n+\tcontain a `\"` symbol are ignored.\n+*\tRedirection to a file (`/path/name`) or pipe (`|command`) is not\n+\tsupported.\n+*\tFile inclusion (`:include: /path/name`) is not supported.\n+*\tWarnings are printed on the standard error output for any\n+\texplicitly unsupported constructs, and any other lines that are not\n+\trecognized by the parser.\n+--\n sendemail.annotate::\n sendemail.bcc::\n sendemail.cc::\n sendemail.ccCmd::\n sendemail.chainReplyTo::\n-sendemail.confirm::\n sendemail.envelopeSender::\n sendemail.from::\n-sendemail.multiEdit::\n sendemail.signedoffbycc::\n sendemail.smtpPass::\n sendemail.suppresscc::\n@@ -44,7 +76,9 @@ sendemail.thread::\n sendemail.transferEncoding::\n sendemail.validate::\n sendemail.xmailer::\n-\tSee linkgit:git-send-email[1] for description.\n+\tThese configuration variables all provide a default for\n+\tlinkgit:git-send-email[1] command-line options. See its\n+\tdocumentation for details.\n \n sendemail.signedoffcc (deprecated)::\n \tDeprecated alias for `sendemail.signedoffbycc`.\ndiff --git a/Documentation/config/uploadpack.txt b/Documentation/config/uploadpack.txt\nindex 32fad5bbe81..d3e5ce7236d 100644\n--- a/Documentation/config/uploadpack.txt\n+++ b/Documentation/config/uploadpack.txt\n@@ -27,31 +27,7 @@ uploadpack.allowAnySHA1InWant::\n \tobject at all.\n \tDefaults to `false`.\n \n-uploadpack.keepAlive::\n-\tWhen `upload-pack` has started `pack-objects`, there may be a\n-\tquiet period while `pack-objects` prepares the pack. Normally\n-\tit would output progress information, but if `--quiet` was used\n-\tfor the fetch, `pack-objects` will output nothing at all until\n-\tthe pack data begins. Some clients and networks may consider\n-\tthe server to be hung and give up. Setting this option instructs\n-\t`upload-pack` to send an empty keepalive packet every\n-\t`uploadpack.keepAlive` seconds. Setting this option to 0\n-\tdisables keepalive packets entirely. The default is 5 seconds.\n-\n-uploadpack.packObjectsHook::\n-\tIf this option is set, when `upload-pack` would run\n-\t`git pack-objects` to create a packfile for a client, it will\n-\trun this shell command instead. The `pack-objects` command and\n-\targuments it _would_ have run (including the `git pack-objects`\n-\tat the beginning) are appended to the shell command. The stdin\n-\tand stdout of the hook are treated as if `pack-objects` itself\n-\twas run. I.e., `upload-pack` will feed input intended for\n-\t`pack-objects` to the hook, and expects a completed packfile on\n-\tstdout.\n-+\n-Note that this configuration variable is ignored if it is seen in the\n-repository-level config (this is a safety measure against fetching from\n-untrusted repositories).\n+include::uploadpack/pack-objects.txt[]\n \n uploadpack.allowFilter::\n \tIf this option is set, `upload-pack` will support partial\ndiff --git a/Documentation/config/uploadpack/pack-objects.txt b/Documentation/config/uploadpack/pack-objects.txt\nnew file mode 100644\nindex 00000000000..0447c112811\n--- /dev/null\n+++ b/Documentation/config/uploadpack/pack-objects.txt\n@@ -0,0 +1,25 @@\n+uploadpack.keepAlive::\n+\tWhen `upload-pack` has started `pack-objects`, there may be a\n+\tquiet period while `pack-objects` prepares the pack. Normally\n+\tit would output progress information, but if `--quiet` was used\n+\tfor the fetch, `pack-objects` will output nothing at all until\n+\tthe pack data begins. Some clients and networks may consider\n+\tthe server to be hung and give up. Setting this option instructs\n+\t`upload-pack` to send an empty keepalive packet every\n+\t`uploadpack.keepAlive` seconds. Setting this option to 0\n+\tdisables keepalive packets entirely. The default is 5 seconds.\n+\n+uploadpack.packObjectsHook::\n+\tIf this option is set, when `upload-pack` would run\n+\t`git pack-objects` to create a packfile for a client, it will\n+\trun this shell command instead. The `pack-objects` command and\n+\targuments it _would_ have run (including the `git pack-objects`\n+\tat the beginning) are appended to the shell command. The stdin\n+\tand stdout of the hook are treated as if `pack-objects` itself\n+\twas run. I.e., `upload-pack` will feed input intended for\n+\t`pack-objects` to the hook, and expects a completed packfile on\n+\tstdout.\n++\n+Note that this configuration variable is ignored if it is seen in the\n+repository-level config (this is a safety measure against fetching from\n+untrusted repositories).\ndiff --git a/Documentation/git-add.txt b/Documentation/git-add.txt\nindex 11eb70f16c7..36da8bf9b51 100644\n--- a/Documentation/git-add.txt\n+++ b/Documentation/git-add.txt\n@@ -431,6 +431,13 @@ they will make the patch impossible to apply:\n * deleting context or removal lines\n * modifying the contents of context or removal lines\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/add.txt[]\n+\n SEE ALSO\n --------\n linkgit:git-status[1]\ndiff --git a/Documentation/git-am.txt b/Documentation/git-am.txt\nindex 09107fb1067..7ccb46b660c 100644\n--- a/Documentation/git-am.txt\n+++ b/Documentation/git-am.txt\n@@ -261,6 +261,13 @@ This command can run `applypatch-msg`, `pre-applypatch`,\n and `post-applypatch` hooks. See linkgit:githooks[5] for more\n information.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/am.txt[]\n+\n SEE ALSO\n --------\n linkgit:git-apply[1].\ndiff --git a/Documentation/git-apply.txt b/Documentation/git-apply.txt\nindex b6d77f42068..1d478cbe9b5 100644\n--- a/Documentation/git-apply.txt\n+++ b/Documentation/git-apply.txt\n@@ -263,13 +263,9 @@ has no effect when `--index` or `--cached` is in use.\n CONFIGURATION\n -------------\n \n-apply.ignoreWhitespace::\n-\tSet to 'change' if you want changes in whitespace to be ignored by default.\n-\tSet to one of: no, none, never, false if you want changes in\n-\twhitespace to be significant.\n-apply.whitespace::\n-\tWhen no `--whitespace` flag is given from the command\n-\tline, this configuration item is used as the default.\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/apply.txt[]\n \n SUBMODULES\n ----------\ndiff --git a/Documentation/git-blame.txt b/Documentation/git-blame.txt\nindex d7a46cc6744..67942b6964d 100644\n--- a/Documentation/git-blame.txt\n+++ b/Documentation/git-blame.txt\n@@ -242,6 +242,16 @@ MAPPING AUTHORS\n See linkgit:gitmailmap[5].\n \n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/blame.txt[]\n+\n+include::config/color/blame.txt[]\n+\n+\n SEE ALSO\n --------\n linkgit:git-annotate[1]\ndiff --git a/Documentation/git-branch.txt b/Documentation/git-branch.txt\nindex c8b4f9ce3c7..601a84451ed 100644\n--- a/Documentation/git-branch.txt\n+++ b/Documentation/git-branch.txt\n@@ -332,6 +332,12 @@ CONFIGURATION\n `--list` is used or implied. The default is to use a pager.\n See linkgit:git-config[1].\n \n+include::includes/cmd-config-section-rest.txt[]\n+\n+include::config/branch.txt[]\n+\n+include::config/color/branch.txt[]\n+\n EXAMPLES\n --------\n \ndiff --git a/Documentation/git-checkout.txt b/Documentation/git-checkout.txt\nindex 9f37e22e130..4cb9d555b4b 100644\n--- a/Documentation/git-checkout.txt\n+++ b/Documentation/git-checkout.txt\n@@ -600,6 +600,13 @@ $ edit frotz\n $ git add frotz\n ------------\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/checkout.txt[]\n+\n SEE ALSO\n --------\n linkgit:git-switch[1],\ndiff --git a/Documentation/git-clean.txt b/Documentation/git-clean.txt\nindex a7f309dff5a..91742633fa8 100644\n--- a/Documentation/git-clean.txt\n+++ b/Documentation/git-clean.txt\n@@ -133,6 +133,13 @@ help::\n \n Show brief usage of interactive git-clean.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/clean.txt[]\n+\n SEE ALSO\n --------\n linkgit:gitignore[5]\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 984d194934f..7547cbd5266 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -358,6 +358,15 @@ $ cd my-linux\n $ git clone --bare -l /home/proj/.git /pub/scm/proj.git\n ------------\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/init.txt[]\n+\n+include::config/clone.txt[]\n+\n \n GIT\n ---\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex e1f48c95b3c..c8143e9a3ba 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -143,6 +143,14 @@ $ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n ------------------------------------------------\n \n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/commitgraph.txt[]\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-commit.txt b/Documentation/git-commit.txt\nindex 6c60bf98f9f..225c6c9f2e5 100644\n--- a/Documentation/git-commit.txt\n+++ b/Documentation/git-commit.txt\n@@ -557,6 +557,10 @@ The editor used to edit the commit log message will be chosen from the\n `VISUAL` environment variable, or the `EDITOR` environment variable (in that\n order). See linkgit:git-var[1] for details.\n \n+include::includes/cmd-config-section-rest.txt[]\n+\n+include::config/commit.txt[]\n+\n HOOKS\n -----\n This command can run `commit-msg`, `prepare-commit-msg`, `pre-commit`,\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex f18673017f5..d80d9f671f9 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -160,6 +160,13 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/credential.txt[]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-diff.txt b/Documentation/git-diff.txt\nindex 6236c75c9b8..c9f39225fb1 100644\n--- a/Documentation/git-diff.txt\n+++ b/Documentation/git-diff.txt\n@@ -213,6 +213,15 @@ $ git diff -R <2>\n rewrites (very expensive).\n <2> Output diff in reverse.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/diff.txt[]\n+\n+include::config/color/diff.txt[]\n+\n SEE ALSO\n --------\n diff(1),\ndiff --git a/Documentation/git-difftool.txt b/Documentation/git-difftool.txt\nindex 143b0c49d73..9f6fae615a4 100644\n--- a/Documentation/git-difftool.txt\n+++ b/Documentation/git-difftool.txt\n@@ -113,33 +113,17 @@ instead. `--no-symlinks` is the default on Windows.\n \n See linkgit:git-diff[1] for the full list of supported options.\n \n-CONFIG VARIABLES\n-----------------\n+CONFIGURATION\n+-------------\n+\n 'git difftool' falls back to 'git mergetool' config variables when the\n difftool equivalents have not been defined.\n \n-diff.tool::\n-\tThe default diff tool to use.\n-\n-diff.guitool::\n-\tThe default diff tool to use when `--gui` is specified.\n+include::includes/cmd-config-section-rest.txt[]\n \n-difftool..path::\n-\tOverride the path for the given tool. This is useful in case\n-\tyour tool is not in the PATH.\n-\n-difftool..cmd::\n-\tSpecify the command to invoke the specified diff tool.\n-+\n-See the `--tool=` option above for more details.\n+include::config/diff/difftool.txt[]\n \n-difftool.prompt::\n-\tPrompt before each invocation of the diff tool.\n-\n-difftool.trustExitCode::\n-\tExit difftool if the invoked diff tool returns a non-zero exit status.\n-+\n-See the `--trust-exit-code` option above for more details.\n+include::config/difftool.txt[]\n \n SEE ALSO\n --------\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 39cfa05b28b..8b5dd6add00 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -1564,6 +1564,13 @@ operator can use this facility to peek at the objects and refs from an\n import in progress, at the cost of some added running time and worse\n compression.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/fastimport.txt[]\n+\n SEE ALSO\n --------\n linkgit:git-fast-export[1]\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex 550c16ca616..3c66dd584ff 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -285,6 +285,15 @@ linkgit:git-gc[1]).\n \n include::transfer-data-leaks.txt[]\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-rest.txt[]\n+\n+include::config/fetch.txt[]\n+\n+include::config/remote.txt[]\n+\n BUGS\n ----\n Using --recurse-submodules can only fetch new commits in already checked\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 853967dea01..2d2b142058e 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -103,14 +103,6 @@ more time, and the resulting space/delta optimization may or may not\n be worth it. Not using this at all is the right trade-off for most\n users and their repositories.\n \n-CONFIGURATION\n--------------\n-\n-The below documentation is the same as what's found in\n-linkgit:git-config[1]:\n-\n-include::config/gc.txt[]\n-\n NOTES\n -----\n \n@@ -150,6 +142,12 @@ HOOKS\n The 'git gc --auto' command will run the 'pre-auto-gc' hook. See\n linkgit:githooks[5] for more information.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/gc.txt[]\n \n SEE ALSO\n --------\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 3d393fbac1b..70f3f7993e7 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -334,34 +334,9 @@ performance in this case, it might be desirable to use `--threads=1`.\n CONFIGURATION\n -------------\n \n-grep.lineNumber::\n-\tIf set to true, enable `-n` option by default.\n-\n-grep.column::\n-\tIf set to true, enable the `--column` option by default.\n-\n-grep.patternType::\n-\tSet the default matching behavior. Using a value of 'basic', 'extended',\n-\t'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`,\n-\t`--fixed-strings`, or `--perl-regexp` option accordingly, while the\n-\tvalue 'default' will return to the default matching behavior.\n-\n-grep.extendedRegexp::\n-\tIf set to true, enable `--extended-regexp` option by default. This\n-\toption is ignored when the `grep.patternType` option is set to a value\n-\tother than 'default'.\n-\n-grep.threads::\n-\tNumber of grep worker threads to use. If unset (or set to 0), Git will\n-\tuse as many threads as the number of logical cores available.\n-\n-grep.fullName::\n-\tIf set to true, enable `--full-name` option by default.\n-\n-grep.fallbackToNoIndex::\n-\tIf set to true, fall back to git grep --no-index if git grep\n-\tis executed outside of a git repository. Defaults to false.\n+include::includes/cmd-config-section-all.txt[]\n \n+include::config/grep.txt[]\n \n GIT\n ---\ndiff --git a/Documentation/git-log.txt b/Documentation/git-log.txt\nindex 20e87cecf49..cb66f0a9cac 100644\n--- a/Documentation/git-log.txt\n+++ b/Documentation/git-log.txt\n@@ -42,6 +42,10 @@ OPTIONS\n \tshown. The option `--decorate` is short-hand for `--decorate=short`.\n \tDefault to configuration value of `log.decorate` if configured,\n \totherwise, `auto`.\n++\n+This output may be colored. See `color.ui` in linkgit:git-config[1]\n+and `color.logDecorate` (also known as `color.decorate`) there and in\n+the \"CONFIGURATION\" section below.\n \n --decorate-refs=::\n --decorate-refs-exclude=::\n@@ -199,47 +203,11 @@ i18n.logOutputEncoding::\n \tDefaults to the value of `i18n.commitEncoding` if set, and UTF-8\n \totherwise.\n \n-log.date::\n-\tDefault format for human-readable dates. (Compare the\n-\t`--date` option.) Defaults to \"default\", which means to write\n-\tdates like `Sat May 8 19:35:34 2010 -0500`.\n-+\n-If the format is set to \"auto:foo\" and the pager is in use, format\n-\"foo\" will be the used for the date format. Otherwise \"default\" will\n-be used.\n-\n-log.follow::\n-\tIf `true`, `git log` will act as if the `--follow` option was used when\n-\ta single is given. This has the same limitations as `--follow`,\n-\ti.e. it cannot be used to follow multiple files and does not work well\n-\ton non-linear history.\n-\n-log.showRoot::\n-\tIf `false`, `git log` and related commands will not treat the\n-\tinitial commit as a big creation event. Any root commits in\n-\t`git log -p` output would be shown without a diff attached.\n-\tThe default is `true`.\n-\n-log.showSignature::\n-\tIf `true`, `git log` and related commands will act as if the\n-\t`--show-signature` option was passed to them.\n-\n-mailmap.*::\n-\tSee linkgit:git-shortlog[1].\n-\n-notes.displayRef::\n-\tWhich refs, in addition to the default set by `core.notesRef`\n-\tor `GIT_NOTES_REF`, to read notes from when showing commit\n-\tmessages with the `log` family of commands. See\n-\tlinkgit:git-notes[1].\n-+\n-May be an unabbreviated ref name or a glob and may be specified\n-multiple times. A warning will be issued for refs that do not exist,\n-but a glob that does not match any refs is silently ignored.\n-+\n-This setting can be disabled by the `--no-notes` option,\n-overridden by the `GIT_NOTES_DISPLAY_REF` environment variable,\n-and overridden by the `--notes=` option.\n+include::includes/cmd-config-section-rest.txt[]\n+\n+include::config/log.txt[]\n+\n+include::config/notes.txt[]\n \n GIT\n ---\ndiff --git a/Documentation/git-mergetool.txt b/Documentation/git-mergetool.txt\nindex e587c7763a7..5a9d94d3d17 100644\n--- a/Documentation/git-mergetool.txt\n+++ b/Documentation/git-mergetool.txt\n@@ -101,6 +101,9 @@ success of the resolution after the custom tool has exited.\n \n CONFIGURATION\n -------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n include::config/mergetool.txt[]\n \n TEMPORARY FILES\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nindex 0a4200674c4..68d547e2474 100644\n--- a/Documentation/git-notes.txt\n+++ b/Documentation/git-notes.txt\n@@ -44,7 +44,7 @@ using the `--notes` option. Such notes are added as a patch commentary\n after a three dash separator line.\n \n To change which notes are shown by 'git log', see the\n-\"notes.displayRef\" configuration in linkgit:git-log[1].\n+\"notes.displayRef\" configuration in linkgit:git-config[1].\n \n See the \"notes.rewrite.\" configuration for a way to carry\n notes across commands that rewrite commits.\n@@ -316,57 +316,9 @@ core.notesRef::\n \tThis setting can be overridden through the environment and\n \tcommand line.\n \n-notes.mergeStrategy::\n-\tWhich merge strategy to choose by default when resolving notes\n-\tconflicts. Must be one of `manual`, `ours`, `theirs`, `union`, or\n-\t`cat_sort_uniq`. Defaults to `manual`. See \"NOTES MERGE STRATEGIES\"\n-\tsection above for more information on each strategy.\n-+\n-This setting can be overridden by passing the `--strategy` option.\n-\n-notes..mergeStrategy::\n-\tWhich merge strategy to choose when doing a notes merge into\n-\trefs/notes/. This overrides the more general\n-\t\"notes.mergeStrategy\". See the \"NOTES MERGE STRATEGIES\" section above\n-\tfor more information on each available strategy.\n-\n-notes.displayRef::\n-\tWhich ref (or refs, if a glob or specified more than once), in\n-\taddition to the default set by `core.notesRef` or\n-\t`GIT_NOTES_REF`, to read notes from when showing commit\n-\tmessages with the 'git log' family of commands.\n-\tThis setting can be overridden on the command line or by the\n-\t`GIT_NOTES_DISPLAY_REF` environment variable.\n-\tSee linkgit:git-log[1].\n-\n-notes.rewrite.::\n-\tWhen rewriting commits with (currently `amend` or\n-\t`rebase`), if this variable is `false`, git will not copy\n-\tnotes from the original to the rewritten commit. Defaults to\n-\t`true`. See also \"`notes.rewriteRef`\" below.\n-+\n-This setting can be overridden by the `GIT_NOTES_REWRITE_REF`\n-environment variable.\n+include::includes/cmd-config-section-rest.txt[]\n \n-notes.rewriteMode::\n-\tWhen copying notes during a rewrite, what to do if the target\n-\tcommit already has a note. Must be one of `overwrite`,\n-\t`concatenate`, `cat_sort_uniq`, or `ignore`. Defaults to\n-\t`concatenate`.\n-+\n-This setting can be overridden with the `GIT_NOTES_REWRITE_MODE`\n-environment variable.\n-\n-notes.rewriteRef::\n-\tWhen copying notes during a rewrite, specifies the (fully\n-\tqualified) ref whose notes should be copied. May be a glob,\n-\tin which case notes in all matching refs will be copied. You\n-\tmay also specify this configuration several times.\n-+\n-Does not have a default value; you must configure this variable to\n-enable note rewriting.\n-+\n-Can be overridden with the `GIT_NOTES_REWRITE_REF` environment variable.\n+include::config/notes.txt[]\n \n \n ENVIRONMENT\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex f8344e1e5ba..41042ebcc7d 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -414,12 +414,10 @@ over user-wide config, and so forth).\n CONFIGURATION\n -------------\n \n-Various configuration variables affect packing, see\n-linkgit:git-config[1] (search for \"pack\" and \"delta\").\n+include::templates/configuration-include-pack.txt[]\n+\n+include::config/uploadpack/pack-objects.txt[]\n \n-Notably, delta compression is not used on objects larger than the\n-`core.bigFileThreshold` configuration variable and on files with the\n-attribute `delta` set to false.\n \n SEE ALSO\n --------\ndiff --git a/Documentation/git-remote.txt b/Documentation/git-remote.txt\nindex 2bebc32566b..03ae8399bae 100644\n--- a/Documentation/git-remote.txt\n+++ b/Documentation/git-remote.txt\n@@ -196,13 +196,6 @@ be updated. (See linkgit:git-config[1]).\n With `--prune` option, run pruning against all the remotes that are updated.\n \n \n-DISCUSSION\n-----------\n-\n-The remote configuration is achieved using the `remote.origin.url` and\n-`remote.origin.fetch` configuration variables. (See\n-linkgit:git-config[1]).\n-\n EXIT STATUS\n -----------\n \n@@ -255,6 +248,14 @@ $ git remote add -f -t master -m master origin git://example.com/git.git/\n $ git merge origin\n ------------\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/remote.txt[]\n+\n+include::config/remotes.txt[]\n \n SEE ALSO\n --------\ndiff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt\nindex ee30edc178a..893a2a7a281 100644\n--- a/Documentation/git-repack.txt\n+++ b/Documentation/git-repack.txt\n@@ -204,22 +204,9 @@ linkgit:git-multi-pack-index[1]).\n CONFIGURATION\n -------------\n \n-Various configuration variables affect packing, see\n-linkgit:git-config[1] (search for \"pack\" and \"delta\").\n-\n-By default, the command passes `--delta-base-offset` option to\n-'git pack-objects'; this typically results in slightly smaller packs,\n-but the generated packs are incompatible with versions of Git older than\n-version 1.4.4. If you need to share your repository with such ancient Git\n-versions, either directly or via the dumb http protocol, then you\n-need to set the configuration variable `repack.UseDeltaBaseOffset` to\n-\"false\" and repack. Access from old Git versions over the native protocol\n-is unaffected by this option as the conversion is performed on the fly\n-as needed in that case.\n-\n-Delta compression is not used on objects larger than the\n-`core.bigFileThreshold` configuration variable and on files with the\n-attribute `delta` set to false.\n+include::templates/configuration-include-pack.txt[]\n+\n+include::config/repack.txt[]\n \n SEE ALSO\n --------\ndiff --git a/Documentation/git-rerere.txt b/Documentation/git-rerere.txt\nindex 4cfc8833780..1a32ca368c1 100644\n--- a/Documentation/git-rerere.txt\n+++ b/Documentation/git-rerere.txt\n@@ -217,6 +217,15 @@ same as lines with conflict markers, 'git rerere' may fail to record a\n conflict resolution. To work around this, the `conflict-marker-size`\n setting in linkgit:gitattributes[5] can be used.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/rerere.txt[]\n+\n+include::config/gc.txt[]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-reset.txt b/Documentation/git-reset.txt\nindex 6f7685f53d5..a193024d39f 100644\n--- a/Documentation/git-reset.txt\n+++ b/Documentation/git-reset.txt\n@@ -499,6 +499,13 @@ working index HEAD target working index HEAD\n \n `X` means any state and `U` means an unmerged index.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/reset.txt[]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/git-send-email.txt b/Documentation/git-send-email.txt\nindex 41cd8cb4247..3290043053a 100644\n--- a/Documentation/git-send-email.txt\n+++ b/Documentation/git-send-email.txt\n@@ -456,41 +456,9 @@ Information\n CONFIGURATION\n -------------\n \n-sendemail.aliasesFile::\n-\tTo avoid typing long email addresses, point this to one or more\n-\temail aliases files. You must also supply `sendemail.aliasFileType`.\n+include::includes/cmd-config-section-all.txt[]\n \n-sendemail.aliasFileType::\n-\tFormat of the file(s) specified in sendemail.aliasesFile. Must be\n-\tone of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'.\n-+\n-What an alias file in each format looks like can be found in\n-the documentation of the email program of the same name. The\n-differences and limitations from the standard formats are\n-described below:\n-+\n---\n-sendmail;;\n-*\tQuoted aliases and quoted addresses are not supported: lines that\n-\tcontain a `\"` symbol are ignored.\n-*\tRedirection to a file (`/path/name`) or pipe (`|command`) is not\n-\tsupported.\n-*\tFile inclusion (`:include: /path/name`) is not supported.\n-*\tWarnings are printed on the standard error output for any\n-\texplicitly unsupported constructs, and any other lines that are not\n-\trecognized by the parser.\n---\n-\n-sendemail.multiEdit::\n-\tIf true (default), a single editor instance will be spawned to edit\n-\tfiles you have to edit (patches when `--annotate` is used, and the\n-\tsummary when `--compose` is used). If false, files will be edited one\n-\tafter the other, spawning a new editor each time.\n-\n-sendemail.confirm::\n-\tSets the default for whether to confirm before sending. Must be\n-\tone of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm`\n-\tin the previous section for the meaning of these values.\n+include::config/sendemail.txt[]\n \n EXAMPLES\n --------\ndiff --git a/Documentation/git-show-branch.txt b/Documentation/git-show-branch.txt\nindex 5cc2fcefbab..e5ec6b467f9 100644\n--- a/Documentation/git-show-branch.txt\n+++ b/Documentation/git-show-branch.txt\n@@ -199,6 +199,13 @@ shows 10 reflog entries going back from the tip as of 1 hour ago.\n Without `--list`, the output also shows how these tips are\n topologically related with each other.\n \n+CONFIGURATION\n+-------------\n+\n+include::includes/cmd-config-section-all.txt[]\n+\n+include::config/showbranch.txt[]\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex a16e62bc8c8..fa1ebb13dbe 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -702,6 +702,10 @@ SEE ALSO\n --------\n linkgit:git-hook[1]\n \n+The `uploadpack.packObjectsHook` configuration variable can configure\n+a hook for linkgit:git-pack-objects[1] to use, but not in a way that\n+uses the hook facility discussed above.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Documentation/includes/cmd-config-section-all.txt b/Documentation/includes/cmd-config-section-all.txt\nnew file mode 100644\nindex 00000000000..296a239f2ab\n--- /dev/null\n+++ b/Documentation/includes/cmd-config-section-all.txt\n@@ -0,0 +1,3 @@\n+Everything below this line in this section is selectively included\n+from the linkgit:git-config[1] documentation. The content is the same\n+as what's found there:\ndiff --git a/Documentation/includes/cmd-config-section-rest.txt b/Documentation/includes/cmd-config-section-rest.txt\nnew file mode 100644\nindex 00000000000..fe4721dac76\n--- /dev/null\n+++ b/Documentation/includes/cmd-config-section-rest.txt\n@@ -0,0 +1,3 @@\n+The discussion above this line is unique to this documentation, but\n+everything below is included from section(s) in linkgit:git-config[1]\n+aplicable to this command:\ndiff --git a/Documentation/templates/configuration-include-pack.txt b/Documentation/templates/configuration-include-pack.txt\nnew file mode 100644\nindex 00000000000..fa80f68b658\n--- /dev/null\n+++ b/Documentation/templates/configuration-include-pack.txt\n@@ -0,0 +1,10 @@\n+Various configuration variables affect packing. Those variables and\n+more are discussed in linkgit:git-config[1].\n+\n+The below documentation is selectively included from\n+linkgit:git-config[1]. It should be all the configuration variables\n+relevant to this command, but some may have been missed. When in doubt\n+doubt consult linkgit:git-config[1] for the full listing:\n+\n+include::../config/core/bigFileThreshold.txt[]\n+include::../config/pack.txt[]\ndiff --git a/builtin/help.c b/builtin/help.c\nindex d387131dd83..d03fc7aaac4 100644\n--- a/builtin/help.c\n+++ b/builtin/help.c\n@@ -94,7 +94,7 @@ static void list_config_help(enum show_config_type type)\n \tstruct slot_expansion slot_expansions[] = {\n \t\t{ \"advice\", \"*\", list_config_advices },\n \t\t{ \"color.branch\", \"\", list_config_color_branch_slots },\n-\t\t{ \"color.decorate\", \"\", list_config_color_decorate_slots },\n+\t\t{ \"color.logDecorate\", \"\", list_config_color_decorate_slots },\n \t\t{ \"color.diff\", \"\", list_config_color_diff_slots },\n \t\t{ \"color.grep\", \"\", list_config_color_grep_slots },\n \t\t{ \"color.interactive\", \"\", list_config_color_interactive_slots },\ndiff --git a/builtin/log.c b/builtin/log.c\nindex 093d0d26553..3240785a7ee 100644\n--- a/builtin/log.c\n+++ b/builtin/log.c\n@@ -522,7 +522,8 @@ static int git_log_config(const char *var, const char *value, void *cb)\n \t\tdefault_follow = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n-\tif (skip_prefix(var, \"color.decorate.\", &slot_name))\n+\tif (skip_prefix(var, \"color.logDecorate.\", &slot_name) ||\n+\t skip_prefix(var, \"color.decorate.\", &slot_name))\n \t\treturn parse_decorate_color_config(var, slot_name, value);\n \tif (!strcmp(var, \"log.mailmap\")) {\n \t\tuse_mailmap_config = git_config_bool(var, value);\ndiff --git a/generate-configlist.sh b/generate-configlist.sh\nindex 8692fe5cf4d..de418d2b8bd 100755\n--- a/generate-configlist.sh\n+++ b/generate-configlist.sh\n@@ -7,7 +7,7 @@ print_config_list () {\n \tcat <out &&\n+\tgit log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw &&\n+\tsed \"s/[0-9a-f]\\{10,10\\}/COMMIT_ID/\" out.color &&\n+\ttest_decode_color out &&\n \ttest_cmp expected out\n '\n \n+test_expect_success 'color.logDecorate.* is preferred to color.decorate.*' '\n+\tgit config --rename-section color.decorate color.logDecorate &&\n+\tgit log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw2 &&\n+\ttest_cmp out.raw out.raw2\n+'\n+\n test_done\n"},{"id":"449654","messageId":"CABPp-BH_xzYRHfVwOEQE8-g_QEQgHURYsqr+S-cxsRCtnaAofg@mail.gmail.com","threadId":"57444","inReplyTo":"xmqq35k8nmwj.fsf@gitster.g","subject":"Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T05:58:44Z","receivedAt":"2022-02-26T05:59:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 24, 2022 at 10:24 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Typically with sparse checkouts, we expect files outside the sparsity\n> > patterns to be marked as SKIP_WORKTREE and be missing from the working\n> > tree. In edge cases, this can be violated and cause confusion, so in a\n> > sparse checkout, since 11d46a399d (\"repo_read_index: clear SKIP_WORKTREE\n>\n> I think this refers to af6a5187 (repo_read_index: clear\n> SKIP_WORKTREE bit from files present in worktree, 2022-01-14).\n\nYes, I'm usually pretty good about grabbing the commits you created\nand merged rather than the local copies I submitted, but I messed it\nup here. Thanks for catching.\n\n>\n> > bit from files present in worktree\", 2022-01-06), Git automatically\n> > clears the SKIP_WORKTREE bit at read time for entries corresponding to\n> > files that are present in the working tree.\n>\n> So, this is a workflow where the user deliberately \"creates\" these\n> files outside the sparsity cone or pattern (by various non-automated\n> means like editing, copying/renaming, or untarring). If they did so\n> on purpose, they may be interested in comparing them with existing\n> commits, or even including them as a newer version in the next\n> commit they create. To help that workflow, clearing the bit makes\n> sense.\n>\n> Am I on the right path? I am wondering if mentioning some of that\n> would help understanding by the reader when it is contrasted with\n> the (competing) goal of supporting VFS use case mentioned next.\n\nYes, this is one of three ways that things can get out of sync. Since\nthis commit was being added to en/present-despite-skipped which\nspelled this out in detail and thus would appear just a few commits\nbefore, I thought it wasn't worth repeating these details, but I\nguessed wrong. I'll include them here again.\n\n> > However, there is a more atypical situation where this situation would\n>\n> I wonder if that is \"more atypical\" (read: makes me wonder if it\n> depends on who the reader is what is typoical), and more importantly,\n> if it helps understanding of the reader (read: whether which one is\n> more common, we'd want to support both camps anyway).\n>\n> There is another workflow, however, that it is expected that\n> paths outside the sparsity patterns appear to exist in the\n> working tree and that they do not lose the SKIP_WORKTREE bit, at\n> least until they get modified.\n>\n> or something?\n\nI like it.\n\n> > be expected. A Git-aware virtual file system[1] takes advantage of its\n> > position as a file system driver to expose all files in the working\n> > tree, fetch them on demand using partial clone on access, and tell Git\n> > to pay attention to them on demand by updating the sparse checkout\n> > pattern on writes. This means that commands like \"git status\" only have\n> > to examine files that have potentially been modified, whereas commands\n> > like \"ls\" are able to show the entire codebase without requiring manual\n> > updates to the sparse checkout pattern.\n>\n> Well explained.\n>\n> > Thus since 11d46a399d, Git with such Git-aware virtual file systems\n>\n> The same stale reference.\n>\n> > unsets the SKIP_WORKTREE bit for all files and commands like \"git\n> > status\" have to fetch and examine them all.\n> >\n> > Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to\n> > allow limiting the tracked set of files to a small set once again. A\n> > Git-aware virtual file system or other application that wants to\n> > maintain files outside of the sparse checkout can set this in a\n> > repository to instruct Git not to check for the presence of\n> > SKIP_WORKTREE files. The setting defaults to false, so most users of\n> > sparse checkout will still get the benefit of an automatically updating\n> > index to recover from the variety of difficult issues detailed in\n> > 11d46a399d for paths with SKIP_WORKTREE set despite the path being\n>\n> Ditto.\n\nWill fix all three.\n\n> > I'm guessing that since there are no code (only documentation) changes since\n> > Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs.\n> > I'm also guessing it matches what Stolee and Dscho stated in their comments\n> > on v1. But it'd be nice to have an ack from each side just to make sure.\n>\n> True. Let me queue but leave it just outside 'next' until that\n> happens.\n>\n> I think the name of the knob is what Jonathan suggested, so I\n> presume that their side would be fine with it, but I am curious (I\n> do not wonder, though) what the plan on the Microsoft's side going\n> forward. When they update the version of Git bundled in their vfsd,\n> would this be reverted and an equivalent they have (and they may\n> have more such \"workaround\" in other areas as well?) will be kept,\n> so whatever we do here will add a minor inconvenience to them but\n> will not hurt them otherwise?\n>\n> > diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\n> > new file mode 100644\n> > index 0000000000..fba504173c\n> > --- /dev/null\n> > +++ b/Documentation/config/sparse.txt\n> > @@ -0,0 +1,28 @@\n> > +sparse.expectFilesOutsideOfPatterns::\n> > + Typically with sparse checkouts, files not matching any\n> > + sparsity patterns are marked as such in the index file and\n>\n> s/index file/index/ perhaps.\n\nWill fix.\n\n> > + missing from the working tree. Accordingly, Git will\n> > + ordinarily check whether files that the index indicates are\n> > + outside of the sparse area are present in the working tree and\n> > + mark them as present in the index if so. This option can be\n>\n> Just an observation. According to this sentence, \"sparse area\" is\n> \"paths that ought to be present in the working tree\", so paths\n> \"outside of the sparse area\" that are present need to be corrected\n> to be \"in\" the sparse area by futzing bits. I always get confused\n> when I hear \"sparse area\" if the author meant \"paths that ought to\n> be missing\" or \"present\", but maybe it is just me.\n\nI reworded this based on a combination of the feedback from you and\nJonathan. I think it's clearer now; I'll resubmit soon.\n\n>\n> > + used to tell Git that such present-but-unmatching files are\n> > + expected and to stop checking for them.\n>\n> OK.\n>\n> > ++\n> > +The default is `false`. Paths which are marked as SKIP_WORKTREE\n> > +despite being present (which can occur for a few different reasons)\n> > +typically present a range of problems which are difficult for users to\n>\n> s/typically // perhaps.\n\nSure.\n\n> > +discover and recover from. The default setting avoids such issues.\n> > ++\n> > +A Git-based virtual file system (VFS) can turn the usual expectation\n> > +on its head: files are present in the working copy but do not take\n> > +up much disk space because their contents are not downloaded until\n> > +they are accessed. With such a virtual file system layer, most files\n> > +do not match the sparsity patterns at first, and the VFS layer\n> > +updates the sparsity patterns to add more files whenever files are\n> > +written. Setting this to `true` supports such a setup where files are\n> > +expected to be present outside the sparse area and a separate, robust\n> > +mechanism is responsible for keeping the sparsity patterns up to date.\n>\n> s/separate, robust/separate/ I would think.\n>\n> We make the outside mechanism that makes these files appear to be\n> present to also be responsible for maintaining the sparse bit and\n> patterns.\n>\n> When the user (or IDE) sets this knob to 'true', do we even have to\n> expect that files appear to be present? In the use case we intend\n> to support with this feature, i.e. some VFS, we might expect all\n> paths to appear to be present, but if that VFS also allows users to\n> configure to expose only a subset of paths, not all paths may appear\n> to be present. And we are perfectly OK with that, becuase we do not\n> expect anything about the working tree paths outside the sparsity\n> pattern. Am I mistaken?\n>\n> So, \"... supports such a setup where some external system releaves\n> us of the responsibility of maintaining the consistency between the\n> presence of working tree files and sparsity patterns, so we stop\n> expecting whether files are present or missing outside the sparse\n> area\", might be closer to the truth?\n\nGood point, and thanks for the suggested wording.\n\n> > +Note that the checking and clearing of the SKIP_WORKTREE bit only\n> > +happens when core.sparseCheckout is true, so this config option has no\n> > +effect unless core.sparseCheckout is true.\n>\n> Good note to have. There is no mention of \"cone\" mode in the entire\n> description; it is unclear if this only applies to \"pattern\" mode or\n> to both \"pattern\" and \"cone\" modes, which may want to be clarified.\n\nYeah, it applies to both pattern and cone modes. I went with\nJonathan's wording, which I think sounded more precise and suggested\nthat only core.sparseCheckout=true matters:\n\n\"\"\"\nRegardless of this setting, Git does not check for\n present-but-unmatching files unless sparse checkout is enabled, so\n this config option has no effect unless `core.sparseCheckout` is\n `true`.\n\"\"\"\n"},{"id":"449655","messageId":"CABPp-BFpOWAuaA7DpS8FBty=LdNu3UFbnkq1zsM68fnE9m93eA@mail.gmail.com","threadId":"57444","inReplyTo":"YhkE2vxI4nM3ut0K@google.com","subject":"Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T06:01:28Z","receivedAt":"2022-02-26T06:01:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Feb 25, 2022 at 8:33 AM Jonathan Nieder wrote:\n>\n> Hi,\n>\n> Elijah Newren wrote:\n>\n> > Signed-off-by: Elijah Newren \n>\n> Reviewed-by: Jonathan Nieder \n>\n> Thanks, and sorry for the slow review. My one remaining area for nits\n> is the documentation, but that can be improved iteratively via patches\n> on top.\n>\n> [...]\n> > --- /dev/null\n> > +++ b/Documentation/config/sparse.txt\n> > @@ -0,0 +1,28 @@\n> > +sparse.expectFilesOutsideOfPatterns::\n> > + Typically with sparse checkouts, files not matching any\n> > + sparsity patterns are marked as such in the index file and\n> > + missing from the working tree. Accordingly, Git will\n> > + ordinarily check whether files that the index indicates are\n> > + outside of the sparse area are present in the working tree and\n>\n> Junio mentioned the \"sparse area\" could suggest that the area is\n> itself sparse and devoid of files, so it might not have been the best\n> choice of words on my part. Perhaps \"whether files that the index\n> indicates are not checked out are present in the working tree\" would\n> work here?\n\nI rewrote the paragraph. I think it's more clear now; I'll resubmit\nit here soon.\n\n> > + mark them as present in the index if so. This option can be\n> > + used to tell Git that such present-but-unmatching files are\n> > + expected and to stop checking for them.\n> > ++\n> > +The default is `false`. Paths which are marked as SKIP_WORKTREE\n> > +despite being present (which can occur for a few different reasons)\n> > +typically present a range of problems which are difficult for users to\n> > +discover and recover from. The default setting avoids such issues.\n>\n> The git-sparse-checkout(1) page never describes what SKIP_WORKTREE\n> means, so it might not be obvious to them what this means. Also, the\n> \"can occur for a few different reasons\" may leave the user wondering\n> whether they are subject to those reasons. What the reader wants to\n> know is \"I should keep using the default because it makes Git work\n> better\", so how about something like\n>\n> The default is `false`, which allows Git to automatically recover\n> from the list of files in the index and working tree falling out of\n> sync.\n> +\n>\n> ?\n\nI like this.\n\n> > ++\n> > +A Git-based virtual file system (VFS) can turn the usual expectation\n> > +on its head: files are present in the working copy but do not take\n> > +up much disk space because their contents are not downloaded until\n> > +they are accessed. With such a virtual file system layer, most files\n> > +do not match the sparsity patterns at first, and the VFS layer\n> > +updates the sparsity patterns to add more files whenever files are\n> > +written. Setting this to `true` supports such a setup where files are\n> > +expected to be present outside the sparse area and a separate, robust\n> > +mechanism is responsible for keeping the sparsity patterns up to date.\n>\n> Here I spent most of the words explaining what a Git-based VFS layer\n> is, which is also not too relevant to most users (who are just\n> interested in \"is `true` the right value for me?\"). How about\n> reducing it to the following?\n>\n> Set this to `true` if you are in a setup where extra files are expected\n> to be present and a separate, robust mechanism is responsible for\n> keeping the sparsity patterns up to date, such as a Git-aware virtual\n> file system.\n>\n> ?\n\nI like this, but I also added in some of the wording suggestions from\nJunio here, so it's\na bit longer but has both some of his suggested wording and yours for\nslightly different aspects that I think works well together.\n\n>\n> > ++\n> > +Note that the checking and clearing of the SKIP_WORKTREE bit only\n> > +happens when core.sparseCheckout is true, so this config option has no\n> > +effect unless core.sparseCheckout is true.\n>\n> Good note. Same nit about the user not necessarily knowing what\n> SKIP_WORKTREE means applies. Also, we can remove the extra words\n> \"Note that\" since the dutiful reader should be noting everything we\n> say. :) I think that would make\n>\n> +\n> Regardless of this setting, Git does not check for\n> present-but-unmatching files unless sparse checkout is enabled, so\n> this config option has no effect unless `core.sparseCheckout` is\n> `true`.\n\nI like this too. Thanks for the suggestions, the proposed changes,\nand the review.\n"},{"id":"449657","messageId":"20220226061222.797107-1-newren@gmail.com","threadId":"57444","inReplyTo":"20220224052259.30498-1-newren@gmail.com","subject":"[PATCH v4] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-02-26T06:12:22Z","receivedAt":"2022-02-26T06:12:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Typically with sparse checkouts, we expect files outside the sparsity\npatterns to be marked as SKIP_WORKTREE and be missing from the working\ntree. Sometimes this expectation would be violated however; including\nin cases such as:\n * users grabbing files from elsewhere and writing them to the worktree\n (perhaps by editing a cached copy in an editor, copying/renaming, or\n even untarring)\n * various git commands having incomplete or no support for the\n SKIP_WORKTREE bit[1,2]\n * users attempting to \"abort\" a sparse-checkout operation with a\n not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n working tree is not atomic)[3].\nWhen the SKIP_WORKTREE bit in the index did not reflect the presence of\nthe file in the working tree, it traditionally caused confusion and was\ndifficult to detect and recover from. So, in a sparse checkout, since\naf6a51875a (\"repo_read_index: clear SKIP_WORKTREE bit from files present\nin worktree\", 2022-01-06), Git automatically clears the SKIP_WORKTREE\nbit at index read time for entries corresponding to files that are\npresent in the working tree.\n\nThere is another workflow, however, where it is expected that paths\noutside the sparsity patterns appear to exist in the working tree and\nthat they do not lose the SKIP_WORKTREE bit, at least until they get\nmodified. A Git-aware virtual file system[4] takes advantage of its\nposition as a file system driver to expose all files in the working\ntree, fetch them on demand using partial clone on access, and tell Git\nto pay attention to them on demand by updating the sparse checkout\npattern on writes. This means that commands like \"git status\" only have\nto examine files that have potentially been modified, whereas commands\nlike \"ls\" are able to show the entire codebase without requiring manual\nupdates to the sparse checkout pattern.\n\nThus since af6a51875a, Git with such Git-aware virtual file systems\nunsets the SKIP_WORKTREE bit for all files and commands like \"git\nstatus\" have to fetch and examine them all.\n\nIntroduce a configuration setting sparse.expectFilesOutsideOfPatterns to\nallow limiting the tracked set of files to a small set once again. A\nGit-aware virtual file system or other application that wants to\nmaintain files outside of the sparse checkout can set this in a\nrepository to instruct Git not to check for the presence of\nSKIP_WORKTREE files. The setting defaults to false, so most users of\nsparse checkout will still get the benefit of an automatically updating\nindex to recover from the variety of difficult issues detailed in\naf6a51875a for paths with SKIP_WORKTREE set despite the path being\npresent.\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n[2] The three long paragraphs in the middle of\n https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n[1] such as the vfsd described in\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Elijah Newren \nReviewed-by: Jonathan Nieder \n---\nChanges since v3:\n * Many wording updates based on feedback and suggestions from both\n Jonathan and Junio. Thanks for providing concrete suggestions!\n\nRange-diff:\n1: 92a13a8f08 ! 1: 5ded1532cd repo_read_index: add config to expect files outside sparse patterns\n @@ Commit message\n \n Typically with sparse checkouts, we expect files outside the sparsity\n patterns to be marked as SKIP_WORKTREE and be missing from the working\n - tree. In edge cases, this can be violated and cause confusion, so in a\n - sparse checkout, since 11d46a399d (\"repo_read_index: clear SKIP_WORKTREE\n - bit from files present in worktree\", 2022-01-06), Git automatically\n - clears the SKIP_WORKTREE bit at read time for entries corresponding to\n - files that are present in the working tree.\n + tree. Sometimes this expectation would be violated however; including\n + in cases such as:\n + * users grabbing files from elsewhere and writing them to the worktree\n + (perhaps by editing a cached copy in an editor, copying/renaming, or\n + even untarring)\n + * various git commands having incomplete or no support for the\n + SKIP_WORKTREE bit[1,2]\n + * users attempting to \"abort\" a sparse-checkout operation with a\n + not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n + working tree is not atomic)[3].\n + When the SKIP_WORKTREE bit in the index did not reflect the presence of\n + the file in the working tree, it traditionally caused confusion and was\n + difficult to detect and recover from. So, in a sparse checkout, since\n + af6a51875a (\"repo_read_index: clear SKIP_WORKTREE bit from files present\n + in worktree\", 2022-01-06), Git automatically clears the SKIP_WORKTREE\n + bit at index read time for entries corresponding to files that are\n + present in the working tree.\n \n - However, there is a more atypical situation where this situation would\n - be expected. A Git-aware virtual file system[1] takes advantage of its\n + There is another workflow, however, where it is expected that paths\n + outside the sparsity patterns appear to exist in the working tree and\n + that they do not lose the SKIP_WORKTREE bit, at least until they get\n + modified. A Git-aware virtual file system[4] takes advantage of its\n position as a file system driver to expose all files in the working\n tree, fetch them on demand using partial clone on access, and tell Git\n to pay attention to them on demand by updating the sparse checkout\n @@ Commit message\n like \"ls\" are able to show the entire codebase without requiring manual\n updates to the sparse checkout pattern.\n \n - Thus since 11d46a399d, Git with such Git-aware virtual file systems\n + Thus since af6a51875a, Git with such Git-aware virtual file systems\n unsets the SKIP_WORKTREE bit for all files and commands like \"git\n status\" have to fetch and examine them all.\n \n @@ Commit message\n SKIP_WORKTREE files. The setting defaults to false, so most users of\n sparse checkout will still get the benefit of an automatically updating\n index to recover from the variety of difficult issues detailed in\n - 11d46a399d for paths with SKIP_WORKTREE set despite the path being\n + af6a51875a for paths with SKIP_WORKTREE set despite the path being\n present.\n \n + [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n + [2] The three long paragraphs in the middle of\n + https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n + [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n [1] such as the vfsd described in\n https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n \n - Helped-by: Jonathan Nieder \n Signed-off-by: Jonathan Nieder \n Signed-off-by: Elijah Newren \n + Reviewed-by: Jonathan Nieder \n \n ## Documentation/config.txt ##\n @@ Documentation/config.txt: include::config/sequencer.txt[]\n @@ Documentation/config/sparse.txt (new)\n @@\n +sparse.expectFilesOutsideOfPatterns::\n +\tTypically with sparse checkouts, files not matching any\n -+\tsparsity patterns are marked as such in the index file and\n -+\tmissing from the working tree. Accordingly, Git will\n -+\tordinarily check whether files that the index indicates are\n -+\toutside of the sparse area are present in the working tree and\n -+\tmark them as present in the index if so. This option can be\n -+\tused to tell Git that such present-but-unmatching files are\n -+\texpected and to stop checking for them.\n ++\tsparsity patterns are marked with a SKIP_WORKTREE bit in the\n ++\tindex and are missing from the working tree. Accordingly, Git\n ++\twill ordinarily check whether files with the SKIP_WORKTREE bit\n ++\tare in fact present in the working tree contrary to\n ++\texpectations. If Git finds any, it marks those paths as\n ++\tpresent by clearing the relevant SKIP_WORKTREE bits. This\n ++\toption can be used to tell Git that such\n ++\tpresent-despite-skipped files are expected and to stop\n ++\tchecking for them.\n ++\n -+The default is `false`. Paths which are marked as SKIP_WORKTREE\n -+despite being present (which can occur for a few different reasons)\n -+typically present a range of problems which are difficult for users to\n -+discover and recover from. The default setting avoids such issues.\n ++The default is `false`, which allows Git to automatically recover\n ++from the list of files in the index and working tree falling out of\n ++sync.\n ++\n -+A Git-based virtual file system (VFS) can turn the usual expectation\n -+on its head: files are present in the working copy but do not take\n -+up much disk space because their contents are not downloaded until\n -+they are accessed. With such a virtual file system layer, most files\n -+do not match the sparsity patterns at first, and the VFS layer\n -+updates the sparsity patterns to add more files whenever files are\n -+written. Setting this to `true` supports such a setup where files are\n -+expected to be present outside the sparse area and a separate, robust\n -+mechanism is responsible for keeping the sparsity patterns up to date.\n ++Set this to `true` if you are in a setup where some external factor\n ++relieves Git of the responsibility for maintaining the consistency\n ++between the presence of working tree files and sparsity patterns. For\n ++example, if you have a Git-aware virtual file system that has a robust\n ++mechanism for keeping the working tree and the sparsity patterns up to\n ++date based on access patterns.\n ++\n -+Note that the checking and clearing of the SKIP_WORKTREE bit only\n -+happens when core.sparseCheckout is true, so this config option has no\n -+effect unless core.sparseCheckout is true.\n ++Regardless of this setting, Git does not check for\n ++present-despite-skipped files unless sparse checkout is enabled, so\n ++this config option has no effect unless `core.sparseCheckout` is\n ++`true`.\n \n ## cache.h ##\n @@ cache.h: extern const char *core_fsmonitor;\n\n Documentation/config.txt | 2 ++\n Documentation/config/sparse.txt | 27 +++++++++++++++++++++++++++\n cache.h | 1 +\n config.c | 14 ++++++++++++++\n environment.c | 1 +\n sparse-index.c | 3 ++-\n t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n 7 files changed, 66 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/sparse.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b168f02dc3..8628ae2634 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -468,6 +468,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..aff49a8d3a\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,27 @@\n+sparse.expectFilesOutsideOfPatterns::\n+\tTypically with sparse checkouts, files not matching any\n+\tsparsity patterns are marked with a SKIP_WORKTREE bit in the\n+\tindex and are missing from the working tree. Accordingly, Git\n+\twill ordinarily check whether files with the SKIP_WORKTREE bit\n+\tare in fact present in the working tree contrary to\n+\texpectations. If Git finds any, it marks those paths as\n+\tpresent by clearing the relevant SKIP_WORKTREE bits. This\n+\toption can be used to tell Git that such\n+\tpresent-despite-skipped files are expected and to stop\n+\tchecking for them.\n++\n+The default is `false`, which allows Git to automatically recover\n+from the list of files in the index and working tree falling out of\n+sync.\n++\n+Set this to `true` if you are in a setup where some external factor\n+relieves Git of the responsibility for maintaining the consistency\n+between the presence of working tree files and sparsity patterns. For\n+example, if you have a Git-aware virtual file system that has a robust\n+mechanism for keeping the working tree and the sparsity patterns up to\n+date based on access patterns.\n++\n+Regardless of this setting, Git does not check for\n+present-despite-skipped files unless sparse checkout is enabled, so\n+this config option has no effect unless `core.sparseCheckout` is\n+`true`.\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b..b6b8e83ae3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\n+extern int sparse_expect_files_outside_of_patterns;\n \n /*\n * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a0..9b9ad1500a 100644\n--- a/config.c\n+++ b/config.c\n@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \treturn platform_core_config(var, value, cb);\n }\n \n+static int git_default_sparse_config(const char *var, const char *value)\n+{\n+\tif (!strcmp(var, \"sparse.expectfilesoutsideofpatterns\")) {\n+\t\tsparse_expect_files_outside_of_patterns = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* Add other config variables here and to Documentation/config/sparse.txt. */\n+\treturn 0;\n+}\n+\n static int git_default_i18n_config(const char *var, const char *value)\n {\n \tif (!strcmp(var, \"i18n.commitencoding\"))\n@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (starts_with(var, \"sparse.\"))\n+\t\treturn git_default_sparse_config(var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a..fb55bf6129 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -70,6 +70,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int core_sparse_checkout_cone;\n+int sparse_expect_files_outside_of_patterns;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/sparse-index.c b/sparse-index.c\nindex eed170cd8f..daeb5112a1 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n \n \tint i;\n \n-\tif (!core_apply_sparse_checkout)\n+\tif (!core_apply_sparse_checkout ||\n+\t sparse_expect_files_outside_of_patterns)\n \t\treturn;\n \n restart:\ndiff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\nindex 3deb490187..d1833c0f31 100755\n--- a/t/t1090-sparse-checkout-scope.sh\n+++ b/t/t1090-sparse-checkout-scope.sh\n@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '\n \ttest \"$(cat b)\" = \"modified\"\n '\n \n+test_expect_success 'skip-worktree on files outside sparse patterns' '\n+\tgit sparse-checkout disable &&\n+\tgit sparse-checkout set --no-cone \"a*\" &&\n+\tgit checkout-index --all --ignore-skip-worktree-bits &&\n+\n+\tgit ls-files -t >output &&\n+\t! grep ^S output >actual &&\n+\ttest_must_be_empty actual &&\n+\n+\ttest_config sparse.expectFilesOutsideOfPatterns true &&\n+\tcat <<-\\EOF >expect &&\n+\tS b\n+\tS c\n+\tEOF\n+\tgit ls-files -t >output &&\n+\tgrep ^S output >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '\n \ttest_create_repo server &&\n \tgit clone \"file://$(pwd)/server\" client &&\n-- \n2.35.1.278.g4f05fa7020.dirty\n\n"},{"id":"450053","messageId":"20220302043301.1620188-1-newren@gmail.com","threadId":"57444","inReplyTo":"20220226061222.797107-1-newren@gmail.com","subject":"[PATCH v5] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-03-02T04:33:01Z","receivedAt":"2022-03-02T04:33:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Typically with sparse checkouts, we expect files outside the sparsity\npatterns to be marked as SKIP_WORKTREE and be missing from the working\ntree. Sometimes this expectation would be violated however; including\nin cases such as:\n * users grabbing files from elsewhere and writing them to the worktree\n (perhaps by editing a cached copy in an editor, copying/renaming, or\n even untarring)\n * various git commands having incomplete or no support for the\n SKIP_WORKTREE bit[1,2]\n * users attempting to \"abort\" a sparse-checkout operation with a\n not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n working tree is not atomic)[3].\nWhen the SKIP_WORKTREE bit in the index did not reflect the presence of\nthe file in the working tree, it traditionally caused confusion and was\ndifficult to detect and recover from. So, in a sparse checkout, since\naf6a51875a (\"repo_read_index: clear SKIP_WORKTREE bit from files present\nin worktree\", 2022-01-06), Git automatically clears the SKIP_WORKTREE\nbit at index read time for entries corresponding to files that are\npresent in the working tree.\n\nThere is another workflow, however, where it is expected that paths\noutside the sparsity patterns appear to exist in the working tree and\nthat they do not lose the SKIP_WORKTREE bit, at least until they get\nmodified. A Git-aware virtual file system[4] takes advantage of its\nposition as a file system driver to expose all files in the working\ntree, fetch them on demand using partial clone on access, and tell Git\nto pay attention to them on demand by updating the sparse checkout\npattern on writes. This means that commands like \"git status\" only have\nto examine files that have potentially been modified, whereas commands\nlike \"ls\" are able to show the entire codebase without requiring manual\nupdates to the sparse checkout pattern.\n\nThus since af6a51875a, Git with such Git-aware virtual file systems\nunsets the SKIP_WORKTREE bit for all files and commands like \"git\nstatus\" have to fetch and examine them all.\n\nIntroduce a configuration setting sparse.expectFilesOutsideOfPatterns to\nallow limiting the tracked set of files to a small set once again. A\nGit-aware virtual file system or other application that wants to\nmaintain files outside of the sparse checkout can set this in a\nrepository to instruct Git not to check for the presence of\nSKIP_WORKTREE files. The setting defaults to false, so most users of\nsparse checkout will still get the benefit of an automatically updating\nindex to recover from the variety of difficult issues detailed in\naf6a51875a for paths with SKIP_WORKTREE set despite the path being\npresent.\n\n[1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n[2] The three long paragraphs in the middle of\n https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n[4] such as the vfsd described in\nhttps://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n\nSigned-off-by: Jonathan Nieder \nSigned-off-by: Elijah Newren \nReviewed-by: Jonathan Nieder \n---\nChanges since v4:\n - Corrected an incorrect footnote\n\nRange-diff:\n1: 5ded1532cd ! 1: d348cc766f repo_read_index: add config to expect files outside sparse patterns\n @@ Commit message\n [2] The three long paragraphs in the middle of\n https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n - [1] such as the vfsd described in\n + [4] such as the vfsd described in\n https://lore.kernel.org/git/20220207190320.2960362-1-jonathantanmy@google.com/\n \n Signed-off-by: Jonathan Nieder \n\n Documentation/config.txt | 2 ++\n Documentation/config/sparse.txt | 27 +++++++++++++++++++++++++++\n cache.h | 1 +\n config.c | 14 ++++++++++++++\n environment.c | 1 +\n sparse-index.c | 3 ++-\n t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++\n 7 files changed, 66 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/sparse.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b168f02dc3..8628ae2634 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -468,6 +468,8 @@ include::config/sequencer.txt[]\n \n include::config/showbranch.txt[]\n \n+include::config/sparse.txt[]\n+\n include::config/splitindex.txt[]\n \n include::config/ssh.txt[]\ndiff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt\nnew file mode 100644\nindex 0000000000..aff49a8d3a\n--- /dev/null\n+++ b/Documentation/config/sparse.txt\n@@ -0,0 +1,27 @@\n+sparse.expectFilesOutsideOfPatterns::\n+\tTypically with sparse checkouts, files not matching any\n+\tsparsity patterns are marked with a SKIP_WORKTREE bit in the\n+\tindex and are missing from the working tree. Accordingly, Git\n+\twill ordinarily check whether files with the SKIP_WORKTREE bit\n+\tare in fact present in the working tree contrary to\n+\texpectations. If Git finds any, it marks those paths as\n+\tpresent by clearing the relevant SKIP_WORKTREE bits. This\n+\toption can be used to tell Git that such\n+\tpresent-despite-skipped files are expected and to stop\n+\tchecking for them.\n++\n+The default is `false`, which allows Git to automatically recover\n+from the list of files in the index and working tree falling out of\n+sync.\n++\n+Set this to `true` if you are in a setup where some external factor\n+relieves Git of the responsibility for maintaining the consistency\n+between the presence of working tree files and sparsity patterns. For\n+example, if you have a Git-aware virtual file system that has a robust\n+mechanism for keeping the working tree and the sparsity patterns up to\n+date based on access patterns.\n++\n+Regardless of this setting, Git does not check for\n+present-despite-skipped files unless sparse checkout is enabled, so\n+this config option has no effect unless `core.sparseCheckout` is\n+`true`.\ndiff --git a/cache.h b/cache.h\nindex 281f00ab1b..b6b8e83ae3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;\n \n extern int core_apply_sparse_checkout;\n extern int core_sparse_checkout_cone;\n+extern int sparse_expect_files_outside_of_patterns;\n \n /*\n * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).\ndiff --git a/config.c b/config.c\nindex 2bffa8d4a0..9b9ad1500a 100644\n--- a/config.c\n+++ b/config.c\n@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \treturn platform_core_config(var, value, cb);\n }\n \n+static int git_default_sparse_config(const char *var, const char *value)\n+{\n+\tif (!strcmp(var, \"sparse.expectfilesoutsideofpatterns\")) {\n+\t\tsparse_expect_files_outside_of_patterns = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\t/* Add other config variables here and to Documentation/config/sparse.txt. */\n+\treturn 0;\n+}\n+\n static int git_default_i18n_config(const char *var, const char *value)\n {\n \tif (!strcmp(var, \"i18n.commitencoding\"))\n@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (starts_with(var, \"sparse.\"))\n+\t\treturn git_default_sparse_config(var, value);\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex fd0501e77a..fb55bf6129 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -70,6 +70,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n int core_sparse_checkout_cone;\n+int sparse_expect_files_outside_of_patterns;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/sparse-index.c b/sparse-index.c\nindex eed170cd8f..daeb5112a1 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)\n \n \tint i;\n \n-\tif (!core_apply_sparse_checkout)\n+\tif (!core_apply_sparse_checkout ||\n+\t sparse_expect_files_outside_of_patterns)\n \t\treturn;\n \n restart:\ndiff --git a/t/t1090-sparse-checkout-scope.sh b/t/t1090-sparse-checkout-scope.sh\nindex 3deb490187..d1833c0f31 100755\n--- a/t/t1090-sparse-checkout-scope.sh\n+++ b/t/t1090-sparse-checkout-scope.sh\n@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' '\n \ttest \"$(cat b)\" = \"modified\"\n '\n \n+test_expect_success 'skip-worktree on files outside sparse patterns' '\n+\tgit sparse-checkout disable &&\n+\tgit sparse-checkout set --no-cone \"a*\" &&\n+\tgit checkout-index --all --ignore-skip-worktree-bits &&\n+\n+\tgit ls-files -t >output &&\n+\t! grep ^S output >actual &&\n+\ttest_must_be_empty actual &&\n+\n+\ttest_config sparse.expectFilesOutsideOfPatterns true &&\n+\tcat <<-\\EOF >expect &&\n+\tS b\n+\tS c\n+\tEOF\n+\tgit ls-files -t >output &&\n+\tgrep ^S output >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' '\n \ttest_create_repo server &&\n \tgit clone \"file://$(pwd)/server\" client &&\n-- \n2.35.1.355.g49d31b1b5a.dirty\n\n"},{"id":"450063","messageId":"xmqq1qzkolet.fsf@gitster.g","threadId":"57444","inReplyTo":"20220302043301.1620188-1-newren@gmail.com","subject":"Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-02T07:36:58Z","receivedAt":"2022-03-02T07:37:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Typically with sparse checkouts, we expect files outside the sparsity\n> patterns to be marked as SKIP_WORKTREE and be missing from the working\n> tree. Sometimes this expectation would be violated however; including\n> in cases such as:\n> * users grabbing files from elsewhere and writing them to the worktree\n> (perhaps by editing a cached copy in an editor, copying/renaming, or\n> even untarring)\n> * various git commands having incomplete or no support for the\n> SKIP_WORKTREE bit[1,2]\n> * users attempting to \"abort\" a sparse-checkout operation with a\n> not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n> working tree is not atomic)[3].\n> When the SKIP_WORKTREE bit in the index did not reflect the presence of\n> the file in the working tree, it traditionally caused confusion and was\n> difficult to detect and recover from. So, in a sparse checkout, since\n> af6a51875a (\"repo_read_index: clear SKIP_WORKTREE bit from files present\n> in worktree\", 2022-01-06), Git automatically clears the SKIP_WORKTREE\n\nThe reference is a bit off here. Here is what I get locally:\n\n af6a51875a (repo_read_index: clear SKIP_WORKTREE bit from files\n present in worktree, 2022-01-14)\n\nand that is in the version I have locally in 'next'.\n\n> [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n> [2] The three long paragraphs in the middle of\n> https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n> [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n> [4] such as the vfsd described in\n\nHere is another difference from the version I have locally in\n'next', which I didn't notice that this [4] was misspelt as [1]\nbefore applying.\n\nEverything else seems the same, so let's not bother reverting the\nold one out of 'next' and merging this version after fixing this\nversion up. What we have is good enough modulo [4] vs [1].\n\nThanks.\n\n"},{"id":"450066","messageId":"CABPp-BFqJGHDMz744Q7FV22tW6dpGNoBwAW=P1HZE6PtZeHN6A@mail.gmail.com","threadId":"57444","inReplyTo":"xmqq1qzkolet.fsf@gitster.g","subject":"Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-03-02T08:01:34Z","receivedAt":"2022-03-02T08:01:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 1, 2022 at 11:37 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Typically with sparse checkouts, we expect files outside the sparsity\n> > patterns to be marked as SKIP_WORKTREE and be missing from the working\n> > tree. Sometimes this expectation would be violated however; including\n> > in cases such as:\n> > * users grabbing files from elsewhere and writing them to the worktree\n> > (perhaps by editing a cached copy in an editor, copying/renaming, or\n> > even untarring)\n> > * various git commands having incomplete or no support for the\n> > SKIP_WORKTREE bit[1,2]\n> > * users attempting to \"abort\" a sparse-checkout operation with a\n> > not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the\n> > working tree is not atomic)[3].\n> > When the SKIP_WORKTREE bit in the index did not reflect the presence of\n> > the file in the working tree, it traditionally caused confusion and was\n> > difficult to detect and recover from. So, in a sparse checkout, since\n> > af6a51875a (\"repo_read_index: clear SKIP_WORKTREE bit from files present\n> > in worktree\", 2022-01-06), Git automatically clears the SKIP_WORKTREE\n>\n> The reference is a bit off here. Here is what I get locally:\n>\n> af6a51875a (repo_read_index: clear SKIP_WORKTREE bit from files\n> present in worktree, 2022-01-14)\n>\n> and that is in the version I have locally in 'next'.\n\nUgh, forgot to update the date when I updated the reference when you\npointed that out.\n\n> > [1] https://lore.kernel.org/git/xmqqbmb1a7ga.fsf@gitster-ct.c.googlers.com/\n> > [2] The three long paragraphs in the middle of\n> > https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/\n> > [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/\n> > [4] such as the vfsd described in\n>\n> Here is another difference from the version I have locally in\n> 'next', which I didn't notice that this [4] was misspelt as [1]\n> before applying.\n\nSorry, I hadn't noticed you merging to next, and I saw in the irc logs\nthe discussion about this 1 vs. 4 between you and jrnieder so I\nthought I'd fix it.\n\n> Everything else seems the same, so let's not bother reverting the\n> old one out of 'next' and merging this version after fixing this\n> version up. What we have is good enough modulo [4] vs [1].\n\nSounds good.\n"},{"id":"450100","messageId":"7f8b3ed3-1836-0e8d-e6f4-77928ad62e40@github.com","threadId":"57444","inReplyTo":"20220226061222.797107-1-newren@gmail.com","subject":"Re: [PATCH v4] repo_read_index: add config to expect files outside sparse patterns","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-02T13:37:12Z","receivedAt":"2022-03-02T13:37:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/26/2022 1:12 AM, Elijah Newren wrote:\n> Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to\n> allow limiting the tracked set of files to a small set once again. A\n> Git-aware virtual file system or other application that wants to\n> maintain files outside of the sparse checkout can set this in a\n> repository to instruct Git not to check for the presence of\n> SKIP_WORKTREE files. The setting defaults to false, so most users of\n> sparse checkout will still get the benefit of an automatically updating\n> index to recover from the variety of difficult issues detailed in\n> af6a51875a for paths with SKIP_WORKTREE set despite the path being\n> present.\n\nSorry that I lost track of this thread recently. I re-read this\nversion that made it to 'next' already and have no issues.\n\nThis approach will work well for our needs in the microsoft/git\nfork. Likely, we'll enable the sparse_expect_files_outside_of_patterns\nglobal when we initialize our core_virtualfilesystem global.\nThen, we'll probably replace some of the uses of core_virtualfilesytem\nwith this new global in some of the conditions that we inserted to\nwork with previous restrictions in this area. Then, those changes will\nbe fit for submitting here. If anyone needs those changes sooner, then\ndo not hesitate to make similar changes. We will adapt in our fork to\nwhatever the Git community thinks is best.\n\nThanks,\n-Stolee\n"}]}