{"thread":{"id":"52995","subject":"[PATCH 0/7] Sparse checkout improvements -- improved sparsity updating","startedAt":"2020-03-15T01:33:37Z","lastAt":"2020-03-27T13:22:12Z","messageCount":69,"participants":["Elijah Newren via GitGitGadget","Derrick Stolee","Elijah Newren","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"393239","messageId":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":null,"subject":"[PATCH 0/7] Sparse checkout improvements -- improved sparsity updating","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:26Z","receivedAt":"2020-03-15T01:33:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series provides a replacement for the git read-tree -mu HEAD-based\nsparsity updating used by sparse-checkout, fixing the myriad problems it\ncauses. The most important bit is patch 5 and its lengthy commit message\nexplaining the current state and rationale.\n\nConsider the series RFC for now; I probably need to add some tests, get some\npeople at work who are willing to test it in real cases, and get some\ncommunity feedback particularly on the choices/comments highlighted in the\ncommit message to patch 5. But it's at least basically functional so I\nwanted to send it out to the community for feedback.\n\nElijah Newren (7):\n unpack-trees: fix minor typo in comment\n t1091: make some tests a little more defensive against failures\n unpack-trees: allow check_updates() to work on a different index\n unpack-trees: do not mark a dirty path with SKIP_WORKTREE\n unpack-trees: add a new update_sparsity() function\n sparse-checkout: use new update_sparsity() function\n sparse-checkout: provide a new update subcommand\n\n Documentation/git-sparse-checkout.txt | 10 +++\n builtin/sparse-checkout.c | 49 ++++-------\n t/t1091-sparse-checkout-builtin.sh | 34 ++++++--\n unpack-trees.c | 119 ++++++++++++++++++++++++--\n unpack-trees.h | 2 +\n 5 files changed, 169 insertions(+), 45 deletions(-)\n\n\nbase-commit: 30e9940356dc67959877f4b2417da33ebdefbb79\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-726%2Fnewren%2Fsparse-checkout-improvements-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-726/newren/sparse-checkout-improvements-v1\nPull-Request: https://github.com/git/git/pull/726\n-- \ngitgitgadget\n"},{"id":"393248","messageId":"650db6863426ae2b324ba717f898247f44279cb8.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 7/7] sparse-checkout: provide a new update subcommand","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:33Z","receivedAt":"2020-03-15T01:57:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf commands like merge or rebase materialize files as part of their work,\nor a previous sparse-checkout command failed to update individual files\ndue to dirty changes, users may want a command to simply 'reapply' the\nsparsity rules. Provide one.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-sparse-checkout.txt | 10 ++++++++++\n builtin/sparse-checkout.c | 10 +++++++++-\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex c0342e53938..27f4392489f 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -70,6 +70,16 @@ C-style quoted strings.\n \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n \tas directory names as in the 'set' subcommand.\n \n+'update'::\n+\tUpdate the sparseness of paths in the working tree based on the\n+\texisting patterns. Commands like merge or rebase can materialize\n+\tpaths to do their work (e.g. in order to show you a conflict), and\n+\tother sparse-checkout commands might fail to sparsify an individual\n+\tfile (e.g. because it has unstaged changes or conflicts). In such\n+\tcases, it can make sense to run `git sparse-checkout update` later\n+\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n+\tor committing changes, etc.).\n+\n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n \tworking directory to include all files. Leaves the sparse-checkout\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 5d3ec2e6be9..2ae21011dfd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -18,7 +18,7 @@\n static const char *empty_base = \"\";\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n+\tN_(\"git sparse-checkout (init|list|set|add|update|disable) \"),\n \tNULL\n };\n \n@@ -552,6 +552,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \treturn modify_pattern_list(argc, argv, m);\n }\n \n+static int sparse_checkout_update(int argc, const char **argv)\n+{\n+\trepo_read_index(the_repository);\n+\treturn update_working_directory(NULL);\n+}\n+\n static int sparse_checkout_disable(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -601,6 +607,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n \t\tif (!strcmp(argv[0], \"add\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n+\t\tif (!strcmp(argv[0], \"update\"))\n+\t\t\treturn sparse_checkout_update(argc, argv);\n \t\tif (!strcmp(argv[0], \"disable\"))\n \t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n-- \ngitgitgadget\n"},{"id":"393252","messageId":"9ab9a9d60371d0a54b2a8fc10512ad6d8daaad43.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 1/7] unpack-trees: fix minor typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:27Z","receivedAt":"2020-03-15T02:26:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1ecdab33040..0d0eec0221e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1618,7 +1618,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t\t/*\n \t\t * Sparse checkout loop #2: set NEW_SKIP_WORKTREE on entries not in loop #1\n-\t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n+\t\t * If they will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n \t\tmark_new_skip_worktree(o->pl, &o->result,\n-- \ngitgitgadget\n\n"},{"id":"393254","messageId":"02afa9c13c1cd299d67003e6984da0b38a53790b.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 3/7] unpack-trees: allow check_updates() to work on a different index","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:29Z","receivedAt":"2020-03-15T03:13:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncheck_updates() previously assumed it was working on o->result. We want\nto use this function in combination with a different index_state, so\ntake the intended index_state as a parameter.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0d0eec0221e..853d843b17a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -357,12 +357,12 @@ static void report_collided_checkout(struct index_state *index)\n \tstring_list_clear(&list, 0);\n }\n \n-static int check_updates(struct unpack_trees_options *o)\n+static int check_updates(struct unpack_trees_options *o,\n+\t\t\t struct index_state *index)\n {\n \tunsigned cnt = 0;\n \tint errs = 0;\n \tstruct progress *progress;\n-\tstruct index_state *index = &o->result;\n \tstruct checkout state = CHECKOUT_INIT;\n \tint i;\n \n@@ -1667,7 +1667,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t}\n \t}\n \n-\tret = check_updates(o) ? (-2) : 0;\n+\tret = check_updates(o, &o->result) ? (-2) : 0;\n \tif (o->dst_index) {\n \t\tmove_index_extensions(&o->result, o->src_index);\n \t\tif (!ret) {\n-- \ngitgitgadget\n\n"},{"id":"393256","messageId":"baa4f23421b6bc245b790cdfc1fc64f5e064bca0.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 4/7] unpack-trees: do not mark a dirty path with SKIP_WORKTREE","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:30Z","receivedAt":"2020-03-15T03:41:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf a path is dirty, removing from the working tree risks losing data.\nAs such, we want to make sure any such path is not marked with\nSKIP_WORKTREE. While the current callers of this code detect this case\nand re-populate with a previous set of sparsity patterns, we want to\nallow some paths to be marked with SKIP_WORKTREE while others are left\nunmarked without it being considered an error. The reason this\nshouldn't be considered an error is that SKIP_WORKTREE has always been\nan advisory-only setting; merge and rebase for example were free to\nmaterialize paths and clear the SKIP_WORKTREE bit in order to accomplish\ntheir work even though they kept the SKIP_WORKTREE bit set for other\npaths. Leaving dirty working files in the working tree is thus a\nnatural extension of what we have already been doing.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 853d843b17a..8f10ac91ce1 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -504,8 +504,11 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\t * also stat info may have lost after merged_entry() so calling\n \t\t * verify_uptodate() again may fail\n \t\t */\n-\t\tif (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))\n+\t\tif (!(ce->ce_flags & CE_UPDATE) &&\n+\t\t verify_uptodate_sparse(ce, o)) {\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \t\t\treturn -1;\n+\t\t}\n \t\tce->ce_flags |= CE_WT_REMOVE;\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"393257","messageId":"9f13a2e8c64d47a5b3a4a2a96451decaf548ca4e.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 5/7] unpack-trees: add a new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:31Z","receivedAt":"2020-03-15T03:46:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPreviously, the only way to update the SKIP_WORKTREE bits for various\npaths was invoking `git read-tree -mu HEAD` or calling the same code\nthat this codepath invoked. This however had a number of problems.\nLet's look at all the various cases:\n\n Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files):\n A) There is no file in the way\n B) There is an untracked file in the way\n Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files):\n C) The path is clean\n D) The path is unmerged\n E) The path has unstaged changes\n F) The path has staged changes (differs from HEAD)\n\nFor `git read-tree -mu HEAD`, the behavior is as follows:\n A) Materialize the file, as expected\n B) Refuse to make *any* changes (to this path or any others)\n C) Remove the file, as expected\n D) Refuse to make *any* changes (to this path or any others)\n E) Refuse to make *any* changes (to this path or any others)\n F) Removes the file from the working copy AND resets the index\n to match HEAD\n\nRefusing to modify files that could result in data loss as in cases B,\nD, and E is good, but refusing to work on any other paths is very\nproblematic for users. If they are in the middle of a rebase or have\nmade modifications to files that bring in more dependencies, then for\ntheir build to work they need to update the sparse paths -- which they\nare prevented from doing. Sometimes they stage the files and re-try,\nat which point they run into case F and believe that their changes are\nlost. (Even if they come ask an expert, they have to trawl through\ntheir loose objects looking for which ones match which files.)\n\nAdd a new update_sparsity() function which behaves as follows in these\ncases:\n A) Materialize the file, as expected\n B) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n bit, and print a warning (thus leaving them in a state where\n status will report the file as modified, which seems logical).\n C) Remove the file, as expected\n D) Do NOT mark this path as SKIP_WORKTREE, but allow others paths\n to be updated.\n E) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n the dirty path, but allow other paths to be updated.\n F) Mark the path as SKIP_WORKTREE, but do not revert the version\n stored in the index to match HEAD; leave the contents alone.\n\nI tried a different behavior for B (leave the SKIP_WORKTREE bit set),\nbut found it very surprising and counter-intuitive (e.g. the user sees\nit is present along with all the other files in that directory, tries to\nstage it, but git add ignores it since the SKIP_WORKTREE bit is set).\nA, B, C, and E all seem like optimal behavior to me. D may be as well,\nthough I wonder if printing a warning would be an improvement. Some\nmight be slightly surprised by F at first, but given that it does the\nright thing with `git commit` and even `git commit -a` (add ignores\nentries that are marked SKIP_WORKTREE and thus doesn't delete them, and\ncommit -a is similar), it seems logical to me.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 106 +++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.h | 2 +\n 2 files changed, 108 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8f10ac91ce1..21234635573 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -255,6 +255,25 @@ static void display_error_msgs(struct unpack_trees_options *o)\n \t\tfprintf(stderr, _(\"Aborting\\n\"));\n }\n \n+/*\n+ * display all the \"error\" messages as warnings\n+ */\n+static void display_warning_msgs(struct unpack_trees_options *o)\n+{\n+\tint e, i;\n+\tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n+\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n+\t\tif (rejects->nr > 0) {\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\t\t\tfor (i = 0; i < rejects->nr; i++)\n+\t\t\t\tstrbuf_addstr(&path, rejects->items[i].string);\n+\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n+\t\t\tstrbuf_release(&path);\n+\t\t}\n+\t\tstring_list_clear(rejects, 0);\n+\t}\n+}\n+\n static int check_submodule_move_head(const struct cache_entry *ce,\n \t\t\t\t const char *old_id,\n \t\t\t\t const char *new_id,\n@@ -1708,6 +1727,93 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tgoto done;\n }\n \n+/*\n+ * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n+ * working directory to match.\n+ *\n+ * Returns\n+ * 0: success with no warnings\n+ * 1: success with warnings (warnings come either from (a) dirty changes\n+ * present in which case we avoid marking those paths as\n+ * SKIP_WORKTREE, or (b) from untracked files being in the way us\n+ * checking a file out of the index, in which case we leave the file\n+ * in the working tree alone while clearing SKIP_WORKTREE)\n+ * -1: failure to manipulate the resulting index\n+ * -2: failure to reflect the changes to the work tree.\n+ *\n+ * CE_NEW_SKIP_WORKTREE is used internally.\n+ */\n+int update_sparsity(struct unpack_trees_options *o)\n+{\n+\tstruct pattern_list pl;\n+\tint i, empty_worktree, ret = 0;\n+\tunsigned old_show_all_errors;\n+\n+\told_show_all_errors = o->show_all_errors;\n+\to->show_all_errors = 1;\n+\n+\t/* Sanity checks */\n+\tif (!o->update || o->index_only || o->skip_sparse_checkout)\n+\t\tBUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n+\tif (o->src_index != o->dst_index || o->fn)\n+\t\tBUG(\"update_sparsity() called wrong\");\n+\n+\ttrace_performance_enter();\n+\n+\tif (!o->pl) {\n+\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tmemset(&pl, 0, sizeof(pl));\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n+\t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0) {\n+\t\t\t/* FIXME: Skip to check_updates()?? */\n+\t\t\to->skip_sparse_checkout = 1;\n+\t\t\tgoto skip_sparse_checkout;\n+\t\t} else\n+\t\t\to->pl = &pl;\n+\t\tfree(sparse);\n+\t}\n+\n+\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n+\tmark_all_ce_unused(o->src_index);\n+\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n+\n+\t/* Then loop over entries and update/remove as needed */\n+\tret = 0;\n+\tempty_worktree = 1;\n+\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = o->src_index->cache[i];\n+\n+\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n+\t\t\tret = 1;\n+\n+\t\tif (!ce_skip_worktree(ce))\n+\t\t\tempty_worktree = 0;\n+\n+\t}\n+\n+\t/*\n+\t * Sparse checkout is meant to narrow down checkout area\n+\t * but it does not make sense to narrow down to empty working\n+\t * tree. This is usually a mistake in sparse checkout rules.\n+\t * Do not allow users to do that.\n+\t */\n+\tif (o->src_index->cache_nr && empty_worktree) {\n+\t\tret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n+\t\tgoto done;\n+\t}\n+\n+skip_sparse_checkout:\n+\tif (check_updates(o, o->src_index))\n+\t\tret = -2;\n+\n+done:\n+\tdisplay_warning_msgs(o);\n+\to->show_all_errors = old_show_all_errors;\n+\ttrace_performance_leave(\"update_sparsity\");\n+\treturn ret;\n+}\n+\n /* Here come the merge functions */\n \n static int reject_merge(const struct cache_entry *ce,\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex ae1557fb804..5d1a77ae249 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -90,6 +90,8 @@ struct unpack_trees_options {\n int unpack_trees(unsigned n, struct tree_desc *t,\n \t\t struct unpack_trees_options *options);\n \n+int update_sparsity(struct unpack_trees_options *options);\n+\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o);\n \n-- \ngitgitgadget\n\n"},{"id":"393259","messageId":"b07fdab55aae88cf30d9be41b81414c0664e5c31.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 2/7] t1091: make some tests a little more defensive against failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:28Z","receivedAt":"2020-03-15T04:25:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 44a91205d60..8607a8e6d1a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -278,6 +278,7 @@ test_expect_success 'cone mode: add parent path' '\n '\n \n test_expect_success 'revert to old sparse-checkout on bad update' '\n+\ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n \techo update >repo/deep/deeper2/a &&\n@@ -328,6 +329,7 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n '\n \n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n+\trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n \tcat >expect <<-\\EOF &&\n-- \ngitgitgadget\n\n"},{"id":"393260","messageId":"892933752c638a9ed938e046e466dd1e927058a6.1584169893.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH 6/7] sparse-checkout: use new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-14T07:11:32Z","receivedAt":"2020-03-15T04:55:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nRemove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\ncodepaths for setting the SKIP_WORKTREE bits and instead use the new\nupdate_sparsity() function.\n\nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 39 +++++++-----------------------\n t/t1091-sparse-checkout-builtin.sh | 32 +++++++++++++++++-------\n 2 files changed, 32 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 740da4b6d54..5d3ec2e6be9 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -97,47 +97,32 @@ static int update_working_directory(struct pattern_list *pl)\n \tint result = 0;\n \tstruct unpack_trees_options o;\n \tstruct lock_file lock_file = LOCK_INIT;\n-\tstruct object_id oid;\n-\tstruct tree *tree;\n-\tstruct tree_desc t;\n \tstruct repository *r = the_repository;\n \n-\tif (repo_read_index_unmerged(r))\n-\t\tdie(_(\"you need to resolve your current index first\"));\n-\n-\tif (get_oid(\"HEAD\", &oid))\n-\t\treturn 0;\n-\n-\ttree = parse_tree_indirect(&oid);\n-\tparse_tree(tree);\n-\tinit_tree_desc(&t, tree->buffer, tree->size);\n-\n \tmemset(&o, 0, sizeof(o));\n \to.verbose_update = isatty(2);\n-\to.merge = 1;\n \to.update = 1;\n-\to.fn = oneway_merge;\n \to.head_idx = -1;\n \to.src_index = r->index;\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n-\to.keep_pattern_list = !!pl;\n \n-\tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\n \n-\tcache_tree_free(&r->index->cache_tree);\n-\n \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n \n-\tcore_apply_sparse_checkout = 1;\n-\tresult = unpack_trees(1, &t, &o);\n+\tresult = update_sparsity(&o);\n \n-\tif (!result) {\n-\t\tprime_cache_tree(r, r->index, tree);\n+\tif (result == 1)\n+\t\t/*\n+\t\t * We don't do any special handling of warnings from untracked\n+\t\t * files in the way or dirty entries that can't be removed.\n+\t\t */\n+\t\tresult = 0;\n+\tif (!result)\n \t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n-\t} else\n+\telse\n \t\trollback_lock_file(&lock_file);\n \n \treturn result;\n@@ -304,8 +289,6 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_sparse_checkout_init_options,\n@@ -560,8 +543,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_sparse_checkout_set_options,\n@@ -577,8 +558,6 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tstruct strbuf match_all = STRBUF_INIT;\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n \n \tmemset(&pl, 0, sizeof(pl));\n \thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 8607a8e6d1a..0d93d3983e0 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n \tcheck_files repo a deep folder1\n '\n \n-test_expect_success 'revert to old sparse-checkout on bad update' '\n+test_expect_success 'not-up-to-date does not block rest of sparsification' '\n \ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n+\n \techo update >repo/deep/deeper2/a &&\n \tcp repo/.git/info/sparse-checkout expect &&\n-\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n-\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n-\ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tcheck_files repo/deep a deeper1 deeper2\n+\ttest_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n+\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tcheck_files repo/deep a deeper1 deeper2 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n+\tcheck_files repo/deep/deeper1/deepest a &&\n+\tcheck_files repo/deep/deeper2 a\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -316,12 +323,19 @@ test_expect_success '.gitignore should not warn about cone mode' '\n \ttest_i18ngrep ! \"disabling cone patterns\" err\n '\n \n-test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n+test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n \tgit clone repo dirty &&\n \techo dirty >dirty/folder1/a &&\n-\ttest_must_fail git -C dirty sparse-checkout init &&\n-\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_must_fail git -C dirty sparse-checkout disable &&\n+\n+\tgit -C dirty sparse-checkout init 2>err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\n+\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\n+\tgit -C dirty sparse-checkout disable &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-- \ngitgitgadget\n\n"},{"id":"393266","messageId":"f99f5379-0e27-4cce-de80-8f780b9544d6@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Sparse checkout improvements -- improved sparsity updating","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-15T12:26:50Z","receivedAt":"2020-03-15T12:26:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> This series provides a replacement for the git read-tree -mu HEAD-based\n> sparsity updating used by sparse-checkout, fixing the myriad problems it\n> causes. The most important bit is patch 5 and its lengthy commit message\n> explaining the current state and rationale.\n\nI'm excited to dig into this change! I'm pleasantly surprised you have\nsomething ready so soon after we talked about it at the summit.\n\n> Consider the series RFC for now; I probably need to add some tests, get some\n> people at work who are willing to test it in real cases, and get some\n> community feedback particularly on the choices/comments highlighted in the\n> commit message to patch 5. But it's at least basically functional so I\n> wanted to send it out to the community for feedback.\n\nCount me in for testing and reviewing. Just expect me to be a little\nslow this week as I'll likely be juggling work time with kid time as\nthings shut down. I'm very excited for this feature, and I'll try to\nhelp as much as I can.\n\nThanks,\n-Stolee\n"},{"id":"393269","messageId":"76845bbf-720f-114a-6941-fcdde536732f@gmail.com","threadId":"52995","inReplyTo":"baa4f23421b6bc245b790cdfc1fc64f5e064bca0.1584169893.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/7] unpack-trees: do not mark a dirty path with SKIP_WORKTREE","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-15T14:39:15Z","receivedAt":"2020-03-15T14:39:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"cOn 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> If a path is dirty, removing from the working tree risks losing data.\n> As such, we want to make sure any such path is not marked with\n> SKIP_WORKTREE. While the current callers of this code detect this case\n> and re-populate with a previous set of sparsity patterns, we want to\n> allow some paths to be marked with SKIP_WORKTREE while others are left\n> unmarked without it being considered an error. The reason this\n> shouldn't be considered an error is that SKIP_WORKTREE has always been\n> an advisory-only setting; merge and rebase for example were free to\n> materialize paths and clear the SKIP_WORKTREE bit in order to accomplish\n> their work even though they kept the SKIP_WORKTREE bit set for other\n> paths. Leaving dirty working files in the working tree is thus a\n> natural extension of what we have already been doing.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> unpack-trees.c | 5 ++++-\n> 1 file changed, 4 insertions(+), 1 deletion(-)\n> \n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 853d843b17a..8f10ac91ce1 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -504,8 +504,11 @@ static int apply_sparse_checkout(struct index_state *istate,\n> \t\t * also stat info may have lost after merged_entry() so calling\n> \t\t * verify_uptodate() again may fail\n> \t\t */\n> -\t\tif (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))\n> +\t\tif (!(ce->ce_flags & CE_UPDATE) &&\n> +\t\t verify_uptodate_sparse(ce, o)) {\n> +\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n> \t\t\treturn -1;\n> +\t\t}\n> \t\tce->ce_flags |= CE_WT_REMOVE;\n> \t\tce->ce_flags &= ~CE_UPDATE;\n> \t}\n\nI initially thought that you needed to update some\nblocks in clear_ce_flags_dir() around cone mode where\nwe skip sections that are outside of the cone. However,\nthis seems to be working on the results after the\nclear_ce_flags() has completed. Should work.\n\nThanks,\n-Stolee\n"},{"id":"393270","messageId":"1c83fb68-519a-88f2-77b9-ab443ed6b4cf@gmail.com","threadId":"52995","inReplyTo":"892933752c638a9ed938e046e466dd1e927058a6.1584169893.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/7] sparse-checkout: use new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-15T16:19:34Z","receivedAt":"2020-03-15T16:19:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Remove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\n> codepaths for setting the SKIP_WORKTREE bits and instead use the new\n> update_sparsity() function.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> builtin/sparse-checkout.c | 39 +++++++-----------------------\n> t/t1091-sparse-checkout-builtin.sh | 32 +++++++++++++++++-------\n> 2 files changed, 32 insertions(+), 39 deletions(-)\n> \n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 740da4b6d54..5d3ec2e6be9 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -97,47 +97,32 @@ static int update_working_directory(struct pattern_list *pl)\n> \tint result = 0;\n> \tstruct unpack_trees_options o;\n> \tstruct lock_file lock_file = LOCK_INIT;\n> -\tstruct object_id oid;\n> -\tstruct tree *tree;\n> -\tstruct tree_desc t;\n> \tstruct repository *r = the_repository;\n> \n> -\tif (repo_read_index_unmerged(r))\n> -\t\tdie(_(\"you need to resolve your current index first\"));\n> -\n> -\tif (get_oid(\"HEAD\", &oid))\n> -\t\treturn 0;\n> -\n> -\ttree = parse_tree_indirect(&oid);\n> -\tparse_tree(tree);\n> -\tinit_tree_desc(&t, tree->buffer, tree->size);\n> -\n> \tmemset(&o, 0, sizeof(o));\n> \to.verbose_update = isatty(2);\n> -\to.merge = 1;\n> \to.update = 1;\n> -\to.fn = oneway_merge;\n\nI'm glad this is getting much simpler.\n\n> \to.head_idx = -1;\n> \to.src_index = r->index;\n> \to.dst_index = r->index;\n> \to.skip_sparse_checkout = 0;\n> \to.pl = pl;\n> -\to.keep_pattern_list = !!pl;\n\nI'm making a mental note to see if this is necessary or not. I'm\nguessing that update_sparsity() expects a pattern list as input and\nwill never free it, so this option is useless.\n\n> -\tresolve_undo_clear_index(r->index);\n> \tsetup_work_tree();\n> \n> -\tcache_tree_free(&r->index->cache_tree);\n> -\n> \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n> \n> -\tcore_apply_sparse_checkout = 1;\n> -\tresult = unpack_trees(1, &t, &o);\n> +\tresult = update_sparsity(&o);\n> \n> -\tif (!result) {\n> -\t\tprime_cache_tree(r, r->index, tree);\n> +\tif (result == 1)\nI wonder if it is appropriate to return an enum in update_sparsity()\nso this statement is clearer out-of-context. For example, I'm looking\nat this patch before looking closely at the update_sparsity() patch\nso I know how it will be used before I check its implementation.\n\n> +\t\t/*\n> +\t\t * We don't do any special handling of warnings from untracked\n> +\t\t * files in the way or dirty entries that can't be removed.\n> +\t\t */\n> +\t\tresult = 0;\n> +\tif (!result)\n> \t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n> -\t} else\n> +\telse\n> \t\trollback_lock_file(&lock_file);\n> \n> \treturn result;\n> @@ -304,8 +289,6 @@ static int sparse_checkout_init(int argc, const char **argv)\n> \t};\n> \n> \trepo_read_index(the_repository);\n> -\trequire_clean_work_tree(the_repository,\n> -\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n> \n> \targc = parse_options(argc, argv, NULL,\n> \t\t\t builtin_sparse_checkout_init_options,\n> @@ -560,8 +543,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n> \t};\n> \n> \trepo_read_index(the_repository);\n> -\trequire_clean_work_tree(the_repository,\n> -\t\t\t\tN_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n> \n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_sparse_checkout_set_options,\n> @@ -577,8 +558,6 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> \tstruct strbuf match_all = STRBUF_INIT;\n> \n> \trepo_read_index(the_repository);\n> -\trequire_clean_work_tree(the_repository,\n> -\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n\nIn the three hunks above, do we still need repo_read_index()?\n\n> \tmemset(&pl, 0, sizeof(pl));\n> \thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 8607a8e6d1a..0d93d3983e0 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n> \tcheck_files repo a deep folder1\n> '\n> \n> -test_expect_success 'revert to old sparse-checkout on bad update' '\n> +test_expect_success 'not-up-to-date does not block rest of sparsification' '\n> \ttest_when_finished git -C repo sparse-checkout disable &&\n> \ttest_when_finished git -C repo reset --hard &&\n> \tgit -C repo sparse-checkout set deep &&\n> +\n> \techo update >repo/deep/deeper2/a &&\n> \tcp repo/.git/info/sparse-checkout expect &&\n> -\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n> -\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n> -\ttest_cmp repo/.git/info/sparse-checkout expect &&\n> -\tcheck_files repo/deep a deeper1 deeper2\n> +\ttest_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n> +\n> +\tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n> +\n> +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tcheck_files repo/deep a deeper1 deeper2 &&\n> +\tcheck_files repo/deep/deeper1 a deepest &&\n> +\tcheck_files repo/deep/deeper1/deepest a &&\n> +\tcheck_files repo/deep/deeper2 a\n> '\n> \n> test_expect_success 'revert to old sparse-checkout on empty update' '\n> @@ -316,12 +323,19 @@ test_expect_success '.gitignore should not warn about cone mode' '\n> \ttest_i18ngrep ! \"disabling cone patterns\" err\n> '\n> \n> -test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n> +test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n> \tgit clone repo dirty &&\n> \techo dirty >dirty/folder1/a &&\n> -\ttest_must_fail git -C dirty sparse-checkout init &&\n> -\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> -\ttest_must_fail git -C dirty sparse-checkout disable &&\n> +\n> +\tgit -C dirty sparse-checkout init 2>err &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> +\n> +\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> +\n> +\tgit -C dirty sparse-checkout disable &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> +\n> \tgit -C dirty reset --hard &&\n> \tgit -C dirty sparse-checkout init &&\n> \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n\nI like that the test changes here are limited to updating existing\ntests. You mentioned adding more tests, which I would also like, but\nthose can come in a follow-up patch that is dedicated to all the fun\nsituations we can get into, and how we should respond.\n\nHere are a few that I've tested manually:\n\n1. Start: no sparse-checkout\n a. Get into rebase conflict in a directory.\n b. \"git sparse-checkout init --cone\"\n c. See that the file is on-disk with conflict markers.\n d. Resolve conflict and \"git add\"\n e. \"git sparse-checkout set \"\n f. Sparse-checkout is set as expected, file is not on disk, but still staged.\n g. \"git rebase --continue\" succeeds.\n\n2. Start: sparse-checkout set to \"dir1\"\n a. Get into merge conflict in \"dir2\"\n b. Resolve conflict, \"git add\"\n c. \"git sparse-checkout update\" (available after next patch)\n d. dir1 gone, dir1/file staged.\n e. git merge --continue succeeds\n\nThose are the really big ones to me, but I'm sure we will find others.\nI'm really happy that this is working as we expect.\n\nThanks!\n-Stolee\n"},{"id":"393271","messageId":"ee9e8bc6-252f-8ac5-e769-26302f3eb1eb@gmail.com","threadId":"52995","inReplyTo":"650db6863426ae2b324ba717f898247f44279cb8.1584169893.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/7] sparse-checkout: provide a new update subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-15T16:24:22Z","receivedAt":"2020-03-15T16:24:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> If commands like merge or rebase materialize files as part of their work,\n> or a previous sparse-checkout command failed to update individual files\n> due to dirty changes, users may want a command to simply 'reapply' the\n> sparsity rules. Provide one.\n\nI was actually thinking \"refresh\" would be a better name, but also you\nuse \"reapply\" which is good, too. I'm concerned that \"update\" may imply\nthat the sparse-checkout patterns can change, but you really mean to\nre-do the work from a previous \"git sparse-checkout (set|add)\".\n\nI also thought of \"reset\" but that would be a confusing overload.\n\n> Signed-off-by: Elijah Newren \n> ---\n> Documentation/git-sparse-checkout.txt | 10 ++++++++++\n> builtin/sparse-checkout.c | 10 +++++++++-\n> 2 files changed, 19 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index c0342e53938..27f4392489f 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -70,6 +70,16 @@ C-style quoted strings.\n> \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n> \tas directory names as in the 'set' subcommand.\n> \n> +'update'::\n> +\tUpdate the sparseness of paths in the working tree based on the\n> +\texisting patterns. Commands like merge or rebase can materialize\n> +\tpaths to do their work (e.g. in order to show you a conflict), and\n> +\tother sparse-checkout commands might fail to sparsify an individual\n> +\tfile (e.g. because it has unstaged changes or conflicts). In such\n> +\tcases, it can make sense to run `git sparse-checkout update` later\n> +\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n> +\tor committing changes, etc.).\n> +\n> 'disable'::\n> \tDisable the `core.sparseCheckout` config setting, and restore the\n> \tworking directory to include all files. Leaves the sparse-checkout\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 5d3ec2e6be9..2ae21011dfd 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -18,7 +18,7 @@\n> static const char *empty_base = \"\";\n> \n> static char const * const builtin_sparse_checkout_usage[] = {\n> -\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n> +\tN_(\"git sparse-checkout (init|list|set|add|update|disable) \"),\n> \tNULL\n> };\n> \n> @@ -552,6 +552,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n> \treturn modify_pattern_list(argc, argv, m);\n> }\n> \n> +static int sparse_checkout_update(int argc, const char **argv)\n> +{\n> +\trepo_read_index(the_repository);\n> +\treturn update_working_directory(NULL);\n> +}\n> +\n\nShort and sweet! I suppose my earlier comment about whether\nrepo_read_index() was necessary is answered here. Perhaps it\nshould be part of update_working_directory()? (And pass a\nrepository pointer to it?)\n\n> static int sparse_checkout_disable(int argc, const char **argv)\n> {\n> \tstruct pattern_list pl;\n> @@ -601,6 +607,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n> \t\tif (!strcmp(argv[0], \"add\"))\n> \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n> +\t\tif (!strcmp(argv[0], \"update\"))\n> +\t\t\treturn sparse_checkout_update(argc, argv);\n> \t\tif (!strcmp(argv[0], \"disable\"))\n> \t\t\treturn sparse_checkout_disable(argc, argv);\n> \t}\n\nThanks,\n-Stolee\n\n\n"},{"id":"393276","messageId":"a2189235-d16a-cccc-a529-08cb854616b3@gmail.com","threadId":"52995","inReplyTo":"9f13a2e8c64d47a5b3a4a2a96451decaf548ca4e.1584169893.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] unpack-trees: add a new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-15T18:17:55Z","receivedAt":"2020-03-15T18:17:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Previously, the only way to update the SKIP_WORKTREE bits for various\n> paths was invoking `git read-tree -mu HEAD` or calling the same code\n> that this codepath invoked. This however had a number of problems.\n> Let's look at all the various cases:\n> \n> Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files):\n> A) There is no file in the way\n> B) There is an untracked file in the way\n> Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files):\n> C) The path is clean\n> D) The path is unmerged\n> E) The path has unstaged changes\n> F) The path has staged changes (differs from HEAD)\n\nPlease also note that cases D, E, and F can appear in any combination.\nThe lack of any of D, E, and F is case \"C\".\n\nPerhaps instead we should think of the following independent\nstates that can toggle orthogonally on a path:\n\n i. Current SKIP_WORKTREE value\n ii. Intended SKIP_WORKTREE value\niii. Path has staged changes.\n iv. Path has unstaged changes.\n v. Path is unmerged.\n\nwith the additional conditional case of:\n\n vi. There is a collision when changing from SKIP_WORKTREE to\n !SKIP_WORKTREE. This could be file content difference or\n a file/directory type conflict.\n\nI believe the intention when refreshing or changing the sparse-\ncheckout definition is to:\n\n 0. If a merge conflict happens outside the sparse cone, then\n the conflicting files are created with the conflict markers.\n This puts those paths in the state of having unstaged changes.\n\n 1. If the path exists and we intend to set SKIP_WORKTREE, then\n delete the file only if there are no unstaged changes.\n\n 2. If the path does not exist and we intend to remove SKIP_WORKTREE,\n then check for case (vi). Fail with a helpful error if any such\n case exists, and do not update any other paths.\n\n(This is all before I read the rest, so I'm sorry if I repeat what\nyou have already described.)\n\n> For `git read-tree -mu HEAD`, the behavior is as follows:\n> A) Materialize the file, as expected\n> B) Refuse to make *any* changes (to this path or any others)\n> C) Remove the file, as expected\n> D) Refuse to make *any* changes (to this path or any others)\n> E) Refuse to make *any* changes (to this path or any others)\n> F) Removes the file from the working copy AND resets the index\n> to match HEAD\n\nThis case F is particularly bad, as we lose a staged change and\nits copy on disk. I believe that was the case that convinced me\nto put in the requirement of a clean status before changing the\nsparse-checkout definition.\n\n> Refusing to modify files that could result in data loss as in cases B,\n> D, and E is good, but refusing to work on any other paths is very\n> problematic for users. If they are in the middle of a rebase or have\n> made modifications to files that bring in more dependencies, then for\n> their build to work they need to update the sparse paths -- which they\n> are prevented from doing. Sometimes they stage the files and re-try,\n> at which point they run into case F and believe that their changes are\n> lost. (Even if they come ask an expert, they have to trawl through\n> their loose objects looking for which ones match which files.)\n> \n> Add a new update_sparsity() function which behaves as follows in these\n> cases:\n> A) Materialize the file, as expected\n> B) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n> bit, and print a warning (thus leaving them in a state where\n> status will report the file as modified, which seems logical).\n> C) Remove the file, as expected\n> D) Do NOT mark this path as SKIP_WORKTREE, but allow others paths\n> to be updated.\n> E) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n> the dirty path, but allow other paths to be updated.\n> F) Mark the path as SKIP_WORKTREE, but do not revert the version\n> stored in the index to match HEAD; leave the contents alone.\n> \n> I tried a different behavior for B (leave the SKIP_WORKTREE bit set),\n> but found it very surprising and counter-intuitive (e.g. the user sees\n> it is present along with all the other files in that directory, tries to\n> stage it, but git add ignores it since the SKIP_WORKTREE bit is set).\n\nThat sounds bad! I think you picked the right approach here.\n\n> A, B, C, and E all seem like optimal behavior to me. D may be as well,\n> though I wonder if printing a warning would be an improvement. Some\n> might be slightly surprised by F at first, but given that it does the\n> right thing with `git commit` and even `git commit -a` (add ignores\n> entries that are marked SKIP_WORKTREE and thus doesn't delete them, and\n> commit -a is similar), it seems logical to me.\n\nThe established usage pattern is that a user can create \"colliding\"\ndirectories and files outside of the sparse cone and Git is expected\nto ignore them. The problem happens when the user adds those paths\nto the sparse cone. I think you handle the situations quite well with\nthis plan.\n\n> +/*\n> + * display all the \"error\" messages as warnings\n> + */\n> +static void display_warning_msgs(struct unpack_trees_options *o)\n> +{\n> +\tint e, i;\n> +\tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n> +\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n> +\t\tif (rejects->nr > 0) {\n> +\t\t\tstruct strbuf path = STRBUF_INIT;\n\nnit: Define 'i' within the inner-most scope that it matters.\n\n> +\t\t\tfor (i = 0; i < rejects->nr; i++)\n> +\t\t\t\tstrbuf_addstr(&path, rejects->items[i].string);\n> +\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n> +\t\t\tstrbuf_release(&path);\n> +\t\t}\n> +\t\tstring_list_clear(rejects, 0);\n> +\t}\n> +}\n> +/*\n> + * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n> + * working directory to match.\n> + *\n> + * Returns\n> + * 0: success with no warnings\n> + * 1: success with warnings (warnings come either from (a) dirty changes\n> + * present in which case we avoid marking those paths as\n> + * SKIP_WORKTREE, or (b) from untracked files being in the way us\n> + * checking a file out of the index, in which case we leave the file\n> + * in the working tree alone while clearing SKIP_WORKTREE)\n> + * -1: failure to manipulate the resulting index\n> + * -2: failure to reflect the changes to the work tree.\n> + *\n> + * CE_NEW_SKIP_WORKTREE is used internally.\n> + */\n\nI think I mentioned using an enum for this function in another patch, and\nreading the documentation here makes that even more clear. Anything other\nthan {-1,0,1} return types really could use stronger typing. Even using\n{-1,0,1} instead of just {0,1} is debatable if it should use an enum instead.\n\nI wonder if this patch could benefit a split where we first return\n{-2, -1, 0} and don't include any warnings, followed by a patch where\nthe warnings are added (and tested).\n\n> +int update_sparsity(struct unpack_trees_options *o)\n> +{\n> +\tstruct pattern_list pl;\n> +\tint i, empty_worktree, ret = 0;\n> +\tunsigned old_show_all_errors;\n> +\n> +\told_show_all_errors = o->show_all_errors;\n> +\to->show_all_errors = 1;\n> +\n> +\t/* Sanity checks */\n> +\tif (!o->update || o->index_only || o->skip_sparse_checkout)\n> +\t\tBUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n> +\tif (o->src_index != o->dst_index || o->fn)\n> +\t\tBUG(\"update_sparsity() called wrong\");\n\nI was going to add something about split indexes at the end,\nbut I think this precondition has it covered.\n\n> +\n> +\ttrace_performance_enter();\n> +\n> +\tif (!o->pl) {\n> +\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n> +\t\tmemset(&pl, 0, sizeof(pl));\n> +\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n> +\t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0) {\n> +\t\t\t/* FIXME: Skip to check_updates()?? */\n> +\t\t\to->skip_sparse_checkout = 1;\n> +\t\t\tgoto skip_sparse_checkout;\n> +\t\t} else\n> +\t\t\to->pl = &pl;\n> +\t\tfree(sparse);\n> +\t}\n\nOutside of the goto, this part looks identical to other parts\nof the same file where we \"fallback\" to the sparse-checkout file.\nPerhaps that could be extracted to a helper method? It could\nreturn failure when there is no in-memory pattern and the\nsparse-checkout file fails to load, and in response we can do\nthe goto.\n\n> +\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n> +\tmark_all_ce_unused(o->src_index);\n> +\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n> +\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n> +\n> +\t/* Then loop over entries and update/remove as needed */\n> +\tret = 0;\n> +\tempty_worktree = 1;\n> +\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n> +\t\tstruct cache_entry *ce = o->src_index->cache[i];\n> +\n> +\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n> +\t\t\tret = 1;\n\nSo here, we report that at least one path has a warning.\n\n> +\t\tif (!ce_skip_worktree(ce))\n> +\t\t\tempty_worktree = 0;\n> +\n> +\t}\n> +\n> +\t/*\n> +\t * Sparse checkout is meant to narrow down checkout area\n> +\t * but it does not make sense to narrow down to empty working\n> +\t * tree. This is usually a mistake in sparse checkout rules.\n> +\t * Do not allow users to do that.\n> +\t */\n> +\tif (o->src_index->cache_nr && empty_worktree) {\n> +\t\tret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n\nInteresting that we are using the result of unpack_failed() here.\nI suppose that could be an argument for using an int instead of\nan enum, but perhaps we translate the result of unpack_failed()\nhere? What sorts of values could it be, anyway? Is this a case\nwhere unpack_failed() could return either -1 or -2? Or should\nwe simply set 'ret' to the \"-1\" case ourselves?\n\n> +\t\tgoto done;\n> +\t}\n> +\n> +skip_sparse_checkout:\n> +\tif (check_updates(o, o->src_index))\n> +\t\tret = -2;\n> +\n> +done:\n> +\tdisplay_warning_msgs(o);\n> +\to->show_all_errors = old_show_all_errors;\n> +\ttrace_performance_leave(\"update_sparsity\");\n> +\treturn ret;\n> +}\n> +\n> /* Here come the merge functions */\n> \n> static int reject_merge(const struct cache_entry *ce,\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index ae1557fb804..5d1a77ae249 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -90,6 +90,8 @@ struct unpack_trees_options {\n> int unpack_trees(unsigned n, struct tree_desc *t,\n> \t\t struct unpack_trees_options *options);\n> \n> +int update_sparsity(struct unpack_trees_options *options);\n> +\n\nWould it not be the pattern to include the documentation comment\nhere in the header file?\n\nIn some ways, your update_sparsity() repeats some of the patterns\nof unpack_trees(), especially with its order of mark_all_ce_unused(),\nmark_new_skip_worktree(), and unpack_failed(). I think there may be\nsome benefits to adding some of the performance tracing patterns from\nthat method, too. These could be added as follow-up patches.\n\nI'm happy that you decided to create a new method instead of adding\nyet another mode to unpack_trees(), especially because your version\ndoes not involve calling traverse_trees().\n\nAfter careful inspection of your patch, I cannot find anything wrong.\nI'm pretty happy with your organization and the approach seems clear.\nHopefully my comments in the commit message could help improve that\npart of the discussion.\n\nAt this point, the only thing we really need are tests that demonstrate\nas many of these cases as possible.\n\nThanks,\n-Stolee\n\n"},{"id":"393305","messageId":"CABPp-BEPCpN=gW=LC7JBbJNmqhYxEhQJi6ycV21zhmuFAta3gw@mail.gmail.com","threadId":"52995","inReplyTo":"1c83fb68-519a-88f2-77b9-ab443ed6b4cf@gmail.com","subject":"Re: [PATCH 6/7] sparse-checkout: use new update_sparsity() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-16T17:02:26Z","receivedAt":"2020-03-16T17:02:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Mar 15, 2020 at 9:19 AM Derrick Stolee wrote:\n>\n> On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Remove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\n> > codepaths for setting the SKIP_WORKTREE bits and instead use the new\n> > update_sparsity() function.\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > builtin/sparse-checkout.c | 39 +++++++-----------------------\n> > t/t1091-sparse-checkout-builtin.sh | 32 +++++++++++++++++-------\n> > 2 files changed, 32 insertions(+), 39 deletions(-)\n> >\n> > diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> > index 740da4b6d54..5d3ec2e6be9 100644\n> > --- a/builtin/sparse-checkout.c\n> > +++ b/builtin/sparse-checkout.c\n> > @@ -97,47 +97,32 @@ static int update_working_directory(struct pattern_list *pl)\n> > int result = 0;\n> > struct unpack_trees_options o;\n> > struct lock_file lock_file = LOCK_INIT;\n> > - struct object_id oid;\n> > - struct tree *tree;\n> > - struct tree_desc t;\n> > struct repository *r = the_repository;\n> >\n> > - if (repo_read_index_unmerged(r))\n> > - die(_(\"you need to resolve your current index first\"));\n> > -\n> > - if (get_oid(\"HEAD\", &oid))\n> > - return 0;\n> > -\n> > - tree = parse_tree_indirect(&oid);\n> > - parse_tree(tree);\n> > - init_tree_desc(&t, tree->buffer, tree->size);\n> > -\n> > memset(&o, 0, sizeof(o));\n> > o.verbose_update = isatty(2);\n> > - o.merge = 1;\n> > o.update = 1;\n> > - o.fn = oneway_merge;\n>\n> I'm glad this is getting much simpler.\n>\n> > o.head_idx = -1;\n> > o.src_index = r->index;\n> > o.dst_index = r->index;\n> > o.skip_sparse_checkout = 0;\n> > o.pl = pl;\n> > - o.keep_pattern_list = !!pl;\n>\n> I'm making a mental note to see if this is necessary or not. I'm\n> guessing that update_sparsity() expects a pattern list as input and\n> will never free it, so this option is useless.\n\nYes, although your comment made me realize that I've got a memory leak\nif we pass o->pl==NULL to update_sparsity(), which happens whenever we\npass NULL to update_working_directory(). I should fix that.\n\n> > - resolve_undo_clear_index(r->index);\n> > setup_work_tree();\n> >\n> > - cache_tree_free(&r->index->cache_tree);\n> > -\n> > repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n> >\n> > - core_apply_sparse_checkout = 1;\n> > - result = unpack_trees(1, &t, &o);\n> > + result = update_sparsity(&o);\n> >\n> > - if (!result) {\n> > - prime_cache_tree(r, r->index, tree);\n> > + if (result == 1)\n> I wonder if it is appropriate to return an enum in update_sparsity()\n> so this statement is clearer out-of-context. For example, I'm looking\n> at this patch before looking closely at the update_sparsity() patch\n> so I know how it will be used before I check its implementation.\n\nYes, that'd be better.\n\n> > + /*\n> > + * We don't do any special handling of warnings from untracked\n> > + * files in the way or dirty entries that can't be removed.\n> > + */\n> > + result = 0;\n> > + if (!result)\n> > write_locked_index(r->index, &lock_file, COMMIT_LOCK);\n> > - } else\n> > + else\n> > rollback_lock_file(&lock_file);\n> >\n> > return result;\n> > @@ -304,8 +289,6 @@ static int sparse_checkout_init(int argc, const char **argv)\n> > };\n> >\n> > repo_read_index(the_repository);\n> > - require_clean_work_tree(the_repository,\n> > - N_(\"initialize sparse-checkout\"), NULL, 1, 0);\n> >\n> > argc = parse_options(argc, argv, NULL,\n> > builtin_sparse_checkout_init_options,\n> > @@ -560,8 +543,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n> > };\n> >\n> > repo_read_index(the_repository);\n> > - require_clean_work_tree(the_repository,\n> > - N_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n> >\n> > argc = parse_options(argc, argv, prefix,\n> > builtin_sparse_checkout_set_options,\n> > @@ -577,8 +558,6 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> > struct strbuf match_all = STRBUF_INIT;\n> >\n> > repo_read_index(the_repository);\n> > - require_clean_work_tree(the_repository,\n> > - N_(\"disable sparse-checkout\"), NULL, 1, 0);\n>\n> In the three hunks above, do we still need repo_read_index()?\n>\n> > memset(&pl, 0, sizeof(pl));\n> > hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n> > diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> > index 8607a8e6d1a..0d93d3983e0 100755\n> > --- a/t/t1091-sparse-checkout-builtin.sh\n> > +++ b/t/t1091-sparse-checkout-builtin.sh\n> > @@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n> > check_files repo a deep folder1\n> > '\n> >\n> > -test_expect_success 'revert to old sparse-checkout on bad update' '\n> > +test_expect_success 'not-up-to-date does not block rest of sparsification' '\n> > test_when_finished git -C repo sparse-checkout disable &&\n> > test_when_finished git -C repo reset --hard &&\n> > git -C repo sparse-checkout set deep &&\n> > +\n> > echo update >repo/deep/deeper2/a &&\n> > cp repo/.git/info/sparse-checkout expect &&\n> > - test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n> > - test_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n> > - test_cmp repo/.git/info/sparse-checkout expect &&\n> > - check_files repo/deep a deeper1 deeper2\n> > + test_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n> > +\n> > + git -C repo sparse-checkout set deep/deeper1 2>err &&\n> > +\n> > + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> > + test_cmp expect repo/.git/info/sparse-checkout &&\n> > + check_files repo/deep a deeper1 deeper2 &&\n> > + check_files repo/deep/deeper1 a deepest &&\n> > + check_files repo/deep/deeper1/deepest a &&\n> > + check_files repo/deep/deeper2 a\n> > '\n> >\n> > test_expect_success 'revert to old sparse-checkout on empty update' '\n> > @@ -316,12 +323,19 @@ test_expect_success '.gitignore should not warn about cone mode' '\n> > test_i18ngrep ! \"disabling cone patterns\" err\n> > '\n> >\n> > -test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n> > +test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n> > git clone repo dirty &&\n> > echo dirty >dirty/folder1/a &&\n> > - test_must_fail git -C dirty sparse-checkout init &&\n> > - test_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> > - test_must_fail git -C dirty sparse-checkout disable &&\n> > +\n> > + git -C dirty sparse-checkout init 2>err &&\n> > + test_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> > +\n> > + git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> > + test_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> > +\n> > + git -C dirty sparse-checkout disable &&\n> > + test_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> > +\n> > git -C dirty reset --hard &&\n> > git -C dirty sparse-checkout init &&\n> > git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n>\n> I like that the test changes here are limited to updating existing\n> tests. You mentioned adding more tests, which I would also like, but\n> those can come in a follow-up patch that is dedicated to all the fun\n> situations we can get into, and how we should respond.\n>\n> Here are a few that I've tested manually:\n>\n> 1. Start: no sparse-checkout\n> a. Get into rebase conflict in a directory.\n> b. \"git sparse-checkout init --cone\"\n> c. See that the file is on-disk with conflict markers.\n> d. Resolve conflict and \"git add\"\n> e. \"git sparse-checkout set \"\n> f. Sparse-checkout is set as expected, file is not on disk, but still staged.\n> g. \"git rebase --continue\" succeeds.\n>\n> 2. Start: sparse-checkout set to \"dir1\"\n> a. Get into merge conflict in \"dir2\"\n> b. Resolve conflict, \"git add\"\n> c. \"git sparse-checkout update\" (available after next patch)\n> d. dir1 gone, dir1/file staged.\n> e. git merge --continue succeeds\n>\n> Those are the really big ones to me, but I'm sure we will find others.\n> I'm really happy that this is working as we expect.\n\nThanks for taking a look.\n"},{"id":"393306","messageId":"CABPp-BE=VdSAfAWR_2KivXbGOYknhJ=4sbRjVPAS+URNgjLFPQ@mail.gmail.com","threadId":"52995","inReplyTo":"ee9e8bc6-252f-8ac5-e769-26302f3eb1eb@gmail.com","subject":"Re: [PATCH 7/7] sparse-checkout: provide a new update subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-16T17:05:37Z","receivedAt":"2020-03-16T17:05:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Mar 15, 2020 at 9:24 AM Derrick Stolee wrote:\n>\n> On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > If commands like merge or rebase materialize files as part of their work,\n> > or a previous sparse-checkout command failed to update individual files\n> > due to dirty changes, users may want a command to simply 'reapply' the\n> > sparsity rules. Provide one.\n>\n> I was actually thinking \"refresh\" would be a better name, but also you\n> use \"reapply\" which is good, too. I'm concerned that \"update\" may imply\n> that the sparse-checkout patterns can change, but you really mean to\n> re-do the work from a previous \"git sparse-checkout (set|add)\".\n>\n> I also thought of \"reset\" but that would be a confusing overload.\n\nMakes sense; I'll switch it over to \"reapply\".\n\n> > Signed-off-by: Elijah Newren \n> > ---\n> > Documentation/git-sparse-checkout.txt | 10 ++++++++++\n> > builtin/sparse-checkout.c | 10 +++++++++-\n> > 2 files changed, 19 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> > index c0342e53938..27f4392489f 100644\n> > --- a/Documentation/git-sparse-checkout.txt\n> > +++ b/Documentation/git-sparse-checkout.txt\n> > @@ -70,6 +70,16 @@ C-style quoted strings.\n> > `core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n> > as directory names as in the 'set' subcommand.\n> >\n> > +'update'::\n> > + Update the sparseness of paths in the working tree based on the\n> > + existing patterns. Commands like merge or rebase can materialize\n> > + paths to do their work (e.g. in order to show you a conflict), and\n> > + other sparse-checkout commands might fail to sparsify an individual\n> > + file (e.g. because it has unstaged changes or conflicts). In such\n> > + cases, it can make sense to run `git sparse-checkout update` later\n> > + after cleaning up affected paths (e.g. resolving conflicts, undoing\n> > + or committing changes, etc.).\n> > +\n> > 'disable'::\n> > Disable the `core.sparseCheckout` config setting, and restore the\n> > working directory to include all files. Leaves the sparse-checkout\n> > diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> > index 5d3ec2e6be9..2ae21011dfd 100644\n> > --- a/builtin/sparse-checkout.c\n> > +++ b/builtin/sparse-checkout.c\n> > @@ -18,7 +18,7 @@\n> > static const char *empty_base = \"\";\n> >\n> > static char const * const builtin_sparse_checkout_usage[] = {\n> > - N_(\"git sparse-checkout (init|list|set|add|disable) \"),\n> > + N_(\"git sparse-checkout (init|list|set|add|update|disable) \"),\n> > NULL\n> > };\n> >\n> > @@ -552,6 +552,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n> > return modify_pattern_list(argc, argv, m);\n> > }\n> >\n> > +static int sparse_checkout_update(int argc, const char **argv)\n> > +{\n> > + repo_read_index(the_repository);\n> > + return update_working_directory(NULL);\n> > +}\n> > +\n>\n> Short and sweet! I suppose my earlier comment about whether\n> repo_read_index() was necessary is answered here. Perhaps it\n> should be part of update_working_directory()? (And pass a\n> repository pointer to it?)\n\nGood question. Is there a chance we want to make\nupdate_working_directory() available to other areas of git outside of\nsparse-checkout.c? If so, potentially re-reading the index might not\nbe friendly, but if sparse-checkout.c is going to remain the only\ncaller then it probably makes sense to move it inside.\n\n> > static int sparse_checkout_disable(int argc, const char **argv)\n> > {\n> > struct pattern_list pl;\n> > @@ -601,6 +607,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> > return sparse_checkout_set(argc, argv, prefix, REPLACE);\n> > if (!strcmp(argv[0], \"add\"))\n> > return sparse_checkout_set(argc, argv, prefix, ADD);\n> > + if (!strcmp(argv[0], \"update\"))\n> > + return sparse_checkout_update(argc, argv);\n> > if (!strcmp(argv[0], \"disable\"))\n> > return sparse_checkout_disable(argc, argv);\n> > }\n"},{"id":"393308","messageId":"137d9910-452b-9c96-b3a4-2b19053f6a67@gmail.com","threadId":"52995","inReplyTo":"CABPp-BE=VdSAfAWR_2KivXbGOYknhJ=4sbRjVPAS+URNgjLFPQ@mail.gmail.com","subject":"Re: [PATCH 7/7] sparse-checkout: provide a new update subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-16T17:18:14Z","receivedAt":"2020-03-16T17:18:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/16/2020 1:05 PM, Elijah Newren wrote:\n> On Sun, Mar 15, 2020 at 9:24 AM Derrick Stolee wrote:\n>>\n>> On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>> +static int sparse_checkout_update(int argc, const char **argv)\n>>> +{\n>>> + repo_read_index(the_repository);\n>>> + return update_working_directory(NULL);\n>>> +}\n>>> +\n>>\n>> Short and sweet! I suppose my earlier comment about whether\n>> repo_read_index() was necessary is answered here. Perhaps it\n>> should be part of update_working_directory()? (And pass a\n>> repository pointer to it?)\n> \n> Good question. Is there a chance we want to make\n> update_working_directory() available to other areas of git outside of\n> sparse-checkout.c? If so, potentially re-reading the index might not\n> be friendly, but if sparse-checkout.c is going to remain the only\n> caller then it probably makes sense to move it inside.\n\nMinh had an interesting idea during side-conversations at the summit:\nhave a way for an in-tree description of some sparse-checkout cones.\nThe idea was to be able to automatically update the sparse-checkout\nwhile moving between commits that may have different dependency\nconfigurations. In the world of Office it would mean that there is\nsome file \".sparse/word\" that describes the directories required to\nbuild Word, and \".sparse/ppt\" for building PowerPoint. Then, based\non local Git config, we would see that we want our sparse-checkout\ncone to match the union of the directories in .sparse/word and\n.sparse/ppt. As we move HEAD, we would want to automatically update\nthe sparse cone when those files change.\n\nI'm working on a design document for how this idea would work,\nrealistically, that I plan to share here and with the Office team\nto see if it is actually a helpful plan. I think it would reduce\nthe performance cost of the hook we plan to use for this, and\nwould reduce the investment needed for a project to adopt\nsparse-checkout.\n\nAll that is to say, yes we may want to add other callers to\nupdate_working_directory() outside of the sparse-checkout\nbuiltin. With that in mind, perhaps its name should reflect\nthe fact that we are only updating it according to the sparse\ncone?\n\nThanks,\n-Stolee\n\n"},{"id":"393325","messageId":"CABPp-BEk66Z1iK00SpnQ-Tz6pzwOFBzaHukO=WAEPwaXH0dVUg@mail.gmail.com","threadId":"52995","inReplyTo":"137d9910-452b-9c96-b3a4-2b19053f6a67@gmail.com","subject":"Re: [PATCH 7/7] sparse-checkout: provide a new update subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-16T19:23:49Z","receivedAt":"2020-03-16T19:24:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 16, 2020 at 10:18 AM Derrick Stolee wrote:\n>\n> On 3/16/2020 1:05 PM, Elijah Newren wrote:\n> > On Sun, Mar 15, 2020 at 9:24 AM Derrick Stolee wrote:\n> >>\n> >> On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> >>> From: Elijah Newren \n> >>> +static int sparse_checkout_update(int argc, const char **argv)\n> >>> +{\n> >>> + repo_read_index(the_repository);\n> >>> + return update_working_directory(NULL);\n> >>> +}\n> >>> +\n> >>\n> >> Short and sweet! I suppose my earlier comment about whether\n> >> repo_read_index() was necessary is answered here. Perhaps it\n> >> should be part of update_working_directory()? (And pass a\n> >> repository pointer to it?)\n> >\n> > Good question. Is there a chance we want to make\n> > update_working_directory() available to other areas of git outside of\n> > sparse-checkout.c? If so, potentially re-reading the index might not\n> > be friendly, but if sparse-checkout.c is going to remain the only\n> > caller then it probably makes sense to move it inside.\n>\n> Minh had an interesting idea during side-conversations at the summit:\n> have a way for an in-tree description of some sparse-checkout cones.\n> The idea was to be able to automatically update the sparse-checkout\n> while moving between commits that may have different dependency\n> configurations. In the world of Office it would mean that there is\n> some file \".sparse/word\" that describes the directories required to\n> build Word, and \".sparse/ppt\" for building PowerPoint. Then, based\n> on local Git config, we would see that we want our sparse-checkout\n> cone to match the union of the directories in .sparse/word and\n> .sparse/ppt. As we move HEAD, we would want to automatically update\n> the sparse cone when those files change.\n>\n> I'm working on a design document for how this idea would work,\n> realistically, that I plan to share here and with the Office team\n> to see if it is actually a helpful plan. I think it would reduce\n> the performance cost of the hook we plan to use for this, and\n> would reduce the investment needed for a project to adopt\n> sparse-checkout.\n>\n> All that is to say, yes we may want to add other callers to\n> update_working_directory() outside of the sparse-checkout\n> builtin. With that in mind, perhaps its name should reflect\n> the fact that we are only updating it according to the sparse\n> cone?\n>\n> Thanks,\n> -Stolee\n\nInteresting. Some context on another usecase (which may not modify\nyour plans but I'll throw it out there for consideration):\n\nFor us, we have a bunch of modules/* directories. Each has a file\nwhich lists the other modules it directly depends upon. Thus to get\nall dependencies both direct and indirect, something has to walk that\nDAG. Being required to list the dependencies in both some place that\nthe build system understand, and one that git understands, doesn't\nsound like fun. Also, requiring users to list all transitive\ndependencies or remembering to run some script to do so sounds\nproblematic.\n\nWe do have a special file that defines teams, e.g. team-1 means these\nthree modules (plus implicitly any of their direct and indirect\ndependencies), team-2 means this one module, etc.\n\nAlso, we do record the user's specification of the modules/teams they\nwant already, but not within the repo as you're doing in e.g.\n.sparse/team-1, .sparse/team-2. If the user runs './sparsify\n--modules A B', we record the modules in\n.git/info/sparse-module-specification. This differs from\n.git/info/sparse-checkout because the latter has full path (\"module/A\"\nand \"module/B\" instead of just \"A\" and \"B\") and because it has\ntransitive dependencies (thus may have hundreds of directories even if\nthe user just specified two).\n\ngit would thus be unable to use our\n.git/info/sparse-module-specification to do updates, and as above we\ndon't want to have to store the dependencies in another place, and the\nfully resolved ones at that. However, we do get partial auto-updating\nbecause the build system has a pre-build hook that essentially runs\n`git sparse-checkout reapply` whenever any relevant\ndependency-declaration file is newer than .git/info/sparse-checkout.\n\nOf course, waiting until a build may be good enough for us, but others\nmight want updates when they switch branches or do other operations\n(merge, rebase, cherry-pick, revert, am, reset, etc.). In such a\ncase, maybe we could use some kind of hook? Is this what\npost-index-change is for? (If not, I certainly don't want to try to\nnavigate post-checkout and post-merge and add post-* for all the other\noperations).\n\nAnyway, some food for thought while you're working in this area...\n"},{"id":"393329","messageId":"CABPp-BEOw6HXY8nwaKs8t5Zmp5nk-HL6+FOG1dgtx2Lc9qXUbQ@mail.gmail.com","threadId":"52995","inReplyTo":"a2189235-d16a-cccc-a529-08cb854616b3@gmail.com","subject":"Re: [PATCH 5/7] unpack-trees: add a new update_sparsity() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-16T20:24:12Z","receivedAt":"2020-03-16T20:24:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Mar 15, 2020 at 11:17 AM Derrick Stolee wrote:\n>\n> On 3/14/2020 3:11 AM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Previously, the only way to update the SKIP_WORKTREE bits for various\n> > paths was invoking `git read-tree -mu HEAD` or calling the same code\n> > that this codepath invoked. This however had a number of problems.\n> > Let's look at all the various cases:\n> >\n> > Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files):\n> > A) There is no file in the way\n> > B) There is an untracked file in the way\n> > Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files):\n> > C) The path is clean\n> > D) The path is unmerged\n> > E) The path has unstaged changes\n> > F) The path has staged changes (differs from HEAD)\n>\n> Please also note that cases D, E, and F can appear in any combination.\n> The lack of any of D, E, and F is case \"C\".\n>\n> Perhaps instead we should think of the following independent\n> states that can toggle orthogonally on a path:\n>\n> i. Current SKIP_WORKTREE value\n> ii. Intended SKIP_WORKTREE value\n> iii. Path has staged changes.\n> iv. Path has unstaged changes.\n> v. Path is unmerged.\n\nI like this way of thinking of it.\n\n> with the additional conditional case of:\n>\n> vi. There is a collision when changing from SKIP_WORKTREE to\n> !SKIP_WORKTREE. This could be file content difference or\n> a file/directory type conflict.\n\nI think it'd be better to shorten this to:\n vi. There is a path in the way when trying to change from\nSKIP_WORKTREE to !SKIP_WORKTREE.\n\nThe path in the way could be a file, a symlink, a directory, a\nsubmodule, or whatever future types might be possible. We don't\nreally care what kind of path is in the way, just that there is one.\n\n> I believe the intention when refreshing or changing the sparse-\n> checkout definition is to:\n>\n> 0. If a merge conflict happens outside the sparse cone, then\n> the conflicting files are created with the conflict markers.\n> This puts those paths in the state of having unstaged changes.\n\nYou seem to be thinking only of content conflicts that are\nrepresentable with conflict markers. There are content conflicts that\nare not representable with conflict markers, and non-content conflicts\nof various types.[1] So, we need to keep unmerged as distinct from\nhaving unstaged changes.\n\n> 1. If the path exists and we intend to set SKIP_WORKTREE, then\n> delete the file only if there are no unstaged changes.\n\nAs above, we need to fix this to:\n\n1. If the path exists and we intend to set SKIP_WORKTREE, then delete\nthe file only if the file is not unmerged and there are no unstaged\nchanges.\n\n> 2. If the path does not exist and we intend to remove SKIP_WORKTREE,\n> then check for case (vi). Fail with a helpful error if any such\n> case exists, and do not update any other paths.\n\nWhy an error and why not update any other paths?\n\n> (This is all before I read the rest, so I'm sorry if I repeat what\n> you have already described.)\n>\n> > For `git read-tree -mu HEAD`, the behavior is as follows:\n> > A) Materialize the file, as expected\n> > B) Refuse to make *any* changes (to this path or any others)\n> > C) Remove the file, as expected\n> > D) Refuse to make *any* changes (to this path or any others)\n> > E) Refuse to make *any* changes (to this path or any others)\n> > F) Removes the file from the working copy AND resets the index\n> > to match HEAD\n>\n> This case F is particularly bad, as we lose a staged change and\n> its copy on disk. I believe that was the case that convinced me\n\nby \"on disk\", I think you mean \"from the working tree\", yes? It does\nstill exist on disk in the git object directory, if people are willing\nto look through all the loose/dangling objects to find one that \"looks\nright\". I helped a user or two do so precisely because of case F\nhitting them.\n\n> to put in the requirement of a clean status before changing the\n> sparse-checkout definition.\n>\n> > Refusing to modify files that could result in data loss as in cases B,\n> > D, and E is good, but refusing to work on any other paths is very\n> > problematic for users. If they are in the middle of a rebase or have\n> > made modifications to files that bring in more dependencies, then for\n> > their build to work they need to update the sparse paths -- which they\n> > are prevented from doing. Sometimes they stage the files and re-try,\n> > at which point they run into case F and believe that their changes are\n> > lost. (Even if they come ask an expert, they have to trawl through\n> > their loose objects looking for which ones match which files.)\n> >\n> > Add a new update_sparsity() function which behaves as follows in these\n> > cases:\n> > A) Materialize the file, as expected\n> > B) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n> > bit, and print a warning (thus leaving them in a state where\n> > status will report the file as modified, which seems logical).\n> > C) Remove the file, as expected\n> > D) Do NOT mark this path as SKIP_WORKTREE, but allow others paths\n> > to be updated.\n> > E) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n> > the dirty path, but allow other paths to be updated.\n> > F) Mark the path as SKIP_WORKTREE, but do not revert the version\n> > stored in the index to match HEAD; leave the contents alone.\n> >\n> > I tried a different behavior for B (leave the SKIP_WORKTREE bit set),\n> > but found it very surprising and counter-intuitive (e.g. the user sees\n> > it is present along with all the other files in that directory, tries to\n> > stage it, but git add ignores it since the SKIP_WORKTREE bit is set).\n>\n> That sounds bad! I think you picked the right approach here.\n\nOk, I'll ignore your earlier comments about erroring out and not\nupdating any other paths then.\n\n> > A, B, C, and E all seem like optimal behavior to me. D may be as well,\n> > though I wonder if printing a warning would be an improvement. Some\n> > might be slightly surprised by F at first, but given that it does the\n> > right thing with `git commit` and even `git commit -a` (add ignores\n> > entries that are marked SKIP_WORKTREE and thus doesn't delete them, and\n> > commit -a is similar), it seems logical to me.\n>\n> The established usage pattern is that a user can create \"colliding\"\n> directories and files outside of the sparse cone and Git is expected\n> to ignore them. The problem happens when the user adds those paths\n> to the sparse cone. I think you handle the situations quite well with\n> this plan.\n\nCool.\n\n> > +/*\n> > + * display all the \"error\" messages as warnings\n> > + */\n> > +static void display_warning_msgs(struct unpack_trees_options *o)\n> > +{\n> > + int e, i;\n> > + for (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n> > + struct string_list *rejects = &o->unpack_rejects[e];\n> > + if (rejects->nr > 0) {\n> > + struct strbuf path = STRBUF_INIT;\n>\n> nit: Define 'i' within the inner-most scope that it matters.\n\nAh, yeah. I also wondered if I should have done something to share\nmore code with display_error_msgs(), which this function was mostly a\ncopy of (and which has the same problem).\n\n> > + for (i = 0; i < rejects->nr; i++)\n> > + strbuf_addstr(&path, rejects->items[i].string);\n> > + warning(ERRORMSG(o, e), super_prefixed(path.buf));\n> > + strbuf_release(&path);\n> > + }\n> > + string_list_clear(rejects, 0);\n> > + }\n> > +}\n> > +/*\n> > + * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n> > + * working directory to match.\n> > + *\n> > + * Returns\n> > + * 0: success with no warnings\n> > + * 1: success with warnings (warnings come either from (a) dirty changes\n> > + * present in which case we avoid marking those paths as\n> > + * SKIP_WORKTREE, or (b) from untracked files being in the way us\n> > + * checking a file out of the index, in which case we leave the file\n> > + * in the working tree alone while clearing SKIP_WORKTREE)\n> > + * -1: failure to manipulate the resulting index\n> > + * -2: failure to reflect the changes to the work tree.\n> > + *\n> > + * CE_NEW_SKIP_WORKTREE is used internally.\n> > + */\n>\n> I think I mentioned using an enum for this function in another patch, and\n> reading the documentation here makes that even more clear. Anything other\n> than {-1,0,1} return types really could use stronger typing. Even using\n> {-1,0,1} instead of just {0,1} is debatable if it should use an enum instead.\n>\n> I wonder if this patch could benefit a split where we first return\n> {-2, -1, 0} and don't include any warnings, followed by a patch where\n> the warnings are added (and tested).\n\nAn enum would definitely make it better. I'll take a look, and see if\nit makes sense to split it up.\n\n> > +int update_sparsity(struct unpack_trees_options *o)\n> > +{\n> > + struct pattern_list pl;\n> > + int i, empty_worktree, ret = 0;\n> > + unsigned old_show_all_errors;\n> > +\n> > + old_show_all_errors = o->show_all_errors;\n> > + o->show_all_errors = 1;\n> > +\n> > + /* Sanity checks */\n> > + if (!o->update || o->index_only || o->skip_sparse_checkout)\n> > + BUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n> > + if (o->src_index != o->dst_index || o->fn)\n> > + BUG(\"update_sparsity() called wrong\");\n>\n> I was going to add something about split indexes at the end,\n> but I think this precondition has it covered.\n\nI'm not sure how this condition covers split\nindices/indexes/however-it-should-be-spelled, but I think split\nindexes don't actually matter to this function because I don't create\nor delete any index entries -- I merely toggle a single bit\n(SKIP_WORKTREE) for an existing index.\n\n> > +\n> > + trace_performance_enter();\n> > +\n> > + if (!o->pl) {\n> > + char *sparse = git_pathdup(\"info/sparse-checkout\");\n> > + memset(&pl, 0, sizeof(pl));\n> > + pl.use_cone_patterns = core_sparse_checkout_cone;\n> > + if (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0) {\n> > + /* FIXME: Skip to check_updates()?? */\n> > + o->skip_sparse_checkout = 1;\n> > + goto skip_sparse_checkout;\n> > + } else\n> > + o->pl = &pl;\n> > + free(sparse);\n> > + }\n>\n> Outside of the goto, this part looks identical to other parts\n> of the same file where we \"fallback\" to the sparse-checkout file.\n> Perhaps that could be extracted to a helper method? It could\n> return failure when there is no in-memory pattern and the\n> sparse-checkout file fails to load, and in response we can do\n> the goto.\n\nMakes sense.\n\n> > + /* Set NEW_SKIP_WORKTREE on existing entries. */\n> > + mark_all_ce_unused(o->src_index);\n> > + mark_new_skip_worktree(o->pl, o->src_index, 0,\n> > + CE_NEW_SKIP_WORKTREE, o->verbose_update);\n> > +\n> > + /* Then loop over entries and update/remove as needed */\n> > + ret = 0;\n> > + empty_worktree = 1;\n> > + for (i = 0; i < o->src_index->cache_nr; i++) {\n> > + struct cache_entry *ce = o->src_index->cache[i];\n> > +\n> > + if (apply_sparse_checkout(o->src_index, ce, o))\n> > + ret = 1;\n>\n> So here, we report that at least one path has a warning.\n>\n> > + if (!ce_skip_worktree(ce))\n> > + empty_worktree = 0;\n> > +\n> > + }\n> > +\n> > + /*\n> > + * Sparse checkout is meant to narrow down checkout area\n> > + * but it does not make sense to narrow down to empty working\n> > + * tree. This is usually a mistake in sparse checkout rules.\n> > + * Do not allow users to do that.\n> > + */\n> > + if (o->src_index->cache_nr && empty_worktree) {\n> > + ret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n>\n> Interesting that we are using the result of unpack_failed() here.\n> I suppose that could be an argument for using an int instead of\n> an enum, but perhaps we translate the result of unpack_failed()\n> here? What sorts of values could it be, anyway? Is this a case\n> where unpack_failed() could return either -1 or -2? Or should\n> we simply set 'ret' to the \"-1\" case ourselves?\n\nI think we can map its return values to the enum we add.\n\n>\n> > + goto done;\n> > + }\n> > +\n> > +skip_sparse_checkout:\n> > + if (check_updates(o, o->src_index))\n> > + ret = -2;\n> > +\n> > +done:\n> > + display_warning_msgs(o);\n> > + o->show_all_errors = old_show_all_errors;\n> > + trace_performance_leave(\"update_sparsity\");\n> > + return ret;\n> > +}\n> > +\n> > /* Here come the merge functions */\n> >\n> > static int reject_merge(const struct cache_entry *ce,\n> > diff --git a/unpack-trees.h b/unpack-trees.h\n> > index ae1557fb804..5d1a77ae249 100644\n> > --- a/unpack-trees.h\n> > +++ b/unpack-trees.h\n> > @@ -90,6 +90,8 @@ struct unpack_trees_options {\n> > int unpack_trees(unsigned n, struct tree_desc *t,\n> > struct unpack_trees_options *options);\n> >\n> > +int update_sparsity(struct unpack_trees_options *options);\n> > +\n>\n> Would it not be the pattern to include the documentation comment\n> here in the header file?\n\nGood point.\n\n> In some ways, your update_sparsity() repeats some of the patterns\n> of unpack_trees(), especially with its order of mark_all_ce_unused(),\n> mark_new_skip_worktree(), and unpack_failed(). I think there may be\n> some benefits to adding some of the performance tracing patterns from\n> that method, too. These could be added as follow-up patches.\n\nThis also makes sense, though I'd be a bit surprised if performance\ntracing for this function shows up anywhere.\n\n> I'm happy that you decided to create a new method instead of adding\n> yet another mode to unpack_trees(), especially because your version\n> does not involve calling traverse_trees().\n>\n> After careful inspection of your patch, I cannot find anything wrong.\n> I'm pretty happy with your organization and the approach seems clear.\n> Hopefully my comments in the commit message could help improve that\n> part of the discussion.\n>\n> At this point, the only thing we really need are tests that demonstrate\n> as many of these cases as possible.\n\nThanks for reviewing it!\n\n> Thanks,\n> -Stolee\n\n\n\n[1] A whole bunch of examples of conflicts that are not representable\nwith conflict markers, and non-content conflicts of various types:\n\nIf a symlink points at different paths, then attempting to represent\nthe contents of the symlinks with conflict markers results in a\nsymlink that is unusable and confusing; instead the version of the\nsymlink from HEAD is left in the directory and we know it is unmerged\nnot by having unstaged changes but the higher order index entries.\n\nIf a binary path has been modified on both sides of history, then\nadding conflict markers is not compatible with the binary nature of\nthe file. Instead, the version of the binary file from HEAD is left\nin the directory and we know it is unmerged due to the presence of\nhigher order index entries.\n\nIf a simple script did not exist in the merge base but was added on\nboth sides with identical contents, with one side of history marking\nit as executable and the other not marking it executable, then there\nis nothing for us to put conflict markers around. Further, we can't\nmark the mode on disk as conflicted in some fashion, we just have to\npick whether to mark the copy in the working directory as executable\nor not. Storing higher order entries in the index is the only way for\nus to know it is unmerged.\n\nIf a simple non-binary, non-symlink file exists and has never been\nmarked executable and existed in the merge base and HEAD but was\ndeleted on the other side of history, there's not much for us to do\nwith conflict markers that would make sense. We just leave the file\nfrom HEAD in the working directory, and instead note the conflict via\nhigher order index entries (and in this case, there will be 2 instead\nof the normal 3 higher order entries).\n\nIf a simple non-binary, non-symlink file was renamed by both sides of\nhistory but did not have any content change (and wasn't ever marked\nexecutable), then there is again nothing for us to do with conflict\nmarkers. We'll instead have 3 higher order index entries, but exactly\none for each path. This case is like the others above in that we can\nlook at the presence of higher order index entries, but slightly\ndifferent because there will only be one higher order (stage > 0)\nentry in the index for any affected path.\n"},{"id":"393631","messageId":"3c183727c983fc733d83301cd289c24bd0e8bc65.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 01/18] unpack-trees: fix minor typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:52Z","receivedAt":"2020-03-21T18:00:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1ecdab33040..0d0eec0221e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1618,7 +1618,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t\t/*\n \t\t * Sparse checkout loop #2: set NEW_SKIP_WORKTREE on entries not in loop #1\n-\t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n+\t\t * If they will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n \t\tmark_new_skip_worktree(o->pl, &o->result,\n-- \ngitgitgadget\n\n"},{"id":"393632","messageId":"739b52393fe595f11df557c28ef9f8aea60eca01.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 03/18] unpack-trees: simplify verify_absent_sparse()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:54Z","receivedAt":"2020-03-21T18:00:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nverify_absent_sparse() was introduced in commit 08402b0409\n(\"merge-recursive: distinguish \"removed\" and \"overwritten\" messages\",\n2010-08-11), and has always had exactly one caller which always passes\nerror_type == ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN. This function\nthen checks whether error_type is this value, and if so, sets it instead\nto ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN. It has been nearly a decade\nand no other caller has been created, and no other value has ever been\npassed, so just pass the expected value to begin with.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f72a7a21f9c..3af2e126abf 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -506,7 +506,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n \tif (was_skip_worktree && !ce_skip_worktree(ce)) {\n-\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n+\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN, o))\n \t\t\treturn -1;\n \t\tce->ce_flags |= CE_UPDATE;\n \t}\n@@ -2026,11 +2026,7 @@ static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\tenum unpack_trees_error_types orphaned_error = error_type;\n-\tif (orphaned_error == ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN)\n-\t\torphaned_error = ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN;\n-\n-\treturn verify_absent_1(ce, orphaned_error, o);\n+\treturn verify_absent_1(ce, error_type, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n-- \ngitgitgadget\n\n"},{"id":"393633","messageId":"e6beb1c5810391274676e4ed0cac78e7e02b804b.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 02/18] unpack-trees: remove unused error type","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:53Z","receivedAt":"2020-03-21T18:00:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncommit 08402b0409 (\"merge-recursive: distinguish \"removed\" and\n\"overwritten\" messages\", 2010-08-11) split\n ERROR_WOULD_LOSE_UNTRACKED\ninto both\n ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN\n ERROR_WOULD_LOSE_UNTRACKED_REMOVED\nand also split\n ERROR_WOULD_LOSE_ORPHANED\ninto both\n ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN\n ERROR_WOULD_LOSE_ORPHANED_REMOVED\n\nHowever, despite the split only three of these four types were used.\nERROR_WOULD_LOSE_ORPHANED_REMOVED was not put into use when it was\nintroduced and nothing else has used it in the intervening decade\neither. Remove it.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 4 ----\n unpack-trees.h | 1 -\n 2 files changed, 5 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0d0eec0221e..f72a7a21f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -49,8 +49,6 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n \n-\t/* ERROR_WOULD_LOSE_ORPHANED_REMOVED */\n-\t\"Working tree file '%s' would be removed by sparse checkout update.\",\n \n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n@@ -172,8 +170,6 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_ORPHANED_REMOVED] =\n-\t\t_(\"The following working tree files would be removed by sparse checkout update:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n \t\t_(\"Cannot update submodule:\\n%s\");\n \ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex ae1557fb804..6d7c7b6c2e0 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -24,7 +24,6 @@ enum unpack_trees_error_types {\n \tERROR_BIND_OVERLAP,\n \tERROR_SPARSE_NOT_UPTODATE_FILE,\n \tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tERROR_WOULD_LOSE_ORPHANED_REMOVED,\n \tERROR_WOULD_LOSE_SUBMODULE,\n \tNB_UNPACK_TREES_ERROR_TYPES\n };\n-- \ngitgitgadget\n\n"},{"id":"393634","messageId":"65772dd46df12d79a41e7ed2d6e1fc197b80d379.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 04/18] unpack-trees: simplify pattern_list freeing","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:55Z","receivedAt":"2020-03-21T18:00:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncommit e091228e17 (\"sparse-checkout: update working directory\nin-process\", 2019-11-21) allowed passing a pre-defined set of patterns\nto unpack_trees(). However, if o->pl was NULL, it would still read the\nexisting patterns and use those. If those patterns were read into a\ndata structure that was allocated, naturally they needed to be free'd.\nHowever, despite the same function being responsible for knowing about\nboth the allocation and the free'ing, the logic for tracking whether to\nfree the pattern_list was hoisted to an outer function with an\nadditional flag in unpack_trees_options. Put the logic back in the\nrelevant function and discard the now unnecessary flag.\n\nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 1 -\n unpack-trees.c | 6 ++++--\n unpack-trees.h | 3 +--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 740da4b6d54..d102a9697fd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -122,7 +122,6 @@ static int update_working_directory(struct pattern_list *pl)\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n-\to.keep_pattern_list = !!pl;\n \n \tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3af2e126abf..d2863fa0310 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1503,6 +1503,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tint i, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n+\tint free_pattern_list = 0;\n \n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n@@ -1519,6 +1520,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\telse\n \t\t\to->pl = &pl;\n \t\tfree(sparse);\n+\t\tfree_pattern_list = 1;\n \t}\n \n \tmemset(&o->result, 0, sizeof(o->result));\n@@ -1686,9 +1688,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->src_index = NULL;\n \n done:\n-\ttrace_performance_leave(\"unpack_trees\");\n-\tif (!o->keep_pattern_list)\n+\tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\n+\ttrace_performance_leave(\"unpack_trees\");\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 6d7c7b6c2e0..d3516267f36 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -58,8 +58,7 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run,\n-\t\t keep_pattern_list;\n+\t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"393635","messageId":"5fea19f0136a849c70f18591d88abae8cee99841.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 06/18] unpack-trees: allow check_updates() to work on a different index","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:57Z","receivedAt":"2020-03-21T18:00:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncheck_updates() previously assumed it was working on o->result. We want\nto use this function in combination with a different index_state, so\ntake the intended index_state as a parameter.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d2863fa0310..dde50047a82 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -353,12 +353,12 @@ static void report_collided_checkout(struct index_state *index)\n \tstring_list_clear(&list, 0);\n }\n \n-static int check_updates(struct unpack_trees_options *o)\n+static int check_updates(struct unpack_trees_options *o,\n+\t\t\t struct index_state *index)\n {\n \tunsigned cnt = 0;\n \tint errs = 0;\n \tstruct progress *progress;\n-\tstruct index_state *index = &o->result;\n \tstruct checkout state = CHECKOUT_INIT;\n \tint i;\n \n@@ -1665,7 +1665,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t}\n \t}\n \n-\tret = check_updates(o) ? (-2) : 0;\n+\tret = check_updates(o, &o->result) ? (-2) : 0;\n \tif (o->dst_index) {\n \t\tmove_index_extensions(&o->result, o->src_index);\n \t\tif (!ret) {\n-- \ngitgitgadget\n\n"},{"id":"393636","messageId":"5cbab2a7a560cd1a2459d63d34d19e5abd265f9a.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 05/18] t1091: make some tests a little more defensive against failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:56Z","receivedAt":"2020-03-21T18:00:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 44a91205d60..8607a8e6d1a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -278,6 +278,7 @@ test_expect_success 'cone mode: add parent path' '\n '\n \n test_expect_success 'revert to old sparse-checkout on bad update' '\n+\ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n \techo update >repo/deep/deeper2/a &&\n@@ -328,6 +329,7 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n '\n \n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n+\trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n \tcat >expect <<-\\EOF &&\n-- \ngitgitgadget\n\n"},{"id":"393637","messageId":"27d6304d33f568c185219f76e47c53ecf9fe8b43.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 08/18] unpack-trees: pull sparse-checkout pattern reading into a new function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:59Z","receivedAt":"2020-03-21T18:00:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate a populate_from_existing_patterns() function for reading the\npath_patterns from $GIT_DIR/info/sparse-checkout so that we can re-use\nit elsewhere.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e8e794880ab..4733e7eaf89 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1492,6 +1492,20 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n+static void populate_from_existing_patterns(struct unpack_trees_options *o,\n+\t\t\t\t\t struct pattern_list *pl)\n+{\n+\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\n+\tpl->use_cone_patterns = core_sparse_checkout_cone;\n+\tif (add_patterns_from_file_to_list(sparse, \"\", 0, pl, NULL) < 0)\n+\t\to->skip_sparse_checkout = 1;\n+\telse\n+\t\to->pl = pl;\n+\tfree(sparse);\n+}\n+\n+\n static int verify_absent(const struct cache_entry *,\n \t\t\t enum unpack_trees_error_types,\n \t\t\t struct unpack_trees_options *);\n@@ -1512,18 +1526,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n \ttrace_performance_enter();\n-\tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n-\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n-\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n-\t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n-\t\t\to->skip_sparse_checkout = 1;\n-\t\telse\n-\t\t\to->pl = &pl;\n-\t\tfree(sparse);\n+\t\tmemset(&pl, 0, sizeof(pl));\n \t\tfree_pattern_list = 1;\n+\t\tpopulate_from_existing_patterns(o, &pl);\n \t}\n \n \tmemset(&o->result, 0, sizeof(o->result));\n-- \ngitgitgadget\n\n"},{"id":"393638","messageId":"a46439c8536f912ad4a1e1751852cf477d3d7dc7.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:00Z","receivedAt":"2020-03-21T18:00:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPreviously, the only way to update the SKIP_WORKTREE bits for various\npaths was invoking `git read-tree -mu HEAD` or calling the same code\nthat this codepath invoked. This however had a number of problems if\nthe index or working directory were not clean. First, let's consider\nthe case:\n\n Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files)\n\nIf the working tree was clean this was fine, but if there were files or\ndirectories or symlinks or whatever already present at the given path\nthen the operation would abort with an error. Let's label this case\nfor later discussion:\n\n A) There is an untracked path in the way\n\nNow let's consider the opposite case:\n\n Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files)\n\nIf the index and working tree was clean this was fine, but if there were\nany unclean paths we would run into problems. There are three different\ncases to consider:\n\n B) The path is unmerged\n C) The path has unstaged changes\n D) The path has staged changes (differs from HEAD)\n\nIf any path fell into case B or C, then the whole operation would be\naborted with an error. With sparse-checkout, the whole operation would\nbe aborted for case D as well, but for its predecessor of using `git\nread-tree -mu HEAD` directly, any paths that fell into case D would be\nremoved from the working copy and the index entry for that path would be\nreset to match HEAD -- which looks and feels like data loss to users\n(only a few are even aware to ask whether it can be recovered, and even\nthen it requires walking through loose objects trying to match up the\nright ones).\n\nRefusing to remove files that have unsaved user changes is good, but\nrefusing to work on any other paths is very problematic for users. If\nthe user is in the middle of a rebase or has made modifications to files\nthat bring in more dependencies, then for their build to work they need\nto update the sparse paths. This logic has been preventing them from\ndoing so. Sometimes in response, the user will stage the files and\nre-try, to no avail with sparse-checkout or to the horror of losing\ntheir changes if they are using its predecessor of `git read-tree -mu\nHEAD`.\n\nAdd a new update_sparsity() function which will not error out in any of\nthese cases but behaves as follows for the special cases:\n A) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n bit, and print a warning (thus leaving the path in a state where\n status will report the file as modified, which seems logical).\n B) Do NOT mark this path as SKIP_WORKTREE, and leave it as unmerged.\n C) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n the dirty path.\n D) Mark the path as SKIP_WORKTREE, but do not revert the version\n stored in the index to match HEAD; leave the contents alone.\n\nI tried a different behavior for A (leave the SKIP_WORKTREE bit set),\nbut found it very surprising and counter-intuitive (e.g. the user sees\nit is present along with all the other files in that directory, tries to\nstage it, but git add ignores it since the SKIP_WORKTREE bit is set). A\n& C seem like optimal behavior to me. B may be as well, though I wonder\nif printing a warning would be an improvement. Some might be slightly\nsurprised by D at first, but given that it does the right thing with\n`git commit` and even `git commit -a` (`git add` ignores entries that\nare marked SKIP_WORKTREE and thus doesn't delete them, and `commit -a`\nis similar), it seems logical to me.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.h | 9 ++++++\n 2 files changed, 87 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 4733e7eaf89..6abea555929 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1714,6 +1714,84 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tgoto done;\n }\n \n+/*\n+ * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n+ * working directory to match.\n+ *\n+ * CE_NEW_SKIP_WORKTREE is used internally.\n+ */\n+enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n+{\n+\tenum update_sparsity_result ret = UPDATE_SPARSITY_SUCCESS;\n+\tstruct pattern_list pl;\n+\tint i, empty_worktree;\n+\tunsigned old_show_all_errors;\n+\tint free_pattern_list = 0;\n+\n+\told_show_all_errors = o->show_all_errors;\n+\to->show_all_errors = 1;\n+\n+\t/* Sanity checks */\n+\tif (!o->update || o->index_only || o->skip_sparse_checkout)\n+\t\tBUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n+\tif (o->src_index != o->dst_index || o->fn)\n+\t\tBUG(\"update_sparsity() called wrong\");\n+\n+\ttrace_performance_enter();\n+\n+\t/* If we weren't given patterns, use the recorded ones */\n+\tif (!o->pl) {\n+\t\tmemset(&pl, 0, sizeof(pl));\n+\t\tfree_pattern_list = 1;\n+\t\tpopulate_from_existing_patterns(o, &pl);\n+\t\tif (o->skip_sparse_checkout)\n+\t\t\tgoto skip_sparse_checkout;\n+\t}\n+\n+\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n+\tmark_all_ce_unused(o->src_index);\n+\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n+\n+\t/* Then loop over entries and update/remove as needed */\n+\tret = UPDATE_SPARSITY_SUCCESS;\n+\tempty_worktree = 1;\n+\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = o->src_index->cache[i];\n+\n+\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n+\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n+\n+\t\tif (!ce_skip_worktree(ce))\n+\t\t\tempty_worktree = 0;\n+\n+\t}\n+\n+\t/*\n+\t * Sparse checkout is meant to narrow down checkout area\n+\t * but it does not make sense to narrow down to empty working\n+\t * tree. This is usually a mistake in sparse checkout rules.\n+\t * Do not allow users to do that.\n+\t */\n+\tif (o->src_index->cache_nr && empty_worktree) {\n+\t\tunpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n+\t\tret = UPDATE_SPARSITY_INDEX_UPDATE_FAILURES;\n+\t\tgoto done;\n+\t}\n+\n+skip_sparse_checkout:\n+\tif (check_updates(o, o->src_index))\n+\t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n+\n+done:\n+\tdisplay_error_msgs(o);\n+\to->show_all_errors = old_show_all_errors;\n+\tif (free_pattern_list)\n+\t\tclear_pattern_list(&pl);\n+\ttrace_performance_leave(\"update_sparsity\");\n+\treturn ret;\n+}\n+\n /* Here come the merge functions */\n \n static int reject_merge(const struct cache_entry *ce,\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex d3516267f36..2c5d54cae9f 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -28,6 +28,13 @@ enum unpack_trees_error_types {\n \tNB_UNPACK_TREES_ERROR_TYPES\n };\n \n+enum update_sparsity_result {\n+\tUPDATE_SPARSITY_SUCCESS = 0,\n+\tUPDATE_SPARSITY_WARNINGS = 1,\n+\tUPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n+\tUPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n+};\n+\n /*\n * Sets the list of user-friendly error messages to be used by the\n * command \"cmd\" (either merge or checkout), and show_all_errors to 1.\n@@ -88,6 +95,8 @@ struct unpack_trees_options {\n int unpack_trees(unsigned n, struct tree_desc *t,\n \t\t struct unpack_trees_options *options);\n \n+int update_sparsity(struct unpack_trees_options *options);\n+\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o);\n \n-- \ngitgitgadget\n\n"},{"id":"393639","messageId":"79b9cc1df55c7fab9b3c0e8dcc88ca19f23e771c.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 11/18] sparse-checkout: use improved unpack_trees porcelain messages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:02Z","receivedAt":"2020-03-21T18:00:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nsetup_unpack_trees_porcelain() provides much improved error/warning\nmessages; instead of a message that assumes that there is only one path\nwith a given problem despite being used by code that intentionally is\ngrouping and showing errors together, it uses a message designed to be\nused with groups of paths. For example, this transforms\n\n error: Entry '\tfolder1/a\n\tfolder2/a\n ' not uptodate. Cannot update sparse checkout.\n\ninto\n\n error: Cannot update sparse checkout: the following entries are not up to date:\n\tfolder1/a\n\tfolder2/a\n\nIn the past the suboptimal messages were never actually triggered\nbecause we would error out if the working directory wasn't clean before\nwe even called unpack_trees(). The previous commit changed that,\nthough, so let's use the better error messages.\n\nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 2 ++\n t/t1091-sparse-checkout-builtin.sh | 9 +++------\n 2 files changed, 5 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex a55c60d7594..aa81199f85d 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -112,7 +112,9 @@ static int update_working_directory(struct pattern_list *pl)\n \n \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n \n+\tsetup_unpack_trees_porcelain(&o, \"sparse-checkout\");\n \tresult = update_sparsity(&o);\n+\tclear_unpack_trees_porcelain(&o);\n \n \tif (result == UPDATE_SPARSITY_WARNINGS)\n \t\t/*\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 86ae422ff5c..93c650ac038 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -328,16 +328,13 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"error\" err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_i18ngrep \"error\" err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout disable &&\n-\ttest_i18ngrep \"error\" err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n-- \ngitgitgadget\n\n"},{"id":"393640","messageId":"203ece0ec2fc7b0e214f8bcf356e299d1bb8f202.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 13/18] unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:04Z","receivedAt":"2020-03-21T18:00:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to treat issues with setting the SKIP_WORKTREE bit as a warning\nrather than an error; rename the enum values to reflect this intent as\na simple step towards that goal.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 12 ++++++------\n unpack-trees.h | 8 +++++---\n 2 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 5c99d588dc3..0554842580b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -46,10 +46,10 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n \n-\t/* ERROR_SPARSE_NOT_UPTODATE_FILE */\n+\t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n-\t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n+\t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n };\n \n@@ -168,9 +168,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n \t\t_(\"Cannot update submodule:\\n%s\");\n \n-\tmsgs[ERROR_SPARSE_NOT_UPTODATE_FILE] =\n+\tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n+\tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n \n \topts->show_all_errors = 1;\n@@ -509,7 +509,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n \tif (was_skip_worktree && !ce_skip_worktree(ce)) {\n-\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN, o))\n+\t\tif (verify_absent_sparse(ce, WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN, o))\n \t\t\treturn -1;\n \t\tce->ce_flags |= CE_UPDATE;\n \t}\n@@ -1876,7 +1876,7 @@ int verify_uptodate(const struct cache_entry *ce,\n static int verify_uptodate_sparse(const struct cache_entry *ce,\n \t\t\t\t struct unpack_trees_options *o)\n {\n-\treturn verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE);\n+\treturn verify_uptodate_1(ce, o, WARNING_SPARSE_NOT_UPTODATE_FILE);\n }\n \n /*\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex a656bbf810b..3c6452fe9e5 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -23,9 +23,11 @@ enum unpack_trees_error_types {\n \tERROR_WOULD_LOSE_UNTRACKED_REMOVED,\n \tERROR_BIND_OVERLAP,\n \tERROR_WOULD_LOSE_SUBMODULE,\n-\tERROR_SPARSE_NOT_UPTODATE_FILE,\n-\tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tNB_UNPACK_TREES_ERROR_TYPES\n+\n+\tWARNING_SPARSE_NOT_UPTODATE_FILE,\n+\tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n+\n+\tNB_UNPACK_TREES_ERROR_TYPES,\n };\n \n enum update_sparsity_result {\n-- \ngitgitgadget\n\n"},{"id":"393641","messageId":"27ed4a3f7a28c66d8185cb4c484e8b0af81b15c6.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 10/18] sparse-checkout: use new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:01Z","receivedAt":"2020-03-21T18:00:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nRemove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\ncodepaths for setting the SKIP_WORKTREE bits and instead use the new\nupdate_sparsity() function.\n\nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 40 ++++++++----------------------\n t/t1091-sparse-checkout-builtin.sh | 35 +++++++++++++++++++-------\n 2 files changed, 36 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex d102a9697fd..a55c60d7594 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -94,49 +94,35 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n static int update_working_directory(struct pattern_list *pl)\n {\n-\tint result = 0;\n+\tenum update_sparsity_result result;\n \tstruct unpack_trees_options o;\n \tstruct lock_file lock_file = LOCK_INIT;\n-\tstruct object_id oid;\n-\tstruct tree *tree;\n-\tstruct tree_desc t;\n \tstruct repository *r = the_repository;\n \n-\tif (repo_read_index_unmerged(r))\n-\t\tdie(_(\"you need to resolve your current index first\"));\n-\n-\tif (get_oid(\"HEAD\", &oid))\n-\t\treturn 0;\n-\n-\ttree = parse_tree_indirect(&oid);\n-\tparse_tree(tree);\n-\tinit_tree_desc(&t, tree->buffer, tree->size);\n-\n \tmemset(&o, 0, sizeof(o));\n \to.verbose_update = isatty(2);\n-\to.merge = 1;\n \to.update = 1;\n-\to.fn = oneway_merge;\n \to.head_idx = -1;\n \to.src_index = r->index;\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n \n-\tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\n \n-\tcache_tree_free(&r->index->cache_tree);\n-\n \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n \n-\tcore_apply_sparse_checkout = 1;\n-\tresult = unpack_trees(1, &t, &o);\n+\tresult = update_sparsity(&o);\n \n-\tif (!result) {\n-\t\tprime_cache_tree(r, r->index, tree);\n+\tif (result == UPDATE_SPARSITY_WARNINGS)\n+\t\t/*\n+\t\t * We don't do any special handling of warnings from untracked\n+\t\t * files in the way or dirty entries that can't be removed.\n+\t\t */\n+\t\tresult = UPDATE_SPARSITY_SUCCESS;\n+\tif (result == UPDATE_SPARSITY_SUCCESS)\n \t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n-\t} else\n+\telse\n \t\trollback_lock_file(&lock_file);\n \n \treturn result;\n@@ -303,8 +289,6 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_sparse_checkout_init_options,\n@@ -559,8 +543,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_sparse_checkout_set_options,\n@@ -576,8 +558,6 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tstruct strbuf match_all = STRBUF_INIT;\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n \n \tmemset(&pl, 0, sizeof(pl));\n \thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 8607a8e6d1a..86ae422ff5c 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n \tcheck_files repo a deep folder1\n '\n \n-test_expect_success 'revert to old sparse-checkout on bad update' '\n+test_expect_success 'not-up-to-date does not block rest of sparsification' '\n \ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n+\n \techo update >repo/deep/deeper2/a &&\n \tcp repo/.git/info/sparse-checkout expect &&\n-\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n-\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n-\ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tcheck_files repo/deep a deeper1 deeper2\n+\ttest_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n+\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tcheck_files repo/deep a deeper1 deeper2 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n+\tcheck_files repo/deep/deeper1/deepest a &&\n+\tcheck_files repo/deep/deeper2 a\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -316,12 +323,22 @@ test_expect_success '.gitignore should not warn about cone mode' '\n \ttest_i18ngrep ! \"disabling cone patterns\" err\n '\n \n-test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n+test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n \tgit clone repo dirty &&\n \techo dirty >dirty/folder1/a &&\n-\ttest_must_fail git -C dirty sparse-checkout init &&\n-\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_must_fail git -C dirty sparse-checkout disable &&\n+\n+\tgit -C dirty sparse-checkout init 2>err &&\n+\ttest_i18ngrep \"error\" err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\n+\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\ttest_i18ngrep \"error\" err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\n+\tgit -C dirty sparse-checkout disable &&\n+\ttest_i18ngrep \"error\" err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-- \ngitgitgadget\n\n"},{"id":"393642","messageId":"02a8e01f35b7864dbeccd76e2969397b1a84d29a.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 15/18] unpack-trees: make sparse path messages sound like warnings","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:06Z","receivedAt":"2020-03-21T18:00:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe messages for problems with sparse paths are phrased as errors that\ncause the operation to abort, even though we are not making the\noperation abort. Reword the messages to make sense in their new\ncontext.\n\nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 8 ++++----\n unpack-trees.c | 8 ++++----\n 2 files changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0d93d3983e0..11eb567f3fb 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -288,7 +288,7 @@ test_expect_success 'not-up-to-date does not block rest of sparsification' '\n \n \tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n \n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"The following paths are not up to date\" err &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo/deep a deeper1 deeper2 &&\n \tcheck_files repo/deep/deeper1 a deepest &&\n@@ -328,13 +328,13 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \n \tgit -C dirty sparse-checkout disable &&\n-\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 9ee04992ac6..7c24933016b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -50,10 +50,10 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t\"\",\n \n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n-\t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n+\t\"Path '%s' not uptodate; will not remove from working tree.\",\n \n \t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n-\t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n+\t\"Path '%s' already present; will not overwrite with sparse update.\",\n };\n \n #define ERRORMSG(o,type) \\\n@@ -172,9 +172,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t\t_(\"Cannot update submodule:\\n%s\");\n \n \tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n-\t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n+\t\t_(\"The following paths are not up to date and were left despite sparse patterns:\\n%s\");\n \tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n-\t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n+\t\t_(\"The following paths were already present and thus not updated despite sparse patterns:\\n%s\");\n \n \topts->show_all_errors = 1;\n \t/* rejected paths may not have a static buffer */\n-- \ngitgitgadget\n\n"},{"id":"393643","messageId":"5730f7d250a2025ae62edaf8779ff39bfbaa5ad6.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 18/18] sparse-checkout: provide a new reapply subcommand","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:09Z","receivedAt":"2020-03-21T18:00:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf commands like merge or rebase materialize files as part of their work,\nor a previous sparse-checkout command failed to update individual files\ndue to dirty changes, users may want a command to simply 'reapply' the\nsparsity rules. Provide one.\n\nSigned-off-by: Elijah Newren \n---\n Documentation/git-sparse-checkout.txt | 10 ++++++++++\n builtin/sparse-checkout.c | 10 +++++++++-\n 2 files changed, 19 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex c0342e53938..1a3ace60820 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -70,6 +70,16 @@ C-style quoted strings.\n \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n \tas directory names as in the 'set' subcommand.\n \n+'reapply::\n+\tReapply the sparsity pattern rules to paths in the working tree.\n+\tCommands like merge or rebase can materialize paths to do their\n+\twork (e.g. in order to show you a conflict), and other\n+\tsparse-checkout commands might fail to sparsify an individual file\n+\t(e.g. because it has unstaged changes or conflicts). In such\n+\tcases, it can make sense to run `git sparse-checkout reapply` later\n+\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n+\tor committing changes, etc.).\n+\n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n \tworking directory to include all files. Leaves the sparse-checkout\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex aa81199f85d..95d08824172 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -18,7 +18,7 @@\n static const char *empty_base = \"\";\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n+\tN_(\"git sparse-checkout (init|list|set|add|reapply|disable) \"),\n \tNULL\n };\n \n@@ -554,6 +554,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \treturn modify_pattern_list(argc, argv, m);\n }\n \n+static int sparse_checkout_reapply(int argc, const char **argv)\n+{\n+\trepo_read_index(the_repository);\n+\treturn update_working_directory(NULL);\n+}\n+\n static int sparse_checkout_disable(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -603,6 +609,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n \t\tif (!strcmp(argv[0], \"add\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n+\t\tif (!strcmp(argv[0], \"reapply\"))\n+\t\t\treturn sparse_checkout_reapply(argc, argv);\n \t\tif (!strcmp(argv[0], \"disable\"))\n \t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n-- \ngitgitgadget\n"},{"id":"393644","messageId":"fa997a1c72f8ff15a6487475679ad9d20a9d3a0e.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 17/18] unpack-trees: failure to set SKIP_WORKTREE bits always just a warning","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:08Z","receivedAt":"2020-03-21T18:00:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSetting and clearing of the SKIP_WORKTREE bit is not only done when\nusers run 'sparse-checkout'; other commands such as 'checkout' also run\nthrough unpack_trees() which has logic for handling this special bit.\nAs such, we need to consider how they handle special cases. A couple\ncomparison points should help explain the rationale for changing how\nunpack_trees() handles these bits:\n\n Ignoring sparse checkouts for a moment, if you are switching\n branches and have dirty changes, it is only considered an error that\n will prevent the branch switching from being successful if the dirty\n file happens to be one of the paths with different contents.\n\n SKIP_WORKTREE has always been considered advisory; for example, if\n rebase or merge need or even want to materialize a path as part of\n their work, they have always been allowed to do so regardless of the\n SKIP_WORKTREE setting. This has been used for unmerged paths, but\n it was often used for paths it wasn't needed just because it made\n the code simpler. It was a best-effort consideration, and when it\n materialized paths contrary to the SKIP_WORKTREE setting, it was\n never required to even print a warning message.\n\nIn the past if you trying to run e.g. 'git checkout' and:\n 1) you had a path that was materialized and had some dirty changes\n 2) the path was listed in $GITDIR/info/sparse-checkout\n 3) this path did not different between the current and target branches\nthen despite the comparison points above, the inability to set\nSKIP_WORKTREE was treated as a *hard* error that would abort the\ncheckout operation. This is completely inconsistent with how\nSKIP_WORKTREE is handled elsewhere, and rather annoying for users as\nleaving the paths materialized in the working copy (with a simple\nwarning) should present no problem at all.\n\nDowngrade any errors from inability to toggle the SKIP_WORKTREE bit to a\nwarning and allow the operations to continue.\n\nSigned-off-by: Elijah Newren \n---\n t/t1011-read-tree-sparse-checkout.sh | 11 +++++-----\n t/t2018-checkout-branch.sh | 22 ++++++++++++++++++++\n unpack-trees.c | 31 ++++++++++++++--------------\n 3 files changed, 43 insertions(+), 21 deletions(-)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex eb44bafb593..63223e13bd1 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -233,18 +233,19 @@ test_expect_success 'read-tree --reset removes outside worktree' '\n \ttest_must_be_empty result\n '\n \n-test_expect_success 'print errors when failed to update worktree' '\n+test_expect_success 'print warnings when some worktree updates disabled' '\n \techo sub >.git/info/sparse-checkout &&\n \tgit checkout -f init &&\n \tmkdir sub &&\n \ttouch sub/added sub/addedtoo &&\n-\ttest_must_fail git checkout top 2>actual &&\n+\t# Use -q to suppress \"Previous HEAD position\" and \"Head is now at\" msgs\n+\tgit checkout -q top 2>actual &&\n \tcat >expected <<\\EOF &&\n-error: The following untracked working tree files would be overwritten by checkout:\n+warning: The following paths were already present and thus not updated despite sparse patterns:\n \tsub/added\n \tsub/addedtoo\n-Please move or remove them before you switch branches.\n-Aborting\n+\n+After fixing the above paths, you may want to run `git sparse-checkout reapply`.\n EOF\n \ttest_i18ncmp expected actual\n '\ndiff --git a/t/t2018-checkout-branch.sh b/t/t2018-checkout-branch.sh\nindex bbca7ef8da6..21583154d8e 100755\n--- a/t/t2018-checkout-branch.sh\n+++ b/t/t2018-checkout-branch.sh\n@@ -238,4 +238,26 @@ test_expect_success 'checkout -b after clone --no-checkout does a checkout of HE\n \ttest_path_is_file dest/a.t\n '\n \n+test_expect_success 'checkout -b to a new branch preserves mergeable changes despite sparse-checkout' '\n+\ttest_when_finished \"\n+\t\tgit reset --hard &&\n+\t\tgit checkout branch1-scratch &&\n+\t\ttest_might_fail git branch -D branch3 &&\n+\t\tgit config core.sparseCheckout false &&\n+\t\trm .git/info/sparse-checkout\" &&\n+\n+\ttest_commit file2 &&\n+\n+\techo stuff >>file1 &&\n+\techo file2 >.git/info/sparse-checkout &&\n+\tgit config core.sparseCheckout true &&\n+\n+\tCURHEAD=$(git rev-parse HEAD) &&\n+\tdo_checkout branch3 $CURHEAD &&\n+\n+\techo file1 >expect &&\n+\tgit diff --name-only >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 298241a5e0e..cab2177c951 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1701,23 +1701,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t * correct CE_NEW_SKIP_WORKTREE\n \t\t\t */\n \t\t\tif (ce->ce_flags & CE_ADDED &&\n-\t\t\t verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\t\tif (!o->show_all_errors)\n-\t\t\t\t\tgoto return_failed;\n-\t\t\t\tret = -1;\n-\t\t\t}\n+\t\t\t verify_absent(ce, WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN, o))\n+\t\t\t\tret = 1;\n+\n+\t\t\tif (apply_sparse_checkout(&o->result, ce, o))\n+\t\t\t\tret = 1;\n \n-\t\t\tif (apply_sparse_checkout(&o->result, ce, o)) {\n-\t\t\t\tif (!o->show_all_errors)\n-\t\t\t\t\tgoto return_failed;\n-\t\t\t\tret = -1;\n-\t\t\t}\n \t\t\tif (!ce_skip_worktree(ce))\n \t\t\t\tempty_worktree = 0;\n-\n \t\t}\n-\t\tif (ret < 0)\n-\t\t\tgoto return_failed;\n \t\t/*\n \t\t * Sparse checkout is meant to narrow down checkout area\n \t\t * but it does not make sense to narrow down to empty working\n@@ -1728,6 +1720,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\tret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n \t\t\tgoto done;\n \t\t}\n+\t\tif (ret == 1) {\n+\t\t\t/*\n+\t\t\t * Inability to sparsify or de-sparsify individual\n+\t\t\t * paths is not an error, but just a warning.\n+\t\t\t */\n+\t\t\tif (o->show_all_errors)\n+\t\t\t\tdisplay_warning_msgs(o);\n+\t\t\tret = 0;\n+\t\t}\n \t}\n \n \tret = check_updates(o, &o->result) ? (-2) : 0;\n@@ -1759,10 +1760,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \treturn ret;\n \n return_failed:\n-\tif (o->show_all_errors) {\n+\tif (o->show_all_errors)\n \t\tdisplay_error_msgs(o);\n-\t\tdisplay_warning_msgs(o);\n-\t}\n \tmark_all_ce_unused(o->src_index);\n \tret = unpack_failed(o, NULL);\n \tif (o->exiting_early)\n-- \ngitgitgadget\n\n"},{"id":"393645","messageId":"57679c8e292ceb58cea2b5b7d893d5f47e1e2de0.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 14/18] unpack-trees: split display_error_msgs() into two","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:05Z","receivedAt":"2020-03-21T18:00:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndisplay_error_msgs() is never called to show messages of both ERROR_*\nand WARNING_* types at the same time; it is instead called multiple\ntimes, separately for each type. Since we want to display these types\ndifferently, make two slightly different versions of this function.\n\nA subsequent commit will further modify unpack_trees() and how it calls\nthe new display_warning_msgs().\n\nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 6 ++--\n unpack-trees.c | 50 +++++++++++++++++++++++++-----\n unpack-trees.h | 8 +++--\n 3 files changed, 50 insertions(+), 14 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 93c650ac038..0d93d3983e0 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -328,13 +328,13 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout disable &&\n-\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0554842580b..9ee04992ac6 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -24,7 +24,7 @@\n * situation better. See how \"git checkout\" and \"git merge\" replaces\n * them using setup_unpack_trees_porcelain(), for example.\n */\n-static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n+static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t/* ERROR_WOULD_OVERWRITE */\n \t\"Entry '%s' would be overwritten by merge. Cannot merge.\",\n \n@@ -46,6 +46,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n \n+\t/* NB_UNPACK_TREES_ERROR_TYPES; just a meta value */\n+\t\"\",\n+\n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n@@ -222,7 +225,7 @@ static int add_rejected_path(struct unpack_trees_options *o,\n \n \t/*\n \t * Otherwise, insert in a list for future display by\n-\t * display_error_msgs()\n+\t * display_(error|warning)_msgs()\n \t */\n \tstring_list_append(&o->unpack_rejects[e], path);\n \treturn -1;\n@@ -233,13 +236,16 @@ static int add_rejected_path(struct unpack_trees_options *o,\n */\n static void display_error_msgs(struct unpack_trees_options *o)\n {\n-\tint e, i;\n-\tint something_displayed = 0;\n+\tint e;\n+\tunsigned error_displayed = 0;\n \tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n \t\tstruct string_list *rejects = &o->unpack_rejects[e];\n+\n \t\tif (rejects->nr > 0) {\n+\t\t\tint i;\n \t\t\tstruct strbuf path = STRBUF_INIT;\n-\t\t\tsomething_displayed = 1;\n+\n+\t\t\terror_displayed = 1;\n \t\t\tfor (i = 0; i < rejects->nr; i++)\n \t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n \t\t\terror(ERRORMSG(o, e), super_prefixed(path.buf));\n@@ -247,10 +253,36 @@ static void display_error_msgs(struct unpack_trees_options *o)\n \t\t}\n \t\tstring_list_clear(rejects, 0);\n \t}\n-\tif (something_displayed)\n+\tif (error_displayed)\n \t\tfprintf(stderr, _(\"Aborting\\n\"));\n }\n \n+/*\n+ * display all the warning messages stored in a nice way\n+ */\n+static void display_warning_msgs(struct unpack_trees_options *o)\n+{\n+\tint e;\n+\tunsigned warning_displayed = 0;\n+\tfor (e = NB_UNPACK_TREES_ERROR_TYPES+1;\n+\t e < NB_UNPACK_TREES_WARNING_TYPES; e++) {\n+\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n+\n+\t\tif (rejects->nr > 0) {\n+\t\t\tint i;\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\t\twarning_displayed = 1;\n+\t\t\tfor (i = 0; i < rejects->nr; i++)\n+\t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n+\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n+\t\t\tstrbuf_release(&path);\n+\t\t}\n+\t\tstring_list_clear(rejects, 0);\n+\t}\n+\tif (warning_displayed)\n+\t\tfprintf(stderr, _(\"After fixing the above paths, you may want to run `git sparse-checkout reapply`.\\n\"));\n+}\n static int check_submodule_move_head(const struct cache_entry *ce,\n \t\t\t\t const char *old_id,\n \t\t\t\t const char *new_id,\n@@ -1705,8 +1737,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \treturn ret;\n \n return_failed:\n-\tif (o->show_all_errors)\n+\tif (o->show_all_errors) {\n \t\tdisplay_error_msgs(o);\n+\t\tdisplay_warning_msgs(o);\n+\t}\n \tmark_all_ce_unused(o->src_index);\n \tret = unpack_failed(o, NULL);\n \tif (o->exiting_early)\n@@ -1784,7 +1818,7 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n \t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n \n done:\n-\tdisplay_error_msgs(o);\n+\tdisplay_warning_msgs(o);\n \to->show_all_errors = old_show_all_errors;\n \tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 3c6452fe9e5..d91c65ae453 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -24,10 +24,12 @@ enum unpack_trees_error_types {\n \tERROR_BIND_OVERLAP,\n \tERROR_WOULD_LOSE_SUBMODULE,\n \n+\tNB_UNPACK_TREES_ERROR_TYPES,\n+\n \tWARNING_SPARSE_NOT_UPTODATE_FILE,\n \tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n \n-\tNB_UNPACK_TREES_ERROR_TYPES,\n+\tNB_UNPACK_TREES_WARNING_TYPES,\n };\n \n enum update_sparsity_result {\n@@ -73,13 +75,13 @@ struct unpack_trees_options {\n \tstruct dir_struct *dir;\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n-\tconst char *msgs[NB_UNPACK_TREES_ERROR_TYPES];\n+\tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n \tstruct argv_array msgs_to_free;\n \t/*\n \t * Store error messages in an array, each case\n \t * corresponding to a error message type\n \t */\n-\tstruct string_list unpack_rejects[NB_UNPACK_TREES_ERROR_TYPES];\n+\tstruct string_list unpack_rejects[NB_UNPACK_TREES_WARNING_TYPES];\n \n \tint head_idx;\n \tint merge_size;\n-- \ngitgitgadget\n\n"},{"id":"393647","messageId":"f664a6dcbb3c3369aa7b94386894da1d437e3295.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 16/18] unpack-trees: provide warnings on sparse updates for unmerged paths too","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:07Z","receivedAt":"2020-03-21T18:00:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen sparse-checkout runs to update the list of sparsity patterns, it\ngives warnings if it can't remove paths from the working tree because\nthose files have dirty changes. Add a similar warning for unmerged\npaths as well.\n\nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 24 ++++++++++++++++++++++++\n unpack-trees.c | 30 ++++++++++++++++++++++++++++++\n unpack-trees.h | 1 +\n 3 files changed, 55 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 11eb567f3fb..39093bcd5ec 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -342,6 +342,30 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \tgit -C dirty sparse-checkout disable\n '\n \n+test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged status' '\n+\tgit clone repo unmerged &&\n+\n+\tcat >input <<-EOF &&\n+\t0 0000000000000000000000000000000000000000\tfolder1/a\n+\t100644 $(git -C unmerged rev-parse HEAD:folder1/a) 1\tfolder1/a\n+\tEOF\n+\tgit -C unmerged update-index --index-info err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C unmerged sparse-checkout disable &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C unmerged reset --hard &&\n+\tgit -C unmerged sparse-checkout init &&\n+\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\tgit -C unmerged sparse-checkout disable\n+'\n+\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7c24933016b..298241a5e0e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -52,6 +52,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Path '%s' not uptodate; will not remove from working tree.\",\n \n+\t/* WARNING_SPARSE_UNMERGED_FILE */\n+\t\"Path '%s' unmerged; will not remove from working tree.\",\n+\n \t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n \t\"Path '%s' already present; will not overwrite with sparse update.\",\n };\n@@ -173,6 +176,8 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \n \tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"The following paths are not up to date and were left despite sparse patterns:\\n%s\");\n+\tmsgs[WARNING_SPARSE_UNMERGED_FILE] =\n+\t\t_(\"The following paths are unmerged and were left despite sparse patterns:\\n%s\");\n \tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n \t\t_(\"The following paths were already present and thus not updated despite sparse patterns:\\n%s\");\n \n@@ -548,6 +553,23 @@ static int apply_sparse_checkout(struct index_state *istate,\n \treturn 0;\n }\n \n+static int warn_conflicted_path(struct index_state *istate,\n+\t\t\t\tint i,\n+\t\t\t\tstruct unpack_trees_options *o)\n+{\n+\tchar *conflicting_path = istate->cache[i]->name;\n+\tint count = 0;\n+\n+\tadd_rejected_path(o, WARNING_SPARSE_UNMERGED_FILE, conflicting_path);\n+\n+\t/* Find out how many higher stage entries at same path */\n+\twhile (++count < istate->cache_nr &&\n+\t !strcmp(conflicting_path,\n+\t\t istate->cache[i+count]->name))\n+\t\t/* do nothing */;\n+\treturn count;\n+}\n+\n static inline int call_unpack_fn(const struct cache_entry * const *src,\n \t\t\t\t struct unpack_trees_options *o)\n {\n@@ -1793,6 +1815,14 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n \tfor (i = 0; i < o->src_index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = o->src_index->cache[i];\n \n+\n+\t\tif (ce_stage(ce)) {\n+\t\t\t/* -1 because for loop will increment by 1 */\n+\t\t\ti += warn_conflicted_path(o->src_index, i, o) - 1;\n+\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (apply_sparse_checkout(o->src_index, ce, o))\n \t\t\tret = UPDATE_SPARSITY_WARNINGS;\n \ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex d91c65ae453..f970fd6c2f4 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -27,6 +27,7 @@ enum unpack_trees_error_types {\n \tNB_UNPACK_TREES_ERROR_TYPES,\n \n \tWARNING_SPARSE_NOT_UPTODATE_FILE,\n+\tWARNING_SPARSE_UNMERGED_FILE,\n \tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n \n \tNB_UNPACK_TREES_WARNING_TYPES,\n-- \ngitgitgadget\n\n"},{"id":"393646","messageId":"3d2531ca8187ac6929f1c3b293682706e6da668d.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 07/18] unpack-trees: do not mark a dirty path with SKIP_WORKTREE","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:58Z","receivedAt":"2020-03-21T18:00:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf a path is dirty, removing from the working tree risks losing data.\nAs such, we want to make sure any such path is not marked with\nSKIP_WORKTREE. While the current callers of this code detect this case\nand re-populate with a previous set of sparsity patterns, we want to\nallow some paths to be marked with SKIP_WORKTREE while others are left\nunmarked without it being considered an error. The reason this\nshouldn't be considered an error is that SKIP_WORKTREE has always been\nan advisory-only setting; merge and rebase for example were free to\nmaterialize paths and clear the SKIP_WORKTREE bit in order to accomplish\ntheir work even though they kept the SKIP_WORKTREE bit set for other\npaths. Leaving dirty working files in the working tree is thus a\nnatural extension of what we have already been doing.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dde50047a82..e8e794880ab 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -500,8 +500,11 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\t * also stat info may have lost after merged_entry() so calling\n \t\t * verify_uptodate() again may fail\n \t\t */\n-\t\tif (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))\n+\t\tif (!(ce->ce_flags & CE_UPDATE) &&\n+\t\t verify_uptodate_sparse(ce, o)) {\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \t\t\treturn -1;\n+\t\t}\n \t\tce->ce_flags |= CE_WT_REMOVE;\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"393648","messageId":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.git.git.1584169893.gitgitgadget@gmail.com","subject":"[PATCH v2 00/18] Sparse checkout improvements -- improved sparsity updating","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T17:59:51Z","receivedAt":"2020-03-21T18:00:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series provides a replacement for the sparsity updating in\nsparse-checkout that is based on the logic from git read-tree -mu HEAD. The\nmost important bit is patch 9 and its lengthy commit message explaining the\ncurrent state and rationale for most the series, though patches 16 and 17\nhave additional related directions and rationale for the series. Those three\npatches are the most important to review.\n\nChanges since v1:\n\n * addressed several cleanups highlighted by Stolee (and I picked 'reapply'\n for the new subcommand name)\n * added three new minor cleanup commits (patches 2-4)\n * several new patches to adjust the warning messages to look like warnings\n (patches 11-15)\n * display warning messages when paths are unmerged (patch 16)\n * also make checkout and other unpack_trees()-using commands not error out\n when unable to set the SKIP_WORKTREE bit (patch 17)\n\nAlso, the loop range in display_warning_msgs() in patch 14 might seem\nslightly weird, but I felt adding another unpack_rejects array of some sort\nand plumbing through an extra bit everywhere to notify the system which\nunpack_rejects array to use would have been more invasive. Opinions either\nway (or alternative approaches) welcome.\n\nI was also curious if I should have squashed all the patches to adjust the\nwarning messages (patches 11-15, they adjust the messages to actually look\nlike warnings instead of errors), since they are individually kind of small,\nbut it looked like it'd be much easier to review if I were to split them\napart, so I did so.\n\nElijah Newren (18):\n unpack-trees: fix minor typo in comment\n unpack-trees: remove unused error type\n unpack-trees: simplify verify_absent_sparse()\n unpack-trees: simplify pattern_list freeing\n t1091: make some tests a little more defensive against failures\n unpack-trees: allow check_updates() to work on a different index\n unpack-trees: do not mark a dirty path with SKIP_WORKTREE\n unpack-trees: pull sparse-checkout pattern reading into a new function\n unpack-trees: add a new update_sparsity() function\n sparse-checkout: use new update_sparsity() function\n sparse-checkout: use improved unpack_trees porcelain messages\n unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier\n unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*\n unpack-trees: split display_error_msgs() into two\n unpack-trees: make sparse path messages sound like warnings\n unpack-trees: provide warnings on sparse updates for unmerged paths\n too\n unpack-trees: failure to set SKIP_WORKTREE bits always just a warning\n sparse-checkout: provide a new reapply subcommand\n\n Documentation/git-sparse-checkout.txt | 10 +\n builtin/sparse-checkout.c | 55 +++---\n t/t1011-read-tree-sparse-checkout.sh | 11 +-\n t/t1091-sparse-checkout-builtin.sh | 58 +++++-\n t/t2018-checkout-branch.sh | 22 +++\n unpack-trees.c | 256 ++++++++++++++++++++------\n unpack-trees.h | 28 ++-\n 7 files changed, 330 insertions(+), 110 deletions(-)\n\n\nbase-commit: be8661a3286c67a5d4088f4226cbd7f8b76544b0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-726%2Fnewren%2Fsparse-checkout-improvements-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-726/newren/sparse-checkout-improvements-v2\nPull-Request: https://github.com/git/git/pull/726\n\nRange-diff vs v1:\n\n 1: 9ab9a9d6037 = 1: 3c183727c98 unpack-trees: fix minor typo in comment\n -: ----------- > 2: e6beb1c5810 unpack-trees: remove unused error type\n -: ----------- > 3: 739b52393fe unpack-trees: simplify verify_absent_sparse()\n -: ----------- > 4: 65772dd46df unpack-trees: simplify pattern_list freeing\n 2: b07fdab55aa = 5: 5cbab2a7a56 t1091: make some tests a little more defensive against failures\n 3: 02afa9c13c1 = 6: 5fea19f0136 unpack-trees: allow check_updates() to work on a different index\n 4: baa4f23421b = 7: 3d2531ca818 unpack-trees: do not mark a dirty path with SKIP_WORKTREE\n -: ----------- > 8: 27d6304d33f unpack-trees: pull sparse-checkout pattern reading into a new function\n 5: 9f13a2e8c64 ! 9: a46439c8536 unpack-trees: add a new update_sparsity() function\n @@ -4,93 +4,78 @@\n \n Previously, the only way to update the SKIP_WORKTREE bits for various\n paths was invoking `git read-tree -mu HEAD` or calling the same code\n - that this codepath invoked. This however had a number of problems.\n - Let's look at all the various cases:\n -\n - Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files):\n - A) There is no file in the way\n - B) There is an untracked file in the way\n - Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files):\n - C) The path is clean\n - D) The path is unmerged\n - E) The path has unstaged changes\n - F) The path has staged changes (differs from HEAD)\n -\n - For `git read-tree -mu HEAD`, the behavior is as follows:\n - A) Materialize the file, as expected\n - B) Refuse to make *any* changes (to this path or any others)\n - C) Remove the file, as expected\n - D) Refuse to make *any* changes (to this path or any others)\n - E) Refuse to make *any* changes (to this path or any others)\n - F) Removes the file from the working copy AND resets the index\n - to match HEAD\n -\n - Refusing to modify files that could result in data loss as in cases B,\n - D, and E is good, but refusing to work on any other paths is very\n - problematic for users. If they are in the middle of a rebase or have\n - made modifications to files that bring in more dependencies, then for\n - their build to work they need to update the sparse paths -- which they\n - are prevented from doing. Sometimes they stage the files and re-try,\n - at which point they run into case F and believe that their changes are\n - lost. (Even if they come ask an expert, they have to trawl through\n - their loose objects looking for which ones match which files.)\n -\n - Add a new update_sparsity() function which behaves as follows in these\n - cases:\n - A) Materialize the file, as expected\n - B) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n - bit, and print a warning (thus leaving them in a state where\n + that this codepath invoked. This however had a number of problems if\n + the index or working directory were not clean. First, let's consider\n + the case:\n +\n + Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files)\n +\n + If the working tree was clean this was fine, but if there were files or\n + directories or symlinks or whatever already present at the given path\n + then the operation would abort with an error. Let's label this case\n + for later discussion:\n +\n + A) There is an untracked path in the way\n +\n + Now let's consider the opposite case:\n +\n + Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files)\n +\n + If the index and working tree was clean this was fine, but if there were\n + any unclean paths we would run into problems. There are three different\n + cases to consider:\n +\n + B) The path is unmerged\n + C) The path has unstaged changes\n + D) The path has staged changes (differs from HEAD)\n +\n + If any path fell into case B or C, then the whole operation would be\n + aborted with an error. With sparse-checkout, the whole operation would\n + be aborted for case D as well, but for its predecessor of using `git\n + read-tree -mu HEAD` directly, any paths that fell into case D would be\n + removed from the working copy and the index entry for that path would be\n + reset to match HEAD -- which looks and feels like data loss to users\n + (only a few are even aware to ask whether it can be recovered, and even\n + then it requires walking through loose objects trying to match up the\n + right ones).\n +\n + Refusing to remove files that have unsaved user changes is good, but\n + refusing to work on any other paths is very problematic for users. If\n + the user is in the middle of a rebase or has made modifications to files\n + that bring in more dependencies, then for their build to work they need\n + to update the sparse paths. This logic has been preventing them from\n + doing so. Sometimes in response, the user will stage the files and\n + re-try, to no avail with sparse-checkout or to the horror of losing\n + their changes if they are using its predecessor of `git read-tree -mu\n + HEAD`.\n +\n + Add a new update_sparsity() function which will not error out in any of\n + these cases but behaves as follows for the special cases:\n + A) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n + bit, and print a warning (thus leaving the path in a state where\n status will report the file as modified, which seems logical).\n - C) Remove the file, as expected\n - D) Do NOT mark this path as SKIP_WORKTREE, but allow others paths\n - to be updated.\n - E) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n - the dirty path, but allow other paths to be updated.\n - F) Mark the path as SKIP_WORKTREE, but do not revert the version\n + B) Do NOT mark this path as SKIP_WORKTREE, and leave it as unmerged.\n + C) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n + the dirty path.\n + D) Mark the path as SKIP_WORKTREE, but do not revert the version\n stored in the index to match HEAD; leave the contents alone.\n \n - I tried a different behavior for B (leave the SKIP_WORKTREE bit set),\n + I tried a different behavior for A (leave the SKIP_WORKTREE bit set),\n but found it very surprising and counter-intuitive (e.g. the user sees\n it is present along with all the other files in that directory, tries to\n - stage it, but git add ignores it since the SKIP_WORKTREE bit is set).\n - A, B, C, and E all seem like optimal behavior to me. D may be as well,\n - though I wonder if printing a warning would be an improvement. Some\n - might be slightly surprised by F at first, but given that it does the\n - right thing with `git commit` and even `git commit -a` (add ignores\n - entries that are marked SKIP_WORKTREE and thus doesn't delete them, and\n - commit -a is similar), it seems logical to me.\n + stage it, but git add ignores it since the SKIP_WORKTREE bit is set). A\n + & C seem like optimal behavior to me. B may be as well, though I wonder\n + if printing a warning would be an improvement. Some might be slightly\n + surprised by D at first, but given that it does the right thing with\n + `git commit` and even `git commit -a` (`git add` ignores entries that\n + are marked SKIP_WORKTREE and thus doesn't delete them, and `commit -a`\n + is similar), it seems logical to me.\n \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n --- a/unpack-trees.c\n +++ b/unpack-trees.c\n -@@\n - \t\tfprintf(stderr, _(\"Aborting\\n\"));\n - }\n - \n -+/*\n -+ * display all the \"error\" messages as warnings\n -+ */\n -+static void display_warning_msgs(struct unpack_trees_options *o)\n -+{\n -+\tint e, i;\n -+\tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n -+\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n -+\t\tif (rejects->nr > 0) {\n -+\t\t\tstruct strbuf path = STRBUF_INIT;\n -+\t\t\tfor (i = 0; i < rejects->nr; i++)\n -+\t\t\t\tstrbuf_addstr(&path, rejects->items[i].string);\n -+\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n -+\t\t\tstrbuf_release(&path);\n -+\t\t}\n -+\t\tstring_list_clear(rejects, 0);\n -+\t}\n -+}\n -+\n - static int check_submodule_move_head(const struct cache_entry *ce,\n - \t\t\t\t const char *old_id,\n - \t\t\t\t const char *new_id,\n @@\n \tgoto done;\n }\n @@ -99,23 +84,15 @@\n + * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n + * working directory to match.\n + *\n -+ * Returns\n -+ * 0: success with no warnings\n -+ * 1: success with warnings (warnings come either from (a) dirty changes\n -+ * present in which case we avoid marking those paths as\n -+ * SKIP_WORKTREE, or (b) from untracked files being in the way us\n -+ * checking a file out of the index, in which case we leave the file\n -+ * in the working tree alone while clearing SKIP_WORKTREE)\n -+ * -1: failure to manipulate the resulting index\n -+ * -2: failure to reflect the changes to the work tree.\n -+ *\n + * CE_NEW_SKIP_WORKTREE is used internally.\n + */\n -+int update_sparsity(struct unpack_trees_options *o)\n ++enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n +{\n ++\tenum update_sparsity_result ret = UPDATE_SPARSITY_SUCCESS;\n +\tstruct pattern_list pl;\n -+\tint i, empty_worktree, ret = 0;\n ++\tint i, empty_worktree;\n +\tunsigned old_show_all_errors;\n ++\tint free_pattern_list = 0;\n +\n +\told_show_all_errors = o->show_all_errors;\n +\to->show_all_errors = 1;\n @@ -128,17 +105,13 @@\n +\n +\ttrace_performance_enter();\n +\n ++\t/* If we weren't given patterns, use the recorded ones */\n +\tif (!o->pl) {\n -+\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n +\t\tmemset(&pl, 0, sizeof(pl));\n -+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n -+\t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0) {\n -+\t\t\t/* FIXME: Skip to check_updates()?? */\n -+\t\t\to->skip_sparse_checkout = 1;\n ++\t\tfree_pattern_list = 1;\n ++\t\tpopulate_from_existing_patterns(o, &pl);\n ++\t\tif (o->skip_sparse_checkout)\n +\t\t\tgoto skip_sparse_checkout;\n -+\t\t} else\n -+\t\t\to->pl = &pl;\n -+\t\tfree(sparse);\n +\t}\n +\n +\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n @@ -147,13 +120,13 @@\n +\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n +\n +\t/* Then loop over entries and update/remove as needed */\n -+\tret = 0;\n ++\tret = UPDATE_SPARSITY_SUCCESS;\n +\tempty_worktree = 1;\n +\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n +\t\tstruct cache_entry *ce = o->src_index->cache[i];\n +\n +\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n -+\t\t\tret = 1;\n ++\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n +\n +\t\tif (!ce_skip_worktree(ce))\n +\t\t\tempty_worktree = 0;\n @@ -167,17 +140,20 @@\n +\t * Do not allow users to do that.\n +\t */\n +\tif (o->src_index->cache_nr && empty_worktree) {\n -+\t\tret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n ++\t\tunpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n ++\t\tret = UPDATE_SPARSITY_INDEX_UPDATE_FAILURES;\n +\t\tgoto done;\n +\t}\n +\n +skip_sparse_checkout:\n +\tif (check_updates(o, o->src_index))\n -+\t\tret = -2;\n ++\t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n +\n +done:\n -+\tdisplay_warning_msgs(o);\n ++\tdisplay_error_msgs(o);\n +\to->show_all_errors = old_show_all_errors;\n ++\tif (free_pattern_list)\n ++\t\tclear_pattern_list(&pl);\n +\ttrace_performance_leave(\"update_sparsity\");\n +\treturn ret;\n +}\n @@ -189,6 +165,20 @@\n diff --git a/unpack-trees.h b/unpack-trees.h\n --- a/unpack-trees.h\n +++ b/unpack-trees.h\n +@@\n + \tNB_UNPACK_TREES_ERROR_TYPES\n + };\n + \n ++enum update_sparsity_result {\n ++\tUPDATE_SPARSITY_SUCCESS = 0,\n ++\tUPDATE_SPARSITY_WARNINGS = 1,\n ++\tUPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n ++\tUPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n ++};\n ++\n + /*\n + * Sets the list of user-friendly error messages to be used by the\n + * command \"cmd\" (either merge or checkout), and show_all_errors to 1.\n @@\n int unpack_trees(unsigned n, struct tree_desc *t,\n \t\t struct unpack_trees_options *options);\n 6: 892933752c6 ! 10: 27ed4a3f7a2 sparse-checkout: use new update_sparsity() function\n @@ -12,7 +12,11 @@\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n @@\n - \tint result = 0;\n + \n + static int update_working_directory(struct pattern_list *pl)\n + {\n +-\tint result = 0;\n ++\tenum update_sparsity_result result;\n \tstruct unpack_trees_options o;\n \tstruct lock_file lock_file = LOCK_INIT;\n -\tstruct object_id oid;\n @@ -40,7 +44,6 @@\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n --\to.keep_pattern_list = !!pl;\n \n -\tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\n @@ -55,13 +58,13 @@\n \n -\tif (!result) {\n -\t\tprime_cache_tree(r, r->index, tree);\n -+\tif (result == 1)\n ++\tif (result == UPDATE_SPARSITY_WARNINGS)\n +\t\t/*\n +\t\t * We don't do any special handling of warnings from untracked\n +\t\t * files in the way or dirty entries that can't be removed.\n +\t\t */\n -+\t\tresult = 0;\n -+\tif (!result)\n ++\t\tresult = UPDATE_SPARSITY_SUCCESS;\n ++\tif (result == UPDATE_SPARSITY_SUCCESS)\n \t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n -\t} else\n +\telse\n @@ -141,13 +144,16 @@\n -\ttest_must_fail git -C dirty sparse-checkout disable &&\n +\n +\tgit -C dirty sparse-checkout init 2>err &&\n -+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n ++\ttest_i18ngrep \"error\" err &&\n ++\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\n +\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n -+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n ++\ttest_i18ngrep \"error\" err &&\n ++\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\n +\tgit -C dirty sparse-checkout disable &&\n -+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n ++\ttest_i18ngrep \"error\" err &&\n ++\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n -: ----------- > 11: 79b9cc1df55 sparse-checkout: use improved unpack_trees porcelain messages\n -: ----------- > 12: 2ba7bbdaee3 unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier\n -: ----------- > 13: 203ece0ec2f unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*\n -: ----------- > 14: 57679c8e292 unpack-trees: split display_error_msgs() into two\n -: ----------- > 15: 02a8e01f35b unpack-trees: make sparse path messages sound like warnings\n -: ----------- > 16: f664a6dcbb3 unpack-trees: provide warnings on sparse updates for unmerged paths too\n -: ----------- > 17: fa997a1c72f unpack-trees: failure to set SKIP_WORKTREE bits always just a warning\n 7: 650db686342 ! 18: 5730f7d250a sparse-checkout: provide a new update subcommand\n @@ -1,6 +1,6 @@\n Author: Elijah Newren \n \n - sparse-checkout: provide a new update subcommand\n + sparse-checkout: provide a new reapply subcommand\n \n If commands like merge or rebase materialize files as part of their work,\n or a previous sparse-checkout command failed to update individual files\n @@ -16,13 +16,13 @@\n \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n \tas directory names as in the 'set' subcommand.\n \n -+'update'::\n -+\tUpdate the sparseness of paths in the working tree based on the\n -+\texisting patterns. Commands like merge or rebase can materialize\n -+\tpaths to do their work (e.g. in order to show you a conflict), and\n -+\tother sparse-checkout commands might fail to sparsify an individual\n -+\tfile (e.g. because it has unstaged changes or conflicts). In such\n -+\tcases, it can make sense to run `git sparse-checkout update` later\n ++'reapply::\n ++\tReapply the sparsity pattern rules to paths in the working tree.\n ++\tCommands like merge or rebase can materialize paths to do their\n ++\twork (e.g. in order to show you a conflict), and other\n ++\tsparse-checkout commands might fail to sparsify an individual file\n ++\t(e.g. because it has unstaged changes or conflicts). In such\n ++\tcases, it can make sense to run `git sparse-checkout reapply` later\n +\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n +\tor committing changes, etc.).\n +\n @@ -38,7 +38,7 @@\n \n static char const * const builtin_sparse_checkout_usage[] = {\n -\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n -+\tN_(\"git sparse-checkout (init|list|set|add|update|disable) \"),\n ++\tN_(\"git sparse-checkout (init|list|set|add|reapply|disable) \"),\n \tNULL\n };\n \n @@ -46,7 +46,7 @@\n \treturn modify_pattern_list(argc, argv, m);\n }\n \n -+static int sparse_checkout_update(int argc, const char **argv)\n ++static int sparse_checkout_reapply(int argc, const char **argv)\n +{\n +\trepo_read_index(the_repository);\n +\treturn update_working_directory(NULL);\n @@ -59,8 +59,8 @@\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n \t\tif (!strcmp(argv[0], \"add\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n -+\t\tif (!strcmp(argv[0], \"update\"))\n -+\t\t\treturn sparse_checkout_update(argc, argv);\n ++\t\tif (!strcmp(argv[0], \"reapply\"))\n ++\t\t\treturn sparse_checkout_reapply(argc, argv);\n \t\tif (!strcmp(argv[0], \"disable\"))\n \t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n\n-- \ngitgitgadget\n"},{"id":"393649","messageId":"2ba7bbdaee39fb9b8714fdc1918fa69a20d5d1e0.1584813609.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v2 12/18] unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-21T18:00:03Z","receivedAt":"2020-03-21T18:00:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA minor change, but we want to convert the sparse messages to warnings\nand this allows us to group warnings and errors.\n\nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 12 ++++++------\n unpack-trees.h | 2 +-\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 6abea555929..5c99d588dc3 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -43,15 +43,14 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_BIND_OVERLAP */\n \t\"Entry '%s' overlaps with '%s'. Cannot bind.\",\n \n+\t/* ERROR_WOULD_LOSE_SUBMODULE */\n+\t\"Submodule '%s' cannot checkout new HEAD.\",\n+\n \t/* ERROR_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n \t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n-\n-\n-\t/* ERROR_WOULD_LOSE_SUBMODULE */\n-\t\"Submodule '%s' cannot checkout new HEAD.\",\n };\n \n #define ERRORMSG(o,type) \\\n@@ -166,12 +165,13 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t */\n \tmsgs[ERROR_BIND_OVERLAP] = _(\"Entry '%s' overlaps with '%s'. Cannot bind.\");\n \n+\tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n+\t\t_(\"Cannot update submodule:\\n%s\");\n+\n \tmsgs[ERROR_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n-\t\t_(\"Cannot update submodule:\\n%s\");\n \n \topts->show_all_errors = 1;\n \t/* rejected paths may not have a static buffer */\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 2c5d54cae9f..a656bbf810b 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -22,9 +22,9 @@ enum unpack_trees_error_types {\n \tERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN,\n \tERROR_WOULD_LOSE_UNTRACKED_REMOVED,\n \tERROR_BIND_OVERLAP,\n+\tERROR_WOULD_LOSE_SUBMODULE,\n \tERROR_SPARSE_NOT_UPTODATE_FILE,\n \tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tERROR_WOULD_LOSE_SUBMODULE,\n \tNB_UNPACK_TREES_ERROR_TYPES\n };\n \n-- \ngitgitgadget\n\n"},{"id":"393778","messageId":"d99a17b6-49ab-51d9-621d-adf6eb0b7ffa@gmail.com","threadId":"52995","inReplyTo":"65772dd46df12d79a41e7ed2d6e1fc197b80d379.1584813609.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/18] unpack-trees: simplify pattern_list freeing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T15:57:21Z","receivedAt":"2020-03-23T15:57:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 1:59 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> commit e091228e17 (\"sparse-checkout: update working directory\n> in-process\", 2019-11-21) allowed passing a pre-defined set of patterns\n> to unpack_trees(). However, if o->pl was NULL, it would still read the\n> existing patterns and use those. If those patterns were read into a\n> data structure that was allocated, naturally they needed to be free'd.\n> However, despite the same function being responsible for knowing about\n> both the allocation and the free'ing, the logic for tracking whether to\n> free the pattern_list was hoisted to an outer function with an\n> additional flag in unpack_trees_options. Put the logic back in the\n> relevant function and discard the now unnecessary flag.\n\nYour approach here is much cleaner. Thanks!\n\n> Signed-off-by: Elijah Newren \n> ---\n> builtin/sparse-checkout.c | 1 -\n> unpack-trees.c | 6 ++++--\n> unpack-trees.h | 3 +--\n> 3 files changed, 5 insertions(+), 5 deletions(-)\n> \n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 740da4b6d54..d102a9697fd 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -122,7 +122,6 @@ static int update_working_directory(struct pattern_list *pl)\n> \to.dst_index = r->index;\n> \to.skip_sparse_checkout = 0;\n> \to.pl = pl;\n> -\to.keep_pattern_list = !!pl;\n> \n> \tresolve_undo_clear_index(r->index);\n> \tsetup_work_tree();\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 3af2e126abf..d2863fa0310 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1503,6 +1503,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \tint i, ret;\n> \tstatic struct cache_entry *dfc;\n> \tstruct pattern_list pl;\n> +\tint free_pattern_list = 0;\n> \n> \tif (len > MAX_UNPACK_TREES)\n> \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n> @@ -1519,6 +1520,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \t\telse\n> \t\t\to->pl = &pl;\n> \t\tfree(sparse);\n> +\t\tfree_pattern_list = 1;\n> \t}\n> \n> \tmemset(&o->result, 0, sizeof(o->result));\n> @@ -1686,9 +1688,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \to->src_index = NULL;\n> \n> done:\n> -\ttrace_performance_leave(\"unpack_trees\");\n> -\tif (!o->keep_pattern_list)\n> +\tif (free_pattern_list)\n> \t\tclear_pattern_list(&pl);\n> +\ttrace_performance_leave(\"unpack_trees\");\n> \treturn ret;\n> \n> return_failed:\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 6d7c7b6c2e0..d3516267f36 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -58,8 +58,7 @@ struct unpack_trees_options {\n> \t\t quiet,\n> \t\t exiting_early,\n> \t\t show_all_errors,\n> -\t\t dry_run,\n> -\t\t keep_pattern_list;\n> +\t\t dry_run;\n> \tconst char *prefix;\n> \tint cache_bottom;\n> \tstruct dir_struct *dir;\n> \n\n"},{"id":"393789","messageId":"9501e018-fe87-16b8-b0d8-913e4331778d@gmail.com","threadId":"52995","inReplyTo":"a46439c8536f912ad4a1e1751852cf477d3d7dc7.1584813609.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:02:22Z","receivedAt":"2020-03-23T18:02:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Previously, the only way to update the SKIP_WORKTREE bits for various\n> paths was invoking `git read-tree -mu HEAD` or calling the same code\n> that this codepath invoked. This however had a number of problems if\n> the index or working directory were not clean. First, let's consider\n> the case:\n> \n> Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files)\n> \n> If the working tree was clean this was fine, but if there were files or\n> directories or symlinks or whatever already present at the given path\n> then the operation would abort with an error. Let's label this case\n> for later discussion:\n> \n> A) There is an untracked path in the way\n> \n> Now let's consider the opposite case:\n> \n> Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files)\n> \n> If the index and working tree was clean this was fine, but if there were\n> any unclean paths we would run into problems. There are three different\n> cases to consider:\n> \n> B) The path is unmerged\n> C) The path has unstaged changes\n> D) The path has staged changes (differs from HEAD)\n> \n> If any path fell into case B or C, then the whole operation would be\n> aborted with an error. With sparse-checkout, the whole operation would\n> be aborted for case D as well, but for its predecessor of using `git\n> read-tree -mu HEAD` directly, any paths that fell into case D would be\n> removed from the working copy and the index entry for that path would be\n> reset to match HEAD -- which looks and feels like data loss to users\n> (only a few are even aware to ask whether it can be recovered, and even\n> then it requires walking through loose objects trying to match up the\n> right ones).\n> \n> Refusing to remove files that have unsaved user changes is good, but\n> refusing to work on any other paths is very problematic for users. If\n> the user is in the middle of a rebase or has made modifications to files\n> that bring in more dependencies, then for their build to work they need\n> to update the sparse paths. This logic has been preventing them from\n> doing so. Sometimes in response, the user will stage the files and\n> re-try, to no avail with sparse-checkout or to the horror of losing\n> their changes if they are using its predecessor of `git read-tree -mu\n> HEAD`.\n> \n> Add a new update_sparsity() function which will not error out in any of\n> these cases but behaves as follows for the special cases:\n> A) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n> bit, and print a warning (thus leaving the path in a state where\n> status will report the file as modified, which seems logical).\n> B) Do NOT mark this path as SKIP_WORKTREE, and leave it as unmerged.\n> C) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n> the dirty path.\n> D) Mark the path as SKIP_WORKTREE, but do not revert the version\n> stored in the index to match HEAD; leave the contents alone.\n> \n> I tried a different behavior for A (leave the SKIP_WORKTREE bit set),\n> but found it very surprising and counter-intuitive (e.g. the user sees\n> it is present along with all the other files in that directory, tries to\n> stage it, but git add ignores it since the SKIP_WORKTREE bit is set). A\n> & C seem like optimal behavior to me. B may be as well, though I wonder\n> if printing a warning would be an improvement. Some might be slightly\n> surprised by D at first, but given that it does the right thing with\n> `git commit` and even `git commit -a` (`git add` ignores entries that\n> are marked SKIP_WORKTREE and thus doesn't delete them, and `commit -a`\n> is similar), it seems logical to me.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> unpack-trees.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++\n> unpack-trees.h | 9 ++++++\n> 2 files changed, 87 insertions(+)\n> \n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 4733e7eaf89..6abea555929 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1714,6 +1714,84 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \tgoto done;\n> }\n> \n> +/*\n> + * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n> + * working directory to match.\n> + *\n> + * CE_NEW_SKIP_WORKTREE is used internally.\n> + */\n> +enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n> +{\n> +\tenum update_sparsity_result ret = UPDATE_SPARSITY_SUCCESS;\n> +\tstruct pattern_list pl;\n> +\tint i, empty_worktree;\n> +\tunsigned old_show_all_errors;\n> +\tint free_pattern_list = 0;\n> +\n> +\told_show_all_errors = o->show_all_errors;\n> +\to->show_all_errors = 1;\n> +\n> +\t/* Sanity checks */\n> +\tif (!o->update || o->index_only || o->skip_sparse_checkout)\n> +\t\tBUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n> +\tif (o->src_index != o->dst_index || o->fn)\n> +\t\tBUG(\"update_sparsity() called wrong\");\n> +\n> +\ttrace_performance_enter();\n\nI was about to say \"why didn't you use the trace2 regions like in\nunpack_trees()?\" when I discovered that we haven't sent them [1]\nupstream yet. I'll put that on my TODO list.\n\n[1] https://github.com/microsoft/git/commit/9a04644e14fe4aeb556dfc30cb2220b799f53448\n\n> +\t/* If we weren't given patterns, use the recorded ones */\n> +\tif (!o->pl) {\n> +\t\tmemset(&pl, 0, sizeof(pl));\n> +\t\tfree_pattern_list = 1;\n\nI notice you are using the same free_pattern_list pattern as your\nearlier commit. Good.\n\n> +\t\tpopulate_from_existing_patterns(o, &pl);\n> +\t\tif (o->skip_sparse_checkout)\n> +\t\t\tgoto skip_sparse_checkout;\n> +\t}\n> +\n> +\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n> +\tmark_all_ce_unused(o->src_index);\n> +\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n> +\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n> +\n> +\t/* Then loop over entries and update/remove as needed */\n> +\tret = UPDATE_SPARSITY_SUCCESS;\n> +\tempty_worktree = 1;\n> +\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n> +\t\tstruct cache_entry *ce = o->src_index->cache[i];\n> +\n> +\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n> +\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n> +\n> +\t\tif (!ce_skip_worktree(ce))\n> +\t\t\tempty_worktree = 0;\n> +\n\nnit: extra whitespace-only line\n\n> +\t}\n> +\n> +\t/*\n> +\t * Sparse checkout is meant to narrow down checkout area\n> +\t * but it does not make sense to narrow down to empty working\n> +\t * tree. This is usually a mistake in sparse checkout rules.\n> +\t * Do not allow users to do that.\n> +\t */\n> +\tif (o->src_index->cache_nr && empty_worktree) {\n> +\t\tunpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n> +\t\tret = UPDATE_SPARSITY_INDEX_UPDATE_FAILURES;\n> +\t\tgoto done;\n> +\t}\n> +\n> +skip_sparse_checkout:\n> +\tif (check_updates(o, o->src_index))\n> +\t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n> +\n> +done:\n> +\tdisplay_error_msgs(o);\n> +\to->show_all_errors = old_show_all_errors;\n> +\tif (free_pattern_list)\n> +\t\tclear_pattern_list(&pl);\n> +\ttrace_performance_leave(\"update_sparsity\");\n> +\treturn ret;\n> +}\n> +\n> /* Here come the merge functions */\n> \n> static int reject_merge(const struct cache_entry *ce,\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index d3516267f36..2c5d54cae9f 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -28,6 +28,13 @@ enum unpack_trees_error_types {\n> \tNB_UNPACK_TREES_ERROR_TYPES\n> };\n> \n> +enum update_sparsity_result {\n> +\tUPDATE_SPARSITY_SUCCESS = 0,\n> +\tUPDATE_SPARSITY_WARNINGS = 1,\n> +\tUPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n> +\tUPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n> +};\n> +\n\nIs there a reason this isn't located just before\nupdate_sparsity()?\n\n> /*\n> * Sets the list of user-friendly error messages to be used by the\n> * command \"cmd\" (either merge or checkout), and show_all_errors to 1.\n> @@ -88,6 +95,8 @@ struct unpack_trees_options {\n> int unpack_trees(unsigned n, struct tree_desc *t,\n> \t\t struct unpack_trees_options *options);\n> \n> +int update_sparsity(struct unpack_trees_options *options);\n> +\n\nThis appears to not use the enum as it should.\n\nThanks,\n-Stolee\n\n"},{"id":"393791","messageId":"45021041-eab4-6e0e-5966-11db06333ab2@gmail.com","threadId":"52995","inReplyTo":"27ed4a3f7a28c66d8185cb4c484e8b0af81b15c6.1584813609.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/18] sparse-checkout: use new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:07:55Z","receivedAt":"2020-03-23T18:08:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Remove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\n> codepaths for setting the SKIP_WORKTREE bits and instead use the new\n> update_sparsity() function.\n> \n[snip]\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 8607a8e6d1a..86ae422ff5c 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n> \tcheck_files repo a deep folder1\n> '\n> \n> -test_expect_success 'revert to old sparse-checkout on bad update' '\n> +test_expect_success 'not-up-to-date does not block rest of sparsification' '\n> \ttest_when_finished git -C repo sparse-checkout disable &&\n> \ttest_when_finished git -C repo reset --hard &&\n> \tgit -C repo sparse-checkout set deep &&\n> +\n> \techo update >repo/deep/deeper2/a &&\n> \tcp repo/.git/info/sparse-checkout expect &&\n> -\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n> -\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n> -\ttest_cmp repo/.git/info/sparse-checkout expect &&\n> -\tcheck_files repo/deep a deeper1 deeper2\n> +\ttest_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n> +\n> +\tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n> +\n> +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tcheck_files repo/deep a deeper1 deeper2 &&\n> +\tcheck_files repo/deep/deeper1 a deepest &&\n> +\tcheck_files repo/deep/deeper1/deepest a &&\n> +\tcheck_files repo/deep/deeper2 a\n> '\n\nThis demonstrates some of the value of using update_sparsity.\nI expect some more specific tests are coming later in the\nseries.\n\n> test_expect_success 'revert to old sparse-checkout on empty update' '\n> @@ -316,12 +323,22 @@ test_expect_success '.gitignore should not warn about cone mode' '\n> \ttest_i18ngrep ! \"disabling cone patterns\" err\n> '\n> \n> -test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n> +test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n> \tgit clone repo dirty &&\n> \techo dirty >dirty/folder1/a &&\n> -\ttest_must_fail git -C dirty sparse-checkout init &&\n> -\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> -\ttest_must_fail git -C dirty sparse-checkout disable &&\n> +\n> +\tgit -C dirty sparse-checkout init 2>err &&\n> +\ttest_i18ngrep \"error\" err &&\n> +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n> +\n> +\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> +\ttest_i18ngrep \"error\" err &&\n> +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n> +\n> +\tgit -C dirty sparse-checkout disable &&\n> +\ttest_i18ngrep \"error\" err &&\n> +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n> +\n\nShould the \"error\" messages we grep for be more specific? Are they\nthe same as the \"Cannot update sparse checkout\"?\n\n\n"},{"id":"393792","messageId":"CABPp-BF7=jJ2fWFcriC3E0=ZrUWawXoLbz659jtYdtkTJ4we_w@mail.gmail.com","threadId":"52995","inReplyTo":"9501e018-fe87-16b8-b0d8-913e4331778d@gmail.com","subject":"Re: [PATCH v2 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-23T18:10:55Z","receivedAt":"2020-03-23T18:11:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 23, 2020 at 11:02 AM Derrick Stolee wrote:\n>\n> On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n\n> > + for (i = 0; i < o->src_index->cache_nr; i++) {\n> > + struct cache_entry *ce = o->src_index->cache[i];\n> > +\n> > + if (apply_sparse_checkout(o->src_index, ce, o))\n> > + ret = UPDATE_SPARSITY_WARNINGS;\n> > +\n> > + if (!ce_skip_worktree(ce))\n> > + empty_worktree = 0;\n> > +\n>\n> nit: extra whitespace-only line\n\nWill clean it up.\n\n> > + }\n> > +\n> > + /*\n> > + * Sparse checkout is meant to narrow down checkout area\n> > + * but it does not make sense to narrow down to empty working\n> > + * tree. This is usually a mistake in sparse checkout rules.\n> > + * Do not allow users to do that.\n> > + */\n> > + if (o->src_index->cache_nr && empty_worktree) {\n> > + unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n> > + ret = UPDATE_SPARSITY_INDEX_UPDATE_FAILURES;\n> > + goto done;\n> > + }\n> > +\n> > +skip_sparse_checkout:\n> > + if (check_updates(o, o->src_index))\n> > + ret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n> > +\n> > +done:\n> > + display_error_msgs(o);\n> > + o->show_all_errors = old_show_all_errors;\n> > + if (free_pattern_list)\n> > + clear_pattern_list(&pl);\n> > + trace_performance_leave(\"update_sparsity\");\n> > + return ret;\n> > +}\n> > +\n> > /* Here come the merge functions */\n> >\n> > static int reject_merge(const struct cache_entry *ce,\n> > diff --git a/unpack-trees.h b/unpack-trees.h\n> > index d3516267f36..2c5d54cae9f 100644\n> > --- a/unpack-trees.h\n> > +++ b/unpack-trees.h\n> > @@ -28,6 +28,13 @@ enum unpack_trees_error_types {\n> > NB_UNPACK_TREES_ERROR_TYPES\n> > };\n> >\n> > +enum update_sparsity_result {\n> > + UPDATE_SPARSITY_SUCCESS = 0,\n> > + UPDATE_SPARSITY_WARNINGS = 1,\n> > + UPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n> > + UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n> > +};\n> > +\n>\n> Is there a reason this isn't located just before\n> update_sparsity()?\n\nYou mean move it to unpack-trees.c? If I did that, the code in\nsparse-checkout.c that uses two of these values would fail to compile.\n\n> > /*\n> > * Sets the list of user-friendly error messages to be used by the\n> > * command \"cmd\" (either merge or checkout), and show_all_errors to 1.\n> > @@ -88,6 +95,8 @@ struct unpack_trees_options {\n> > int unpack_trees(unsigned n, struct tree_desc *t,\n> > struct unpack_trees_options *options);\n> >\n> > +int update_sparsity(struct unpack_trees_options *options);\n> > +\n>\n> This appears to not use the enum as it should.\n\nWhoops! Will fix. (Interesting that the compiler didn't flag any\nkind of warning based on mismatch of declared function return types\nfor update_sparsity() in the .c and .h files...)\n\nThanks,\nElijah\n"},{"id":"393794","messageId":"CABPp-BH1BMZjLN91F5Ece6v30dw8RczrBY+9GnmPWCCdeH6Stg@mail.gmail.com","threadId":"52995","inReplyTo":"45021041-eab4-6e0e-5966-11db06333ab2@gmail.com","subject":"Re: [PATCH v2 10/18] sparse-checkout: use new update_sparsity() function","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-23T18:14:00Z","receivedAt":"2020-03-23T18:14:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Mar 23, 2020 at 11:07 AM Derrick Stolee wrote:\n>\n> On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Remove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\n> > codepaths for setting the SKIP_WORKTREE bits and instead use the new\n> > update_sparsity() function.\n> >\n> [snip]\n> > diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> > index 8607a8e6d1a..86ae422ff5c 100755\n> > --- a/t/t1091-sparse-checkout-builtin.sh\n> > +++ b/t/t1091-sparse-checkout-builtin.sh\n> > @@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n> > check_files repo a deep folder1\n> > '\n> >\n> > -test_expect_success 'revert to old sparse-checkout on bad update' '\n> > +test_expect_success 'not-up-to-date does not block rest of sparsification' '\n> > test_when_finished git -C repo sparse-checkout disable &&\n> > test_when_finished git -C repo reset --hard &&\n> > git -C repo sparse-checkout set deep &&\n> > +\n> > echo update >repo/deep/deeper2/a &&\n> > cp repo/.git/info/sparse-checkout expect &&\n> > - test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n> > - test_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n> > - test_cmp repo/.git/info/sparse-checkout expect &&\n> > - check_files repo/deep a deeper1 deeper2\n> > + test_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n> > +\n> > + git -C repo sparse-checkout set deep/deeper1 2>err &&\n> > +\n> > + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> > + test_cmp expect repo/.git/info/sparse-checkout &&\n> > + check_files repo/deep a deeper1 deeper2 &&\n> > + check_files repo/deep/deeper1 a deepest &&\n> > + check_files repo/deep/deeper1/deepest a &&\n> > + check_files repo/deep/deeper2 a\n> > '\n>\n> This demonstrates some of the value of using update_sparsity.\n> I expect some more specific tests are coming later in the\n> series.\n>\n> > test_expect_success 'revert to old sparse-checkout on empty update' '\n> > @@ -316,12 +323,22 @@ test_expect_success '.gitignore should not warn about cone mode' '\n> > test_i18ngrep ! \"disabling cone patterns\" err\n> > '\n> >\n> > -test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n> > +test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n> > git clone repo dirty &&\n> > echo dirty >dirty/folder1/a &&\n> > - test_must_fail git -C dirty sparse-checkout init &&\n> > - test_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> > - test_must_fail git -C dirty sparse-checkout disable &&\n> > +\n> > + git -C dirty sparse-checkout init 2>err &&\n> > + test_i18ngrep \"error\" err &&\n> > + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> > +\n> > + git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> > + test_i18ngrep \"error\" err &&\n> > + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> > +\n> > + git -C dirty sparse-checkout disable &&\n> > + test_i18ngrep \"error\" err &&\n> > + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> > +\n>\n> Should the \"error\" messages we grep for be more specific? Are they\n> the same as the \"Cannot update sparse checkout\"?\n\nThe two lines are checking for the same message, it's just that the\nerror message is split across multiple lines and thus requires two\ngreps. The error messages are actually a bit of a mess, particularly\nwhen they are triggered for multiple paths. That's something that\nsubsequent commits will clean up.\n\nI can add a note about this to the commit message.\n"},{"id":"393795","messageId":"5a83259e-af03-1930-a62b-75ad37b05f37@gmail.com","threadId":"52995","inReplyTo":"CABPp-BF7=jJ2fWFcriC3E0=ZrUWawXoLbz659jtYdtkTJ4we_w@mail.gmail.com","subject":"Re: [PATCH v2 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:21:27Z","receivedAt":"2020-03-23T18:21:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/23/2020 2:10 PM, Elijah Newren wrote:\n> On Mon, Mar 23, 2020 at 11:02 AM Derrick Stolee wrote:\n>>\n>> On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n>>> +enum update_sparsity_result {\n>>> + UPDATE_SPARSITY_SUCCESS = 0,\n>>> + UPDATE_SPARSITY_WARNINGS = 1,\n>>> + UPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n>>> + UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n>>> +};\n>>> +\n>>\n>> Is there a reason this isn't located just before\n>> update_sparsity()?\n> \n> You mean move it to unpack-trees.c? If I did that, the code in\n> sparse-checkout.c that uses two of these values would fail to compile.\n\nNo, I just meant the function declaration below in unpack-trees.h.\n\n>>> +int update_sparsity(struct unpack_trees_options *options);\n>>> +\n>>\n>> This appears to not use the enum as it should.\n> \n> Whoops! Will fix. (Interesting that the compiler didn't flag any\n> kind of warning based on mismatch of declared function return types\n> for update_sparsity() in the .c and .h files...)\n\n*shrug* enums are essentially decoration over an int, so I'm not\nsurprised it can happen.\n\n-Stolee\n"},{"id":"393796","messageId":"bae7c3ba-d835-864f-4f6f-51dbde0e338b@gmail.com","threadId":"52995","inReplyTo":"CABPp-BH1BMZjLN91F5Ece6v30dw8RczrBY+9GnmPWCCdeH6Stg@mail.gmail.com","subject":"Re: [PATCH v2 10/18] sparse-checkout: use new update_sparsity() function","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:22:52Z","receivedAt":"2020-03-23T18:22:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/23/2020 2:14 PM, Elijah Newren wrote:\n> On Mon, Mar 23, 2020 at 11:07 AM Derrick Stolee wrote:\n>>\n>> Should the \"error\" messages we grep for be more specific? Are they\n>> the same as the \"Cannot update sparse checkout\"?\n> \n> The two lines are checking for the same message, it's just that the\n> error message is split across multiple lines and thus requires two\n> greps. The error messages are actually a bit of a mess, particularly\n> when they are triggered for multiple paths. That's something that\n> subsequent commits will clean up.\n\nI see that now! The very next commit cleans this up nicely.\n\n> I can add a note about this to the commit message.\n\nThanks!\n\n"},{"id":"393797","messageId":"c384a788-9ce8-7a33-9d0d-e5dde746046e@gmail.com","threadId":"52995","inReplyTo":"57679c8e292ceb58cea2b5b7d893d5f47e1e2de0.1584813609.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 14/18] unpack-trees: split display_error_msgs() into two","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:32:47Z","receivedAt":"2020-03-23T18:32:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> display_error_msgs() is never called to show messages of both ERROR_*\n> and WARNING_* types at the same time; it is instead called multiple\n> times, separately for each type. Since we want to display these types\n> differently, make two slightly different versions of this function.\n> \n> A subsequent commit will further modify unpack_trees() and how it calls\n> the new display_warning_msgs().\n> \n> Signed-off-by: Elijah Newren \n> ---\n> t/t1091-sparse-checkout-builtin.sh | 6 ++--\n> unpack-trees.c | 50 +++++++++++++++++++++++++-----\n> unpack-trees.h | 8 +++--\n> 3 files changed, 50 insertions(+), 14 deletions(-)\n> \n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 93c650ac038..0d93d3983e0 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -328,13 +328,13 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n> \techo dirty >dirty/folder1/a &&\n> \n> \tgit -C dirty sparse-checkout init 2>err &&\n> -\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> \n> \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n> -\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n> \n> \tgit -C dirty sparse-checkout disable &&\n> -\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n> +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n\nI'm very happy this change is so clearly demonstrated by\na simple test change like this.\n\n> \tgit -C dirty reset --hard &&\n> \tgit -C dirty sparse-checkout init &&\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 0554842580b..9ee04992ac6 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -24,7 +24,7 @@\n> * situation better. See how \"git checkout\" and \"git merge\" replaces\n> * them using setup_unpack_trees_porcelain(), for example.\n> */\n> -static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n> +static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n> \t/* ERROR_WOULD_OVERWRITE */\n> \t\"Entry '%s' would be overwritten by merge. Cannot merge.\",\n> \n> @@ -46,6 +46,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n> \t/* ERROR_WOULD_LOSE_SUBMODULE */\n> \t\"Submodule '%s' cannot checkout new HEAD.\",\n> \n> +\t/* NB_UNPACK_TREES_ERROR_TYPES; just a meta value */\n> +\t\"\",\n> +\n> \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n> \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n> \n> @@ -222,7 +225,7 @@ static int add_rejected_path(struct unpack_trees_options *o,\n> \n> \t/*\n> \t * Otherwise, insert in a list for future display by\n> -\t * display_error_msgs()\n> +\t * display_(error|warning)_msgs()\n> \t */\n> \tstring_list_append(&o->unpack_rejects[e], path);\n> \treturn -1;\n> @@ -233,13 +236,16 @@ static int add_rejected_path(struct unpack_trees_options *o,\n> */\n> static void display_error_msgs(struct unpack_trees_options *o)\n> {\n> -\tint e, i;\n> -\tint something_displayed = 0;\n> +\tint e;\n> +\tunsigned error_displayed = 0;\n> \tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n> \t\tstruct string_list *rejects = &o->unpack_rejects[e];\n> +\n> \t\tif (rejects->nr > 0) {\n> +\t\t\tint i;\n> \t\t\tstruct strbuf path = STRBUF_INIT;\n> -\t\t\tsomething_displayed = 1;\n> +\n> +\t\t\terror_displayed = 1;\n> \t\t\tfor (i = 0; i < rejects->nr; i++)\n> \t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n> \t\t\terror(ERRORMSG(o, e), super_prefixed(path.buf));\n> @@ -247,10 +253,36 @@ static void display_error_msgs(struct unpack_trees_options *o)\n> \t\t}\n> \t\tstring_list_clear(rejects, 0);\n> \t}\n> -\tif (something_displayed)\n> +\tif (error_displayed)\n> \t\tfprintf(stderr, _(\"Aborting\\n\"));\n> }\n> \n> +/*\n> + * display all the warning messages stored in a nice way\n> + */\n> +static void display_warning_msgs(struct unpack_trees_options *o)\n> +{\n> +\tint e;\n> +\tunsigned warning_displayed = 0;\n> +\tfor (e = NB_UNPACK_TREES_ERROR_TYPES+1;\n\nsuper-nit: spaces around '+'\n\n> +\t e < NB_UNPACK_TREES_WARNING_TYPES; e++) {\n> +\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n> +\n> +\t\tif (rejects->nr > 0) {\n> +\t\t\tint i;\n> +\t\t\tstruct strbuf path = STRBUF_INIT;\n> +\n> +\t\t\twarning_displayed = 1;\n> +\t\t\tfor (i = 0; i < rejects->nr; i++)\n> +\t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n> +\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n> +\t\t\tstrbuf_release(&path);\n> +\t\t}\n> +\t\tstring_list_clear(rejects, 0);\n> +\t}\n> +\tif (warning_displayed)\n> +\t\tfprintf(stderr, _(\"After fixing the above paths, you may want to run `git sparse-checkout reapply`.\\n\"));\n> +}\n> static int check_submodule_move_head(const struct cache_entry *ce,\n> \t\t\t\t const char *old_id,\n> \t\t\t\t const char *new_id,\n> @@ -1705,8 +1737,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> \treturn ret;\n> \n> return_failed:\n> -\tif (o->show_all_errors)\n> +\tif (o->show_all_errors) {\n> \t\tdisplay_error_msgs(o);\n> +\t\tdisplay_warning_msgs(o);\n> +\t}\n> \tmark_all_ce_unused(o->src_index);\n> \tret = unpack_failed(o, NULL);\n> \tif (o->exiting_early)\n> @@ -1784,7 +1818,7 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n> \t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n> \n> done:\n> -\tdisplay_error_msgs(o);\n> +\tdisplay_warning_msgs(o);\n> \to->show_all_errors = old_show_all_errors;\n> \tif (free_pattern_list)\n> \t\tclear_pattern_list(&pl);\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index 3c6452fe9e5..d91c65ae453 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -24,10 +24,12 @@ enum unpack_trees_error_types {\n> \tERROR_BIND_OVERLAP,\n> \tERROR_WOULD_LOSE_SUBMODULE,\n> \n> +\tNB_UNPACK_TREES_ERROR_TYPES,\n> +\n\nI think this is a very clever way to partition the enum.\n\n> \tWARNING_SPARSE_NOT_UPTODATE_FILE,\n> \tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n> \n> -\tNB_UNPACK_TREES_ERROR_TYPES,\n> +\tNB_UNPACK_TREES_WARNING_TYPES,\n> };\n> \n> enum update_sparsity_result {\n> @@ -73,13 +75,13 @@ struct unpack_trees_options {\n> \tstruct dir_struct *dir;\n> \tstruct pathspec *pathspec;\n> \tmerge_fn_t fn;\n> -\tconst char *msgs[NB_UNPACK_TREES_ERROR_TYPES];\n> +\tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n> \tstruct argv_array msgs_to_free;\n> \t/*\n> \t * Store error messages in an array, each case\n> \t * corresponding to a error message type\n> \t */\n> -\tstruct string_list unpack_rejects[NB_UNPACK_TREES_ERROR_TYPES];\n> +\tstruct string_list unpack_rejects[NB_UNPACK_TREES_WARNING_TYPES];\n> \n> \tint head_idx;\n> \tint merge_size;\n\nLGTM other than my really nitty nit-pick. Feel free to ignore.\n\n-Stolee\n\n"},{"id":"393798","messageId":"0394a965-892b-6fb1-ca95-17d88913d656@gmail.com","threadId":"52995","inReplyTo":"5730f7d250a2025ae62edaf8779ff39bfbaa5ad6.1584813609.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 18/18] sparse-checkout: provide a new reapply subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:40:07Z","receivedAt":"2020-03-23T18:40:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 2:00 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> If commands like merge or rebase materialize files as part of their work,\n> or a previous sparse-checkout command failed to update individual files\n> due to dirty changes, users may want a command to simply 'reapply' the\n> sparsity rules. Provide one.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> Documentation/git-sparse-checkout.txt | 10 ++++++++++\n> builtin/sparse-checkout.c | 10 +++++++++-\n> 2 files changed, 19 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index c0342e53938..1a3ace60820 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -70,6 +70,16 @@ C-style quoted strings.\n> \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n> \tas directory names as in the 'set' subcommand.\n> \n> +'reapply::\n> +\tReapply the sparsity pattern rules to paths in the working tree.\n> +\tCommands like merge or rebase can materialize paths to do their\n> +\twork (e.g. in order to show you a conflict), and other\n> +\tsparse-checkout commands might fail to sparsify an individual file\n> +\t(e.g. because it has unstaged changes or conflicts). In such\n> +\tcases, it can make sense to run `git sparse-checkout reapply` later\n> +\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n> +\tor committing changes, etc.).\n> +\n> 'disable'::\n> \tDisable the `core.sparseCheckout` config setting, and restore the\n> \tworking directory to include all files. Leaves the sparse-checkout\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index aa81199f85d..95d08824172 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -18,7 +18,7 @@\n> static const char *empty_base = \"\";\n> \n> static char const * const builtin_sparse_checkout_usage[] = {\n> -\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n> +\tN_(\"git sparse-checkout (init|list|set|add|reapply|disable) \"),\n> \tNULL\n> };\n> \n> @@ -554,6 +554,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n> \treturn modify_pattern_list(argc, argv, m);\n> }\n> \n> +static int sparse_checkout_reapply(int argc, const char **argv)\n> +{\n> +\trepo_read_index(the_repository);\n> +\treturn update_working_directory(NULL);\n> +}\n> +\n> static int sparse_checkout_disable(int argc, const char **argv)\n> {\n> \tstruct pattern_list pl;\n> @@ -603,6 +609,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n> \t\tif (!strcmp(argv[0], \"add\"))\n> \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n> +\t\tif (!strcmp(argv[0], \"reapply\"))\n> +\t\t\treturn sparse_checkout_reapply(argc, argv);\n> \t\tif (!strcmp(argv[0], \"disable\"))\n> \t\t\treturn sparse_checkout_disable(argc, argv);\n> \t}\n\nI feel it necessary to commend you for how simple this patch is.\nYou did all the hard work in earlier patches to make this one\nextremely simple.\n\nNice!\n-Stolee\n\n"},{"id":"393799","messageId":"1a58492d-4e09-c12b-ff53-2f2b9cb2e28b@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/18] Sparse checkout improvements -- improved sparsity updating","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-23T18:41:49Z","receivedAt":"2020-03-23T18:41:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/21/2020 1:59 PM, Elijah Newren via GitGitGadget wrote:\n> This series provides a replacement for the sparsity updating in\n> sparse-checkout that is based on the logic from git read-tree -mu HEAD. The\n> most important bit is patch 9 and its lengthy commit message explaining the\n> current state and rationale for most the series, though patches 16 and 17\n> have additional related directions and rationale for the series. Those three\n> patches are the most important to review.\n> \n> Changes since v1:\n> \n> * addressed several cleanups highlighted by Stolee (and I picked 'reapply'\n> for the new subcommand name)\n> * added three new minor cleanup commits (patches 2-4)\n> * several new patches to adjust the warning messages to look like warnings\n> (patches 11-15)\n> * display warning messages when paths are unmerged (patch 16)\n> * also make checkout and other unpack_trees()-using commands not error out\n> when unable to set the SKIP_WORKTREE bit (patch 17)\n\nI'm very happy with the updates in this version.\n\n> Also, the loop range in display_warning_msgs() in patch 14 might seem\n> slightly weird, but I felt adding another unpack_rejects array of some sort\n> and plumbing through an extra bit everywhere to notify the system which\n> unpack_rejects array to use would have been more invasive. Opinions either\n> way (or alternative approaches) welcome.\n> \n> I was also curious if I should have squashed all the patches to adjust the\n> warning messages (patches 11-15, they adjust the messages to actually look\n> like warnings instead of errors), since they are individually kind of small,\n> but it looked like it'd be much easier to review if I were to split them\n> apart, so I did so.\n\nI found them very easy to review. I had the one comment that was my fault\nfor not looking one patch ahead. In general, it was simple to see that you\nwere doing good updates with good motivations.\n\nThis is going to help the UX around sparse-checkout quite a bit!\n\nReviewed-by: Derrick Stolee \n\n-Stolee\n\n"},{"id":"393813","messageId":"xmqqo8sm4y6w.fsf@gitster.c.googlers.com","threadId":"52995","inReplyTo":"5a83259e-af03-1930-a62b-75ad37b05f37@gmail.com","subject":"Re: [PATCH v2 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-23T20:24:07Z","receivedAt":"2020-03-23T20:24:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>>> +int update_sparsity(struct unpack_trees_options *options);\n>>>> +\n>>>\n>>> This appears to not use the enum as it should.\n>> \n>> Whoops! Will fix. (Interesting that the compiler didn't flag any\n>> kind of warning based on mismatch of declared function return types\n>> for update_sparsity() in the .c and .h files...)\n>\n> *shrug* enums are essentially decoration over an int, so I'm not\n> surprised it can happen.\n\n;-) \n\nYeah, the value of preferring enum over preprocessor macros because\nit helps debuggers to show symbolic constants, is oversold, I think.\nTypes of variables and functions' return values need to be set to\nthe enum or the benefit won't come to us X-<.\n\n"},{"id":"393814","messageId":"xmqqh7ye4y3o.fsf@gitster.c.googlers.com","threadId":"52995","inReplyTo":"1a58492d-4e09-c12b-ff53-2f2b9cb2e28b@gmail.com","subject":"Re: [PATCH v2 00/18] Sparse checkout improvements -- improved sparsity updating","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-23T20:26:03Z","receivedAt":"2020-03-23T20:26:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> Changes since v1:\n>> \n>> * addressed several cleanups highlighted by Stolee (and I picked 'reapply'\n>> for the new subcommand name)\n>> * added three new minor cleanup commits (patches 2-4)\n>> * several new patches to adjust the warning messages to look like warnings\n>> (patches 11-15)\n>> * display warning messages when paths are unmerged (patch 16)\n>> * also make checkout and other unpack_trees()-using commands not error out\n>> when unable to set the SKIP_WORKTREE bit (patch 17)\n>\n> I'm very happy with the updates in this version.\n> ...\n> This is going to help the UX around sparse-checkout quite a bit!\n>\n> Reviewed-by: Derrick Stolee \n\nThanks, both of you. Will queue.\n"},{"id":"394133","messageId":"bb41a2a52f48263a5b5c6fad955f506cb6390a22.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 01/18] unpack-trees: fix minor typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:44Z","receivedAt":"2020-03-27T00:49:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 1ecdab33040..0d0eec0221e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1618,7 +1618,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \t\t/*\n \t\t * Sparse checkout loop #2: set NEW_SKIP_WORKTREE on entries not in loop #1\n-\t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n+\t\t * If they will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n \t\tmark_new_skip_worktree(o->pl, &o->result,\n-- \ngitgitgadget\n\n"},{"id":"394134","messageId":"1dc36eb33f7879f5553c2bb28c516022d040e12b.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 02/18] unpack-trees: remove unused error type","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:45Z","receivedAt":"2020-03-27T00:49:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncommit 08402b0409 (\"merge-recursive: distinguish \"removed\" and\n\"overwritten\" messages\", 2010-08-11) split\n ERROR_WOULD_LOSE_UNTRACKED\ninto both\n ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN\n ERROR_WOULD_LOSE_UNTRACKED_REMOVED\nand also split\n ERROR_WOULD_LOSE_ORPHANED\ninto both\n ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN\n ERROR_WOULD_LOSE_ORPHANED_REMOVED\n\nHowever, despite the split only three of these four types were used.\nERROR_WOULD_LOSE_ORPHANED_REMOVED was not put into use when it was\nintroduced and nothing else has used it in the intervening decade\neither. Remove it.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 4 ----\n unpack-trees.h | 1 -\n 2 files changed, 5 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 0d0eec0221e..f72a7a21f9c 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -49,8 +49,6 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n \n-\t/* ERROR_WOULD_LOSE_ORPHANED_REMOVED */\n-\t\"Working tree file '%s' would be removed by sparse checkout update.\",\n \n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n@@ -172,8 +170,6 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_ORPHANED_REMOVED] =\n-\t\t_(\"The following working tree files would be removed by sparse checkout update:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n \t\t_(\"Cannot update submodule:\\n%s\");\n \ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex ae1557fb804..6d7c7b6c2e0 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -24,7 +24,6 @@ enum unpack_trees_error_types {\n \tERROR_BIND_OVERLAP,\n \tERROR_SPARSE_NOT_UPTODATE_FILE,\n \tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tERROR_WOULD_LOSE_ORPHANED_REMOVED,\n \tERROR_WOULD_LOSE_SUBMODULE,\n \tNB_UNPACK_TREES_ERROR_TYPES\n };\n-- \ngitgitgadget\n\n"},{"id":"394135","messageId":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v2.git.git.1584813609.gitgitgadget@gmail.com","subject":"[PATCH v3 00/18] Sparse checkout improvements -- improved sparsity updating","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:43Z","receivedAt":"2020-03-27T00:49:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series provides a replacement for the sparsity updating in\nsparse-checkout that is based on the logic from git read-tree -mu HEAD. The\nmost important bit is patch 9 and its lengthy commit message explaining the\ncurrent state and rationale for most the series, though patches 16 and 17\nhave additional related directions and rationale for the series. Those three\npatches are the most important to review.\n\nChanges since v2:\n\n * addressed Stolee's cleanups and added his Reviewed-by to the series\n (hopefully the next two changes don't invalidate that)\n * added a test for the new 'reapply' subcommand (noticed it was missing\n with Stolee's test coverage report)\n * fix a minor issue with two of the other tests I modified -- when I'm\n grepping through stderr for a command, I need to make sure to actually\n record stderr from that command (otherwise my grep is looking through the\n stderr of a previous command that was recorded). Also, since I messed\n this up, I added one or two more sanity checks while I was at it.\n\nElijah Newren (18):\n unpack-trees: fix minor typo in comment\n unpack-trees: remove unused error type\n unpack-trees: simplify verify_absent_sparse()\n unpack-trees: simplify pattern_list freeing\n t1091: make some tests a little more defensive against failures\n unpack-trees: allow check_updates() to work on a different index\n unpack-trees: do not mark a dirty path with SKIP_WORKTREE\n unpack-trees: pull sparse-checkout pattern reading into a new function\n unpack-trees: add a new update_sparsity() function\n sparse-checkout: use new update_sparsity() function\n sparse-checkout: use improved unpack_trees porcelain messages\n unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier\n unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*\n unpack-trees: split display_error_msgs() into two\n unpack-trees: make sparse path messages sound like warnings\n unpack-trees: provide warnings on sparse updates for unmerged paths\n too\n unpack-trees: failure to set SKIP_WORKTREE bits always just a warning\n sparse-checkout: provide a new reapply subcommand\n\n Documentation/git-sparse-checkout.txt | 10 +\n builtin/sparse-checkout.c | 55 +++---\n t/t1011-read-tree-sparse-checkout.sh | 11 +-\n t/t1091-sparse-checkout-builtin.sh | 105 ++++++++++-\n t/t2018-checkout-branch.sh | 22 +++\n unpack-trees.c | 255 ++++++++++++++++++++------\n unpack-trees.h | 28 ++-\n 7 files changed, 375 insertions(+), 111 deletions(-)\n\n\nbase-commit: be8661a3286c67a5d4088f4226cbd7f8b76544b0\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-726%2Fnewren%2Fsparse-checkout-improvements-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-726/newren/sparse-checkout-improvements-v3\nPull-Request: https://github.com/git/git/pull/726\n\nRange-diff vs v2:\n\n 1: 3c183727c98 ! 1: bb41a2a52f4 unpack-trees: fix minor typo in comment\n @@ -2,6 +2,7 @@\n \n unpack-trees: fix minor typo in comment\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 2: e6beb1c5810 ! 2: 1dc36eb33f7 unpack-trees: remove unused error type\n @@ -19,6 +19,7 @@\n introduced and nothing else has used it in the intervening decade\n either. Remove it.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 3: 739b52393fe ! 3: 97a95e91d8f unpack-trees: simplify verify_absent_sparse()\n @@ -11,6 +11,7 @@\n and no other caller has been created, and no other value has ever been\n passed, so just pass the expected value to begin with.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 4: 65772dd46df ! 4: 1ce5dbe9c68 unpack-trees: simplify pattern_list freeing\n @@ -13,6 +13,7 @@\n additional flag in unpack_trees_options. Put the logic back in the\n relevant function and discard the now unnecessary flag.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n 5: 5cbab2a7a56 ! 5: da4b74093cd t1091: make some tests a little more defensive against failures\n @@ -2,6 +2,7 @@\n \n t1091: make some tests a little more defensive against failures\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n 6: 5fea19f0136 ! 6: 555eedf5380 unpack-trees: allow check_updates() to work on a different index\n @@ -6,6 +6,7 @@\n to use this function in combination with a different index_state, so\n take the intended index_state as a parameter.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 7: 3d2531ca818 ! 7: 10ec70fa697 unpack-trees: do not mark a dirty path with SKIP_WORKTREE\n @@ -15,6 +15,7 @@\n paths. Leaving dirty working files in the working tree is thus a\n natural extension of what we have already been doing.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 8: 27d6304d33f ! 8: 8ed2f4e49ed unpack-trees: pull sparse-checkout pattern reading into a new function\n @@ -6,6 +6,7 @@\n path_patterns from $GIT_DIR/info/sparse-checkout so that we can re-use\n it elsewhere.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 9: a46439c8536 ! 9: 3a1f084641e unpack-trees: add a new update_sparsity() function\n @@ -71,6 +71,7 @@\n are marked SKIP_WORKTREE and thus doesn't delete them, and `commit -a`\n is similar), it seems logical to me.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n @@ -130,7 +131,6 @@\n +\n +\t\tif (!ce_skip_worktree(ce))\n +\t\t\tempty_worktree = 0;\n -+\n +\t}\n +\n +\t/*\n @@ -166,8 +166,8 @@\n --- a/unpack-trees.h\n +++ b/unpack-trees.h\n @@\n - \tNB_UNPACK_TREES_ERROR_TYPES\n - };\n + int unpack_trees(unsigned n, struct tree_desc *t,\n + \t\t struct unpack_trees_options *options);\n \n +enum update_sparsity_result {\n +\tUPDATE_SPARSITY_SUCCESS = 0,\n @@ -176,14 +176,7 @@\n +\tUPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n +};\n +\n - /*\n - * Sets the list of user-friendly error messages to be used by the\n - * command \"cmd\" (either merge or checkout), and show_all_errors to 1.\n -@@\n - int unpack_trees(unsigned n, struct tree_desc *t,\n - \t\t struct unpack_trees_options *options);\n - \n -+int update_sparsity(struct unpack_trees_options *options);\n ++enum update_sparsity_result update_sparsity(struct unpack_trees_options *options);\n +\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o);\n 10: 27ed4a3f7a2 ! 10: 9040ac97175 sparse-checkout: use new update_sparsity() function\n @@ -6,6 +6,12 @@\n codepaths for setting the SKIP_WORKTREE bits and instead use the new\n update_sparsity() function.\n \n + Note that when an issue is hit, the error message splits 'error' and\n + 'Cannot update sparse checkout' on separate lines. For now, we use two\n + greps to find both pieces of the error message but subsequent commits\n + will clean up the messages reported to the user.\n +\n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n @@ -147,14 +153,21 @@\n +\ttest_i18ngrep \"error\" err &&\n +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\n -+\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n ++\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n +\ttest_i18ngrep \"error\" err &&\n +\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n ++\ttest_path_is_file dirty/folder1/a &&\n +\n -+\tgit -C dirty sparse-checkout disable &&\n -+\ttest_i18ngrep \"error\" err &&\n -+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n ++\tgit -C dirty sparse-checkout disable 2>err &&\n ++\ttest_must_be_empty err &&\n +\n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n +-\tgit -C dirty sparse-checkout disable\n ++\ttest_path_is_missing dirty/folder1/a &&\n ++\tgit -C dirty sparse-checkout disable &&\n ++\ttest_path_is_file dirty/folder1/a\n + '\n + \n + test_expect_success 'cone mode: set with core.ignoreCase=true' '\n 11: 79b9cc1df55 ! 11: e9c7e8ec46f sparse-checkout: use improved unpack_trees porcelain messages\n @@ -23,6 +23,7 @@\n we even called unpack_trees(). The previous commit changed that,\n though, so let's use the better error messages.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n @@ -50,15 +51,10 @@\n -\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \n - \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n + \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n -\ttest_i18ngrep \"error\" err &&\n -\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n + \ttest_path_is_file dirty/folder1/a &&\n \n - \tgit -C dirty sparse-checkout disable &&\n --\ttest_i18ngrep \"error\" err &&\n --\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n -+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n - \n - \tgit -C dirty reset --hard &&\n - \tgit -C dirty sparse-checkout init &&\n + \tgit -C dirty sparse-checkout disable 2>err &&\n 12: 2ba7bbdaee3 ! 12: c3741107a9d unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier\n @@ -5,6 +5,7 @@\n A minor change, but we want to convert the sparse messages to warnings\n and this allows us to group warnings and errors.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n 13: 203ece0ec2f ! 13: 90c514e220b unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*\n @@ -6,6 +6,7 @@\n rather than an error; rename the enum values to reflect this intent as\n a simple step towards that goal.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/unpack-trees.c b/unpack-trees.c\n @@ -72,4 +73,4 @@\n +\tNB_UNPACK_TREES_ERROR_TYPES,\n };\n \n - enum update_sparsity_result {\n + /*\n 14: 57679c8e292 ! 14: 5676e890f65 unpack-trees: split display_error_msgs() into two\n @@ -10,6 +10,7 @@\n A subsequent commit will further modify unpack_trees() and how it calls\n the new display_warning_msgs().\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n @@ -22,16 +23,12 @@\n -\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \n - \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n + \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n -\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n + \ttest_path_is_file dirty/folder1/a &&\n \n - \tgit -C dirty sparse-checkout disable &&\n --\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n -+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n - \n - \tgit -C dirty reset --hard &&\n - \tgit -C dirty sparse-checkout init &&\n + \tgit -C dirty sparse-checkout disable 2>err &&\n \n diff --git a/unpack-trees.c b/unpack-trees.c\n --- a/unpack-trees.c\n @@ -100,7 +97,7 @@\n +{\n +\tint e;\n +\tunsigned warning_displayed = 0;\n -+\tfor (e = NB_UNPACK_TREES_ERROR_TYPES+1;\n ++\tfor (e = NB_UNPACK_TREES_ERROR_TYPES + 1;\n +\t e < NB_UNPACK_TREES_WARNING_TYPES; e++) {\n +\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n +\n @@ -160,7 +157,7 @@\n +\tNB_UNPACK_TREES_WARNING_TYPES,\n };\n \n - enum update_sparsity_result {\n + /*\n @@\n \tstruct dir_struct *dir;\n \tstruct pathspec *pathspec;\n 15: 02a8e01f35b ! 15: 74e13ccee40 unpack-trees: make sparse path messages sound like warnings\n @@ -7,6 +7,7 @@\n operation abort. Reword the messages to make sense in their new\n context.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n @@ -28,16 +29,12 @@\n -\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \n - \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n + \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n -\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n +\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n + \ttest_path_is_file dirty/folder1/a &&\n \n - \tgit -C dirty sparse-checkout disable &&\n --\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n -+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n - \n - \tgit -C dirty reset --hard &&\n - \tgit -C dirty sparse-checkout init &&\n + \tgit -C dirty sparse-checkout disable 2>err &&\n \n diff --git a/unpack-trees.c b/unpack-trees.c\n --- a/unpack-trees.c\n 16: f664a6dcbb3 ! 16: 5d9e8a9d27e unpack-trees: provide warnings on sparse updates for unmerged paths too\n @@ -7,13 +7,14 @@\n those files have dirty changes. Add a similar warning for unmerged\n paths as well.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \tgit -C dirty sparse-checkout disable\n + \ttest_path_is_file dirty/folder1/a\n '\n \n +test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged status' '\n @@ -28,10 +29,11 @@\n +\tgit -C unmerged sparse-checkout init 2>err &&\n +\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n +\n -+\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* &&\n ++\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n +\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n ++\ttest_path_is_file dirty/folder1/a &&\n +\n -+\tgit -C unmerged sparse-checkout disable &&\n ++\tgit -C unmerged sparse-checkout disable 2>err &&\n +\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n +\n +\tgit -C unmerged reset --hard &&\n 17: fa997a1c72f ! 17: 844306c3e86 unpack-trees: failure to set SKIP_WORKTREE bits always just a warning\n @@ -37,6 +37,7 @@\n Downgrade any errors from inability to toggle the SKIP_WORKTREE bit to a\n warning and allow the operations to continue.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\n 18: 5730f7d250a ! 18: 93dbdd78acf sparse-checkout: provide a new reapply subcommand\n @@ -7,6 +7,7 @@\n due to dirty changes, users may want a command to simply 'reapply' the\n sparsity rules. Provide one.\n \n + Reviewed-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n @@ -64,3 +65,55 @@\n \t\tif (!strcmp(argv[0], \"disable\"))\n \t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n +\n + diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n + --- a/t/t1091-sparse-checkout-builtin.sh\n + +++ b/t/t1091-sparse-checkout-builtin.sh\n +@@\n + \tgit -C unmerged sparse-checkout disable\n + '\n + \n ++test_expect_success 'sparse-checkout reapply' '\n ++\tgit clone repo tweak &&\n ++\n ++\techo dirty >tweak/deep/deeper2/a &&\n ++\n ++\tcat >input <<-EOF &&\n ++\t0 0000000000000000000000000000000000000000\tfolder1/a\n ++\t100644 $(git -C tweak rev-parse HEAD:folder1/a) 1\tfolder1/a\n ++\tEOF\n ++\tgit -C tweak update-index --index-info err &&\n ++\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n ++\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n ++\n ++\tgit -C tweak sparse-checkout set folder2 deep/deeper1 2>err &&\n ++\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n ++\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n ++\n ++\tgit -C tweak sparse-checkout reapply 2>err &&\n ++\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n ++\ttest_path_is_file tweak/deep/deeper2/a &&\n ++\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n ++\ttest_path_is_file tweak/folder1/a &&\n ++\n ++\tgit -C tweak checkout HEAD deep/deeper2/a &&\n ++\tgit -C tweak sparse-checkout reapply 2>err &&\n ++\ttest_i18ngrep ! \"warning.*The following paths are not up to date\" err &&\n ++\ttest_path_is_missing tweak/deep/deeper2/a &&\n ++\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n ++\ttest_path_is_file tweak/folder1/a &&\n ++\n ++\tgit -C tweak add folder1/a &&\n ++\tgit -C tweak sparse-checkout reapply 2>err &&\n ++\ttest_must_be_empty err &&\n ++\ttest_path_is_missing tweak/deep/deeper2/a &&\n ++\ttest_path_is_missing tweak/folder1/a &&\n ++\n ++\tgit -C tweak sparse-checkout disable\n ++'\n ++\n + test_expect_success 'cone mode: set with core.ignoreCase=true' '\n + \trm repo/.git/info/sparse-checkout &&\n + \tgit -C repo sparse-checkout init --cone &&\n\n-- \ngitgitgadget\n"},{"id":"394136","messageId":"555eedf538030b9c9560a2f9d3cbd0ad1db026d5.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 06/18] unpack-trees: allow check_updates() to work on a different index","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:49Z","receivedAt":"2020-03-27T00:49:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncheck_updates() previously assumed it was working on o->result. We want\nto use this function in combination with a different index_state, so\ntake the intended index_state as a parameter.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex d2863fa0310..dde50047a82 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -353,12 +353,12 @@ static void report_collided_checkout(struct index_state *index)\n \tstring_list_clear(&list, 0);\n }\n \n-static int check_updates(struct unpack_trees_options *o)\n+static int check_updates(struct unpack_trees_options *o,\n+\t\t\t struct index_state *index)\n {\n \tunsigned cnt = 0;\n \tint errs = 0;\n \tstruct progress *progress;\n-\tstruct index_state *index = &o->result;\n \tstruct checkout state = CHECKOUT_INIT;\n \tint i;\n \n@@ -1665,7 +1665,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t}\n \t}\n \n-\tret = check_updates(o) ? (-2) : 0;\n+\tret = check_updates(o, &o->result) ? (-2) : 0;\n \tif (o->dst_index) {\n \t\tmove_index_extensions(&o->result, o->src_index);\n \t\tif (!ret) {\n-- \ngitgitgadget\n\n"},{"id":"394137","messageId":"1ce5dbe9c68c6a5377bab22297b3c8d684d2b16c.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 04/18] unpack-trees: simplify pattern_list freeing","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:47Z","receivedAt":"2020-03-27T00:49:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ncommit e091228e17 (\"sparse-checkout: update working directory\nin-process\", 2019-11-21) allowed passing a pre-defined set of patterns\nto unpack_trees(). However, if o->pl was NULL, it would still read the\nexisting patterns and use those. If those patterns were read into a\ndata structure that was allocated, naturally they needed to be free'd.\nHowever, despite the same function being responsible for knowing about\nboth the allocation and the free'ing, the logic for tracking whether to\nfree the pattern_list was hoisted to an outer function with an\nadditional flag in unpack_trees_options. Put the logic back in the\nrelevant function and discard the now unnecessary flag.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 1 -\n unpack-trees.c | 6 ++++--\n unpack-trees.h | 3 +--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 740da4b6d54..d102a9697fd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -122,7 +122,6 @@ static int update_working_directory(struct pattern_list *pl)\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n-\to.keep_pattern_list = !!pl;\n \n \tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3af2e126abf..d2863fa0310 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1503,6 +1503,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tint i, ret;\n \tstatic struct cache_entry *dfc;\n \tstruct pattern_list pl;\n+\tint free_pattern_list = 0;\n \n \tif (len > MAX_UNPACK_TREES)\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n@@ -1519,6 +1520,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\telse\n \t\t\to->pl = &pl;\n \t\tfree(sparse);\n+\t\tfree_pattern_list = 1;\n \t}\n \n \tmemset(&o->result, 0, sizeof(o->result));\n@@ -1686,9 +1688,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \to->src_index = NULL;\n \n done:\n-\ttrace_performance_leave(\"unpack_trees\");\n-\tif (!o->keep_pattern_list)\n+\tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\n+\ttrace_performance_leave(\"unpack_trees\");\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 6d7c7b6c2e0..d3516267f36 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -58,8 +58,7 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run,\n-\t\t keep_pattern_list;\n+\t\t dry_run;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"394138","messageId":"da4b74093cd4eb0d91f4472d61fd5e992b8e4d12.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 05/18] t1091: make some tests a little more defensive against failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:48Z","receivedAt":"2020-03-27T00:49:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 44a91205d60..8607a8e6d1a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -278,6 +278,7 @@ test_expect_success 'cone mode: add parent path' '\n '\n \n test_expect_success 'revert to old sparse-checkout on bad update' '\n+\ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n \techo update >repo/deep/deeper2/a &&\n@@ -328,6 +329,7 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n '\n \n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n+\trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n \tcat >expect <<-\\EOF &&\n-- \ngitgitgadget\n\n"},{"id":"394139","messageId":"97a95e91d8f2d71df59d7fa9911d8974d2199a38.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 03/18] unpack-trees: simplify verify_absent_sparse()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:46Z","receivedAt":"2020-03-27T00:49:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nverify_absent_sparse() was introduced in commit 08402b0409\n(\"merge-recursive: distinguish \"removed\" and \"overwritten\" messages\",\n2010-08-11), and has always had exactly one caller which always passes\nerror_type == ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN. This function\nthen checks whether error_type is this value, and if so, sets it instead\nto ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN. It has been nearly a decade\nand no other caller has been created, and no other value has ever been\npassed, so just pass the expected value to begin with.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 8 ++------\n 1 file changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f72a7a21f9c..3af2e126abf 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -506,7 +506,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n \tif (was_skip_worktree && !ce_skip_worktree(ce)) {\n-\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o))\n+\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN, o))\n \t\t\treturn -1;\n \t\tce->ce_flags |= CE_UPDATE;\n \t}\n@@ -2026,11 +2026,7 @@ static int verify_absent_sparse(const struct cache_entry *ce,\n \t\t\t\tenum unpack_trees_error_types error_type,\n \t\t\t\tstruct unpack_trees_options *o)\n {\n-\tenum unpack_trees_error_types orphaned_error = error_type;\n-\tif (orphaned_error == ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN)\n-\t\torphaned_error = ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN;\n-\n-\treturn verify_absent_1(ce, orphaned_error, o);\n+\treturn verify_absent_1(ce, error_type, o);\n }\n \n static int merged_entry(const struct cache_entry *ce,\n-- \ngitgitgadget\n\n"},{"id":"394140","messageId":"10ec70fa697e060ddc65c1245f73e37612edefc2.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 07/18] unpack-trees: do not mark a dirty path with SKIP_WORKTREE","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:50Z","receivedAt":"2020-03-27T00:49:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf a path is dirty, removing from the working tree risks losing data.\nAs such, we want to make sure any such path is not marked with\nSKIP_WORKTREE. While the current callers of this code detect this case\nand re-populate with a previous set of sparsity patterns, we want to\nallow some paths to be marked with SKIP_WORKTREE while others are left\nunmarked without it being considered an error. The reason this\nshouldn't be considered an error is that SKIP_WORKTREE has always been\nan advisory-only setting; merge and rebase for example were free to\nmaterialize paths and clear the SKIP_WORKTREE bit in order to accomplish\ntheir work even though they kept the SKIP_WORKTREE bit set for other\npaths. Leaving dirty working files in the working tree is thus a\nnatural extension of what we have already been doing.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dde50047a82..e8e794880ab 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -500,8 +500,11 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\t * also stat info may have lost after merged_entry() so calling\n \t\t * verify_uptodate() again may fail\n \t\t */\n-\t\tif (!(ce->ce_flags & CE_UPDATE) && verify_uptodate_sparse(ce, o))\n+\t\tif (!(ce->ce_flags & CE_UPDATE) &&\n+\t\t verify_uptodate_sparse(ce, o)) {\n+\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n \t\t\treturn -1;\n+\t\t}\n \t\tce->ce_flags |= CE_WT_REMOVE;\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"394141","messageId":"8ed2f4e49ed80626292013f55dc5164f0eaad98b.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 08/18] unpack-trees: pull sparse-checkout pattern reading into a new function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:51Z","receivedAt":"2020-03-27T00:49:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate a populate_from_existing_patterns() function for reading the\npath_patterns from $GIT_DIR/info/sparse-checkout so that we can re-use\nit elsewhere.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 24 ++++++++++++++++--------\n 1 file changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex e8e794880ab..4733e7eaf89 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1492,6 +1492,20 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n+static void populate_from_existing_patterns(struct unpack_trees_options *o,\n+\t\t\t\t\t struct pattern_list *pl)\n+{\n+\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\n+\tpl->use_cone_patterns = core_sparse_checkout_cone;\n+\tif (add_patterns_from_file_to_list(sparse, \"\", 0, pl, NULL) < 0)\n+\t\to->skip_sparse_checkout = 1;\n+\telse\n+\t\to->pl = pl;\n+\tfree(sparse);\n+}\n+\n+\n static int verify_absent(const struct cache_entry *,\n \t\t\t enum unpack_trees_error_types,\n \t\t\t struct unpack_trees_options *);\n@@ -1512,18 +1526,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\tdie(\"unpack_trees takes at most %d trees\", MAX_UNPACK_TREES);\n \n \ttrace_performance_enter();\n-\tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout && !o->pl) {\n-\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n-\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n-\t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n-\t\t\to->skip_sparse_checkout = 1;\n-\t\telse\n-\t\t\to->pl = &pl;\n-\t\tfree(sparse);\n+\t\tmemset(&pl, 0, sizeof(pl));\n \t\tfree_pattern_list = 1;\n+\t\tpopulate_from_existing_patterns(o, &pl);\n \t}\n \n \tmemset(&o->result, 0, sizeof(o->result));\n-- \ngitgitgadget\n\n"},{"id":"394142","messageId":"3a1f084641eb47515b5a41ed4409a36128913309.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 09/18] unpack-trees: add a new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:52Z","receivedAt":"2020-03-27T00:49:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPreviously, the only way to update the SKIP_WORKTREE bits for various\npaths was invoking `git read-tree -mu HEAD` or calling the same code\nthat this codepath invoked. This however had a number of problems if\nthe index or working directory were not clean. First, let's consider\nthe case:\n\n Flipping SKIP_WORKTREE -> !SKIP_WORKTREE (materializing files)\n\nIf the working tree was clean this was fine, but if there were files or\ndirectories or symlinks or whatever already present at the given path\nthen the operation would abort with an error. Let's label this case\nfor later discussion:\n\n A) There is an untracked path in the way\n\nNow let's consider the opposite case:\n\n Flipping !SKIP_WORKTREE -> SKIP_WORKTREE (removing files)\n\nIf the index and working tree was clean this was fine, but if there were\nany unclean paths we would run into problems. There are three different\ncases to consider:\n\n B) The path is unmerged\n C) The path has unstaged changes\n D) The path has staged changes (differs from HEAD)\n\nIf any path fell into case B or C, then the whole operation would be\naborted with an error. With sparse-checkout, the whole operation would\nbe aborted for case D as well, but for its predecessor of using `git\nread-tree -mu HEAD` directly, any paths that fell into case D would be\nremoved from the working copy and the index entry for that path would be\nreset to match HEAD -- which looks and feels like data loss to users\n(only a few are even aware to ask whether it can be recovered, and even\nthen it requires walking through loose objects trying to match up the\nright ones).\n\nRefusing to remove files that have unsaved user changes is good, but\nrefusing to work on any other paths is very problematic for users. If\nthe user is in the middle of a rebase or has made modifications to files\nthat bring in more dependencies, then for their build to work they need\nto update the sparse paths. This logic has been preventing them from\ndoing so. Sometimes in response, the user will stage the files and\nre-try, to no avail with sparse-checkout or to the horror of losing\ntheir changes if they are using its predecessor of `git read-tree -mu\nHEAD`.\n\nAdd a new update_sparsity() function which will not error out in any of\nthese cases but behaves as follows for the special cases:\n A) Leave the file in the working copy alone, clear the SKIP_WORKTREE\n bit, and print a warning (thus leaving the path in a state where\n status will report the file as modified, which seems logical).\n B) Do NOT mark this path as SKIP_WORKTREE, and leave it as unmerged.\n C) Do NOT mark this path as SKIP_WORKTREE and print a warning about\n the dirty path.\n D) Mark the path as SKIP_WORKTREE, but do not revert the version\n stored in the index to match HEAD; leave the contents alone.\n\nI tried a different behavior for A (leave the SKIP_WORKTREE bit set),\nbut found it very surprising and counter-intuitive (e.g. the user sees\nit is present along with all the other files in that directory, tries to\nstage it, but git add ignores it since the SKIP_WORKTREE bit is set). A\n& C seem like optimal behavior to me. B may be as well, though I wonder\nif printing a warning would be an improvement. Some might be slightly\nsurprised by D at first, but given that it does the right thing with\n`git commit` and even `git commit -a` (`git add` ignores entries that\nare marked SKIP_WORKTREE and thus doesn't delete them, and `commit -a`\nis similar), it seems logical to me.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.h | 9 ++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 4733e7eaf89..a5bc0a3a16d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1714,6 +1714,83 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tgoto done;\n }\n \n+/*\n+ * Update SKIP_WORKTREE bits according to sparsity patterns, and update\n+ * working directory to match.\n+ *\n+ * CE_NEW_SKIP_WORKTREE is used internally.\n+ */\n+enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n+{\n+\tenum update_sparsity_result ret = UPDATE_SPARSITY_SUCCESS;\n+\tstruct pattern_list pl;\n+\tint i, empty_worktree;\n+\tunsigned old_show_all_errors;\n+\tint free_pattern_list = 0;\n+\n+\told_show_all_errors = o->show_all_errors;\n+\to->show_all_errors = 1;\n+\n+\t/* Sanity checks */\n+\tif (!o->update || o->index_only || o->skip_sparse_checkout)\n+\t\tBUG(\"update_sparsity() is for reflecting sparsity patterns in working directory\");\n+\tif (o->src_index != o->dst_index || o->fn)\n+\t\tBUG(\"update_sparsity() called wrong\");\n+\n+\ttrace_performance_enter();\n+\n+\t/* If we weren't given patterns, use the recorded ones */\n+\tif (!o->pl) {\n+\t\tmemset(&pl, 0, sizeof(pl));\n+\t\tfree_pattern_list = 1;\n+\t\tpopulate_from_existing_patterns(o, &pl);\n+\t\tif (o->skip_sparse_checkout)\n+\t\t\tgoto skip_sparse_checkout;\n+\t}\n+\n+\t/* Set NEW_SKIP_WORKTREE on existing entries. */\n+\tmark_all_ce_unused(o->src_index);\n+\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n+\n+\t/* Then loop over entries and update/remove as needed */\n+\tret = UPDATE_SPARSITY_SUCCESS;\n+\tempty_worktree = 1;\n+\tfor (i = 0; i < o->src_index->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = o->src_index->cache[i];\n+\n+\t\tif (apply_sparse_checkout(o->src_index, ce, o))\n+\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n+\n+\t\tif (!ce_skip_worktree(ce))\n+\t\t\tempty_worktree = 0;\n+\t}\n+\n+\t/*\n+\t * Sparse checkout is meant to narrow down checkout area\n+\t * but it does not make sense to narrow down to empty working\n+\t * tree. This is usually a mistake in sparse checkout rules.\n+\t * Do not allow users to do that.\n+\t */\n+\tif (o->src_index->cache_nr && empty_worktree) {\n+\t\tunpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n+\t\tret = UPDATE_SPARSITY_INDEX_UPDATE_FAILURES;\n+\t\tgoto done;\n+\t}\n+\n+skip_sparse_checkout:\n+\tif (check_updates(o, o->src_index))\n+\t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n+\n+done:\n+\tdisplay_error_msgs(o);\n+\to->show_all_errors = old_show_all_errors;\n+\tif (free_pattern_list)\n+\t\tclear_pattern_list(&pl);\n+\ttrace_performance_leave(\"update_sparsity\");\n+\treturn ret;\n+}\n+\n /* Here come the merge functions */\n \n static int reject_merge(const struct cache_entry *ce,\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex d3516267f36..5cf41ef5b53 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -88,6 +88,15 @@ struct unpack_trees_options {\n int unpack_trees(unsigned n, struct tree_desc *t,\n \t\t struct unpack_trees_options *options);\n \n+enum update_sparsity_result {\n+\tUPDATE_SPARSITY_SUCCESS = 0,\n+\tUPDATE_SPARSITY_WARNINGS = 1,\n+\tUPDATE_SPARSITY_INDEX_UPDATE_FAILURES = -1,\n+\tUPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES = -2\n+};\n+\n+enum update_sparsity_result update_sparsity(struct unpack_trees_options *options);\n+\n int verify_uptodate(const struct cache_entry *ce,\n \t\t struct unpack_trees_options *o);\n \n-- \ngitgitgadget\n\n"},{"id":"394143","messageId":"9040ac97175a8a5e525285b5c0ff41060621a72e.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 10/18] sparse-checkout: use new update_sparsity() function","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:53Z","receivedAt":"2020-03-27T00:49:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nRemove the equivalent of 'git read-tree -mu HEAD' in the sparse-checkout\ncodepaths for setting the SKIP_WORKTREE bits and instead use the new\nupdate_sparsity() function.\n\nNote that when an issue is hit, the error message splits 'error' and\n'Cannot update sparse checkout' on separate lines. For now, we use two\ngreps to find both pieces of the error message but subsequent commits\nwill clean up the messages reported to the user.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 40 ++++++++----------------------\n t/t1091-sparse-checkout-builtin.sh | 39 +++++++++++++++++++++--------\n 2 files changed, 39 insertions(+), 40 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex d102a9697fd..a55c60d7594 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -94,49 +94,35 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n static int update_working_directory(struct pattern_list *pl)\n {\n-\tint result = 0;\n+\tenum update_sparsity_result result;\n \tstruct unpack_trees_options o;\n \tstruct lock_file lock_file = LOCK_INIT;\n-\tstruct object_id oid;\n-\tstruct tree *tree;\n-\tstruct tree_desc t;\n \tstruct repository *r = the_repository;\n \n-\tif (repo_read_index_unmerged(r))\n-\t\tdie(_(\"you need to resolve your current index first\"));\n-\n-\tif (get_oid(\"HEAD\", &oid))\n-\t\treturn 0;\n-\n-\ttree = parse_tree_indirect(&oid);\n-\tparse_tree(tree);\n-\tinit_tree_desc(&t, tree->buffer, tree->size);\n-\n \tmemset(&o, 0, sizeof(o));\n \to.verbose_update = isatty(2);\n-\to.merge = 1;\n \to.update = 1;\n-\to.fn = oneway_merge;\n \to.head_idx = -1;\n \to.src_index = r->index;\n \to.dst_index = r->index;\n \to.skip_sparse_checkout = 0;\n \to.pl = pl;\n \n-\tresolve_undo_clear_index(r->index);\n \tsetup_work_tree();\n \n-\tcache_tree_free(&r->index->cache_tree);\n-\n \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n \n-\tcore_apply_sparse_checkout = 1;\n-\tresult = unpack_trees(1, &t, &o);\n+\tresult = update_sparsity(&o);\n \n-\tif (!result) {\n-\t\tprime_cache_tree(r, r->index, tree);\n+\tif (result == UPDATE_SPARSITY_WARNINGS)\n+\t\t/*\n+\t\t * We don't do any special handling of warnings from untracked\n+\t\t * files in the way or dirty entries that can't be removed.\n+\t\t */\n+\t\tresult = UPDATE_SPARSITY_SUCCESS;\n+\tif (result == UPDATE_SPARSITY_SUCCESS)\n \t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n-\t} else\n+\telse\n \t\trollback_lock_file(&lock_file);\n \n \treturn result;\n@@ -303,8 +289,6 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_sparse_checkout_init_options,\n@@ -559,8 +543,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \t};\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_sparse_checkout_set_options,\n@@ -576,8 +558,6 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tstruct strbuf match_all = STRBUF_INIT;\n \n \trepo_read_index(the_repository);\n-\trequire_clean_work_tree(the_repository,\n-\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n \n \tmemset(&pl, 0, sizeof(pl));\n \thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 8607a8e6d1a..a991e0a80d5 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -277,16 +277,23 @@ test_expect_success 'cone mode: add parent path' '\n \tcheck_files repo a deep folder1\n '\n \n-test_expect_success 'revert to old sparse-checkout on bad update' '\n+test_expect_success 'not-up-to-date does not block rest of sparsification' '\n \ttest_when_finished git -C repo sparse-checkout disable &&\n \ttest_when_finished git -C repo reset --hard &&\n \tgit -C repo sparse-checkout set deep &&\n+\n \techo update >repo/deep/deeper2/a &&\n \tcp repo/.git/info/sparse-checkout expect &&\n-\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n-\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n-\ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tcheck_files repo/deep a deeper1 deeper2\n+\ttest_write_lines \"!/deep/*/\" \"/deep/deeper1/\" >>expect &&\n+\n+\tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n+\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tcheck_files repo/deep a deeper1 deeper2 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n+\tcheck_files repo/deep/deeper1/deepest a &&\n+\tcheck_files repo/deep/deeper2 a\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -316,16 +323,28 @@ test_expect_success '.gitignore should not warn about cone mode' '\n \ttest_i18ngrep ! \"disabling cone patterns\" err\n '\n \n-test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n+test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status' '\n \tgit clone repo dirty &&\n \techo dirty >dirty/folder1/a &&\n-\ttest_must_fail git -C dirty sparse-checkout init &&\n-\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\ttest_must_fail git -C dirty sparse-checkout disable &&\n+\n+\tgit -C dirty sparse-checkout init 2>err &&\n+\ttest_i18ngrep \"error\" err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\n+\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n+\ttest_i18ngrep \"error\" err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_path_is_file dirty/folder1/a &&\n+\n+\tgit -C dirty sparse-checkout disable 2>err &&\n+\ttest_must_be_empty err &&\n+\n \tgit -C dirty reset --hard &&\n \tgit -C dirty sparse-checkout init &&\n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n-\tgit -C dirty sparse-checkout disable\n+\ttest_path_is_missing dirty/folder1/a &&\n+\tgit -C dirty sparse-checkout disable &&\n+\ttest_path_is_file dirty/folder1/a\n '\n \n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n-- \ngitgitgadget\n\n"},{"id":"394144","messageId":"e9c7e8ec46f284d44c5c7451f1175cbf3d9ce622.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 11/18] sparse-checkout: use improved unpack_trees porcelain messages","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:54Z","receivedAt":"2020-03-27T00:49:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nsetup_unpack_trees_porcelain() provides much improved error/warning\nmessages; instead of a message that assumes that there is only one path\nwith a given problem despite being used by code that intentionally is\ngrouping and showing errors together, it uses a message designed to be\nused with groups of paths. For example, this transforms\n\n error: Entry '\tfolder1/a\n\tfolder2/a\n ' not uptodate. Cannot update sparse checkout.\n\ninto\n\n error: Cannot update sparse checkout: the following entries are not up to date:\n\tfolder1/a\n\tfolder2/a\n\nIn the past the suboptimal messages were never actually triggered\nbecause we would error out if the working directory wasn't clean before\nwe even called unpack_trees(). The previous commit changed that,\nthough, so let's use the better error messages.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n builtin/sparse-checkout.c | 2 ++\n t/t1091-sparse-checkout-builtin.sh | 6 ++----\n 2 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex a55c60d7594..aa81199f85d 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -112,7 +112,9 @@ static int update_working_directory(struct pattern_list *pl)\n \n \trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n \n+\tsetup_unpack_trees_porcelain(&o, \"sparse-checkout\");\n \tresult = update_sparsity(&o);\n+\tclear_unpack_trees_porcelain(&o);\n \n \tif (result == UPDATE_SPARSITY_WARNINGS)\n \t\t/*\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a991e0a80d5..9bc65d32f07 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -328,12 +328,10 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"error\" err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n-\ttest_i18ngrep \"error\" err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n \ttest_path_is_file dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout disable 2>err &&\n-- \ngitgitgadget\n\n"},{"id":"394145","messageId":"90c514e220ba22e8fe63c33fd9c8cb9a67d21e64.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 13/18] unpack-trees: rename ERROR_* fields meant for warnings to WARNING_*","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:56Z","receivedAt":"2020-03-27T00:49:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWe want to treat issues with setting the SKIP_WORKTREE bit as a warning\nrather than an error; rename the enum values to reflect this intent as\na simple step towards that goal.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 12 ++++++------\n unpack-trees.h | 8 +++++---\n 2 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex eeac309e30e..2a2306c5c28 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -46,10 +46,10 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n \n-\t/* ERROR_SPARSE_NOT_UPTODATE_FILE */\n+\t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n-\t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n+\t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n };\n \n@@ -168,9 +168,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n \t\t_(\"Cannot update submodule:\\n%s\");\n \n-\tmsgs[ERROR_SPARSE_NOT_UPTODATE_FILE] =\n+\tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n+\tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n \n \topts->show_all_errors = 1;\n@@ -509,7 +509,7 @@ static int apply_sparse_checkout(struct index_state *istate,\n \t\tce->ce_flags &= ~CE_UPDATE;\n \t}\n \tif (was_skip_worktree && !ce_skip_worktree(ce)) {\n-\t\tif (verify_absent_sparse(ce, ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN, o))\n+\t\tif (verify_absent_sparse(ce, WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN, o))\n \t\t\treturn -1;\n \t\tce->ce_flags |= CE_UPDATE;\n \t}\n@@ -1875,7 +1875,7 @@ int verify_uptodate(const struct cache_entry *ce,\n static int verify_uptodate_sparse(const struct cache_entry *ce,\n \t\t\t\t struct unpack_trees_options *o)\n {\n-\treturn verify_uptodate_1(ce, o, ERROR_SPARSE_NOT_UPTODATE_FILE);\n+\treturn verify_uptodate_1(ce, o, WARNING_SPARSE_NOT_UPTODATE_FILE);\n }\n \n /*\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 3e996a6c0a9..aac1ad4b014 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -23,9 +23,11 @@ enum unpack_trees_error_types {\n \tERROR_WOULD_LOSE_UNTRACKED_REMOVED,\n \tERROR_BIND_OVERLAP,\n \tERROR_WOULD_LOSE_SUBMODULE,\n-\tERROR_SPARSE_NOT_UPTODATE_FILE,\n-\tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tNB_UNPACK_TREES_ERROR_TYPES\n+\n+\tWARNING_SPARSE_NOT_UPTODATE_FILE,\n+\tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n+\n+\tNB_UNPACK_TREES_ERROR_TYPES,\n };\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"394146","messageId":"5d9e8a9d27e13670b56fe05dc36c066ec766b390.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 16/18] unpack-trees: provide warnings on sparse updates for unmerged paths too","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:59Z","receivedAt":"2020-03-27T00:49:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nWhen sparse-checkout runs to update the list of sparsity patterns, it\ngives warnings if it can't remove paths from the working tree because\nthose files have dirty changes. Add a similar warning for unmerged\npaths as well.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 25 +++++++++++++++++++++++++\n unpack-trees.c | 30 ++++++++++++++++++++++++++++++\n unpack-trees.h | 1 +\n 3 files changed, 56 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex afbde89e605..8e2976bc7b8 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -345,6 +345,31 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \ttest_path_is_file dirty/folder1/a\n '\n \n+test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged status' '\n+\tgit clone repo unmerged &&\n+\n+\tcat >input <<-EOF &&\n+\t0 0000000000000000000000000000000000000000\tfolder1/a\n+\t100644 $(git -C unmerged rev-parse HEAD:folder1/a) 1\tfolder1/a\n+\tEOF\n+\tgit -C unmerged update-index --index-info err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\ttest_path_is_file dirty/folder1/a &&\n+\n+\tgit -C unmerged sparse-checkout disable 2>err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C unmerged reset --hard &&\n+\tgit -C unmerged sparse-checkout init &&\n+\tgit -C unmerged sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\tgit -C unmerged sparse-checkout disable\n+'\n+\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 484d30a53a7..dec044339df 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -52,6 +52,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Path '%s' not uptodate; will not remove from working tree.\",\n \n+\t/* WARNING_SPARSE_UNMERGED_FILE */\n+\t\"Path '%s' unmerged; will not remove from working tree.\",\n+\n \t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n \t\"Path '%s' already present; will not overwrite with sparse update.\",\n };\n@@ -173,6 +176,8 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \n \tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"The following paths are not up to date and were left despite sparse patterns:\\n%s\");\n+\tmsgs[WARNING_SPARSE_UNMERGED_FILE] =\n+\t\t_(\"The following paths are unmerged and were left despite sparse patterns:\\n%s\");\n \tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n \t\t_(\"The following paths were already present and thus not updated despite sparse patterns:\\n%s\");\n \n@@ -548,6 +553,23 @@ static int apply_sparse_checkout(struct index_state *istate,\n \treturn 0;\n }\n \n+static int warn_conflicted_path(struct index_state *istate,\n+\t\t\t\tint i,\n+\t\t\t\tstruct unpack_trees_options *o)\n+{\n+\tchar *conflicting_path = istate->cache[i]->name;\n+\tint count = 0;\n+\n+\tadd_rejected_path(o, WARNING_SPARSE_UNMERGED_FILE, conflicting_path);\n+\n+\t/* Find out how many higher stage entries at same path */\n+\twhile (++count < istate->cache_nr &&\n+\t !strcmp(conflicting_path,\n+\t\t istate->cache[i+count]->name))\n+\t\t/* do nothing */;\n+\treturn count;\n+}\n+\n static inline int call_unpack_fn(const struct cache_entry * const *src,\n \t\t\t\t struct unpack_trees_options *o)\n {\n@@ -1793,6 +1815,14 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n \tfor (i = 0; i < o->src_index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = o->src_index->cache[i];\n \n+\n+\t\tif (ce_stage(ce)) {\n+\t\t\t/* -1 because for loop will increment by 1 */\n+\t\t\ti += warn_conflicted_path(o->src_index, i, o) - 1;\n+\t\t\tret = UPDATE_SPARSITY_WARNINGS;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (apply_sparse_checkout(o->src_index, ce, o))\n \t\t\tret = UPDATE_SPARSITY_WARNINGS;\n \ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex dae948205f9..0f7424aec61 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -27,6 +27,7 @@ enum unpack_trees_error_types {\n \tNB_UNPACK_TREES_ERROR_TYPES,\n \n \tWARNING_SPARSE_NOT_UPTODATE_FILE,\n+\tWARNING_SPARSE_UNMERGED_FILE,\n \tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n \n \tNB_UNPACK_TREES_WARNING_TYPES,\n-- \ngitgitgadget\n\n"},{"id":"394147","messageId":"93dbdd78acf5f703126695aff0e27024a6b725c0.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 18/18] sparse-checkout: provide a new reapply subcommand","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:49:01Z","receivedAt":"2020-03-27T00:49:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIf commands like merge or rebase materialize files as part of their work,\nor a previous sparse-checkout command failed to update individual files\ndue to dirty changes, users may want a command to simply 'reapply' the\nsparsity rules. Provide one.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n Documentation/git-sparse-checkout.txt | 10 +++++++\n builtin/sparse-checkout.c | 10 ++++++-\n t/t1091-sparse-checkout-builtin.sh | 41 +++++++++++++++++++++++++++\n 3 files changed, 60 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex c0342e53938..1a3ace60820 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -70,6 +70,16 @@ C-style quoted strings.\n \t`core.sparseCheckoutCone` is enabled, the given patterns are interpreted\n \tas directory names as in the 'set' subcommand.\n \n+'reapply::\n+\tReapply the sparsity pattern rules to paths in the working tree.\n+\tCommands like merge or rebase can materialize paths to do their\n+\twork (e.g. in order to show you a conflict), and other\n+\tsparse-checkout commands might fail to sparsify an individual file\n+\t(e.g. because it has unstaged changes or conflicts). In such\n+\tcases, it can make sense to run `git sparse-checkout reapply` later\n+\tafter cleaning up affected paths (e.g. resolving conflicts, undoing\n+\tor committing changes, etc.).\n+\n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n \tworking directory to include all files. Leaves the sparse-checkout\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex aa81199f85d..95d08824172 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -18,7 +18,7 @@\n static const char *empty_base = \"\";\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list|set|add|disable) \"),\n+\tN_(\"git sparse-checkout (init|list|set|add|reapply|disable) \"),\n \tNULL\n };\n \n@@ -554,6 +554,12 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix,\n \treturn modify_pattern_list(argc, argv, m);\n }\n \n+static int sparse_checkout_reapply(int argc, const char **argv)\n+{\n+\trepo_read_index(the_repository);\n+\treturn update_working_directory(NULL);\n+}\n+\n static int sparse_checkout_disable(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -603,6 +609,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, REPLACE);\n \t\tif (!strcmp(argv[0], \"add\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix, ADD);\n+\t\tif (!strcmp(argv[0], \"reapply\"))\n+\t\t\treturn sparse_checkout_reapply(argc, argv);\n \t\tif (!strcmp(argv[0], \"disable\"))\n \t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 8e2976bc7b8..dee99eeec30 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -370,6 +370,47 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with unmerged stat\n \tgit -C unmerged sparse-checkout disable\n '\n \n+test_expect_success 'sparse-checkout reapply' '\n+\tgit clone repo tweak &&\n+\n+\techo dirty >tweak/deep/deeper2/a &&\n+\n+\tcat >input <<-EOF &&\n+\t0 0000000000000000000000000000000000000000\tfolder1/a\n+\t100644 $(git -C tweak rev-parse HEAD:folder1/a) 1\tfolder1/a\n+\tEOF\n+\tgit -C tweak update-index --index-info err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C tweak sparse-checkout set folder2 deep/deeper1 2>err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\n+\tgit -C tweak sparse-checkout reapply 2>err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n+\ttest_path_is_file tweak/deep/deeper2/a &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\ttest_path_is_file tweak/folder1/a &&\n+\n+\tgit -C tweak checkout HEAD deep/deeper2/a &&\n+\tgit -C tweak sparse-checkout reapply 2>err &&\n+\ttest_i18ngrep ! \"warning.*The following paths are not up to date\" err &&\n+\ttest_path_is_missing tweak/deep/deeper2/a &&\n+\ttest_i18ngrep \"warning.*The following paths are unmerged\" err &&\n+\ttest_path_is_file tweak/folder1/a &&\n+\n+\tgit -C tweak add folder1/a &&\n+\tgit -C tweak sparse-checkout reapply 2>err &&\n+\ttest_must_be_empty err &&\n+\ttest_path_is_missing tweak/deep/deeper2/a &&\n+\ttest_path_is_missing tweak/folder1/a &&\n+\n+\tgit -C tweak sparse-checkout disable\n+'\n+\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \trm repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init --cone &&\n-- \ngitgitgadget\n"},{"id":"394148","messageId":"74e13ccee408757beb8ae3bb1dddea8d34ef56bb.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 15/18] unpack-trees: make sparse path messages sound like warnings","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:58Z","receivedAt":"2020-03-27T00:49:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe messages for problems with sparse paths are phrased as errors that\ncause the operation to abort, even though we are not making the\noperation abort. Reword the messages to make sense in their new\ncontext.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 6 +++---\n unpack-trees.c | 8 ++++----\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ed5e9059969..afbde89e605 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -288,7 +288,7 @@ test_expect_success 'not-up-to-date does not block rest of sparsification' '\n \n \tgit -C repo sparse-checkout set deep/deeper1 2>err &&\n \n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"The following paths are not up to date\" err &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo/deep a deeper1 deeper2 &&\n \tcheck_files repo/deep/deeper1 a deepest &&\n@@ -328,10 +328,10 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n-\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*The following paths are not up to date\" err &&\n \ttest_path_is_file dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout disable 2>err &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex f9a5626a670..484d30a53a7 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -50,10 +50,10 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t\"\",\n \n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n-\t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n+\t\"Path '%s' not uptodate; will not remove from working tree.\",\n \n \t/* WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN */\n-\t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n+\t\"Path '%s' already present; will not overwrite with sparse update.\",\n };\n \n #define ERRORMSG(o,type) \\\n@@ -172,9 +172,9 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t\t_(\"Cannot update submodule:\\n%s\");\n \n \tmsgs[WARNING_SPARSE_NOT_UPTODATE_FILE] =\n-\t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n+\t\t_(\"The following paths are not up to date and were left despite sparse patterns:\\n%s\");\n \tmsgs[WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN] =\n-\t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n+\t\t_(\"The following paths were already present and thus not updated despite sparse patterns:\\n%s\");\n \n \topts->show_all_errors = 1;\n \t/* rejected paths may not have a static buffer */\n-- \ngitgitgadget\n\n"},{"id":"394151","messageId":"5676e890f65d3b804e2642c7c57532cbcc1d6673.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 14/18] unpack-trees: split display_error_msgs() into two","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:57Z","receivedAt":"2020-03-27T00:49:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndisplay_error_msgs() is never called to show messages of both ERROR_*\nand WARNING_* types at the same time; it is instead called multiple\ntimes, separately for each type. Since we want to display these types\ndifferently, make two slightly different versions of this function.\n\nA subsequent commit will further modify unpack_trees() and how it calls\nthe new display_warning_msgs().\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n t/t1091-sparse-checkout-builtin.sh | 4 +--\n unpack-trees.c | 50 +++++++++++++++++++++++++-----\n unpack-trees.h | 8 +++--\n 3 files changed, 49 insertions(+), 13 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9bc65d32f07..ed5e9059969 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -328,10 +328,10 @@ test_expect_success 'sparse-checkout (init|set|disable) warns with dirty status'\n \techo dirty >dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout init 2>err &&\n-\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \n \tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* 2>err &&\n-\ttest_i18ngrep \"error.*Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"warning.*Cannot update sparse checkout\" err &&\n \ttest_path_is_file dirty/folder1/a &&\n \n \tgit -C dirty sparse-checkout disable 2>err &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 2a2306c5c28..f9a5626a670 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -24,7 +24,7 @@\n * situation better. See how \"git checkout\" and \"git merge\" replaces\n * them using setup_unpack_trees_porcelain(), for example.\n */\n-static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n+static const char *unpack_plumbing_errors[NB_UNPACK_TREES_WARNING_TYPES] = {\n \t/* ERROR_WOULD_OVERWRITE */\n \t\"Entry '%s' would be overwritten by merge. Cannot merge.\",\n \n@@ -46,6 +46,9 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_WOULD_LOSE_SUBMODULE */\n \t\"Submodule '%s' cannot checkout new HEAD.\",\n \n+\t/* NB_UNPACK_TREES_ERROR_TYPES; just a meta value */\n+\t\"\",\n+\n \t/* WARNING_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n@@ -222,7 +225,7 @@ static int add_rejected_path(struct unpack_trees_options *o,\n \n \t/*\n \t * Otherwise, insert in a list for future display by\n-\t * display_error_msgs()\n+\t * display_(error|warning)_msgs()\n \t */\n \tstring_list_append(&o->unpack_rejects[e], path);\n \treturn -1;\n@@ -233,13 +236,16 @@ static int add_rejected_path(struct unpack_trees_options *o,\n */\n static void display_error_msgs(struct unpack_trees_options *o)\n {\n-\tint e, i;\n-\tint something_displayed = 0;\n+\tint e;\n+\tunsigned error_displayed = 0;\n \tfor (e = 0; e < NB_UNPACK_TREES_ERROR_TYPES; e++) {\n \t\tstruct string_list *rejects = &o->unpack_rejects[e];\n+\n \t\tif (rejects->nr > 0) {\n+\t\t\tint i;\n \t\t\tstruct strbuf path = STRBUF_INIT;\n-\t\t\tsomething_displayed = 1;\n+\n+\t\t\terror_displayed = 1;\n \t\t\tfor (i = 0; i < rejects->nr; i++)\n \t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n \t\t\terror(ERRORMSG(o, e), super_prefixed(path.buf));\n@@ -247,10 +253,36 @@ static void display_error_msgs(struct unpack_trees_options *o)\n \t\t}\n \t\tstring_list_clear(rejects, 0);\n \t}\n-\tif (something_displayed)\n+\tif (error_displayed)\n \t\tfprintf(stderr, _(\"Aborting\\n\"));\n }\n \n+/*\n+ * display all the warning messages stored in a nice way\n+ */\n+static void display_warning_msgs(struct unpack_trees_options *o)\n+{\n+\tint e;\n+\tunsigned warning_displayed = 0;\n+\tfor (e = NB_UNPACK_TREES_ERROR_TYPES + 1;\n+\t e < NB_UNPACK_TREES_WARNING_TYPES; e++) {\n+\t\tstruct string_list *rejects = &o->unpack_rejects[e];\n+\n+\t\tif (rejects->nr > 0) {\n+\t\t\tint i;\n+\t\t\tstruct strbuf path = STRBUF_INIT;\n+\n+\t\t\twarning_displayed = 1;\n+\t\t\tfor (i = 0; i < rejects->nr; i++)\n+\t\t\t\tstrbuf_addf(&path, \"\\t%s\\n\", rejects->items[i].string);\n+\t\t\twarning(ERRORMSG(o, e), super_prefixed(path.buf));\n+\t\t\tstrbuf_release(&path);\n+\t\t}\n+\t\tstring_list_clear(rejects, 0);\n+\t}\n+\tif (warning_displayed)\n+\t\tfprintf(stderr, _(\"After fixing the above paths, you may want to run `git sparse-checkout reapply`.\\n\"));\n+}\n static int check_submodule_move_head(const struct cache_entry *ce,\n \t\t\t\t const char *old_id,\n \t\t\t\t const char *new_id,\n@@ -1705,8 +1737,10 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \treturn ret;\n \n return_failed:\n-\tif (o->show_all_errors)\n+\tif (o->show_all_errors) {\n \t\tdisplay_error_msgs(o);\n+\t\tdisplay_warning_msgs(o);\n+\t}\n \tmark_all_ce_unused(o->src_index);\n \tret = unpack_failed(o, NULL);\n \tif (o->exiting_early)\n@@ -1783,7 +1817,7 @@ enum update_sparsity_result update_sparsity(struct unpack_trees_options *o)\n \t\tret = UPDATE_SPARSITY_WORKTREE_UPDATE_FAILURES;\n \n done:\n-\tdisplay_error_msgs(o);\n+\tdisplay_warning_msgs(o);\n \to->show_all_errors = old_show_all_errors;\n \tif (free_pattern_list)\n \t\tclear_pattern_list(&pl);\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex aac1ad4b014..dae948205f9 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -24,10 +24,12 @@ enum unpack_trees_error_types {\n \tERROR_BIND_OVERLAP,\n \tERROR_WOULD_LOSE_SUBMODULE,\n \n+\tNB_UNPACK_TREES_ERROR_TYPES,\n+\n \tWARNING_SPARSE_NOT_UPTODATE_FILE,\n \tWARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN,\n \n-\tNB_UNPACK_TREES_ERROR_TYPES,\n+\tNB_UNPACK_TREES_WARNING_TYPES,\n };\n \n /*\n@@ -66,13 +68,13 @@ struct unpack_trees_options {\n \tstruct dir_struct *dir;\n \tstruct pathspec *pathspec;\n \tmerge_fn_t fn;\n-\tconst char *msgs[NB_UNPACK_TREES_ERROR_TYPES];\n+\tconst char *msgs[NB_UNPACK_TREES_WARNING_TYPES];\n \tstruct argv_array msgs_to_free;\n \t/*\n \t * Store error messages in an array, each case\n \t * corresponding to a error message type\n \t */\n-\tstruct string_list unpack_rejects[NB_UNPACK_TREES_ERROR_TYPES];\n+\tstruct string_list unpack_rejects[NB_UNPACK_TREES_WARNING_TYPES];\n \n \tint head_idx;\n \tint merge_size;\n-- \ngitgitgadget\n\n"},{"id":"394149","messageId":"844306c3e86ef67591cc086decb2b760e7d710a3.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 17/18] unpack-trees: failure to set SKIP_WORKTREE bits always just a warning","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:49:00Z","receivedAt":"2020-03-27T00:49:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSetting and clearing of the SKIP_WORKTREE bit is not only done when\nusers run 'sparse-checkout'; other commands such as 'checkout' also run\nthrough unpack_trees() which has logic for handling this special bit.\nAs such, we need to consider how they handle special cases. A couple\ncomparison points should help explain the rationale for changing how\nunpack_trees() handles these bits:\n\n Ignoring sparse checkouts for a moment, if you are switching\n branches and have dirty changes, it is only considered an error that\n will prevent the branch switching from being successful if the dirty\n file happens to be one of the paths with different contents.\n\n SKIP_WORKTREE has always been considered advisory; for example, if\n rebase or merge need or even want to materialize a path as part of\n their work, they have always been allowed to do so regardless of the\n SKIP_WORKTREE setting. This has been used for unmerged paths, but\n it was often used for paths it wasn't needed just because it made\n the code simpler. It was a best-effort consideration, and when it\n materialized paths contrary to the SKIP_WORKTREE setting, it was\n never required to even print a warning message.\n\nIn the past if you trying to run e.g. 'git checkout' and:\n 1) you had a path that was materialized and had some dirty changes\n 2) the path was listed in $GITDIR/info/sparse-checkout\n 3) this path did not different between the current and target branches\nthen despite the comparison points above, the inability to set\nSKIP_WORKTREE was treated as a *hard* error that would abort the\ncheckout operation. This is completely inconsistent with how\nSKIP_WORKTREE is handled elsewhere, and rather annoying for users as\nleaving the paths materialized in the working copy (with a simple\nwarning) should present no problem at all.\n\nDowngrade any errors from inability to toggle the SKIP_WORKTREE bit to a\nwarning and allow the operations to continue.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n t/t1011-read-tree-sparse-checkout.sh | 11 +++++-----\n t/t2018-checkout-branch.sh | 22 ++++++++++++++++++++\n unpack-trees.c | 31 ++++++++++++++--------------\n 3 files changed, 43 insertions(+), 21 deletions(-)\n\ndiff --git a/t/t1011-read-tree-sparse-checkout.sh b/t/t1011-read-tree-sparse-checkout.sh\nindex eb44bafb593..63223e13bd1 100755\n--- a/t/t1011-read-tree-sparse-checkout.sh\n+++ b/t/t1011-read-tree-sparse-checkout.sh\n@@ -233,18 +233,19 @@ test_expect_success 'read-tree --reset removes outside worktree' '\n \ttest_must_be_empty result\n '\n \n-test_expect_success 'print errors when failed to update worktree' '\n+test_expect_success 'print warnings when some worktree updates disabled' '\n \techo sub >.git/info/sparse-checkout &&\n \tgit checkout -f init &&\n \tmkdir sub &&\n \ttouch sub/added sub/addedtoo &&\n-\ttest_must_fail git checkout top 2>actual &&\n+\t# Use -q to suppress \"Previous HEAD position\" and \"Head is now at\" msgs\n+\tgit checkout -q top 2>actual &&\n \tcat >expected <<\\EOF &&\n-error: The following untracked working tree files would be overwritten by checkout:\n+warning: The following paths were already present and thus not updated despite sparse patterns:\n \tsub/added\n \tsub/addedtoo\n-Please move or remove them before you switch branches.\n-Aborting\n+\n+After fixing the above paths, you may want to run `git sparse-checkout reapply`.\n EOF\n \ttest_i18ncmp expected actual\n '\ndiff --git a/t/t2018-checkout-branch.sh b/t/t2018-checkout-branch.sh\nindex bbca7ef8da6..21583154d8e 100755\n--- a/t/t2018-checkout-branch.sh\n+++ b/t/t2018-checkout-branch.sh\n@@ -238,4 +238,26 @@ test_expect_success 'checkout -b after clone --no-checkout does a checkout of HE\n \ttest_path_is_file dest/a.t\n '\n \n+test_expect_success 'checkout -b to a new branch preserves mergeable changes despite sparse-checkout' '\n+\ttest_when_finished \"\n+\t\tgit reset --hard &&\n+\t\tgit checkout branch1-scratch &&\n+\t\ttest_might_fail git branch -D branch3 &&\n+\t\tgit config core.sparseCheckout false &&\n+\t\trm .git/info/sparse-checkout\" &&\n+\n+\ttest_commit file2 &&\n+\n+\techo stuff >>file1 &&\n+\techo file2 >.git/info/sparse-checkout &&\n+\tgit config core.sparseCheckout true &&\n+\n+\tCURHEAD=$(git rev-parse HEAD) &&\n+\tdo_checkout branch3 $CURHEAD &&\n+\n+\techo file1 >expect &&\n+\tgit diff --name-only >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex dec044339df..b43f3e775ad 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1701,23 +1701,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\t * correct CE_NEW_SKIP_WORKTREE\n \t\t\t */\n \t\t\tif (ce->ce_flags & CE_ADDED &&\n-\t\t\t verify_absent(ce, ERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN, o)) {\n-\t\t\t\tif (!o->show_all_errors)\n-\t\t\t\t\tgoto return_failed;\n-\t\t\t\tret = -1;\n-\t\t\t}\n+\t\t\t verify_absent(ce, WARNING_SPARSE_ORPHANED_NOT_OVERWRITTEN, o))\n+\t\t\t\tret = 1;\n+\n+\t\t\tif (apply_sparse_checkout(&o->result, ce, o))\n+\t\t\t\tret = 1;\n \n-\t\t\tif (apply_sparse_checkout(&o->result, ce, o)) {\n-\t\t\t\tif (!o->show_all_errors)\n-\t\t\t\t\tgoto return_failed;\n-\t\t\t\tret = -1;\n-\t\t\t}\n \t\t\tif (!ce_skip_worktree(ce))\n \t\t\t\tempty_worktree = 0;\n-\n \t\t}\n-\t\tif (ret < 0)\n-\t\t\tgoto return_failed;\n \t\t/*\n \t\t * Sparse checkout is meant to narrow down checkout area\n \t\t * but it does not make sense to narrow down to empty working\n@@ -1728,6 +1720,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t\tret = unpack_failed(o, \"Sparse checkout leaves no entry on working directory\");\n \t\t\tgoto done;\n \t\t}\n+\t\tif (ret == 1) {\n+\t\t\t/*\n+\t\t\t * Inability to sparsify or de-sparsify individual\n+\t\t\t * paths is not an error, but just a warning.\n+\t\t\t */\n+\t\t\tif (o->show_all_errors)\n+\t\t\t\tdisplay_warning_msgs(o);\n+\t\t\tret = 0;\n+\t\t}\n \t}\n \n \tret = check_updates(o, &o->result) ? (-2) : 0;\n@@ -1759,10 +1760,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \treturn ret;\n \n return_failed:\n-\tif (o->show_all_errors) {\n+\tif (o->show_all_errors)\n \t\tdisplay_error_msgs(o);\n-\t\tdisplay_warning_msgs(o);\n-\t}\n \tmark_all_ce_unused(o->src_index);\n \tret = unpack_failed(o, NULL);\n \tif (o->exiting_early)\n-- \ngitgitgadget\n\n"},{"id":"394150","messageId":"c3741107a9d59ca8c97f1f4cb4768061fd1d2b2d.1585270142.git.gitgitgadget@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"[PATCH v3 12/18] unpack-trees: move ERROR_WOULD_LOSE_SUBMODULE earlier","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-27T00:48:55Z","receivedAt":"2020-03-27T00:49:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA minor change, but we want to convert the sparse messages to warnings\nand this allows us to group warnings and errors.\n\nReviewed-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n unpack-trees.c | 12 ++++++------\n unpack-trees.h | 2 +-\n 2 files changed, 7 insertions(+), 7 deletions(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a5bc0a3a16d..eeac309e30e 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -43,15 +43,14 @@ static const char *unpack_plumbing_errors[NB_UNPACK_TREES_ERROR_TYPES] = {\n \t/* ERROR_BIND_OVERLAP */\n \t\"Entry '%s' overlaps with '%s'. Cannot bind.\",\n \n+\t/* ERROR_WOULD_LOSE_SUBMODULE */\n+\t\"Submodule '%s' cannot checkout new HEAD.\",\n+\n \t/* ERROR_SPARSE_NOT_UPTODATE_FILE */\n \t\"Entry '%s' not uptodate. Cannot update sparse checkout.\",\n \n \t/* ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN */\n \t\"Working tree file '%s' would be overwritten by sparse checkout update.\",\n-\n-\n-\t/* ERROR_WOULD_LOSE_SUBMODULE */\n-\t\"Submodule '%s' cannot checkout new HEAD.\",\n };\n \n #define ERRORMSG(o,type) \\\n@@ -166,12 +165,13 @@ void setup_unpack_trees_porcelain(struct unpack_trees_options *opts,\n \t */\n \tmsgs[ERROR_BIND_OVERLAP] = _(\"Entry '%s' overlaps with '%s'. Cannot bind.\");\n \n+\tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n+\t\t_(\"Cannot update submodule:\\n%s\");\n+\n \tmsgs[ERROR_SPARSE_NOT_UPTODATE_FILE] =\n \t\t_(\"Cannot update sparse checkout: the following entries are not up to date:\\n%s\");\n \tmsgs[ERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN] =\n \t\t_(\"The following working tree files would be overwritten by sparse checkout update:\\n%s\");\n-\tmsgs[ERROR_WOULD_LOSE_SUBMODULE] =\n-\t\t_(\"Cannot update submodule:\\n%s\");\n \n \topts->show_all_errors = 1;\n \t/* rejected paths may not have a static buffer */\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex 5cf41ef5b53..3e996a6c0a9 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -22,9 +22,9 @@ enum unpack_trees_error_types {\n \tERROR_WOULD_LOSE_UNTRACKED_OVERWRITTEN,\n \tERROR_WOULD_LOSE_UNTRACKED_REMOVED,\n \tERROR_BIND_OVERLAP,\n+\tERROR_WOULD_LOSE_SUBMODULE,\n \tERROR_SPARSE_NOT_UPTODATE_FILE,\n \tERROR_WOULD_LOSE_ORPHANED_OVERWRITTEN,\n-\tERROR_WOULD_LOSE_SUBMODULE,\n \tNB_UNPACK_TREES_ERROR_TYPES\n };\n \n-- \ngitgitgadget\n\n"},{"id":"394169","messageId":"9f33833a-43e3-8878-859d-1c116a54de97@gmail.com","threadId":"52995","inReplyTo":"pull.726.v3.git.git.1585270142.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/18] Sparse checkout improvements -- improved sparsity updating","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-27T13:22:07Z","receivedAt":"2020-03-27T13:22:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/26/2020 8:48 PM, Elijah Newren via GitGitGadget wrote:\n> This series provides a replacement for the sparsity updating in\n> sparse-checkout that is based on the logic from git read-tree -mu HEAD. The\n> most important bit is patch 9 and its lengthy commit message explaining the\n> current state and rationale for most the series, though patches 16 and 17\n> have additional related directions and rationale for the series. Those three\n> patches are the most important to review.\n> \n> Changes since v2:\n> \n> * addressed Stolee's cleanups and added his Reviewed-by to the series\n> (hopefully the next two changes don't invalidate that)\n\nThe other changes below look good to me. My Reviewed-by stands.\n\n> * added a test for the new 'reapply' subcommand (noticed it was missing\n> with Stolee's test coverage report)\n> * fix a minor issue with two of the other tests I modified -- when I'm\n> grepping through stderr for a command, I need to make sure to actually\n> record stderr from that command (otherwise my grep is looking through the\n> stderr of a previous command that was recorded). Also, since I messed\n> this up, I added one or two more sanity checks while I was at it.\nThanks!\n\n-Stolee\n"}]}