{"thread":{"id":"52638","subject":"[PATCH 0/8] Harden the sparse-checkout builtin","startedAt":"2020-01-14T19:26:07Z","lastAt":"2020-02-09T17:27:19Z","messageCount":82,"participants":["Derrick Stolee via GitGitGadget","Taylor Blau","Derrick Stolee","Jeff King","Junio C Hamano","Jeff King via GitGitGadget","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"389764","messageId":"pull.513.git.1579029962.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":null,"subject":"[PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:54Z","receivedAt":"2020-01-14T19:26:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/sparse-list-in-cone-mode.\n\nThis series attempts to clean up some rough edges in the sparse-checkout\nfeature, especially around the cone mode.\n\nUnfortunately, after the v2.25.0 release, we noticed an issue with the \"git\nclone --sparse\" option when using a URL instead of a local path. This is\nfixed and properly tested here.\n\nAlso, let's improve Git's response to these more complicated scenarios:\n\n 1. Running \"git sparse-checkout init\" in a worktree would complain because\n the \"info\" dir doesn't exist.\n 2. Tracked paths that include \"*\" and \"\" in their filenames.\n 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n as \"*\" anywhere or \"\" in the wrong place, then we should respond\n appropriately. That is: warn that the patterns are not cone-mode, then\n revert to the old logic.\n\nThanks, -Stolee\n\nDerrick Stolee (8):\n t1091: use check_files to reduce boilerplate\n sparse-checkout: create leading directories\n clone: fix --sparse option with URLs\n sparse-checkout: cone mode does not recognize \"**\"\n sparse-checkout: detect short patterns\n sparse-checkout: warn on incorrect '*' in patterns\n sparse-checkout: properly match escaped characters\n sparse-checkout: write escaped patterns in cone mode\n\n builtin/clone.c | 2 +-\n builtin/sparse-checkout.c | 52 ++++-\n dir.c | 69 ++++++-\n dir.h | 1 +\n t/t1091-sparse-checkout-builtin.sh | 320 ++++++++++++++++-------------\n 5 files changed, 296 insertions(+), 148 deletions(-)\n\n\nbase-commit: 4fd683b6a35eabd23dd5183da7f654a1e1f00325\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-513%2Fderrickstolee%2Fsparse-harden-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-513/derrickstolee/sparse-harden-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/513\n-- \ngitgitgadget\n"},{"id":"389765","messageId":"53a266f9aab5f4331c35b3ff15fb0628e2131c56.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 2/8] sparse-checkout: create leading directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:56Z","receivedAt":"2020-01-14T19:26:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git init' command creates the \".git/info\" directory and fills it\nwith some default files. However, 'git worktree add' does not create\nthe info directory for that worktree. This causes a problem when running\n\"git sparse-checkout init\" inside a worktree. While care was taken to\nallow the sparse-checkout config to be specific to a worktree, this\ninitialization was untested.\n\nSafely create the leading directories for the sparse-checkout file. This\nis the safest thing to do even without worktrees, as a user could delete\ntheir \".git/info\" directory and expect Git to recover safely.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 4 ++++\n t/t1091-sparse-checkout-builtin.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b3bed891cb..3cee8ab46e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -199,6 +199,10 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tint result;\n \n \tsparse_filename = get_sparse_checkout_filename();\n+\n+\tif (safe_create_leading_directories(sparse_filename))\n+\t\tdie(_(\"failed to create directory for sparse-checkout file\"));\n+\n \tfd = hold_lock_file_for_update(&lk, sparse_filename,\n \t\t\t\t LOCK_DIE_ON_ERROR);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 20caefe155..37365dc668 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -295,4 +295,14 @@ test_expect_success 'interaction with submodules' '\n \tcheck_files super/modules/child \"a deep folder1 folder2\"\n '\n \n+test_expect_success 'different sparse-checkouts with worktrees' '\n+\tgit -C repo worktree add --detach ../worktree &&\n+\tcheck_files worktree \"a deep folder1 folder2\" &&\n+\tgit -C worktree sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set folder1 &&\n+\tgit -C worktree sparse-checkout set deep/deeper1 &&\n+\tcheck_files repo \"a folder1\" &&\n+\tcheck_files worktree \"a deep\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"389766","messageId":"9f7791ae5ee8df9d84d25407a014fb2c63e09b38.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 1/8] t1091: use check_files to reduce boilerplate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:55Z","receivedAt":"2020-01-14T19:26:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen testing the sparse-checkout feature, we need to compare the\ncontents of the working-directory against some expected output.\nUsing here-docs was useful in the beginning, but became repetetive\nas the test script grew.\n\nCreate a check_files helper to make the tests simpler and easier\nto extend. It also reduces instances of bad here-doc whitespace.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 215 ++++++++++-------------------\n 1 file changed, 71 insertions(+), 144 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ff7f8f7a1f..20caefe155 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -12,6 +12,13 @@ list_files() {\n \t(cd \"$1\" && printf '%s\\n' *)\n }\n \n+check_files() {\n+\tDIR=$1\n+\tprintf \"%s\\n\" $2 >expect &&\n+\tlist_files $DIR >actual &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'setup' '\n \tgit init repo &&\n \t(\n@@ -39,11 +46,11 @@ test_expect_success 'git sparse-checkout list (empty)' '\n \n test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n-\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n-\t\t/folder1/*\n-\t\t/deep/\n-\t\t**/a\n-\t\t!*bin*\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/folder1/*\n+\t/deep/\n+\t**/a\n+\t!*bin*\n \tEOF\n \tcp repo/.git/info/sparse-checkout expect &&\n \tgit -C repo sparse-checkout list >list &&\n@@ -52,22 +59,20 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \n test_expect_success 'git sparse-checkout init' '\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n-\tlist_files repo >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a\n '\n \n test_expect_success 'git sparse-checkout list after init' '\n \tgit -C repo sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual\n '\n@@ -75,32 +80,24 @@ test_expect_success 'git sparse-checkout list after init' '\n test_expect_success 'init with existing sparse-checkout' '\n \techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'clone --sparse' '\n \tgit clone --sparse repo clone &&\n \tgit -C clone sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n-\tlist_files clone >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files clone a\n '\n \n test_expect_success 'set enables config' '\n@@ -119,41 +116,29 @@ test_expect_success 'set enables config' '\n \n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n-\t\t/folder2/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n+\t/folder2/\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: match patterns' '\n@@ -162,13 +147,7 @@ test_expect_success 'cone mode: match patterns' '\n \tgit -C repo read-tree -mu HEAD 2>err &&\n \ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: warn on bad pattern' '\n@@ -185,14 +164,7 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_path_is_file repo/.git/info/sparse-checkout &&\n \tgit -C repo config --list >config &&\n \ttest_must_fail git config core.sparseCheckout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a deep folder1 folder2\"\n '\n \n test_expect_success 'cone mode: init and set' '\n@@ -204,52 +176,31 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir &&\n \tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n \ttest_must_be_empty err &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep/deeper1 >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeepest\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n-\t\t!/deep/*/\n-\t\t/deep/deeper1/\n-\t\t!/deep/deeper1/*/\n-\t\t/deep/deeper1/deepest/\n+\tcheck_files repo \"a deep\" &&\n+\tcheck_files repo/deep \"a deeper1\" &&\n+\tcheck_files repo/deep/deeper1 \"a deepest\" &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t!/deep/*/\n+\t/deep/deeper1/\n+\t!/deep/deeper1/*/\n+\t/deep/deeper1/deepest/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\tlist_files repo >dir &&\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: list' '\n-\tcat >expect <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tcat >expect <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual 2>err &&\n@@ -260,10 +211,10 @@ test_expect_success 'cone mode: list' '\n test_expect_success 'cone mode: set with nested folders' '\n \tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n \ttest_line_count = 0 err &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n \tEOF\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n@@ -275,13 +226,7 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n \ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n \ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\t\tdeeper2\n-\tEOF\n-\ttest_cmp dir expect\n+\tcheck_files repo/deep \"a deeper1 deeper2\"\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -326,18 +271,13 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1\"\n '\n \n test_expect_success 'interaction with submodules' '\n@@ -351,21 +291,8 @@ test_expect_success 'interaction with submodules' '\n \t\tgit sparse-checkout init --cone &&\n \t\tgit sparse-checkout set folder1\n \t) &&\n-\tlist_files super >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tmodules\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files super/modules/child >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files super \"a folder1 modules\" &&\n+\tcheck_files super/modules/child \"a deep folder1 folder2\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"389767","messageId":"dfa7e204449c8958d35cc01261d2dc8099d7ce0c.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 4/8] sparse-checkout: cone mode does not recognize \"**\"","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:58Z","receivedAt":"2020-01-14T19:26:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen core.sparseCheckoutCone is enabled, the 'git sparse-checkout set'\ncommand creates a restricted set of possible patterns that are used\nby a custom algorithm to quickly match those patterns.\n\nIf a user manually edits the sparse-checkout file, then they could\ncreate patterns that do not match these expectations. The cone-mode\nmatching algorithm can return incorrect results. The solution is to\ndetect these incorrect patterns, warn that we do not recognize them,\nand revert to the standard algorithm.\n\nCheck each pattern for the \"**\" substring, and revert to the old\nlogic if seen. While technically a \"//**\" pattern matches\nthe meaning of \"//\", it is not one that would be written by\nthe sparse-checkout builtin in cone mode. Attempting to accept that\npattern change complicates the logic and instead we punt and do\nnot accept any instance of \"**\".\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 7 ++++++\n t/t1091-sparse-checkout-builtin.sh | 34 ++++++++++++++++++++++++++++++\n 2 files changed, 41 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 22d08e61c2..f8e350dda2 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,6 +651,13 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n+\tif (strstr(given->pattern, \"**\")) {\n+\t\t/* Not a cone pattern. */\n+\t\tpl->use_cone_patterns = 0;\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 58d9c69163..e532a52f89 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -305,4 +305,38 @@ test_expect_success 'different sparse-checkouts with worktrees' '\n \tcheck_files worktree \"a deep\"\n '\n \n+check_read_tree_errors () {\n+\tREPO=$1\n+\tFILES=$2\n+\tERRORS=$3\n+\tgit -C $REPO read-tree -mu HEAD 2>err &&\n+\tif test -z \"$ERRORS\"\n+\tthen\n+\t\ttest_must_be_empty err\n+\telse\n+\t\ttest_i18ngrep \"$ERRORS\" err\n+\tfi &&\n+\tcheck_files $REPO \"$FILES\"\n+}\n+\n+test_expect_success 'pattern-checks: /A/**' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/**\n+\tEOF\n+\tcheck_read_tree_errors repo \"a folder1\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: /A/**/B/' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/**/deepest\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\" &&\n+\tcheck_files repo/deep \"deeper1\" &&\n+\tcheck_files repo/deep/deeper1 \"deepest\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"389768","messageId":"9be49908fd85e27d49c1c23ec6928785eabd7f65.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 5/8] sparse-checkout: detect short patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:59Z","receivedAt":"2020-01-14T19:26:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the shortest pattern the sparse-checkout command will\nwrite into the sparse-checkout file is \"/*\". This is handled carefully\nin add_pattern_to_hashsets(), so warn if any other pattern is this\nshort. This will assist future pattern checks by allowing us to assume\nthere are at least three characters in the pattern.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 3 ++-\n t/t1091-sparse-checkout-builtin.sh | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex f8e350dda2..1c96ddf5e3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,7 +651,8 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n-\tif (strstr(given->pattern, \"**\")) {\n+\tif (given->patternlen <= 2 ||\n+\t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\tpl->use_cone_patterns = 0;\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e532a52f89..974a4fec8f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -339,4 +339,13 @@ test_expect_success 'pattern-checks: /A/**/B/' '\n \tcheck_files repo/deep/deeper1 \"deepest\"\n '\n \n+test_expect_success 'pattern-checks: too short' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"389769","messageId":"3ef8e021a586bd986e5a2a7e1bc956daf2d874d2.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 3/8] clone: fix --sparse option with URLs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:25:57Z","receivedAt":"2020-01-14T19:26:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --sparse option was added to the clone builtin in d89f09c (clone:\nadd --sparse mode, 2019-11-21) and was tested with a local path clone\nin t1091-sparse-checkout-builtin.sh. However, due to a difference in\nhow local paths are handled versus URLs, this mechanism does not work\nwith URLs.\n\nModify the test to use a \"file://\" URL, which would output this error\nbefore the code change:\n\n Cloning into 'clone'...\n fatal: cannot change to 'file://.../repo': No such file or directory\n error: failed to initialize sparse-checkout\n\nThese errors are due to using a \"-C \" option to call 'git -C\n sparse-checkout init' but the URL is being given instead of\nthe target directory.\n\nUpdate that target directory to evaluate this correctly. I have also\nmanually tested that https:// URLs are handled correctly as well.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4348d962c9..2caefc44fb 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1130,7 +1130,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\tif (option_sparse_checkout && git_sparse_checkout_init(dir))\n \t\treturn 1;\n \n \tremote = remote_get(option_origin);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 37365dc668..58d9c69163 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -90,7 +90,7 @@ test_expect_success 'init with existing sparse-checkout' '\n '\n \n test_expect_success 'clone --sparse' '\n-\tgit clone --sparse repo clone &&\n+\tgit clone --sparse \"file://$(pwd)/repo\" clone &&\n \tgit -C clone sparse-checkout list >actual &&\n \tcat >expect <<-\\EOF &&\n \t/*\n-- \ngitgitgadget\n\n"},{"id":"389770","messageId":"09dbe1f9029d173e579b74442b02bc20e1a82437.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 7/8] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:26:01Z","receivedAt":"2020-01-14T19:26:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout feature uses hashset containment\nqueries to match paths. Make this algorithm respect escaped asterisk\n(*) and backslash (\\) characters.\n\nCreate dup_and_filter_pattern() method to convert a pattern by\nremoving escape characters and dropping an optional \"/*\" at the end.\nThis method is available in dir.h as we will use it in\nbuiltin/sparse-chekcout.c in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 31 +++++++++++++++++++++++++++---\n dir.h | 1 +\n t/t1091-sparse-checkout-builtin.sh | 22 +++++++++++++++++----\n 3 files changed, 47 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 150c05f4de..2840bacd40 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -630,6 +630,32 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n }\n \n+char *dup_and_filter_pattern(const char *pattern)\n+{\n+\tchar *set, *read;\n+\tchar *result = xstrdup(pattern);\n+\n+\tset = result;\n+\tread = result;\n+\n+\twhile (*read) {\n+\t\t/* skip escape characters (once) */\n+\t\tif (*read == '\\\\')\n+\t\t\tread++;\n+\n+\t\t*set = *read;\n+\n+\t\tset++;\n+\t\tread++;\n+\t}\n+\t*set = 0;\n+\n+\tif (*(read - 2) == '/' && *(read - 1) == '*')\n+\t\t*(read - 2) = 0;\n+\n+\treturn result;\n+}\n+\n static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n {\n \tstruct pattern_entry *translated;\n@@ -698,8 +724,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\t\tgoto clear_hashmaps;\n \t\t}\n \n-\t\ttruncated = xstrdup(given->pattern);\n-\t\ttruncated[given->patternlen - 2] = 0;\n+\t\ttruncated = dup_and_filter_pattern(given->pattern);\n \n \t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n \t\ttranslated->pattern = truncated;\n@@ -733,7 +758,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \n \ttranslated = xmalloc(sizeof(struct pattern_entry));\n \n-\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->pattern = dup_and_filter_pattern(given->pattern);\n \ttranslated->patternlen = given->patternlen;\n \thashmap_entry_init(&translated->ent,\n \t\t\t ignore_case ?\ndiff --git a/dir.h b/dir.h\nindex 77a43dbf89..6dcd9d33e7 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -304,6 +304,7 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \t\t const struct hashmap_entry *a,\n \t\t const struct hashmap_entry *b,\n \t\t const void *key);\n+char *dup_and_filter_pattern(const char *pattern);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 5b50be53a4..051c1f3bf2 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -366,13 +366,27 @@ test_expect_success 'pattern-checks: starting \"*\"' '\n \tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n '\n \n-test_expect_success 'pattern-checks: escaped \"*\"' '\n-\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n+\tgit clone repo escaped &&\n+\tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n+\tNEWTREE=$(git -C escaped mktree <<-EOF\n+\t$(git -C escaped ls-tree HEAD)\n+\t040000 tree $TREEOID\tzbad\\\\dir\n+\t040000 tree $TREEOID\tzdoes*exist\n+\tEOF\n+\t) &&\n+\tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n+\tgit -C escaped reset --hard $COMMIT &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout init --cone &&\n+\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n \t/*\n \t!/*/\n-\t/does\\*not\\*exist/\n+\t/zbad\\\\dir/\n+\t/zdoes\\*not\\*exist/\n+\t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors repo \"a\" \"\"\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"389771","messageId":"77a514f50bfb73c590aa1cab437d56d862069f66.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 6/8] sparse-checkout: warn on incorrect '*' in patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:26:00Z","receivedAt":"2020-01-14T19:26:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout commmand will write patterns that\nallow faster pattern matching. This matching only works if the patterns\nin the sparse-checkout file are those written by that command. Users\ncan edit the sparse-checkout file and create patterns that cause the\ncone mode matching to fail.\n\nThe cone mode patterns may end in \"/*\" but otherwise an un-escaped\nasterisk is invalid. Add checks to disable cone mode when seeing these\nvalues.\n\nA later change will properly handle escaped asterisks.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 30 ++++++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 27 +++++++++++++++++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 1c96ddf5e3..150c05f4de 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -635,6 +635,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \tstruct pattern_entry *translated;\n \tchar *truncated;\n \tchar *data = NULL;\n+\tconst char *prev, *cur, *next;\n \n \tif (!pl->use_cone_patterns)\n \t\treturn;\n@@ -652,6 +653,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t}\n \n \tif (given->patternlen <= 2 ||\n+\t *given->pattern == '*' ||\n \t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\tpl->use_cone_patterns = 0;\n@@ -659,6 +661,34 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\tgoto clear_hashmaps;\n \t}\n \n+\tprev = given->pattern;\n+\tcur = given->pattern + 1;\n+\tnext = given->pattern + 2;\n+\n+\twhile (*cur) {\n+\t\t/* We care about *cur == '*' */\n+\t\tif (*cur != '*')\n+\t\t\tgoto increment;\n+\n+\t\t/* But only if *prev != '\\\\' */\n+\t\tif (*prev == '\\\\')\n+\t\t\tgoto increment;\n+\n+\t\t/* But a trailing '/' then '*' is fine */\n+\t\tif (*prev == '/' && *next == 0)\n+\t\t\tgoto increment;\n+\n+\t\t/* Not a cone pattern. */\n+\t\tpl->use_cone_patterns = 0;\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\n+\tincrement:\n+\t\tprev++;\n+\t\tcur++;\n+\t\tnext++;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 974a4fec8f..5b50be53a4 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -348,4 +348,31 @@ test_expect_success 'pattern-checks: too short' '\n \tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n '\n \n+test_expect_success 'pattern-checks: trailing \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a*\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: starting \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*eep/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: escaped \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/does\\*not\\*exist/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"389772","messageId":"79b6e9a565fde766954d7cda84a835b9015af6cb.1579029963.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-14T19:26:02Z","receivedAt":"2020-01-14T19:26:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user somehow creates a directory with an asterisk (*) or backslash\n(\\), then the \"git sparse-checkout set\" command will struggle to provide\nthe correct pattern in the sparse-checkout file. When not in cone mode,\nthe provided pattern is written directly into the sparse-checkout file.\nHowever, in cone mode we expect a list of paths to directories and then\nwe convert those into patterns.\n\nEven more specifically, the goal is to always allow the following from\nthe root of a repo:\n\n git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n\nThe ls-tree command provides directory names with an unescaped asterisk.\nIt also quotes the directories that contain an escaped backslash. We\nmust remove these quotes, then keep the escaped backslashes.\n\nHowever, there is some care needed for the timing of these escapes. The\nin-memory pattern list is used to update the working directory before\nwriting the patterns to disk. Thus, we need the command to have the\nunescaped names in the hashsets for the cone comparisons, then escape\nthe patterns later.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 48 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 21 +++++++++++--\n 2 files changed, 64 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 3cee8ab46e..61d2c30036 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -140,6 +140,22 @@ static int update_working_directory(struct pattern_list *pl)\n \treturn result;\n }\n \n+static char *escaped_pattern(char *pattern)\n+{\n+\tchar *p = pattern;\n+\tstruct strbuf final = STRBUF_INIT;\n+\n+\twhile (*p) {\n+\t\tif (*p == '*' || *p == '\\\\')\n+\t\t\tstrbuf_addch(&final, '\\\\');\n+\n+\t\tstrbuf_addch(&final, *p);\n+\t\tp++;\n+\t}\n+\n+\treturn strbuf_detach(&final, NULL);\n+}\n+\n static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n {\n \tint i;\n@@ -164,10 +180,11 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \n \t\tif (strlen(pattern))\n \t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t\tfree(pattern);\n \t}\n \n \tstring_list_clear(&sl, 0);\n@@ -185,8 +202,9 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstring_list_remove_duplicates(&sl, 0);\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t\tfree(pattern);\n \t}\n }\n \n@@ -337,7 +355,9 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat\n {\n \tstruct pattern_entry *e = xmalloc(sizeof(*e));\n \te->patternlen = path->len;\n-\te->pattern = strbuf_detach(path, NULL);\n+\te->pattern = dup_and_filter_pattern(path->buf);\n+\tstrbuf_release(path);\n+\n \thashmap_entry_init(&e->ent,\n \t\t\t ignore_case ?\n \t\t\t strihash(e->pattern) :\n@@ -369,6 +389,7 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat\n \n static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n {\n+\tint i;\n \tstrbuf_trim(line);\n \n \tstrbuf_trim_trailing_dir_sep(line);\n@@ -376,6 +397,27 @@ static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n \tif (!line->len)\n \t\treturn;\n \n+\tfor (i = 0; i < line->len; i++) {\n+\t\tif (line->buf[i] == '*') {\n+\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n+\t\t\ti++;\n+\t\t}\n+\n+\t\tif (line->buf[i] == '\\\\') {\n+\t\t\tif (i < line->len - 1 && line->buf[i + 1] == '\\\\')\n+\t\t\t\ti++;\n+\t\t\telse\n+\t\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n+\n+\t\t\ti++;\n+\t\t}\n+\t}\n+\n+\tif (line->buf[0] == '\"' && line->buf[line->len - 1] == '\"') {\n+\t\tstrbuf_remove(line, 0, 1);\n+\t\tstrbuf_remove(line, line->len - 1, 1);\n+\t}\n+\n \tif (line->buf[0] != '/')\n \t\tstrbuf_insert(line, 0, \"/\", 1);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 051c1f3bf2..3da7c10bd9 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -309,6 +309,9 @@ check_read_tree_errors () {\n \tREPO=$1\n \tFILES=$2\n \tERRORS=$3\n+\tgit -C $REPO -c core.sparseCheckoutCone=false read-tree -mu HEAD 2>err &&\n+\ttest_must_be_empty err &&\n+\tcheck_files $REPO \"$FILES\" &&\n \tgit -C $REPO read-tree -mu HEAD 2>err &&\n \tif test -z \"$ERRORS\"\n \tthen\n@@ -379,14 +382,28 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tgit -C escaped reset --hard $COMMIT &&\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n-\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n+\tgit -C escaped sparse-checkout set zbad\\\\dir zdoes\\*not\\*exist zdoes\\*exist &&\n+\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n \t/zbad\\\\dir/\n+\t/zdoes\\*exist/\n \t/zdoes\\*not\\*exist/\n+\tEOF\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t/folder1/\n+\t/folder2/\n+\t/zbad\\\\dir/\n \t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n"},{"id":"389773","messageId":"20200114193032.GA8593@syl.local","threadId":"52638","inReplyTo":"3ef8e021a586bd986e5a2a7e1bc956daf2d874d2.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] clone: fix --sparse option with URLs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-01-14T19:30:32Z","receivedAt":"2020-01-14T19:30:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Stolee,\n\nOn Tue, Jan 14, 2020 at 07:25:57PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> The --sparse option was added to the clone builtin in d89f09c (clone:\n> add --sparse mode, 2019-11-21) and was tested with a local path clone\n> in t1091-sparse-checkout-builtin.sh. However, due to a difference in\n> how local paths are handled versus URLs, this mechanism does not work\n> with URLs.\n\nAs we discussed off-list, both of us (as well as Peff) were able to\nreproduce this issue. I think that this paragraph is a good description\nof what's going on heee.\n\n> Modify the test to use a \"file://\" URL, which would output this error\n> before the code change:\n>\n> Cloning into 'clone'...\n> fatal: cannot change to 'file://.../repo': No such file or directory\n> error: failed to initialize sparse-checkout\n\nNice, this should give us confidence that there won't be a regression\nhere in the future. I don't think that the explanation is complicated\nenough for a single commit which introduced an expected failure, so\ngrouping it all together in this patch seems good to me.\n\n> These errors are due to using a \"-C \" option to call 'git -C\n> sparse-checkout init' but the URL is being given instead of\n> the target directory.\n>\n> Update that target directory to evaluate this correctly. I have also\n> manually tested that https:// URLs are handled correctly as well.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/clone.c | 2 +-\n> t/t1091-sparse-checkout-builtin.sh | 2 +-\n> 2 files changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 4348d962c9..2caefc44fb 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -1130,7 +1130,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \tif (option_required_reference.nr || option_optional_reference.nr)\n> \t\tsetup_reference();\n>\n> -\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n> +\tif (option_sparse_checkout && git_sparse_checkout_init(dir))\n\nI agree that 'dir' is the right thing to use here. It's the string we\nread from to print \"Cloning into ...\", which always displays the\ndirectory relative to the cwd. Looking at the implementation in\n'git_sparse_checkout_init', this matches my understanding, too.\n\n> \t\treturn 1;\n>\n> \tremote = remote_get(option_origin);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 37365dc668..58d9c69163 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -90,7 +90,7 @@ test_expect_success 'init with existing sparse-checkout' '\n> '\n>\n> test_expect_success 'clone --sparse' '\n> -\tgit clone --sparse repo clone &&\n> +\tgit clone --sparse \"file://$(pwd)/repo\" clone &&\n> \tgit -C clone sparse-checkout list >actual &&\n> \tcat >expect <<-\\EOF &&\n> \t/*\n> --\n> gitgitgadget\n\nThis all looks good to me.\n\n Acked-by: Taylor Blau \n\nThanks,\nTaylor\n"},{"id":"389774","messageId":"20200114193405.GA8674@syl.local","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-01-14T19:34:05Z","receivedAt":"2020-01-14T19:34:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Stolee,\n\nOn Tue, Jan 14, 2020 at 07:25:54PM +0000, Derrick Stolee via GitGitGadget wrote:\n> This series is based on ds/sparse-list-in-cone-mode.\n>\n> This series attempts to clean up some rough edges in the sparse-checkout\n> feature, especially around the cone mode.\n>\n> Unfortunately, after the v2.25.0 release, we noticed an issue with the \"git\n> clone --sparse\" option when using a URL instead of a local path. This is\n> fixed and properly tested here.\n\nI haven't had a chance to look at the other patches (besides the one\nthat I have already reviewed on- and off-list), so take my comments here\nwith a grain of salt.\n\nIt's too bad that 'git clone --sparse' isn't working with URLs in\nv2.25.0, but it happens, and I don't think that this is a grave-enough\nissue to warrant a new release. At least, since '--sparse' is new in\nv2.25.0, we're not breaking existing workflows that have already relied\non it.\n\nAnd, since sparse-checkout is still relatively niche (at, least for now\n;-)), I think that this not handling cloning from URLs is fine until\nv2.26.0.\n\nOf course, if there's ever another need for v2.25.1, I don't think that\nthis would *hurt* to release then, which is to say that we definitely\nshould have these patches in a release, soon, but I don't think that\nthere's a terrible sense of urgency in the meantime.\n\n> Also, let's improve Git's response to these more complicated scenarios:\n>\n> 1. Running \"git sparse-checkout init\" in a worktree would complain because\n> the \"info\" dir doesn't exist.\n> 2. Tracked paths that include \"*\" and \"\" in their filenames.\n> 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n> as \"*\" anywhere or \"\" in the wrong place, then we should respond\n> appropriately. That is: warn that the patterns are not cone-mode, then\n> revert to the old logic.\n>\n> Thanks, -Stolee\n>\n> Derrick Stolee (8):\n> t1091: use check_files to reduce boilerplate\n> sparse-checkout: create leading directories\n> clone: fix --sparse option with URLs\n> sparse-checkout: cone mode does not recognize \"**\"\n> sparse-checkout: detect short patterns\n> sparse-checkout: warn on incorrect '*' in patterns\n> sparse-checkout: properly match escaped characters\n> sparse-checkout: write escaped patterns in cone mode\n>\n> builtin/clone.c | 2 +-\n> builtin/sparse-checkout.c | 52 ++++-\n> dir.c | 69 ++++++-\n> dir.h | 1 +\n> t/t1091-sparse-checkout-builtin.sh | 320 ++++++++++++++++-------------\n> 5 files changed, 296 insertions(+), 148 deletions(-)\n>\n>\n> base-commit: 4fd683b6a35eabd23dd5183da7f654a1e1f00325\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-513%2Fderrickstolee%2Fsparse-harden-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-513/derrickstolee/sparse-harden-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/513\n> --\n> gitgitgadget\nThanks,\nTaylor\n"},{"id":"389775","messageId":"c05c7dcb-43b4-dcf8-bc5d-e162f0b2d1a4@gmail.com","threadId":"52638","inReplyTo":"20200114193405.GA8674@syl.local","subject":"Re: [PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-14T19:44:50Z","receivedAt":"2020-01-14T19:44:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2020 2:34 PM, Taylor Blau wrote:\n> Hi Stolee,\n> \n> On Tue, Jan 14, 2020 at 07:25:54PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> This series is based on ds/sparse-list-in-cone-mode.\n>>\n>> This series attempts to clean up some rough edges in the sparse-checkout\n>> feature, especially around the cone mode.\n>>\n>> Unfortunately, after the v2.25.0 release, we noticed an issue with the \"git\n>> clone --sparse\" option when using a URL instead of a local path. This is\n>> fixed and properly tested here.\n> \n> I haven't had a chance to look at the other patches (besides the one\n> that I have already reviewed on- and off-list), so take my comments here\n> with a grain of salt.\n> \n> It's too bad that 'git clone --sparse' isn't working with URLs in\n> v2.25.0, but it happens, and I don't think that this is a grave-enough\n> issue to warrant a new release. At least, since '--sparse' is new in\n> v2.25.0, we're not breaking existing workflows that have already relied\n> on it.\n> \n> And, since sparse-checkout is still relatively niche (at, least for now\n> ;-)), I think that this not handling cloning from URLs is fine until\n> v2.26.0.\n\nSince we've already worked out the workaround to be:\n\n\tgit clone --no-checkout \n\tcd \n\tgit sparse-checkout init --cone\n\nthere is no rush to fix this. Users _may_ discover the --sparse option\nfrom the clone docs and complain, but we can point them to the above\ndirections for now.\n\n> Of course, if there's ever another need for v2.25.1, I don't think that\n> this would *hurt* to release then, which is to say that we definitely\n> should have these patches in a release, soon, but I don't think that\n> there's a terrible sense of urgency in the meantime.\n\nI wouldn't complain to have patches 1-3 in an otherwise-warranted .1 release.\n\nThanks,\n-Stolee\n"},{"id":"389780","messageId":"20200114211653.GB3949560@coredump.intra.peff.net","threadId":"52638","inReplyTo":"dfa7e204449c8958d35cc01261d2dc8099d7ce0c.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/8] sparse-checkout: cone mode does not recognize \"**\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-14T21:16:53Z","receivedAt":"2020-01-14T21:16:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2020 at 07:25:58PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> When core.sparseCheckoutCone is enabled, the 'git sparse-checkout set'\n> command creates a restricted set of possible patterns that are used\n> by a custom algorithm to quickly match those patterns.\n> \n> If a user manually edits the sparse-checkout file, then they could\n> create patterns that do not match these expectations. The cone-mode\n> matching algorithm can return incorrect results. The solution is to\n> detect these incorrect patterns, warn that we do not recognize them,\n> and revert to the standard algorithm.\n> \n> Check each pattern for the \"**\" substring, and revert to the old\n> logic if seen. While technically a \"//**\" pattern matches\n> the meaning of \"//\", it is not one that would be written by\n> the sparse-checkout builtin in cone mode. Attempting to accept that\n> pattern change complicates the logic and instead we punt and do\n> not accept any instance of \"**\".\n\nThat all makes sense.\n\n> diff --git a/dir.c b/dir.c\n> index 22d08e61c2..f8e350dda2 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -651,6 +651,13 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n> \t\treturn;\n> \t}\n> \n> +\tif (strstr(given->pattern, \"**\")) {\n> +\t\t/* Not a cone pattern. */\n> +\t\tpl->use_cone_patterns = 0;\n> +\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n> +\t\tgoto clear_hashmaps;\n> +\t}\n\nThe clear_hashmaps label already unsets pl->use_cone_patterns, so the\nfirst line is redundant (the same is true of existing goto jumps, as\nwell, though).\n\nI wondered whether this warning could be triggered accidentally by\nsomebody who just happened to add such a pattern. But we'd exit\nimmediately from add_pattern_to_hashsets() immediately unless the user\nhas set core.sparseCheckoutCone. And if that's set, then warning is\ndefinitely the right thing to do.\n\n-Peff\n"},{"id":"389781","messageId":"20200114212120.GC3949560@coredump.intra.peff.net","threadId":"52638","inReplyTo":"09dbe1f9029d173e579b74442b02bc20e1a82437.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/8] sparse-checkout: properly match escaped characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-14T21:21:20Z","receivedAt":"2020-01-14T21:21:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2020 at 07:26:01PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> In cone mode, the sparse-checkout feature uses hashset containment\n> queries to match paths. Make this algorithm respect escaped asterisk\n> (*) and backslash (\\) characters.\n> \n> Create dup_and_filter_pattern() method to convert a pattern by\n> removing escape characters and dropping an optional \"/*\" at the end.\n> This method is available in dir.h as we will use it in\n> builtin/sparse-chekcout.c in a later change.\n\ns/chekcout/checkout/\n\nIt took me a minute to understand the problem here, but I think it's: if\na path in the sparse-checkout file has \"\\*\" in it, we'd try to match a\nliteral \"\\*\" in the hash, not \"*\"?\n\nBut we wouldn't run into that yet because we don't properly _write_ the\nescaped names until patch 8.\n\nIs that right?\n\n-Peff\n"},{"id":"389782","messageId":"20200114212548.GD3949560@coredump.intra.peff.net","threadId":"52638","inReplyTo":"79b6e9a565fde766954d7cda84a835b9015af6cb.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-14T21:25:48Z","receivedAt":"2020-01-14T21:25:52Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2020 at 07:26:02PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> If a user somehow creates a directory with an asterisk (*) or backslash\n> (\\), then the \"git sparse-checkout set\" command will struggle to provide\n> the correct pattern in the sparse-checkout file. When not in cone mode,\n> the provided pattern is written directly into the sparse-checkout file.\n> However, in cone mode we expect a list of paths to directories and then\n> we convert those into patterns.\n> \n> Even more specifically, the goal is to always allow the following from\n> the root of a repo:\n> \n> git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n> \n> The ls-tree command provides directory names with an unescaped asterisk.\n> It also quotes the directories that contain an escaped backslash. We\n> must remove these quotes, then keep the escaped backslashes.\n\nDo we need to document these rules somewhere? Naively I'd expect\n\"--stdin\" to take in literal pathnames. But of course it can't represent\na path with a newline. So perhaps it makes sense to take quoted names by\ndefault, and allow literal NUL-separated input with \"-z\" if anybody\nwants it.\n\n-Peff\n"},{"id":"389784","messageId":"20200114213110.GE3949560@coredump.intra.peff.net","threadId":"52638","inReplyTo":"c05c7dcb-43b4-dcf8-bc5d-e162f0b2d1a4@gmail.com","subject":"Re: [PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-14T21:31:10Z","receivedAt":"2020-01-14T21:31:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2020 at 02:44:50PM -0500, Derrick Stolee wrote:\n\n> Since we've already worked out the workaround to be:\n> \n> \tgit clone --no-checkout \n> \tcd \n> \tgit sparse-checkout init --cone\n> \n> there is no rush to fix this. Users _may_ discover the --sparse option\n> from the clone docs and complain, but we can point them to the above\n> directions for now.\n\nYes, but part of the beauty of the new system is just having to say\n\"--sparse\" to make something useful happen. :)\n\n> > Of course, if there's ever another need for v2.25.1, I don't think that\n> > this would *hurt* to release then, which is to say that we definitely\n> > should have these patches in a release, soon, but I don't think that\n> > there's a terrible sense of urgency in the meantime.\n> \n> I wouldn't complain to have patches 1-3 in an otherwise-warranted .1 release.\n\nAgreed. 1-3 look obviously correct to me. The quoting bits I'm a little\nmore fuzzy on, just because I haven't really looked hard into cone mode.\nDitto for the \"disable cone mode\" checks.\n\nMy gut instinct is that you should be able to deduce whether the pattern\nhashmap can be used purely from the patterns you see, and\ncore.sparseCheckoutCone would not be needed (and so if you violate the\nrules by writing something manual, then it just gets slower; or maybe\nwe're even able to apply the literal cone-mode rules quickly and handle\nthe other separately). But it's much more likely I'm showing off my lack\nof knowledge of the details of the problem space. You can feel free to\neducate me, and/or roll your eyes and ignore me if this was already\ndiscussed earlier.\n\nBy the way, I did notice this while poking about, which could go on top\n(or hopefully be lumped in with the 1-3 as \"obviously correct\"):\n\n-- >8 --\n\nSubject: [PATCH] sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n\nSigned-off-by: Jeff King \n---\n Documentation/git-sparse-checkout.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 974ade2238..285893b069 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -106,7 +106,7 @@ The full pattern set allows for arbitrary pattern matches and complicated\n inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n updating the index, where N is the number of patterns and M is the number\n of paths in the index. To combat this performance issue, a more restricted\n-pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+pattern set is allowed when `core.sparseCheckoutCone` is enabled.\n \n The accepted patterns in the cone pattern set are:\n \n-- \n2.25.0.639.gb9b1511416\n\n"},{"id":"389789","messageId":"79915215-15c1-7e38-6f06-294c719ff260@gmail.com","threadId":"52638","inReplyTo":"20200114212120.GC3949560@coredump.intra.peff.net","subject":"Re: [PATCH 7/8] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-14T22:08:21Z","receivedAt":"2020-01-14T22:08:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2020 4:21 PM, Jeff King wrote:\n> On Tue, Jan 14, 2020 at 07:26:01PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> In cone mode, the sparse-checkout feature uses hashset containment\n>> queries to match paths. Make this algorithm respect escaped asterisk\n>> (*) and backslash (\\) characters.\n>>\n>> Create dup_and_filter_pattern() method to convert a pattern by\n>> removing escape characters and dropping an optional \"/*\" at the end.\n>> This method is available in dir.h as we will use it in\n>> builtin/sparse-chekcout.c in a later change.\n> \n> s/chekcout/checkout/\n\nThanks.\n\n> It took me a minute to understand the problem here, but I think it's: if\n> a path in the sparse-checkout file has \"\\*\" in it, we'd try to match a\n> literal \"\\*\" in the hash, not \"*\"?\n\nYes, the hashset would have the string \"\\*\" instead of the string \"*\". This\nwould lead to missing directories when cone mode is enabled compared to\ncone mode not being enabled.\n \n> But we wouldn't run into that yet because we don't properly _write_ the\n> escaped names until patch 8.\n\nWe wouldn't run into it when using the builtin, but also a user could\nedit their sparse-checkout file manually OR figure out how to get the\n\"right\" pattern by running \"git sparse-checkout set \"my\\\\*dir\" (where the\nescaped backslash is collapsed by the shell and Git sees \"my\\*dir\".\n\nThanks,\n-Stolee\n"},{"id":"389791","messageId":"e325daa4-5a26-3652-5f46-06203403e51d@gmail.com","threadId":"52638","inReplyTo":"20200114212548.GD3949560@coredump.intra.peff.net","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-14T22:11:03Z","receivedAt":"2020-01-14T22:11:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2020 4:25 PM, Jeff King wrote:\n> On Tue, Jan 14, 2020 at 07:26:02PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> If a user somehow creates a directory with an asterisk (*) or backslash\n>> (\\), then the \"git sparse-checkout set\" command will struggle to provide\n>> the correct pattern in the sparse-checkout file. When not in cone mode,\n>> the provided pattern is written directly into the sparse-checkout file.\n>> However, in cone mode we expect a list of paths to directories and then\n>> we convert those into patterns.\n>>\n>> Even more specifically, the goal is to always allow the following from\n>> the root of a repo:\n>>\n>> git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n>>\n>> The ls-tree command provides directory names with an unescaped asterisk.\n>> It also quotes the directories that contain an escaped backslash. We\n>> must remove these quotes, then keep the escaped backslashes.\n> \n> Do we need to document these rules somewhere? Naively I'd expect\n> \"--stdin\" to take in literal pathnames. But of course it can't represent\n> a path with a newline. So perhaps it makes sense to take quoted names by\n> default, and allow literal NUL-separated input with \"-z\" if anybody\n> wants it.\n\nThis is worth thinking about the right way to describe the rules:\n\n1. You don't _need_ quotes. They happen to come along for the ride in\n 'git ls-tree' so it doesn't mess up shell scripts that iterate on\n those entries. At least, that's why I think they are quoted.\n\n2. If you use quotes, the first layer of quotes will be removed.\n\nHow much of this needs to be documented explicitly, or how much should\nwe say \"The input format matches what we would expect from 'git ls-tree\n--name-only'\"?\n\nThanks,\n-Stolee\n"},{"id":"389794","messageId":"20200114224818.GD3957260@coredump.intra.peff.net","threadId":"52638","inReplyTo":"e325daa4-5a26-3652-5f46-06203403e51d@gmail.com","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-14T22:48:18Z","receivedAt":"2020-01-14T22:48:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 14, 2020 at 05:11:03PM -0500, Derrick Stolee wrote:\n\n> > Do we need to document these rules somewhere? Naively I'd expect\n> > \"--stdin\" to take in literal pathnames. But of course it can't represent\n> > a path with a newline. So perhaps it makes sense to take quoted names by\n> > default, and allow literal NUL-separated input with \"-z\" if anybody\n> > wants it.\n> \n> This is worth thinking about the right way to describe the rules:\n> \n> 1. You don't _need_ quotes. They happen to come along for the ride in\n> 'git ls-tree' so it doesn't mess up shell scripts that iterate on\n> those entries. At least, that's why I think they are quoted.\n\nIt's not just shell scripts. Without quoting, the syntax becomes\nambiguous (e.g., imagine a file with a newline in it). So most Git\noutput that shows a filename will quote it if necessary, unless\nNUL separators are being used.\n\n> 2. If you use quotes, the first layer of quotes will be removed.\n\nI take this to mean that anything starting with a double-quote will have\nthe outer layer removed, and backslash escapes inside expanded. And\nanything without a starting double quote (even if it has internal\nbackslash escapes!) will be taken literally.\n\nThat would match how things like \"update-index --index-info\" work.\n\nAs far as implementation, I know you're trying to keep some of the\nescaping, but I think it might make more sense to do use\nunquote_c_style() to parse the input (see update-index's use for some\nprior art), and then re-quote as necessary to put things into the\nsparse-checkout file (I guess quoting more than just quote_c_style()\nwould do, since you need to quote glob metacharacters like '*' and\nprobably \"!\"). But as much as possible, I think you'd want literal\nstrings inside the program, and just quoting/unquoting at the edges.\n\n> How much of this needs to be documented explicitly, or how much should\n> we say \"The input format matches what we would expect from 'git ls-tree\n> --name-only'\"?\n\nI think it's fine to say that, and maybe call attention to the quoting.\nLike:\n\n The input format matches the output of `git ls-tree --name-only`. This\n includes interpreting pathnames that begin with a double quote (\") as\n C-style quoted strings.\n\nDisappointingly, update-index does not seem to explain the rules\nanywhere. fast-import does cover it. Maybe it's something that ought to\nbe hoisted out into gitcli(7) or similar (or maybe it has been and I\njust can't find it).\n\n-Peff\n"},{"id":"389813","messageId":"xmqqzheoil82.fsf@gitster-ct.c.googlers.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-15T19:16:29Z","receivedAt":"2020-01-15T19:16:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Also, let's improve Git's response to these more complicated scenarios:\n>\n> 1. Running \"git sparse-checkout init\" in a worktree would complain because\n> the \"info\" dir doesn't exist.\n> 2. Tracked paths that include \"*\" and \"\" in their filenames.\n> 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n> as \"*\" anywhere or \"\" in the wrong place, then we should respond\n> appropriately. That is: warn that the patterns are not cone-mode, then\n> revert to the old logic.\n\nIt seems somebody ate a letter to make \"\" into \"\" an\nempty string, so I cannot quite grok the above list---two out of\nthree bullet points are not quite readable.\n\n"},{"id":"389822","messageId":"19493656-c7d0-5916-7f9d-46c3183d2d77@gmail.com","threadId":"52638","inReplyTo":"xmqqzheoil82.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/8] Harden the sparse-checkout builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-15T20:32:36Z","receivedAt":"2020-01-15T20:32:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/15/2020 2:16 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> Also, let's improve Git's response to these more complicated scenarios:\n>>\n>> 1. Running \"git sparse-checkout init\" in a worktree would complain because\n>> the \"info\" dir doesn't exist.\n>> 2. Tracked paths that include \"*\" and \"\" in their filenames.\n>> 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n>> as \"*\" anywhere or \"\" in the wrong place, then we should respond\n>> appropriately. That is: warn that the patterns are not cone-mode, then\n>> revert to the old logic.\n> \n> It seems somebody ate a letter to make \"\" into \"\" an\n> empty string, so I cannot quite grok the above list---two out of\n> three bullet points are not quite readable.\n\nIn 2., the \"\" should be \"\\\".\n\nIn 3., the \"*\" and \"\" should be \"**\" and \"*\" respectively.\n\nThese are things that are being collapsed by GitHub's markdown\nprocessing. Sorry that this affects GitGitGadget's cover letter.\n\nBy escaping appropriately, these show up correctly and hopefully\nwill be fixed in v2.\n\n-Stolee\n"},{"id":"389938","messageId":"xmqq8sm7gjvt.fsf@gitster-ct.c.googlers.com","threadId":"52638","inReplyTo":"9f7791ae5ee8df9d84d25407a014fb2c63e09b38.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/8] t1091: use check_files to reduce boilerplate","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-16T21:40:38Z","receivedAt":"2020-01-16T21:40:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When testing the sparse-checkout feature, we need to compare the\n> contents of the working-directory against some expected output.\n> Using here-docs was useful in the beginning, but became repetetive\n> as the test script grew.\n>\n> Create a check_files helper to make the tests simpler and easier\n> to extend. It also reduces instances of bad here-doc whitespace.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1091-sparse-checkout-builtin.sh | 215 ++++++++++-------------------\n> 1 file changed, 71 insertions(+), 144 deletions(-)\n>\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index ff7f8f7a1f..20caefe155 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -12,6 +12,13 @@ list_files() {\n> \t(cd \"$1\" && printf '%s\\n' *)\n> }\n> \n> +check_files() {\n> +\tDIR=$1\n> +\tprintf \"%s\\n\" $2 >expect &&\n> +\tlist_files $DIR >actual &&\n\nIt is unclear if the script is being deliberate or sloppy.\n\nIt turns out that not quoting $2 is deliberate (i.e. it wants to\npass more than one words in $2, have them split at $IFS and show\neach of them on a separate line), at the same time not quoting $DIR\nis simply sloppy.\n\nAnd it is totally unnecessary to confuse readers like this.\n\nUnless you plan to extend this helper further, I think this would be\nmuch less burdensome to the readers:\n\n check_files () {\n list_files \"$1\" >actual &&\n shift &&\n printf \"%s\\n\" \"$@\" >expect &&\n test_cmp expect actual\n }\n\nThis ...\n\n> \ttest_cmp expect repo/.git/info/sparse-checkout &&\n> -\tlist_files repo >dir &&\n> -\tcat >expect <<-EOF &&\n> -\t\ta\n> -\t\tfolder1\n> -\t\tfolder2\n> -\tEOF\n> -\ttest_cmp expect dir\n> +\tcheck_files repo \"a folder1 folder2\"\n\n... is a kind of change that the log message advertises, which is a\nvery nice rewrite.\n\nAnd ...\n\n> test_expect_success 'clone --sparse' '\n> \tgit clone --sparse repo clone &&\n> \tgit -C clone sparse-checkout list >actual &&\n> -\tcat >expect <<-EOF &&\n> -\t\t/*\n> -\t\t!/*/\n> +\tcat >expect <<-\\EOF &&\n> +\t/*\n> +\t!/*/\n> \tEOF\n\n... this is a style-fix that is another nice rewrite but in a\ndifferent category. I wonder if they should be done in separate\ncommits.\n\nOther than that, makes sense.\n\nThanks.\n"},{"id":"389940","messageId":"xmqq4kwvgjmn.fsf@gitster-ct.c.googlers.com","threadId":"52638","inReplyTo":"53a266f9aab5f4331c35b3ff15fb0628e2131c56.1579029963.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] sparse-checkout: create leading directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-01-16T21:46:08Z","receivedAt":"2020-01-16T21:46:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The 'git init' command creates the \".git/info\" directory and fills it\n> with some default files. However, 'git worktree add' does not create\n> the info directory for that worktree. This causes a problem when running\n> \"git sparse-checkout init\" inside a worktree. While care was taken to\n> allow the sparse-checkout config to be specific to a worktree, this\n> initialization was untested.\n>\n> Safely create the leading directories for the sparse-checkout file. This\n> is the safest thing to do even without worktrees, as a user could delete\n> their \".git/info\" directory and expect Git to recover safely.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/sparse-checkout.c | 4 ++++\n> t/t1091-sparse-checkout-builtin.sh | 10 ++++++++++\n> 2 files changed, 14 insertions(+)\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index b3bed891cb..3cee8ab46e 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -199,6 +199,10 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> \tint result;\n> \n> \tsparse_filename = get_sparse_checkout_filename();\n> +\n> +\tif (safe_create_leading_directories(sparse_filename))\n> +\t\tdie(_(\"failed to create directory for sparse-checkout file\"));\n> +\n\nThe use of safe_create_leading_directories() here, which uses\nadjust_shared_perm(), is the right thing to do.\n\nLooks good.\n\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 20caefe155..37365dc668 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -295,4 +295,14 @@ test_expect_success 'interaction with submodules' '\n> \tcheck_files super/modules/child \"a deep folder1 folder2\"\n> '\n> \n> +test_expect_success 'different sparse-checkouts with worktrees' '\n> +\tgit -C repo worktree add --detach ../worktree &&\n> +\tcheck_files worktree \"a deep folder1 folder2\" &&\n> +\tgit -C worktree sparse-checkout init --cone &&\n> +\tgit -C repo sparse-checkout set folder1 &&\n> +\tgit -C worktree sparse-checkout set deep/deeper1 &&\n> +\tcheck_files repo \"a folder1\" &&\n> +\tcheck_files worktree \"a deep\"\n> +'\n> +\n> test_done\n"},{"id":"390396","messageId":"d4fe0b4a-9e3f-6669-9068-d7b4b8934d1b@gmail.com","threadId":"52638","inReplyTo":"20200114224818.GD3957260@coredump.intra.peff.net","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-24T21:10:21Z","receivedAt":"2020-01-24T21:10:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/14/2020 5:48 PM, Jeff King wrote:\n> On Tue, Jan 14, 2020 at 05:11:03PM -0500, Derrick Stolee wrote:\n> \n>>> Do we need to document these rules somewhere? Naively I'd expect\n>>> \"--stdin\" to take in literal pathnames. But of course it can't represent\n>>> a path with a newline. So perhaps it makes sense to take quoted names by\n>>> default, and allow literal NUL-separated input with \"-z\" if anybody\n>>> wants it.\n>>\n>> This is worth thinking about the right way to describe the rules:\n>>\n>> 1. You don't _need_ quotes. They happen to come along for the ride in\n>> 'git ls-tree' so it doesn't mess up shell scripts that iterate on\n>> those entries. At least, that's why I think they are quoted.\n> \n> It's not just shell scripts. Without quoting, the syntax becomes\n> ambiguous (e.g., imagine a file with a newline in it). So most Git\n> output that shows a filename will quote it if necessary, unless\n> NUL separators are being used.\n\nGood to know.\n\n>> 2. If you use quotes, the first layer of quotes will be removed.\n> \n> I take this to mean that anything starting with a double-quote will have\n> the outer layer removed, and backslash escapes inside expanded. And\n> anything without a starting double quote (even if it has internal\n> backslash escapes!) will be taken literally.\n\nHm. Perhaps you are right! The ls-tree output for the test example\nis:\n\n\tdeep\n\tfolder1\n\tfolder2\n\t\"zbad\\\\dir\"\n\tzdoes*exist\n\nso the \"zdoes*exist\" value is not escaped. I believe the current\nlogic does something extra: consider supplying this input to\n'git sparse-checkout set --stdin':\n\n\tdeep\n\tfolder1\n\tfolder2\n\t\"zbad\\\\dir\"\n\tzdoes\\*exist\n\nthen should we un-escape \"\\*\" to \"*\"? Or is this not a valid input\nbecause a backslash should have been quoted into C-style quotes?\n\nThe behavior in the current series allows this output that would\nnever be written by \"git ls-tree\".\n\n> That would match how things like \"update-index --index-info\" work.\n> \n> As far as implementation, I know you're trying to keep some of the\n> escaping, but I think it might make more sense to do use\n> unquote_c_style() to parse the input (see update-index's use for some\n> prior art), and then re-quote as necessary to put things into the\n> sparse-checkout file (I guess quoting more than just quote_c_style()\n> would do, since you need to quote glob metacharacters like '*' and\n> probably \"!\"). But as much as possible, I think you'd want literal\n> strings inside the program, and just quoting/unquoting at the edges.\n\nI was playing around with this, and I think that quote_c_style() is\nnecessary for the output, but we have a strange in-memory situation\nfor the other escaping: we both fill the hashsets with the un-escaped\ndata and fill the pattern list with the escaped patterns.\n\nI'll add a commit with the quote_c_style() calls during the 'list'\nsubcommand.\n\n>> How much of this needs to be documented explicitly, or how much should\n>> we say \"The input format matches what we would expect from 'git ls-tree\n>> --name-only'\"?\n> \n> I think it's fine to say that, and maybe call attention to the quoting.\n> Like:\n> \n> The input format matches the output of `git ls-tree --name-only`. This\n> includes interpreting pathnames that begin with a double quote (\") as\n> C-style quoted strings.\n> \n> Disappointingly, update-index does not seem to explain the rules\n> anywhere. fast-import does cover it. Maybe it's something that ought to\n> be hoisted out into gitcli(7) or similar (or maybe it has been and I\n> just can't find it).\n\nI'll start the process by using your recommended language. I noticed\nalso that the 'set' command doesn't actually document what happens\nwhen in cone mode, so I will correct that, too.\n\nThanks,\n-Stolee\n\n"},{"id":"390399","messageId":"1cc825412f0f2b66f7b89ed5ce33bcda937f4ee6.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 01/12] t1091: use check_files to reduce boilerplate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:31Z","receivedAt":"2020-01-24T21:19:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen testing the sparse-checkout feature, we need to compare the\ncontents of the working-directory against some expected output.\nUsing here-docs was useful in the beginning, but became repetetive\nas the test script grew.\n\nCreate a check_files helper to make the tests simpler and easier\nto extend. It also reduces instances of bad here-doc whitespace.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 117 ++++++-----------------------\n 1 file changed, 22 insertions(+), 95 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ff7f8f7a1f..e058a20ad6 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -12,6 +12,13 @@ list_files() {\n \t(cd \"$1\" && printf '%s\\n' *)\n }\n \n+check_files() {\n+\tlist_files \"$1\" >actual &&\n+\tshift &&\n+\tprintf \"%s\\n\" $@ >expect &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'setup' '\n \tgit init repo &&\n \t(\n@@ -58,9 +65,7 @@ test_expect_success 'git sparse-checkout init' '\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n-\tlist_files repo >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a\n '\n \n test_expect_success 'git sparse-checkout list after init' '\n@@ -81,13 +86,7 @@ test_expect_success 'init with existing sparse-checkout' '\n \t\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'clone --sparse' '\n@@ -98,9 +97,7 @@ test_expect_success 'clone --sparse' '\n \t\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n-\tlist_files clone >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files clone a\n '\n \n test_expect_success 'set enables config' '\n@@ -127,13 +124,7 @@ test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n@@ -147,13 +138,7 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: match patterns' '\n@@ -162,13 +147,7 @@ test_expect_success 'cone mode: match patterns' '\n \tgit -C repo read-tree -mu HEAD 2>err &&\n \ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: warn on bad pattern' '\n@@ -185,14 +164,7 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_path_is_file repo/.git/info/sparse-checkout &&\n \tgit -C repo config --list >config &&\n \ttest_must_fail git config core.sparseCheckout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a deep folder1 folder2\n '\n \n test_expect_success 'cone mode: init and set' '\n@@ -204,24 +176,9 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir &&\n \tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n \ttest_must_be_empty err &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep/deeper1 >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeepest\n-\tEOF\n-\ttest_cmp expect dir &&\n+\tcheck_files repo a deep &&\n+\tcheck_files repo/deep a deeper1 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n \tcat >expect <<-EOF &&\n \t\t/*\n \t\t!/*/\n@@ -237,13 +194,7 @@ test_expect_success 'cone mode: init and set' '\n \t\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\tlist_files repo >dir &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n@@ -275,13 +226,7 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n \ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n \ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\t\tdeeper2\n-\tEOF\n-\ttest_cmp dir expect\n+\tcheck_files repo/deep a deeper1 deeper2\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -332,12 +277,7 @@ test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \t\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1\n '\n \n test_expect_success 'interaction with submodules' '\n@@ -351,21 +291,8 @@ test_expect_success 'interaction with submodules' '\n \t\tgit sparse-checkout init --cone &&\n \t\tgit sparse-checkout set folder1\n \t) &&\n-\tlist_files super >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tmodules\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files super/modules/child >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files super a folder1 modules &&\n+\tcheck_files super/modules/child a deep folder1 folder2\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390400","messageId":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.git.1579029962.gitgitgadget@gmail.com","subject":"[PATCH v2 00/12] Harden the sparse-checkout builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:30Z","receivedAt":"2020-01-24T21:19:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/sparse-list-in-cone-mode.\n\nThis series attempts to clean up some rough edges in the sparse-checkout\nfeature, especially around the cone mode.\n\nUnfortunately, after the v2.25.0 release, we noticed an issue with the \"git\nclone --sparse\" option when using a URL instead of a local path. This is\nfixed and properly tested here.\n\nAlso, let's improve Git's response to these more complicated scenarios:\n\n 1. Running \"git sparse-checkout init\" in a worktree would complain because\n the \"info\" dir doesn't exist.\n 2. Tracked paths that include \"*\" and \"\\\" in their filenames.\n 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n as \"**\" anywhere or \"*\" in the wrong place, then we should respond\n appropriately. That is: warn that the patterns are not cone-mode, then\n revert to the old logic.\n\nUpdates in V2:\n\n * Added C-style quoting to the output of \"git sparse-checkout list\" in cone\n mode.\n * Improved documentation.\n * Responded to most style feedback. Hopefully I didn't miss anything.\n * I was lingering on this a little to see if I could also fix the issue\n raised in [1], but I have not figured that one out, yet.\n\n[1] \nhttps://lore.kernel.org/git/062301d5d0bc$c3e17760$4ba46620$@Frontier.com/\n\nThanks, -Stolee\n\nDerrick Stolee (11):\n t1091: use check_files to reduce boilerplate\n t1091: improve here-docs\n sparse-checkout: create leading directories\n clone: fix --sparse option with URLs\n sparse-checkout: cone mode does not recognize \"**\"\n sparse-checkout: detect short patterns\n sparse-checkout: warn on incorrect '*' in patterns\n sparse-checkout: properly match escaped characters\n sparse-checkout: write escaped patterns in cone mode\n sparse-checkout: use C-style quotes in 'list' subcommand\n sparse-checkout: improve docs around 'set' in cone mode\n\nJeff King (1):\n sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n\n Documentation/git-sparse-checkout.txt | 19 +-\n builtin/clone.c | 2 +-\n builtin/sparse-checkout.c | 59 ++++-\n dir.c | 68 +++++-\n dir.h | 1 +\n t/t1091-sparse-checkout-builtin.sh | 323 +++++++++++++++-----------\n 6 files changed, 317 insertions(+), 155 deletions(-)\n\n\nbase-commit: 4fd683b6a35eabd23dd5183da7f654a1e1f00325\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-513%2Fderrickstolee%2Fsparse-harden-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-513/derrickstolee/sparse-harden-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/513\n\nRange-diff vs v1:\n\n -: ---------- > 1: 1cc825412f t1091: use check_files to reduce boilerplate\n 1: 9f7791ae5e ! 2: b7a6ad145a t1091: use check_files to reduce boilerplate\n @@ -1,14 +1,11 @@\n Author: Derrick Stolee \n \n - t1091: use check_files to reduce boilerplate\n + t1091: improve here-docs\n \n - When testing the sparse-checkout feature, we need to compare the\n - contents of the working-directory against some expected output.\n - Using here-docs was useful in the beginning, but became repetetive\n - as the test script grew.\n -\n - Create a check_files helper to make the tests simpler and easier\n - to extend. It also reduces instances of bad here-doc whitespace.\n + t1091-sparse-checkout-builtin.sh uses here-docs to populate the\n + expected contents of the sparse-checkout file. These do not use\n + shell interpolation, so use \"-\\EOF\" instead of \"-EOF\". Also use\n + proper tabbing.\n \n Signed-off-by: Derrick Stolee \n \n @@ -16,20 +13,6 @@\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \t(cd \"$1\" && printf '%s\\n' *)\n - }\n - \n -+check_files() {\n -+\tDIR=$1\n -+\tprintf \"%s\\n\" $2 >expect &&\n -+\tlist_files $DIR >actual &&\n -+\ttest_cmp expect actual\n -+}\n -+\n - test_expect_success 'setup' '\n - \tgit init repo &&\n - \t(\n -@@\n \n test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n @@ -59,11 +42,7 @@\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n --\tlist_files repo >dir &&\n --\techo a >expect &&\n --\ttest_cmp expect dir\n -+\tcheck_files repo a\n - '\n +@@\n \n test_expect_success 'git sparse-checkout list after init' '\n \tgit -C repo sparse-checkout list >actual &&\n @@ -90,16 +69,8 @@\n +\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1 folder2\"\n - '\n - \n + \tcheck_files repo a folder1 folder2\n +@@\n test_expect_success 'clone --sparse' '\n \tgit clone --sparse repo clone &&\n \tgit -C clone sparse-checkout list >actual &&\n @@ -111,13 +82,7 @@\n +\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n --\tlist_files clone >dir &&\n --\techo a >expect &&\n --\ttest_cmp expect dir\n -+\tcheck_files clone a\n - '\n - \n - test_expect_success 'set enables config' '\n + \tcheck_files clone a\n @@\n \n test_expect_success 'set sparse-checkout using builtin' '\n @@ -133,15 +98,7 @@\n \tEOF\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n - \ttest_cmp expect repo/.git/info/sparse-checkout &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1 folder2\"\n +@@\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n @@ -158,72 +115,10 @@\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual &&\n - \ttest_cmp expect actual &&\n - \ttest_cmp expect repo/.git/info/sparse-checkout &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1 folder2\"\n - '\n - \n - test_expect_success 'cone mode: match patterns' '\n -@@\n - \tgit -C repo read-tree -mu HEAD 2>err &&\n - \ttest_i18ngrep ! \"disabling cone patterns\" err &&\n - \tgit -C repo reset --hard &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1 folder2\"\n - '\n - \n - test_expect_success 'cone mode: warn on bad pattern' '\n @@\n - \ttest_path_is_file repo/.git/info/sparse-checkout &&\n - \tgit -C repo config --list >config &&\n - \ttest_must_fail git config core.sparseCheckout &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tdeep\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a deep folder1 folder2\"\n - '\n - \n - test_expect_success 'cone mode: init and set' '\n -@@\n - \ttest_cmp expect dir &&\n - \tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n - \ttest_must_be_empty err &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tdeep\n --\tEOF\n --\ttest_cmp expect dir &&\n --\tlist_files repo/deep >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tdeeper1\n --\tEOF\n --\ttest_cmp expect dir &&\n --\tlist_files repo/deep/deeper1 >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tdeepest\n --\tEOF\n --\ttest_cmp expect dir &&\n + \tcheck_files repo a deep &&\n + \tcheck_files repo/deep a deeper1 &&\n + \tcheck_files repo/deep/deeper1 a deepest &&\n -\tcat >expect <<-EOF &&\n -\t\t/*\n -\t\t!/*/\n @@ -232,9 +127,6 @@\n -\t\t/deep/deeper1/\n -\t\t!/deep/deeper1/*/\n -\t\t/deep/deeper1/deepest/\n -+\tcheck_files repo \"a deep\" &&\n -+\tcheck_files repo/deep \"a deeper1\" &&\n -+\tcheck_files repo/deep/deeper1 \"a deepest\" &&\n +\tcat >expect <<-\\EOF &&\n +\t/*\n +\t!/*/\n @@ -253,14 +145,7 @@\n +\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\tlist_files repo >dir &&\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1 folder2\"\n + \tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n @@ -288,21 +173,6 @@\n \tEOF\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n -@@\n - \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n - \ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n - \ttest_cmp repo/.git/info/sparse-checkout expect &&\n --\tlist_files repo/deep >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tdeeper1\n --\t\tdeeper2\n --\tEOF\n --\ttest_cmp dir expect\n -+\tcheck_files repo/deep \"a deeper1 deeper2\"\n - '\n - \n - test_expect_success 'revert to old sparse-checkout on empty update' '\n @@\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \tgit -C repo sparse-checkout init --cone &&\n @@ -317,37 +187,4 @@\n +\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n --\tlist_files repo >dir &&\n --\tcat >expect <<-EOF &&\n --\t\ta\n --\t\tfolder1\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files repo \"a folder1\"\n - '\n - \n - test_expect_success 'interaction with submodules' '\n -@@\n - \t\tgit sparse-checkout init --cone &&\n - \t\tgit sparse-checkout set folder1\n - \t) &&\n --\tlist_files super >dir &&\n --\tcat >expect <<-\\EOF &&\n --\t\ta\n --\t\tfolder1\n --\t\tmodules\n --\tEOF\n --\ttest_cmp expect dir &&\n --\tlist_files super/modules/child >dir &&\n --\tcat >expect <<-\\EOF &&\n --\t\ta\n --\t\tdeep\n --\t\tfolder1\n --\t\tfolder2\n --\tEOF\n --\ttest_cmp expect dir\n -+\tcheck_files super \"a folder1 modules\" &&\n -+\tcheck_files super/modules/child \"a deep folder1 folder2\"\n - '\n - \n - test_done\n + \tcheck_files repo a folder1\n 2: 53a266f9aa ! 3: 5497ad8778 sparse-checkout: create leading directories\n @@ -34,7 +34,7 @@\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \tcheck_files super/modules/child \"a deep folder1 folder2\"\n + \tcheck_files super/modules/child a deep folder1 folder2\n '\n \n +test_expect_success 'different sparse-checkouts with worktrees' '\n @@ -43,8 +43,8 @@\n +\tgit -C worktree sparse-checkout init --cone &&\n +\tgit -C repo sparse-checkout set folder1 &&\n +\tgit -C worktree sparse-checkout set deep/deeper1 &&\n -+\tcheck_files repo \"a folder1\" &&\n -+\tcheck_files worktree \"a deep\"\n ++\tcheck_files repo a folder1 &&\n ++\tcheck_files worktree a deep\n +'\n +\n test_done\n 3: 3ef8e021a5 ! 4: 4991a51f6d clone: fix --sparse option with URLs\n @@ -22,6 +22,7 @@\n Update that target directory to evaluate this correctly. I have also\n manually tested that https:// URLs are handled correctly as well.\n \n + Acked-by: Taylor Blau \n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/clone.c b/builtin/clone.c\n -: ---------- > 5: ae78c3069b sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n 4: dfa7e20444 ! 6: 2ad4d3e467 sparse-checkout: cone mode does not recognize \"**\"\n @@ -30,7 +30,6 @@\n \n +\tif (strstr(given->pattern, \"**\")) {\n +\t\t/* Not a cone pattern. */\n -+\t\tpl->use_cone_patterns = 0;\n +\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n +\t\tgoto clear_hashmaps;\n +\t}\n @@ -38,12 +37,17 @@\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n + \t\t\t/* Not a cone pattern. */\n +-\t\t\tpl->use_cone_patterns = 0;\n + \t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n + \t\t\tgoto clear_hashmaps;\n + \t\t}\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \tcheck_files worktree \"a deep\"\n + \tcheck_files worktree a deep\n '\n \n +check_read_tree_errors () {\n @@ -57,7 +61,7 @@\n +\telse\n +\t\ttest_i18ngrep \"$ERRORS\" err\n +\tfi &&\n -+\tcheck_files $REPO \"$FILES\"\n ++\tcheck_files $REPO $FILES\n +}\n +\n +test_expect_success 'pattern-checks: /A/**' '\n 5: 9be49908fd ! 7: aace064510 sparse-checkout: detect short patterns\n @@ -21,8 +21,8 @@\n +\tif (given->patternlen <= 2 ||\n +\t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n - \t\tpl->use_cone_patterns = 0;\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n + \t\tgoto clear_hashmaps;\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n 6: 77a514f50b ! 8: d2a510a3bb sparse-checkout: warn on incorrect '*' in patterns\n @@ -34,8 +34,7 @@\n +\t *given->pattern == '*' ||\n \t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n - \t\tpl->use_cone_patterns = 0;\n -@@\n + \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\n \t}\n \n @@ -57,7 +56,6 @@\n +\t\t\tgoto increment;\n +\n +\t\t/* Not a cone pattern. */\n -+\t\tpl->use_cone_patterns = 0;\n +\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n +\t\tgoto clear_hashmaps;\n +\n 7: 09dbe1f902 ! 9: 65c53d7526 sparse-checkout: properly match escaped characters\n @@ -9,7 +9,7 @@\n Create dup_and_filter_pattern() method to convert a pattern by\n removing escape characters and dropping an optional \"/*\" at the end.\n This method is available in dir.h as we will use it in\n - builtin/sparse-chekcout.c in a later change.\n + builtin/sparse-checkout.c in a later change.\n \n Signed-off-by: Derrick Stolee \n \n 8: 79b6e9a565 = 10: c27a17a2fc sparse-checkout: write escaped patterns in cone mode\n -: ---------- > 11: 526d5becbc sparse-checkout: use C-style quotes in 'list' subcommand\n -: ---------- > 12: 1b5858adee sparse-checkout: improve docs around 'set' in cone mode\n\n-- \ngitgitgadget\n"},{"id":"390401","messageId":"b7a6ad145af6a5560bb07ed7e36f587f0f60f41f.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 02/12] t1091: improve here-docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:32Z","receivedAt":"2020-01-24T21:19:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nt1091-sparse-checkout-builtin.sh uses here-docs to populate the\nexpected contents of the sparse-checkout file. These do not use\nshell interpolation, so use \"-\\EOF\" instead of \"-EOF\". Also use\nproper tabbing.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 98 +++++++++++++++---------------\n 1 file changed, 49 insertions(+), 49 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e058a20ad6..e28e1c797f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -46,11 +46,11 @@ test_expect_success 'git sparse-checkout list (empty)' '\n \n test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n-\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n-\t\t/folder1/*\n-\t\t/deep/\n-\t\t**/a\n-\t\t!*bin*\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/folder1/*\n+\t/deep/\n+\t**/a\n+\t!*bin*\n \tEOF\n \tcp repo/.git/info/sparse-checkout expect &&\n \tgit -C repo sparse-checkout list >list &&\n@@ -59,9 +59,9 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \n test_expect_success 'git sparse-checkout init' '\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n@@ -70,9 +70,9 @@ test_expect_success 'git sparse-checkout init' '\n \n test_expect_success 'git sparse-checkout list after init' '\n \tgit -C repo sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual\n '\n@@ -80,10 +80,10 @@ test_expect_success 'git sparse-checkout list after init' '\n test_expect_success 'init with existing sparse-checkout' '\n \techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1 folder2\n@@ -92,9 +92,9 @@ test_expect_success 'init with existing sparse-checkout' '\n test_expect_success 'clone --sparse' '\n \tgit clone --sparse repo clone &&\n \tgit -C clone sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n \tcheck_files clone a\n@@ -116,10 +116,10 @@ test_expect_success 'set enables config' '\n \n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n@@ -128,11 +128,11 @@ test_expect_success 'set sparse-checkout using builtin' '\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n-\t\t/folder2/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n+\t/folder2/\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual &&\n@@ -179,28 +179,28 @@ test_expect_success 'cone mode: init and set' '\n \tcheck_files repo a deep &&\n \tcheck_files repo/deep a deeper1 &&\n \tcheck_files repo/deep/deeper1 a deepest &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n-\t\t!/deep/*/\n-\t\t/deep/deeper1/\n-\t\t!/deep/deeper1/*/\n-\t\t/deep/deeper1/deepest/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t!/deep/*/\n+\t/deep/deeper1/\n+\t!/deep/deeper1/*/\n+\t/deep/deeper1/deepest/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n \tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n-\tcat >expect <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tcat >expect <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual 2>err &&\n@@ -211,10 +211,10 @@ test_expect_success 'cone mode: list' '\n test_expect_success 'cone mode: set with nested folders' '\n \tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n \ttest_line_count = 0 err &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n \tEOF\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n@@ -271,10 +271,10 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1\n-- \ngitgitgadget\n\n"},{"id":"390402","messageId":"4991a51f6d5d840eaa3bb830e68f1530c2ee08e4.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 04/12] clone: fix --sparse option with URLs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:34Z","receivedAt":"2020-01-24T21:19:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --sparse option was added to the clone builtin in d89f09c (clone:\nadd --sparse mode, 2019-11-21) and was tested with a local path clone\nin t1091-sparse-checkout-builtin.sh. However, due to a difference in\nhow local paths are handled versus URLs, this mechanism does not work\nwith URLs.\n\nModify the test to use a \"file://\" URL, which would output this error\nbefore the code change:\n\n Cloning into 'clone'...\n fatal: cannot change to 'file://.../repo': No such file or directory\n error: failed to initialize sparse-checkout\n\nThese errors are due to using a \"-C \" option to call 'git -C\n sparse-checkout init' but the URL is being given instead of\nthe target directory.\n\nUpdate that target directory to evaluate this correctly. I have also\nmanually tested that https:// URLs are handled correctly as well.\n\nAcked-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4348d962c9..2caefc44fb 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1130,7 +1130,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\tif (option_sparse_checkout && git_sparse_checkout_init(dir))\n \t\treturn 1;\n \n \tremote = remote_get(option_origin);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 43d1f7520c..cf4a595c86 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -90,7 +90,7 @@ test_expect_success 'init with existing sparse-checkout' '\n '\n \n test_expect_success 'clone --sparse' '\n-\tgit clone --sparse repo clone &&\n+\tgit clone --sparse \"file://$(pwd)/repo\" clone &&\n \tgit -C clone sparse-checkout list >actual &&\n \tcat >expect <<-\\EOF &&\n \t/*\n-- \ngitgitgadget\n\n"},{"id":"390404","messageId":"ae78c3069b08557e3acc544601c71ee932ace19d.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 05/12] sparse-checkout: fix documentation typo for core.sparseCheckoutCone","fromName":"Jeff King via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:35Z","receivedAt":"2020-01-24T21:19:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"From: Jeff King \n\nSigned-off-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 3b341cf0fc..4834fb434d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -106,7 +106,7 @@ The full pattern set allows for arbitrary pattern matches and complicated\n inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n updating the index, where N is the number of patterns and M is the number\n of paths in the index. To combat this performance issue, a more restricted\n-pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+pattern set is allowed when `core.sparseCheckoutCone` is enabled.\n \n The accepted patterns in the cone pattern set are:\n \n-- \ngitgitgadget\n\n"},{"id":"390403","messageId":"2ad4d3e467a7e2be679de4084e54f426aa242bd6.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 06/12] sparse-checkout: cone mode does not recognize \"**\"","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:36Z","receivedAt":"2020-01-24T21:19:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen core.sparseCheckoutCone is enabled, the 'git sparse-checkout set'\ncommand creates a restricted set of possible patterns that are used\nby a custom algorithm to quickly match those patterns.\n\nIf a user manually edits the sparse-checkout file, then they could\ncreate patterns that do not match these expectations. The cone-mode\nmatching algorithm can return incorrect results. The solution is to\ndetect these incorrect patterns, warn that we do not recognize them,\nand revert to the standard algorithm.\n\nCheck each pattern for the \"**\" substring, and revert to the old\nlogic if seen. While technically a \"//**\" pattern matches\nthe meaning of \"//\", it is not one that would be written by\nthe sparse-checkout builtin in cone mode. Attempting to accept that\npattern change complicates the logic and instead we punt and do\nnot accept any instance of \"**\".\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 7 +++++-\n t/t1091-sparse-checkout-builtin.sh | 34 ++++++++++++++++++++++++++++++\n 2 files changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 22d08e61c2..40fed73a94 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,11 +651,16 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n+\tif (strstr(given->pattern, \"**\")) {\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n \t\t\t/* Not a cone pattern. */\n-\t\t\tpl->use_cone_patterns = 0;\n \t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\t\tgoto clear_hashmaps;\n \t\t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex cf4a595c86..e2e45dc7fd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -305,4 +305,38 @@ test_expect_success 'different sparse-checkouts with worktrees' '\n \tcheck_files worktree a deep\n '\n \n+check_read_tree_errors () {\n+\tREPO=$1\n+\tFILES=$2\n+\tERRORS=$3\n+\tgit -C $REPO read-tree -mu HEAD 2>err &&\n+\tif test -z \"$ERRORS\"\n+\tthen\n+\t\ttest_must_be_empty err\n+\telse\n+\t\ttest_i18ngrep \"$ERRORS\" err\n+\tfi &&\n+\tcheck_files $REPO $FILES\n+}\n+\n+test_expect_success 'pattern-checks: /A/**' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/**\n+\tEOF\n+\tcheck_read_tree_errors repo \"a folder1\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: /A/**/B/' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/**/deepest\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\" &&\n+\tcheck_files repo/deep \"deeper1\" &&\n+\tcheck_files repo/deep/deeper1 \"deepest\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390405","messageId":"5497ad8778c29ca936206d2ded6773758c236c12.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 03/12] sparse-checkout: create leading directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:33Z","receivedAt":"2020-01-24T21:19:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git init' command creates the \".git/info\" directory and fills it\nwith some default files. However, 'git worktree add' does not create\nthe info directory for that worktree. This causes a problem when running\n\"git sparse-checkout init\" inside a worktree. While care was taken to\nallow the sparse-checkout config to be specific to a worktree, this\ninitialization was untested.\n\nSafely create the leading directories for the sparse-checkout file. This\nis the safest thing to do even without worktrees, as a user could delete\ntheir \".git/info\" directory and expect Git to recover safely.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 4 ++++\n t/t1091-sparse-checkout-builtin.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b3bed891cb..3cee8ab46e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -199,6 +199,10 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tint result;\n \n \tsparse_filename = get_sparse_checkout_filename();\n+\n+\tif (safe_create_leading_directories(sparse_filename))\n+\t\tdie(_(\"failed to create directory for sparse-checkout file\"));\n+\n \tfd = hold_lock_file_for_update(&lk, sparse_filename,\n \t\t\t\t LOCK_DIE_ON_ERROR);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e28e1c797f..43d1f7520c 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -295,4 +295,14 @@ test_expect_success 'interaction with submodules' '\n \tcheck_files super/modules/child a deep folder1 folder2\n '\n \n+test_expect_success 'different sparse-checkouts with worktrees' '\n+\tgit -C repo worktree add --detach ../worktree &&\n+\tcheck_files worktree \"a deep folder1 folder2\" &&\n+\tgit -C worktree sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set folder1 &&\n+\tgit -C worktree sparse-checkout set deep/deeper1 &&\n+\tcheck_files repo a folder1 &&\n+\tcheck_files worktree a deep\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390406","messageId":"65c53d7526ca3cae1a1bfab8c9727221542665f6.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 09/12] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:39Z","receivedAt":"2020-01-24T21:19:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout feature uses hashset containment\nqueries to match paths. Make this algorithm respect escaped asterisk\n(*) and backslash (\\) characters.\n\nCreate dup_and_filter_pattern() method to convert a pattern by\nremoving escape characters and dropping an optional \"/*\" at the end.\nThis method is available in dir.h as we will use it in\nbuiltin/sparse-checkout.c in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 31 +++++++++++++++++++++++++++---\n dir.h | 1 +\n t/t1091-sparse-checkout-builtin.sh | 22 +++++++++++++++++----\n 3 files changed, 47 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 7cb78c8b87..6d8abc09c3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -630,6 +630,32 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n }\n \n+char *dup_and_filter_pattern(const char *pattern)\n+{\n+\tchar *set, *read;\n+\tchar *result = xstrdup(pattern);\n+\n+\tset = result;\n+\tread = result;\n+\n+\twhile (*read) {\n+\t\t/* skip escape characters (once) */\n+\t\tif (*read == '\\\\')\n+\t\t\tread++;\n+\n+\t\t*set = *read;\n+\n+\t\tset++;\n+\t\tread++;\n+\t}\n+\t*set = 0;\n+\n+\tif (*(read - 2) == '/' && *(read - 1) == '*')\n+\t\t*(read - 2) = 0;\n+\n+\treturn result;\n+}\n+\n static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n {\n \tstruct pattern_entry *translated;\n@@ -695,8 +721,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\t\tgoto clear_hashmaps;\n \t\t}\n \n-\t\ttruncated = xstrdup(given->pattern);\n-\t\ttruncated[given->patternlen - 2] = 0;\n+\t\ttruncated = dup_and_filter_pattern(given->pattern);\n \n \t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n \t\ttranslated->pattern = truncated;\n@@ -730,7 +755,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \n \ttranslated = xmalloc(sizeof(struct pattern_entry));\n \n-\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->pattern = dup_and_filter_pattern(given->pattern);\n \ttranslated->patternlen = given->patternlen;\n \thashmap_entry_init(&translated->ent,\n \t\t\t ignore_case ?\ndiff --git a/dir.h b/dir.h\nindex 77a43dbf89..6dcd9d33e7 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -304,6 +304,7 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \t\t const struct hashmap_entry *a,\n \t\t const struct hashmap_entry *b,\n \t\t const void *key);\n+char *dup_and_filter_pattern(const char *pattern);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 470900f6f4..0a21a5e15d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -366,13 +366,27 @@ test_expect_success 'pattern-checks: starting \"*\"' '\n \tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n '\n \n-test_expect_success 'pattern-checks: escaped \"*\"' '\n-\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n+\tgit clone repo escaped &&\n+\tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n+\tNEWTREE=$(git -C escaped mktree <<-EOF\n+\t$(git -C escaped ls-tree HEAD)\n+\t040000 tree $TREEOID\tzbad\\\\dir\n+\t040000 tree $TREEOID\tzdoes*exist\n+\tEOF\n+\t) &&\n+\tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n+\tgit -C escaped reset --hard $COMMIT &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout init --cone &&\n+\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n \t/*\n \t!/*/\n-\t/does\\*not\\*exist/\n+\t/zbad\\\\dir/\n+\t/zdoes\\*not\\*exist/\n+\t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors repo \"a\" \"\"\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390407","messageId":"d2a510a3bb76db4c432187d631d5f6f8085af7e8.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 08/12] sparse-checkout: warn on incorrect '*' in patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:38Z","receivedAt":"2020-01-24T21:19:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout commmand will write patterns that\nallow faster pattern matching. This matching only works if the patterns\nin the sparse-checkout file are those written by that command. Users\ncan edit the sparse-checkout file and create patterns that cause the\ncone mode matching to fail.\n\nThe cone mode patterns may end in \"/*\" but otherwise an un-escaped\nasterisk is invalid. Add checks to disable cone mode when seeing these\nvalues.\n\nA later change will properly handle escaped asterisks.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 29 +++++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 27 +++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex c2e585607e..7cb78c8b87 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -635,6 +635,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \tstruct pattern_entry *translated;\n \tchar *truncated;\n \tchar *data = NULL;\n+\tconst char *prev, *cur, *next;\n \n \tif (!pl->use_cone_patterns)\n \t\treturn;\n@@ -652,12 +653,40 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t}\n \n \tif (given->patternlen <= 2 ||\n+\t *given->pattern == '*' ||\n \t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\n \t}\n \n+\tprev = given->pattern;\n+\tcur = given->pattern + 1;\n+\tnext = given->pattern + 2;\n+\n+\twhile (*cur) {\n+\t\t/* We care about *cur == '*' */\n+\t\tif (*cur != '*')\n+\t\t\tgoto increment;\n+\n+\t\t/* But only if *prev != '\\\\' */\n+\t\tif (*prev == '\\\\')\n+\t\t\tgoto increment;\n+\n+\t\t/* But a trailing '/' then '*' is fine */\n+\t\tif (*prev == '/' && *next == 0)\n+\t\t\tgoto increment;\n+\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\n+\tincrement:\n+\t\tprev++;\n+\t\tcur++;\n+\t\tnext++;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2e57534799..470900f6f4 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -348,4 +348,31 @@ test_expect_success 'pattern-checks: too short' '\n \tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n '\n \n+test_expect_success 'pattern-checks: trailing \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a*\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: starting \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*eep/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: escaped \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/does\\*not\\*exist/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390408","messageId":"aace0645107719f62c611296adc44f8c286d8403.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 07/12] sparse-checkout: detect short patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:37Z","receivedAt":"2020-01-24T21:19:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the shortest pattern the sparse-checkout command will\nwrite into the sparse-checkout file is \"/*\". This is handled carefully\nin add_pattern_to_hashsets(), so warn if any other pattern is this\nshort. This will assist future pattern checks by allowing us to assume\nthere are at least three characters in the pattern.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 3 ++-\n t/t1091-sparse-checkout-builtin.sh | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 40fed73a94..c2e585607e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,7 +651,8 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n-\tif (strstr(given->pattern, \"**\")) {\n+\tif (given->patternlen <= 2 ||\n+\t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e2e45dc7fd..2e57534799 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -339,4 +339,13 @@ test_expect_success 'pattern-checks: /A/**/B/' '\n \tcheck_files repo/deep/deeper1 \"deepest\"\n '\n \n+test_expect_success 'pattern-checks: too short' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390409","messageId":"1b5858adee36e326b7a98f3ca2a3c36c9ba5b85c.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 12/12] sparse-checkout: improve docs around 'set' in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:42Z","receivedAt":"2020-01-24T21:19:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe existing documentation does not clarify how the 'set' subcommand\nchanges when core.sparseCheckoutCone is enabled. Correct this by\nchanging some language around the \"A/B/C\" example. Also include a\ndescription of the input format matching the output of 'git ls-tree\n--name-only'.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 4834fb434d..0914619881 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -50,6 +50,14 @@ To avoid interfering with other worktrees, it first enables the\n +\n When the `--stdin` option is provided, the patterns are read from\n standard in as a newline-delimited list instead of from the arguments.\n++\n+When `core.sparseCheckoutCone` is enabled, the input list is considered a\n+list of directories instead of sparse-checkout patterns. The command writes\n+patterns to the sparse-checkout file to include all files contained in those\n+directories (recursively) as well as files that are siblings of ancestor\n+directories. The input format matches the output of `git ls-tree --name-only`.\n+This includes interpreting pathnames that begin with a double quote (\") as\n+C-style quoted strings.\n \n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n@@ -128,9 +136,12 @@ the following patterns:\n ----------------\n \n This says \"include everything in root, but nothing two levels below root.\"\n-If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n-`A/B` are added as parent patterns. The resulting sparse-checkout file is\n-now\n+\n+When in cone mode, the `git sparse-checkout set` subcommand takes a list of\n+directories instead of a list of sparse-checkout patterns. In this mode,\n+the command `git sparse-checkout set A/B/C` sets the directory `A/B/C` as\n+a recursive pattern, the directories `A` and `A/B` are added as parent\n+patterns. The resulting sparse-checkout file is now\n \n ----------------\n /*\n-- \ngitgitgadget\n"},{"id":"390410","messageId":"c27a17a2fcebbc17253c7740dc7efd9bb1db91a3.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:40Z","receivedAt":"2020-01-24T21:19:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user somehow creates a directory with an asterisk (*) or backslash\n(\\), then the \"git sparse-checkout set\" command will struggle to provide\nthe correct pattern in the sparse-checkout file. When not in cone mode,\nthe provided pattern is written directly into the sparse-checkout file.\nHowever, in cone mode we expect a list of paths to directories and then\nwe convert those into patterns.\n\nEven more specifically, the goal is to always allow the following from\nthe root of a repo:\n\n git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n\nThe ls-tree command provides directory names with an unescaped asterisk.\nIt also quotes the directories that contain an escaped backslash. We\nmust remove these quotes, then keep the escaped backslashes.\n\nHowever, there is some care needed for the timing of these escapes. The\nin-memory pattern list is used to update the working directory before\nwriting the patterns to disk. Thus, we need the command to have the\nunescaped names in the hashsets for the cone comparisons, then escape\nthe patterns later.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 48 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 21 +++++++++++--\n 2 files changed, 64 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 3cee8ab46e..61d2c30036 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -140,6 +140,22 @@ static int update_working_directory(struct pattern_list *pl)\n \treturn result;\n }\n \n+static char *escaped_pattern(char *pattern)\n+{\n+\tchar *p = pattern;\n+\tstruct strbuf final = STRBUF_INIT;\n+\n+\twhile (*p) {\n+\t\tif (*p == '*' || *p == '\\\\')\n+\t\t\tstrbuf_addch(&final, '\\\\');\n+\n+\t\tstrbuf_addch(&final, *p);\n+\t\tp++;\n+\t}\n+\n+\treturn strbuf_detach(&final, NULL);\n+}\n+\n static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n {\n \tint i;\n@@ -164,10 +180,11 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \n \t\tif (strlen(pattern))\n \t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t\tfree(pattern);\n \t}\n \n \tstring_list_clear(&sl, 0);\n@@ -185,8 +202,9 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstring_list_remove_duplicates(&sl, 0);\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t\tfree(pattern);\n \t}\n }\n \n@@ -337,7 +355,9 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat\n {\n \tstruct pattern_entry *e = xmalloc(sizeof(*e));\n \te->patternlen = path->len;\n-\te->pattern = strbuf_detach(path, NULL);\n+\te->pattern = dup_and_filter_pattern(path->buf);\n+\tstrbuf_release(path);\n+\n \thashmap_entry_init(&e->ent,\n \t\t\t ignore_case ?\n \t\t\t strihash(e->pattern) :\n@@ -369,6 +389,7 @@ static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *pat\n \n static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n {\n+\tint i;\n \tstrbuf_trim(line);\n \n \tstrbuf_trim_trailing_dir_sep(line);\n@@ -376,6 +397,27 @@ static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n \tif (!line->len)\n \t\treturn;\n \n+\tfor (i = 0; i < line->len; i++) {\n+\t\tif (line->buf[i] == '*') {\n+\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n+\t\t\ti++;\n+\t\t}\n+\n+\t\tif (line->buf[i] == '\\\\') {\n+\t\t\tif (i < line->len - 1 && line->buf[i + 1] == '\\\\')\n+\t\t\t\ti++;\n+\t\t\telse\n+\t\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n+\n+\t\t\ti++;\n+\t\t}\n+\t}\n+\n+\tif (line->buf[0] == '\"' && line->buf[line->len - 1] == '\"') {\n+\t\tstrbuf_remove(line, 0, 1);\n+\t\tstrbuf_remove(line, line->len - 1, 1);\n+\t}\n+\n \tif (line->buf[0] != '/')\n \t\tstrbuf_insert(line, 0, \"/\", 1);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0a21a5e15d..2bb30cbe29 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -309,6 +309,9 @@ check_read_tree_errors () {\n \tREPO=$1\n \tFILES=$2\n \tERRORS=$3\n+\tgit -C $REPO -c core.sparseCheckoutCone=false read-tree -mu HEAD 2>err &&\n+\ttest_must_be_empty err &&\n+\tcheck_files $REPO \"$FILES\" &&\n \tgit -C $REPO read-tree -mu HEAD 2>err &&\n \tif test -z \"$ERRORS\"\n \tthen\n@@ -379,14 +382,28 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tgit -C escaped reset --hard $COMMIT &&\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n-\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n+\tgit -C escaped sparse-checkout set zbad\\\\dir zdoes\\*not\\*exist zdoes\\*exist &&\n+\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n \t/zbad\\\\dir/\n+\t/zdoes\\*exist/\n \t/zdoes\\*not\\*exist/\n+\tEOF\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t/folder1/\n+\t/folder2/\n+\t/zbad\\\\dir/\n \t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390411","messageId":"526d5becbc87591d7ad37a589a2e44e933bb3e87.1579900782.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v2 11/12] sparse-checkout: use C-style quotes in 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-24T21:19:41Z","receivedAt":"2020-01-24T21:20:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen in cone mode, the 'git sparse-checkout list' subcommand lists\nthe directories included in the sparse cone. When these directories\ncontain odd characters, such as a backslash, then we need to use\nC-style quotes similar to 'git ls-tree'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 7 +++++--\n t/t1091-sparse-checkout-builtin.sh | 7 +++++--\n 2 files changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 61d2c30036..83c8e9bb0c 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -13,6 +13,7 @@\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n #include \"wt-status.h\"\n+#include \"quote.h\"\n \n static const char *empty_base = \"\";\n \n@@ -77,8 +78,10 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n \t\tstring_list_sort(&sl);\n \n-\t\tfor (i = 0; i < sl.nr; i++)\n-\t\t\tprintf(\"%s\\n\", sl.items[i].string);\n+\t\tfor (i = 0; i < sl.nr; i++) {\n+\t\t\tquote_c_style(sl.items[i].string, NULL, stdout, 0);\n+\t\t\tprintf(\"\\n\");\n+\t\t}\n \n \t\treturn 0;\n \t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2bb30cbe29..16dd924291 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -392,7 +392,8 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n \tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n-\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tgit -C escaped ls-tree -d --name-only HEAD >list-expect &&\n+\tgit -C escaped sparse-checkout set --stdin expect <<-\\EOF &&\n \t/*\n \t!/*/\n@@ -403,7 +404,9 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/zdoes\\*exist/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout list >list-actual &&\n+\ttest_cmp list-expect list-actual\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390415","messageId":"20200124214206.GA2211026@coredump.intra.peff.net","threadId":"52638","inReplyTo":"d4fe0b4a-9e3f-6669-9068-d7b4b8934d1b@gmail.com","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-24T21:42:06Z","receivedAt":"2020-01-24T21:42:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 24, 2020 at 04:10:21PM -0500, Derrick Stolee wrote:\n\n> Hm. Perhaps you are right! The ls-tree output for the test example\n> is:\n> \n> \tdeep\n> \tfolder1\n> \tfolder2\n> \t\"zbad\\\\dir\"\n> \tzdoes*exist\n> \n> so the \"zdoes*exist\" value is not escaped. I believe the current\n> logic does something extra: consider supplying this input to\n> 'git sparse-checkout set --stdin':\n> \n> \tdeep\n> \tfolder1\n> \tfolder2\n> \t\"zbad\\\\dir\"\n> \tzdoes\\*exist\n> \n> then should we un-escape \"\\*\" to \"*\"? Or is this not a valid input\n> because a backslash should have been quoted into C-style quotes?\n\nI'd think we should not un-escape anything, because we weren't told that\nthis was a C-style quoted string by the presence of an opening\ndouble-quote. And that's how, say, update-index behaves:\n\n $ blob=$(echo foo | git hash-object -w --stdin)\n $ printf '100644 %s\\t%s\\n' \\\n $blob 'just*asterisk' \\\n $blob 'backslash\\without\\quotes' \\\n $blob '\"backslash\\\\with\\\\quotes\"' |\n git update-index --index-info\n\nwhich results in:\n\n $ git ls-files\n \"backslash\\\\with\\\\quotes\"\n \"backslash\\\\without\\\\quotes\"\n just*asterisk\n\n [same, but without quoting]\n $ git ls-files -z | tr '\\0' '\\n'\n backslash\\with\\quotes\n backslash\\without\\quotes\n just*asterisk\n\n> The behavior in the current series allows this output that would\n> never be written by \"git ls-tree\".\n\nYes, I think we'd never write that, because ls-tree would quote anything\nwith a backslash in it, even though it's not strictly necessary. But it\nwould be valid input to specify a file that has backslashes but not\ndouble-quotes, and I think sparse-checkout should be changed to match\nupdate-index here.\n\n> I was playing around with this, and I think that quote_c_style() is\n> necessary for the output, but we have a strange in-memory situation\n> for the other escaping: we both fill the hashsets with the un-escaped\n> data and fill the pattern list with the escaped patterns.\n\nYeah, but I think that the syntactic escaping on input might not have\nidentical rules to the escaping needed for the patterns.\n\nSo it makes sense to me to handle input as a separate mechanism, get a\npristine copy of what the user was trying to communicate to us, and then\nre-escape whatever we need to put into the pattern list.\n\nAnd ultimately the flow would be something like:\n\n - read input\n - if argument is from command-line, use it verbatim\n - else if reading stdin with \"-z\", use it verbatim\n - else if line starts with double-quote, unquote_c_style()\n - else use line verbatim\n - the result is a single pristine filename\n - fill hashset with pristine filenames\n - generate pattern list to write to sparse file, escaping filenames as\n necessary according to sparse-pattern rules\n\nObviously you don't have a \"-z\" yet, but I think it's something we'd\nprobably want in the long run. And anything coming from the command-line\nshouldn't need quoting to get it to us either (and so we'd need to\nescape before writing to the sparse file).\n\n-Peff\n"},{"id":"390637","messageId":"6864b79c-b22b-1d1e-1f86-1d07a470fdd3@gmail.com","threadId":"52638","inReplyTo":"20200124214206.GA2211026@coredump.intra.peff.net","subject":"Re: [PATCH 8/8] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-28T15:03:28Z","receivedAt":"2020-01-28T15:03:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2020 4:42 PM, Jeff King wrote:\n> And ultimately the flow would be something like:\n> \n> - read input\n> - if argument is from command-line, use it verbatim\n> - else if reading stdin with \"-z\", use it verbatim\n> - else if line starts with double-quote, unquote_c_style()\n> - else use line verbatim\n> - the result is a single pristine filename\n> - fill hashset with pristine filenames\n> - generate pattern list to write to sparse file, escaping filenames as\n> necessary according to sparse-pattern rules\n> \n> Obviously you don't have a \"-z\" yet, but I think it's something we'd\n> probably want in the long run. And anything coming from the command-line\n> shouldn't need quoting to get it to us either (and so we'd need to\n> escape before writing to the sparse file).\n\nThis recommendation came async with my v2, so I'll follow shortly with\na v3 that uses this flow. I have something that I think works, after\nslightly adapting my tests, but now I need to make sure that all the\npatches still make sense and build cleanly.\n\nThanks,\n-Stolee\n\n"},{"id":"390645","messageId":"1cc825412f0f2b66f7b89ed5ce33bcda937f4ee6.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 01/12] t1091: use check_files to reduce boilerplate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:32Z","receivedAt":"2020-01-28T18:26:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen testing the sparse-checkout feature, we need to compare the\ncontents of the working-directory against some expected output.\nUsing here-docs was useful in the beginning, but became repetetive\nas the test script grew.\n\nCreate a check_files helper to make the tests simpler and easier\nto extend. It also reduces instances of bad here-doc whitespace.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 117 ++++++-----------------------\n 1 file changed, 22 insertions(+), 95 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ff7f8f7a1f..e058a20ad6 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -12,6 +12,13 @@ list_files() {\n \t(cd \"$1\" && printf '%s\\n' *)\n }\n \n+check_files() {\n+\tlist_files \"$1\" >actual &&\n+\tshift &&\n+\tprintf \"%s\\n\" $@ >expect &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'setup' '\n \tgit init repo &&\n \t(\n@@ -58,9 +65,7 @@ test_expect_success 'git sparse-checkout init' '\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n-\tlist_files repo >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a\n '\n \n test_expect_success 'git sparse-checkout list after init' '\n@@ -81,13 +86,7 @@ test_expect_success 'init with existing sparse-checkout' '\n \t\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'clone --sparse' '\n@@ -98,9 +97,7 @@ test_expect_success 'clone --sparse' '\n \t\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n-\tlist_files clone >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files clone a\n '\n \n test_expect_success 'set enables config' '\n@@ -127,13 +124,7 @@ test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n@@ -147,13 +138,7 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: match patterns' '\n@@ -162,13 +147,7 @@ test_expect_success 'cone mode: match patterns' '\n \tgit -C repo read-tree -mu HEAD 2>err &&\n \ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: warn on bad pattern' '\n@@ -185,14 +164,7 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_path_is_file repo/.git/info/sparse-checkout &&\n \tgit -C repo config --list >config &&\n \ttest_must_fail git config core.sparseCheckout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a deep folder1 folder2\n '\n \n test_expect_success 'cone mode: init and set' '\n@@ -204,24 +176,9 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir &&\n \tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n \ttest_must_be_empty err &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep/deeper1 >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeepest\n-\tEOF\n-\ttest_cmp expect dir &&\n+\tcheck_files repo a deep &&\n+\tcheck_files repo/deep a deeper1 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n \tcat >expect <<-EOF &&\n \t\t/*\n \t\t!/*/\n@@ -237,13 +194,7 @@ test_expect_success 'cone mode: init and set' '\n \t\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\tlist_files repo >dir &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n@@ -275,13 +226,7 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n \ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n \ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\t\tdeeper2\n-\tEOF\n-\ttest_cmp dir expect\n+\tcheck_files repo/deep a deeper1 deeper2\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -332,12 +277,7 @@ test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \t\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1\n '\n \n test_expect_success 'interaction with submodules' '\n@@ -351,21 +291,8 @@ test_expect_success 'interaction with submodules' '\n \t\tgit sparse-checkout init --cone &&\n \t\tgit sparse-checkout set folder1\n \t) &&\n-\tlist_files super >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tmodules\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files super/modules/child >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files super a folder1 modules &&\n+\tcheck_files super/modules/child a deep folder1 folder2\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390646","messageId":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v2.git.1579900782.gitgitgadget@gmail.com","subject":"[PATCH v3 00/12] Harden the sparse-checkout builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:31Z","receivedAt":"2020-01-28T18:26:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/sparse-list-in-cone-mode.\n\nThis series attempts to clean up some rough edges in the sparse-checkout\nfeature, especially around the cone mode.\n\nUnfortunately, after the v2.25.0 release, we noticed an issue with the \"git\nclone --sparse\" option when using a URL instead of a local path. This is\nfixed and properly tested here.\n\nAlso, let's improve Git's response to these more complicated scenarios:\n\n 1. Running \"git sparse-checkout init\" in a worktree would complain because\n the \"info\" dir doesn't exist.\n 2. Tracked paths that include \"*\" and \"\\\" in their filenames.\n 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n as \"**\" anywhere or \"*\" in the wrong place, then we should respond\n appropriately. That is: warn that the patterns are not cone-mode, then\n revert to the old logic.\n\nUpdates in V2:\n\n * Added C-style quoting to the output of \"git sparse-checkout list\" in cone\n mode.\n * Improved documentation.\n * Responded to most style feedback. Hopefully I didn't miss anything.\n * I was lingering on this a little to see if I could also fix the issue\n raised in [1], but I have not figured that one out, yet.\n\nUpdate in V3:\n\n * Input now uses Peff's recommended pattern: unquote C-style strings over\n stdin and otherwise do not un-escape input.\n\n[1] \nhttps://lore.kernel.org/git/062301d5d0bc$c3e17760$4ba46620$@Frontier.com/\n\nThanks, -Stolee\n\nDerrick Stolee (11):\n t1091: use check_files to reduce boilerplate\n t1091: improve here-docs\n sparse-checkout: create leading directories\n clone: fix --sparse option with URLs\n sparse-checkout: cone mode does not recognize \"**\"\n sparse-checkout: detect short patterns\n sparse-checkout: warn on incorrect '*' in patterns\n sparse-checkout: properly match escaped characters\n sparse-checkout: write escaped patterns in cone mode\n sparse-checkout: use C-style quotes in 'list' subcommand\n sparse-checkout: improve docs around 'set' in cone mode\n\nJeff King (1):\n sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n\n Documentation/git-sparse-checkout.txt | 19 +-\n builtin/clone.c | 2 +-\n builtin/sparse-checkout.c | 48 +++-\n dir.c | 68 +++++-\n t/t1091-sparse-checkout-builtin.sh | 323 +++++++++++++++-----------\n 5 files changed, 305 insertions(+), 155 deletions(-)\n\n\nbase-commit: 4fd683b6a35eabd23dd5183da7f654a1e1f00325\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-513%2Fderrickstolee%2Fsparse-harden-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-513/derrickstolee/sparse-harden-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/513\n\nRange-diff vs v2:\n\n 1: 1cc825412f = 1: 1cc825412f t1091: use check_files to reduce boilerplate\n 2: b7a6ad145a = 2: b7a6ad145a t1091: improve here-docs\n 3: 5497ad8778 = 3: 5497ad8778 sparse-checkout: create leading directories\n 4: 4991a51f6d = 4: 4991a51f6d clone: fix --sparse option with URLs\n 5: ae78c3069b = 5: ae78c3069b sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n 6: 2ad4d3e467 = 6: 2ad4d3e467 sparse-checkout: cone mode does not recognize \"**\"\n 7: aace064510 = 7: aace064510 sparse-checkout: detect short patterns\n 8: d2a510a3bb = 8: d2a510a3bb sparse-checkout: warn on incorrect '*' in patterns\n 9: 65c53d7526 ! 9: 9ea69e9069 sparse-checkout: properly match escaped characters\n @@ -20,7 +20,7 @@\n \treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n }\n \n -+char *dup_and_filter_pattern(const char *pattern)\n ++static char *dup_and_filter_pattern(const char *pattern)\n +{\n +\tchar *set, *read;\n +\tchar *result = xstrdup(pattern);\n @@ -69,18 +69,6 @@\n \thashmap_entry_init(&translated->ent,\n \t\t\t ignore_case ?\n \n - diff --git a/dir.h b/dir.h\n - --- a/dir.h\n - +++ b/dir.h\n -@@\n - \t\t const struct hashmap_entry *a,\n - \t\t const struct hashmap_entry *b,\n - \t\t const void *key);\n -+char *dup_and_filter_pattern(const char *pattern);\n - int hashmap_contains_parent(struct hashmap *map,\n - \t\t\t const char *path,\n - \t\t\t struct strbuf *buffer);\n -\n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n 10: c27a17a2fc ! 10: e2f9afc70c sparse-checkout: write escaped patterns in cone mode\n @@ -24,11 +24,24 @@\n unescaped names in the hashsets for the cone comparisons, then escape\n the patterns later.\n \n + Use unquote_c_style() when parsing lines from stdin. Command-line\n + arguments will be parsed as-is, assuming the user can do the correct\n + level of escaping from their environment to match the exact directory\n + names.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n +@@\n + #include \"resolve-undo.h\"\n + #include \"unpack-trees.h\"\n + #include \"wt-status.h\"\n ++#include \"quote.h\"\n + \n + static const char *empty_base = \"\";\n + \n @@\n \treturn result;\n }\n @@ -77,52 +90,28 @@\n }\n \n @@\n - {\n - \tstruct pattern_entry *e = xmalloc(sizeof(*e));\n - \te->patternlen = path->len;\n --\te->pattern = strbuf_detach(path, NULL);\n -+\te->pattern = dup_and_filter_pattern(path->buf);\n -+\tstrbuf_release(path);\n -+\n - \thashmap_entry_init(&e->ent,\n - \t\t\t ignore_case ?\n - \t\t\t strihash(e->pattern) :\n -@@\n + \t\tpl.use_cone_patterns = 1;\n \n - static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n - {\n -+\tint i;\n - \tstrbuf_trim(line);\n - \n - \tstrbuf_trim_trailing_dir_sep(line);\n -@@\n - \tif (!line->len)\n - \t\treturn;\n - \n -+\tfor (i = 0; i < line->len; i++) {\n -+\t\tif (line->buf[i] == '*') {\n -+\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n -+\t\t\ti++;\n -+\t\t}\n + \t\tif (set_opts.use_stdin) {\n +-\t\t\twhile (!strbuf_getline(&line, stdin))\n ++\t\t\tstruct strbuf unquoted = STRBUF_INIT;\n ++\t\t\twhile (!strbuf_getline(&line, stdin)) {\n ++\t\t\t\tif (line.buf[0] == '\"') {\n ++\t\t\t\t\tstrbuf_setlen(&unquoted, 0);\n ++\t\t\t\t\tif (unquote_c_style(&unquoted, line.buf, NULL))\n ++\t\t\t\t\t\tdie(_(\"unable to unquote C-style string '%s'\"),\n ++\t\t\t\t\t\tline.buf);\n +\n -+\t\tif (line->buf[i] == '\\\\') {\n -+\t\t\tif (i < line->len - 1 && line->buf[i + 1] == '\\\\')\n -+\t\t\t\ti++;\n -+\t\t\telse\n -+\t\t\t\tstrbuf_insert(line, i, \"\\\\\", 1);\n ++\t\t\t\t\tstrbuf_swap(&unquoted, &line);\n ++\t\t\t\t}\n +\n -+\t\t\ti++;\n -+\t\t}\n -+\t}\n + \t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n ++\t\t\t}\n +\n -+\tif (line->buf[0] == '\"' && line->buf[line->len - 1] == '\"') {\n -+\t\tstrbuf_remove(line, 0, 1);\n -+\t\tstrbuf_remove(line, line->len - 1, 1);\n -+\t}\n -+\n - \tif (line->buf[0] != '/')\n - \t\tstrbuf_insert(line, 0, \"/\", 1);\n - \n ++\t\t\tstrbuf_release(&unquoted);\n + \t\t} else {\n + \t\t\tfor (i = 0; i < argc; i++) {\n + \t\t\t\tstrbuf_setlen(&line, 0);\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -142,7 +131,7 @@\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n -\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n -+\tgit -C escaped sparse-checkout set zbad\\\\dir zdoes\\*not\\*exist zdoes\\*exist &&\n ++\tgit -C escaped sparse-checkout set zbad\\\\dir \"zdoes*not*exist\" \"zdoes*exist\" &&\n +\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n 11: 526d5becbc ! 11: ec714a4cf0 sparse-checkout: use C-style quotes in 'list' subcommand\n @@ -12,14 +12,6 @@\n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n -@@\n - #include \"resolve-undo.h\"\n - #include \"unpack-trees.h\"\n - #include \"wt-status.h\"\n -+#include \"quote.h\"\n - \n - static const char *empty_base = \"\";\n - \n @@\n \n \t\tstring_list_sort(&sl);\n 12: 1b5858adee = 12: 1867746d97 sparse-checkout: improve docs around 'set' in cone mode\n\n-- \ngitgitgadget\n"},{"id":"390647","messageId":"b7a6ad145af6a5560bb07ed7e36f587f0f60f41f.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 02/12] t1091: improve here-docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:33Z","receivedAt":"2020-01-28T18:26:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nt1091-sparse-checkout-builtin.sh uses here-docs to populate the\nexpected contents of the sparse-checkout file. These do not use\nshell interpolation, so use \"-\\EOF\" instead of \"-EOF\". Also use\nproper tabbing.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 98 +++++++++++++++---------------\n 1 file changed, 49 insertions(+), 49 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e058a20ad6..e28e1c797f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -46,11 +46,11 @@ test_expect_success 'git sparse-checkout list (empty)' '\n \n test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n-\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n-\t\t/folder1/*\n-\t\t/deep/\n-\t\t**/a\n-\t\t!*bin*\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/folder1/*\n+\t/deep/\n+\t**/a\n+\t!*bin*\n \tEOF\n \tcp repo/.git/info/sparse-checkout expect &&\n \tgit -C repo sparse-checkout list >list &&\n@@ -59,9 +59,9 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \n test_expect_success 'git sparse-checkout init' '\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n@@ -70,9 +70,9 @@ test_expect_success 'git sparse-checkout init' '\n \n test_expect_success 'git sparse-checkout list after init' '\n \tgit -C repo sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual\n '\n@@ -80,10 +80,10 @@ test_expect_success 'git sparse-checkout list after init' '\n test_expect_success 'init with existing sparse-checkout' '\n \techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1 folder2\n@@ -92,9 +92,9 @@ test_expect_success 'init with existing sparse-checkout' '\n test_expect_success 'clone --sparse' '\n \tgit clone --sparse repo clone &&\n \tgit -C clone sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n \tcheck_files clone a\n@@ -116,10 +116,10 @@ test_expect_success 'set enables config' '\n \n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n@@ -128,11 +128,11 @@ test_expect_success 'set sparse-checkout using builtin' '\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n-\t\t/folder2/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n+\t/folder2/\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual &&\n@@ -179,28 +179,28 @@ test_expect_success 'cone mode: init and set' '\n \tcheck_files repo a deep &&\n \tcheck_files repo/deep a deeper1 &&\n \tcheck_files repo/deep/deeper1 a deepest &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n-\t\t!/deep/*/\n-\t\t/deep/deeper1/\n-\t\t!/deep/deeper1/*/\n-\t\t/deep/deeper1/deepest/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t!/deep/*/\n+\t/deep/deeper1/\n+\t!/deep/deeper1/*/\n+\t/deep/deeper1/deepest/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n \tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n-\tcat >expect <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tcat >expect <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual 2>err &&\n@@ -211,10 +211,10 @@ test_expect_success 'cone mode: list' '\n test_expect_success 'cone mode: set with nested folders' '\n \tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n \ttest_line_count = 0 err &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n \tEOF\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n@@ -271,10 +271,10 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1\n-- \ngitgitgadget\n\n"},{"id":"390648","messageId":"4991a51f6d5d840eaa3bb830e68f1530c2ee08e4.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 04/12] clone: fix --sparse option with URLs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:35Z","receivedAt":"2020-01-28T18:26:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --sparse option was added to the clone builtin in d89f09c (clone:\nadd --sparse mode, 2019-11-21) and was tested with a local path clone\nin t1091-sparse-checkout-builtin.sh. However, due to a difference in\nhow local paths are handled versus URLs, this mechanism does not work\nwith URLs.\n\nModify the test to use a \"file://\" URL, which would output this error\nbefore the code change:\n\n Cloning into 'clone'...\n fatal: cannot change to 'file://.../repo': No such file or directory\n error: failed to initialize sparse-checkout\n\nThese errors are due to using a \"-C \" option to call 'git -C\n sparse-checkout init' but the URL is being given instead of\nthe target directory.\n\nUpdate that target directory to evaluate this correctly. I have also\nmanually tested that https:// URLs are handled correctly as well.\n\nAcked-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4348d962c9..2caefc44fb 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1130,7 +1130,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\tif (option_sparse_checkout && git_sparse_checkout_init(dir))\n \t\treturn 1;\n \n \tremote = remote_get(option_origin);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 43d1f7520c..cf4a595c86 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -90,7 +90,7 @@ test_expect_success 'init with existing sparse-checkout' '\n '\n \n test_expect_success 'clone --sparse' '\n-\tgit clone --sparse repo clone &&\n+\tgit clone --sparse \"file://$(pwd)/repo\" clone &&\n \tgit -C clone sparse-checkout list >actual &&\n \tcat >expect <<-\\EOF &&\n \t/*\n-- \ngitgitgadget\n\n"},{"id":"390649","messageId":"ae78c3069b08557e3acc544601c71ee932ace19d.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 05/12] sparse-checkout: fix documentation typo for core.sparseCheckoutCone","fromName":"Jeff King via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:36Z","receivedAt":"2020-01-28T18:26:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"From: Jeff King \n\nSigned-off-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 3b341cf0fc..4834fb434d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -106,7 +106,7 @@ The full pattern set allows for arbitrary pattern matches and complicated\n inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n updating the index, where N is the number of patterns and M is the number\n of paths in the index. To combat this performance issue, a more restricted\n-pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+pattern set is allowed when `core.sparseCheckoutCone` is enabled.\n \n The accepted patterns in the cone pattern set are:\n \n-- \ngitgitgadget\n\n"},{"id":"390650","messageId":"5497ad8778c29ca936206d2ded6773758c236c12.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 03/12] sparse-checkout: create leading directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:34Z","receivedAt":"2020-01-28T18:26:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git init' command creates the \".git/info\" directory and fills it\nwith some default files. However, 'git worktree add' does not create\nthe info directory for that worktree. This causes a problem when running\n\"git sparse-checkout init\" inside a worktree. While care was taken to\nallow the sparse-checkout config to be specific to a worktree, this\ninitialization was untested.\n\nSafely create the leading directories for the sparse-checkout file. This\nis the safest thing to do even without worktrees, as a user could delete\ntheir \".git/info\" directory and expect Git to recover safely.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 4 ++++\n t/t1091-sparse-checkout-builtin.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b3bed891cb..3cee8ab46e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -199,6 +199,10 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tint result;\n \n \tsparse_filename = get_sparse_checkout_filename();\n+\n+\tif (safe_create_leading_directories(sparse_filename))\n+\t\tdie(_(\"failed to create directory for sparse-checkout file\"));\n+\n \tfd = hold_lock_file_for_update(&lk, sparse_filename,\n \t\t\t\t LOCK_DIE_ON_ERROR);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e28e1c797f..43d1f7520c 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -295,4 +295,14 @@ test_expect_success 'interaction with submodules' '\n \tcheck_files super/modules/child a deep folder1 folder2\n '\n \n+test_expect_success 'different sparse-checkouts with worktrees' '\n+\tgit -C repo worktree add --detach ../worktree &&\n+\tcheck_files worktree \"a deep folder1 folder2\" &&\n+\tgit -C worktree sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set folder1 &&\n+\tgit -C worktree sparse-checkout set deep/deeper1 &&\n+\tcheck_files repo a folder1 &&\n+\tcheck_files worktree a deep\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390651","messageId":"2ad4d3e467a7e2be679de4084e54f426aa242bd6.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 06/12] sparse-checkout: cone mode does not recognize \"**\"","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:37Z","receivedAt":"2020-01-28T18:26:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen core.sparseCheckoutCone is enabled, the 'git sparse-checkout set'\ncommand creates a restricted set of possible patterns that are used\nby a custom algorithm to quickly match those patterns.\n\nIf a user manually edits the sparse-checkout file, then they could\ncreate patterns that do not match these expectations. The cone-mode\nmatching algorithm can return incorrect results. The solution is to\ndetect these incorrect patterns, warn that we do not recognize them,\nand revert to the standard algorithm.\n\nCheck each pattern for the \"**\" substring, and revert to the old\nlogic if seen. While technically a \"//**\" pattern matches\nthe meaning of \"//\", it is not one that would be written by\nthe sparse-checkout builtin in cone mode. Attempting to accept that\npattern change complicates the logic and instead we punt and do\nnot accept any instance of \"**\".\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 7 +++++-\n t/t1091-sparse-checkout-builtin.sh | 34 ++++++++++++++++++++++++++++++\n 2 files changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 22d08e61c2..40fed73a94 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,11 +651,16 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n+\tif (strstr(given->pattern, \"**\")) {\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n \t\t\t/* Not a cone pattern. */\n-\t\t\tpl->use_cone_patterns = 0;\n \t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\t\tgoto clear_hashmaps;\n \t\t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex cf4a595c86..e2e45dc7fd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -305,4 +305,38 @@ test_expect_success 'different sparse-checkouts with worktrees' '\n \tcheck_files worktree a deep\n '\n \n+check_read_tree_errors () {\n+\tREPO=$1\n+\tFILES=$2\n+\tERRORS=$3\n+\tgit -C $REPO read-tree -mu HEAD 2>err &&\n+\tif test -z \"$ERRORS\"\n+\tthen\n+\t\ttest_must_be_empty err\n+\telse\n+\t\ttest_i18ngrep \"$ERRORS\" err\n+\tfi &&\n+\tcheck_files $REPO $FILES\n+}\n+\n+test_expect_success 'pattern-checks: /A/**' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/**\n+\tEOF\n+\tcheck_read_tree_errors repo \"a folder1\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: /A/**/B/' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/**/deepest\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\" &&\n+\tcheck_files repo/deep \"deeper1\" &&\n+\tcheck_files repo/deep/deeper1 \"deepest\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390652","messageId":"ec714a4cf04e86f55057e17109a29e74021b73bf.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 11/12] sparse-checkout: use C-style quotes in 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:42Z","receivedAt":"2020-01-28T18:26:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen in cone mode, the 'git sparse-checkout list' subcommand lists\nthe directories included in the sparse cone. When these directories\ncontain odd characters, such as a backslash, then we need to use\nC-style quotes similar to 'git ls-tree'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 6 ++++--\n t/t1091-sparse-checkout-builtin.sh | 7 +++++--\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 61414fef18..b3c1e97dba 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -78,8 +78,10 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n \t\tstring_list_sort(&sl);\n \n-\t\tfor (i = 0; i < sl.nr; i++)\n-\t\t\tprintf(\"%s\\n\", sl.items[i].string);\n+\t\tfor (i = 0; i < sl.nr; i++) {\n+\t\t\tquote_c_style(sl.items[i].string, NULL, stdout, 0);\n+\t\t\tprintf(\"\\n\");\n+\t\t}\n \n \t\treturn 0;\n \t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 459715d541..7617fb027a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -392,7 +392,8 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n \tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n-\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tgit -C escaped ls-tree -d --name-only HEAD >list-expect &&\n+\tgit -C escaped sparse-checkout set --stdin expect <<-\\EOF &&\n \t/*\n \t!/*/\n@@ -403,7 +404,9 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/zdoes\\*exist/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout list >list-actual &&\n+\ttest_cmp list-expect list-actual\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390653","messageId":"1867746d97c5e3d5d738f12aeb0e1ab739a6b8c1.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 12/12] sparse-checkout: improve docs around 'set' in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:43Z","receivedAt":"2020-01-28T18:26:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe existing documentation does not clarify how the 'set' subcommand\nchanges when core.sparseCheckoutCone is enabled. Correct this by\nchanging some language around the \"A/B/C\" example. Also include a\ndescription of the input format matching the output of 'git ls-tree\n--name-only'.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 4834fb434d..0914619881 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -50,6 +50,14 @@ To avoid interfering with other worktrees, it first enables the\n +\n When the `--stdin` option is provided, the patterns are read from\n standard in as a newline-delimited list instead of from the arguments.\n++\n+When `core.sparseCheckoutCone` is enabled, the input list is considered a\n+list of directories instead of sparse-checkout patterns. The command writes\n+patterns to the sparse-checkout file to include all files contained in those\n+directories (recursively) as well as files that are siblings of ancestor\n+directories. The input format matches the output of `git ls-tree --name-only`.\n+This includes interpreting pathnames that begin with a double quote (\") as\n+C-style quoted strings.\n \n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n@@ -128,9 +136,12 @@ the following patterns:\n ----------------\n \n This says \"include everything in root, but nothing two levels below root.\"\n-If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n-`A/B` are added as parent patterns. The resulting sparse-checkout file is\n-now\n+\n+When in cone mode, the `git sparse-checkout set` subcommand takes a list of\n+directories instead of a list of sparse-checkout patterns. In this mode,\n+the command `git sparse-checkout set A/B/C` sets the directory `A/B/C` as\n+a recursive pattern, the directories `A` and `A/B` are added as parent\n+patterns. The resulting sparse-checkout file is now\n \n ----------------\n /*\n-- \ngitgitgadget\n"},{"id":"390654","messageId":"e2f9afc70c92c606a9d93660bce0599b7b7ddf6e.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:41Z","receivedAt":"2020-01-28T18:26:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user somehow creates a directory with an asterisk (*) or backslash\n(\\), then the \"git sparse-checkout set\" command will struggle to provide\nthe correct pattern in the sparse-checkout file. When not in cone mode,\nthe provided pattern is written directly into the sparse-checkout file.\nHowever, in cone mode we expect a list of paths to directories and then\nwe convert those into patterns.\n\nEven more specifically, the goal is to always allow the following from\nthe root of a repo:\n\n git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n\nThe ls-tree command provides directory names with an unescaped asterisk.\nIt also quotes the directories that contain an escaped backslash. We\nmust remove these quotes, then keep the escaped backslashes.\n\nHowever, there is some care needed for the timing of these escapes. The\nin-memory pattern list is used to update the working directory before\nwriting the patterns to disk. Thus, we need the command to have the\nunescaped names in the hashsets for the cone comparisons, then escape\nthe patterns later.\n\nUse unquote_c_style() when parsing lines from stdin. Command-line\narguments will be parsed as-is, assuming the user can do the correct\nlevel of escaping from their environment to match the exact directory\nnames.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 38 +++++++++++++++++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 21 +++++++++++++++--\n 2 files changed, 54 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 3cee8ab46e..61414fef18 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -13,6 +13,7 @@\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n #include \"wt-status.h\"\n+#include \"quote.h\"\n \n static const char *empty_base = \"\";\n \n@@ -140,6 +141,22 @@ static int update_working_directory(struct pattern_list *pl)\n \treturn result;\n }\n \n+static char *escaped_pattern(char *pattern)\n+{\n+\tchar *p = pattern;\n+\tstruct strbuf final = STRBUF_INIT;\n+\n+\twhile (*p) {\n+\t\tif (*p == '*' || *p == '\\\\')\n+\t\t\tstrbuf_addch(&final, '\\\\');\n+\n+\t\tstrbuf_addch(&final, *p);\n+\t\tp++;\n+\t}\n+\n+\treturn strbuf_detach(&final, NULL);\n+}\n+\n static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n {\n \tint i;\n@@ -164,10 +181,11 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \n \t\tif (strlen(pattern))\n \t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t\tfree(pattern);\n \t}\n \n \tstring_list_clear(&sl, 0);\n@@ -185,8 +203,9 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstring_list_remove_duplicates(&sl, 0);\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t\tfree(pattern);\n \t}\n }\n \n@@ -423,8 +442,21 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n-\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\tstruct strbuf unquoted = STRBUF_INIT;\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tif (line.buf[0] == '\"') {\n+\t\t\t\t\tstrbuf_setlen(&unquoted, 0);\n+\t\t\t\t\tif (unquote_c_style(&unquoted, line.buf, NULL))\n+\t\t\t\t\t\tdie(_(\"unable to unquote C-style string '%s'\"),\n+\t\t\t\t\t\tline.buf);\n+\n+\t\t\t\t\tstrbuf_swap(&unquoted, &line);\n+\t\t\t\t}\n+\n \t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&unquoted);\n \t\t} else {\n \t\t\tfor (i = 0; i < argc; i++) {\n \t\t\t\tstrbuf_setlen(&line, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0a21a5e15d..459715d541 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -309,6 +309,9 @@ check_read_tree_errors () {\n \tREPO=$1\n \tFILES=$2\n \tERRORS=$3\n+\tgit -C $REPO -c core.sparseCheckoutCone=false read-tree -mu HEAD 2>err &&\n+\ttest_must_be_empty err &&\n+\tcheck_files $REPO \"$FILES\" &&\n \tgit -C $REPO read-tree -mu HEAD 2>err &&\n \tif test -z \"$ERRORS\"\n \tthen\n@@ -379,14 +382,28 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tgit -C escaped reset --hard $COMMIT &&\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n-\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n+\tgit -C escaped sparse-checkout set zbad\\\\dir \"zdoes*not*exist\" \"zdoes*exist\" &&\n+\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n \t/zbad\\\\dir/\n+\t/zdoes\\*exist/\n \t/zdoes\\*not\\*exist/\n+\tEOF\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t/folder1/\n+\t/folder2/\n+\t/zbad\\\\dir/\n \t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390655","messageId":"d2a510a3bb76db4c432187d631d5f6f8085af7e8.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 08/12] sparse-checkout: warn on incorrect '*' in patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:39Z","receivedAt":"2020-01-28T18:27:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout commmand will write patterns that\nallow faster pattern matching. This matching only works if the patterns\nin the sparse-checkout file are those written by that command. Users\ncan edit the sparse-checkout file and create patterns that cause the\ncone mode matching to fail.\n\nThe cone mode patterns may end in \"/*\" but otherwise an un-escaped\nasterisk is invalid. Add checks to disable cone mode when seeing these\nvalues.\n\nA later change will properly handle escaped asterisks.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 29 +++++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 27 +++++++++++++++++++++++++++\n 2 files changed, 56 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex c2e585607e..7cb78c8b87 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -635,6 +635,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \tstruct pattern_entry *translated;\n \tchar *truncated;\n \tchar *data = NULL;\n+\tconst char *prev, *cur, *next;\n \n \tif (!pl->use_cone_patterns)\n \t\treturn;\n@@ -652,12 +653,40 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t}\n \n \tif (given->patternlen <= 2 ||\n+\t *given->pattern == '*' ||\n \t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\n \t}\n \n+\tprev = given->pattern;\n+\tcur = given->pattern + 1;\n+\tnext = given->pattern + 2;\n+\n+\twhile (*cur) {\n+\t\t/* We care about *cur == '*' */\n+\t\tif (*cur != '*')\n+\t\t\tgoto increment;\n+\n+\t\t/* But only if *prev != '\\\\' */\n+\t\tif (*prev == '\\\\')\n+\t\t\tgoto increment;\n+\n+\t\t/* But a trailing '/' then '*' is fine */\n+\t\tif (*prev == '/' && *next == 0)\n+\t\t\tgoto increment;\n+\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\n+\tincrement:\n+\t\tprev++;\n+\t\tcur++;\n+\t\tnext++;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2e57534799..470900f6f4 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -348,4 +348,31 @@ test_expect_success 'pattern-checks: too short' '\n \tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n '\n \n+test_expect_success 'pattern-checks: trailing \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a*\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: starting \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*eep/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: escaped \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/does\\*not\\*exist/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390656","messageId":"aace0645107719f62c611296adc44f8c286d8403.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 07/12] sparse-checkout: detect short patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:38Z","receivedAt":"2020-01-28T18:27:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the shortest pattern the sparse-checkout command will\nwrite into the sparse-checkout file is \"/*\". This is handled carefully\nin add_pattern_to_hashsets(), so warn if any other pattern is this\nshort. This will assist future pattern checks by allowing us to assume\nthere are at least three characters in the pattern.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 3 ++-\n t/t1091-sparse-checkout-builtin.sh | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 40fed73a94..c2e585607e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,7 +651,8 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n-\tif (strstr(given->pattern, \"**\")) {\n+\tif (given->patternlen <= 2 ||\n+\t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e2e45dc7fd..2e57534799 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -339,4 +339,13 @@ test_expect_success 'pattern-checks: /A/**/B/' '\n \tcheck_files repo/deep/deeper1 \"deepest\"\n '\n \n+test_expect_success 'pattern-checks: too short' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390657","messageId":"9ea69e90694e53842acd68d3ac85c9a00c4bd343.1580236003.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v3 09/12] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-28T18:26:40Z","receivedAt":"2020-01-28T18:27:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout feature uses hashset containment\nqueries to match paths. Make this algorithm respect escaped asterisk\n(*) and backslash (\\) characters.\n\nCreate dup_and_filter_pattern() method to convert a pattern by\nremoving escape characters and dropping an optional \"/*\" at the end.\nThis method is available in dir.h as we will use it in\nbuiltin/sparse-checkout.c in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 31 +++++++++++++++++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 22 +++++++++++++++++----\n 2 files changed, 46 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 7cb78c8b87..579f274d13 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -630,6 +630,32 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n }\n \n+static char *dup_and_filter_pattern(const char *pattern)\n+{\n+\tchar *set, *read;\n+\tchar *result = xstrdup(pattern);\n+\n+\tset = result;\n+\tread = result;\n+\n+\twhile (*read) {\n+\t\t/* skip escape characters (once) */\n+\t\tif (*read == '\\\\')\n+\t\t\tread++;\n+\n+\t\t*set = *read;\n+\n+\t\tset++;\n+\t\tread++;\n+\t}\n+\t*set = 0;\n+\n+\tif (*(read - 2) == '/' && *(read - 1) == '*')\n+\t\t*(read - 2) = 0;\n+\n+\treturn result;\n+}\n+\n static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n {\n \tstruct pattern_entry *translated;\n@@ -695,8 +721,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\t\tgoto clear_hashmaps;\n \t\t}\n \n-\t\ttruncated = xstrdup(given->pattern);\n-\t\ttruncated[given->patternlen - 2] = 0;\n+\t\ttruncated = dup_and_filter_pattern(given->pattern);\n \n \t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n \t\ttranslated->pattern = truncated;\n@@ -730,7 +755,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \n \ttranslated = xmalloc(sizeof(struct pattern_entry));\n \n-\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->pattern = dup_and_filter_pattern(given->pattern);\n \ttranslated->patternlen = given->patternlen;\n \thashmap_entry_init(&translated->ent,\n \t\t\t ignore_case ?\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 470900f6f4..0a21a5e15d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -366,13 +366,27 @@ test_expect_success 'pattern-checks: starting \"*\"' '\n \tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n '\n \n-test_expect_success 'pattern-checks: escaped \"*\"' '\n-\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n+\tgit clone repo escaped &&\n+\tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n+\tNEWTREE=$(git -C escaped mktree <<-EOF\n+\t$(git -C escaped ls-tree HEAD)\n+\t040000 tree $TREEOID\tzbad\\\\dir\n+\t040000 tree $TREEOID\tzdoes*exist\n+\tEOF\n+\t) &&\n+\tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n+\tgit -C escaped reset --hard $COMMIT &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout init --cone &&\n+\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n \t/*\n \t!/*/\n-\t/does\\*not\\*exist/\n+\t/zbad\\\\dir/\n+\t/zdoes\\*not\\*exist/\n+\t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors repo \"a\" \"\"\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390706","messageId":"20200129100309.GA4218@coredump.intra.peff.net","threadId":"52638","inReplyTo":"9ea69e90694e53842acd68d3ac85c9a00c4bd343.1580236003.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 09/12] sparse-checkout: properly match escaped characters","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-29T10:03:09Z","receivedAt":"2020-01-29T10:03:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 28, 2020 at 06:26:40PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> In cone mode, the sparse-checkout feature uses hashset containment\n> queries to match paths. Make this algorithm respect escaped asterisk\n> (*) and backslash (\\) characters.\n\nDo we also need to worry about other glob metacharacters? E.g., \"?\" or\nranges like \"[A-Z]\"?\n\n> +static char *dup_and_filter_pattern(const char *pattern)\n> +{\n> +\tchar *set, *read;\n> +\tchar *result = xstrdup(pattern);\n> +\n> +\tset = result;\n> +\tread = result;\n> +\n> +\twhile (*read) {\n> +\t\t/* skip escape characters (once) */\n> +\t\tif (*read == '\\\\')\n> +\t\t\tread++;\n> +\n> +\t\t*set = *read;\n> +\n> +\t\tset++;\n> +\t\tread++;\n> +\t}\n> +\t*set = 0;\n> +\n> +\tif (*(read - 2) == '/' && *(read - 1) == '*')\n> +\t\t*(read - 2) = 0;\n> +\n> +\treturn result;\n> +}\n\nDo we need to check that the pattern is longer than 1 character here? If\nit's a single character, it seems like this \"*(read - 2)\" will\ndereference the byte before the string.\n\n-Peff\n"},{"id":"390708","messageId":"20200129101713.GB4218@coredump.intra.peff.net","threadId":"52638","inReplyTo":"e2f9afc70c92c606a9d93660bce0599b7b7ddf6e.1580236003.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-29T10:17:13Z","receivedAt":"2020-01-29T10:17:15Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 28, 2020 at 06:26:41PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> If a user somehow creates a directory with an asterisk (*) or backslash\n> (\\), then the \"git sparse-checkout set\" command will struggle to provide\n> the correct pattern in the sparse-checkout file. When not in cone mode,\n> the provided pattern is written directly into the sparse-checkout file.\n> However, in cone mode we expect a list of paths to directories and then\n> we convert those into patterns.\n\nIs this really about cone mode? It seems more like it is about --stdin.\nI.e., what are the rules for when the input is a filename and when it is\na pattern? In our earlier discussion, I had assumed that command-line\narguments to \"sparse-checkout set\" were actual filenames, and \"--stdin\"\njust read them from stdin.\n\nBut looking at the documentation, they are always called \"patterns\" on\nthe command-line. Should the \"--stdin\" documentation make it clear that\nwe are no longer taking patterns, but instead actual filenames?\n\nOr am I confused, and in non-cone-mode the \"ls-tree | sparse-checkout\"\npipeline is not supposed to work at all? (I.e., they really are always\npatterns)?\n\n> Even more specifically, the goal is to always allow the following from\n> the root of a repo:\n> \n> git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n> \n> The ls-tree command provides directory names with an unescaped asterisk.\n> It also quotes the directories that contain an escaped backslash. We\n> must remove these quotes, then keep the escaped backslashes.\n> \n> However, there is some care needed for the timing of these escapes. The\n> in-memory pattern list is used to update the working directory before\n> writing the patterns to disk. Thus, we need the command to have the\n> unescaped names in the hashsets for the cone comparisons, then escape\n> the patterns later.\n\nOK, this part make sense.\n\n> Use unquote_c_style() when parsing lines from stdin. Command-line\n> arguments will be parsed as-is, assuming the user can do the correct\n> level of escaping from their environment to match the exact directory\n> names.\n\nI think there's two issues here: escaping characters from the shell so\nthat they make it intact to Git, and the question of whether Git is\nexpecting patterns or raw filenames. I agree the user is responsible for\nthe shell half, but I think we need to clarify what we're expecting.\nI.e., if I say:\n\n git sparse-checkout set 'f*'\n\nam I trying to match \"foo\", or the literal file \"f*\"?\n\n> +static char *escaped_pattern(char *pattern)\n> +{\n> +\tchar *p = pattern;\n> +\tstruct strbuf final = STRBUF_INIT;\n> +\n> +\twhile (*p) {\n> +\t\tif (*p == '*' || *p == '\\\\')\n> +\t\t\tstrbuf_addch(&final, '\\\\');\n> +\n> +\t\tstrbuf_addch(&final, *p);\n> +\t\tp++;\n> +\t}\n> +\n> +\treturn strbuf_detach(&final, NULL);\n> +}\n\nDo we need to catch other metacharacters here (using is_glob_special()\nperhaps)?\n\n> @@ -423,8 +442,21 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> \t\tpl.use_cone_patterns = 1;\n> \n> \t\tif (set_opts.use_stdin) {\n> -\t\t\twhile (!strbuf_getline(&line, stdin))\n> +\t\t\tstruct strbuf unquoted = STRBUF_INIT;\n> +\t\t\twhile (!strbuf_getline(&line, stdin)) {\n> +\t\t\t\tif (line.buf[0] == '\"') {\n> +\t\t\t\t\tstrbuf_setlen(&unquoted, 0);\n\nA minor nit, but strbuf_reset(&unquoted) would be more idiomatic here.\n\n> +\t\t\t\t\tif (unquote_c_style(&unquoted, line.buf, NULL))\n> +\t\t\t\t\t\tdie(_(\"unable to unquote C-style string '%s'\"),\n> +\t\t\t\t\t\tline.buf);\n> +\n> +\t\t\t\t\tstrbuf_swap(&unquoted, &line);\n> +\t\t\t\t}\n> +\n> \t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n> +\t\t\t}\n\nOK, overall this input procedure makes sense to me.\n\n-Peff\n"},{"id":"390709","messageId":"20200129102326.GC4218@coredump.intra.peff.net","threadId":"52638","inReplyTo":"ec714a4cf04e86f55057e17109a29e74021b73bf.1580236003.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 11/12] sparse-checkout: use C-style quotes in 'list' subcommand","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-29T10:23:26Z","receivedAt":"2020-01-29T10:23:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 28, 2020 at 06:26:42PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> \n> When in cone mode, the 'git sparse-checkout list' subcommand lists\n> the directories included in the sparse cone. When these directories\n> contain odd characters, such as a backslash, then we need to use\n> C-style quotes similar to 'git ls-tree'.\n\nMakes sense, and the code looks correct to me.\n\n-Peff\n"},{"id":"390710","messageId":"20200129103352.GD4218@coredump.intra.peff.net","threadId":"52638","inReplyTo":"20200129101713.GB4218@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-29T10:33:52Z","receivedAt":"2020-01-29T10:33:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 29, 2020 at 05:17:13AM -0500, Jeff King wrote:\n\n> > From: Derrick Stolee \n> > \n> > If a user somehow creates a directory with an asterisk (*) or backslash\n> > (\\), then the \"git sparse-checkout set\" command will struggle to provide\n> > the correct pattern in the sparse-checkout file. When not in cone mode,\n> > the provided pattern is written directly into the sparse-checkout file.\n> > However, in cone mode we expect a list of paths to directories and then\n> > we convert those into patterns.\n> \n> Is this really about cone mode? It seems more like it is about --stdin.\n> I.e., what are the rules for when the input is a filename and when it is\n> a pattern? In our earlier discussion, I had assumed that command-line\n> arguments to \"sparse-checkout set\" were actual filenames, and \"--stdin\"\n> just read them from stdin.\n> \n> But looking at the documentation, they are always called \"patterns\" on\n> the command-line. Should the \"--stdin\" documentation make it clear that\n> we are no longer taking patterns, but instead actual filenames?\n> \n> Or am I confused, and in non-cone-mode the \"ls-tree | sparse-checkout\"\n> pipeline is not supposed to work at all? (I.e., they really are always\n> patterns)?\n\nHmph, sorry, I _was_ just confused. I was reading a copy of the manpage\nwithout your final patch, which made things much clearer.\n\nSo OK, I think the resulting documentation does make things clear. And\nthis is just about cone mode, not --stdin. Please ignore my ramblings in\nthe rest of the replied-to message. But...\n\n> > Even more specifically, the goal is to always allow the following from\n> > the root of a repo:\n> > \n> > git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n> > \n> > The ls-tree command provides directory names with an unescaped asterisk.\n> > It also quotes the directories that contain an escaped backslash. We\n> > must remove these quotes, then keep the escaped backslashes.\n> > \n> > However, there is some care needed for the timing of these escapes. The\n> > in-memory pattern list is used to update the working directory before\n> > writing the patterns to disk. Thus, we need the command to have the\n> > unescaped names in the hashsets for the cone comparisons, then escape\n> > the patterns later.\n> \n> OK, this part make sense.\n\nYou could also demonstrate this even without --stdin with something\nlike:\n\n git config core.sparsecheckoutcone true\n git sparse-checkout set 'foo*bar'\n\nwhich should take that as a literal filename and put the pattern\n'foo\\*bar' in the sparse-checkout file. And your tests do cover that.\n\nSo really there are two separate bugs here, and it might be a little\neasier to explain the \"timing of these escapes\" thing by doing them\nseparately. I.e., the case above needs escaping and we could demonstrate\nthe bug with a command-line \"set\". And then follow up by fixing the\nproblem with correctly de-quoting --stdin.\n\n> > +static char *escaped_pattern(char *pattern)\n> [...]\n> Do we need to catch other metacharacters here (using is_glob_special()\n> perhaps)?\n\nAfter de-confusing myself, I think the individual code comments I wrote\nstill apply though (especially this one).\n\n-Peff\n"},{"id":"390723","messageId":"6003bbf2-ad16-0686-dc58-2010fe02ce05@gmail.com","threadId":"52638","inReplyTo":"20200129100309.GA4218@coredump.intra.peff.net","subject":"Re: [PATCH v3 09/12] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-29T13:58:59Z","receivedAt":"2020-01-29T13:59:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 5:03 AM, Jeff King wrote:\n> On Tue, Jan 28, 2020 at 06:26:40PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> In cone mode, the sparse-checkout feature uses hashset containment\n>> queries to match paths. Make this algorithm respect escaped asterisk\n>> (*) and backslash (\\) characters.\n> \n> Do we also need to worry about other glob metacharacters? E.g., \"?\" or\n> ranges like \"[A-Z]\"?\n\nThese are not part of the .gitignore patterns [1].\n\n[1] https://git-scm.com/docs/gitignore#_pattern_format\n\n>> +static char *dup_and_filter_pattern(const char *pattern)\n>> +{\n>> +\tchar *set, *read;\n>> +\tchar *result = xstrdup(pattern);\n>> +\n>> +\tset = result;\n>> +\tread = result;\n>> +\n>> +\twhile (*read) {\n>> +\t\t/* skip escape characters (once) */\n>> +\t\tif (*read == '\\\\')\n>> +\t\t\tread++;\n>> +\n>> +\t\t*set = *read;\n>> +\n>> +\t\tset++;\n>> +\t\tread++;\n>> +\t}\n>> +\t*set = 0;\n>> +\n>> +\tif (*(read - 2) == '/' && *(read - 1) == '*')\n>> +\t\t*(read - 2) = 0;\n>> +\n>> +\treturn result;\n>> +}\n> \n> Do we need to check that the pattern is longer than 1 character here? If\n> it's a single character, it seems like this \"*(read - 2)\" will\n> dereference the byte before the string.\n\nThis method is only called by add_pattern_to_hashsets(), which\nhas a guard against paths of length less than 2, but thats' no\nexcuse for dangerous pointer arithmetic here.\n\nBut you also point out an even more confusing thing: why are we\nmodifying based on the 'read' pointer, and not the 'set' pointer?\nThis seems to work _accidentally_ only when the pattern has \"/*\"\nand \"\" has no escape characters.\n\nI had to recall exactly why we are dropping this \"/*\", but it's because\nthe pattern _actually_ ends with \"/*/\" but the in-memory pattern has\nalready dropped that last slash and applied PATTERN_FLAG_MUSTBEDIR.\n\nHere is a diff that I can apply to this patch to fix this problem\n_and_ demonstrate it in the tests:\n\ndiff --git a/dir.c b/dir.c\nindex 579f274d13..277577c8bf 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -633,6 +633,7 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n static char *dup_and_filter_pattern(const char *pattern)\n {\n char *set, *read;\n+ size_t count = 0;\n char *result = xstrdup(pattern);\n \n set = result;\n@@ -647,11 +648,14 @@ static char *dup_and_filter_pattern(const char *pattern)\n \n set++;\n read++;\n+ count++;\n }\n *set = 0;\n \n- if (*(read - 2) == '/' && *(read - 1) == '*')\n- *(read - 2) = 0;\n+ if (count > 2 &&\n+ *(set - 1) == '*' &&\n+ *(set - 2) == '/')\n+ *(set - 2) = 0;\n \n return result;\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0a21a5e15d..20b0465f77 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -383,6 +383,7 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n /*\n !/*/\n /zbad\\\\dir/\n+ !/zbad\\\\dir/*/\n /zdoes\\*not\\*exist/\n /zdoes\\*exist/\n EOF\n\nWith this extra line in the test, but compiling the old version of this patch,\nthe test fails with:\n\n'err' is not empty, it contains:\n+ cat err\nwarning: unrecognized negative pattern: '/zbad\\\\dir/*'\nwarning: disabling cone pattern matching\n\nTo ensure this negative pattern exists in the later patch where we set\nthe patterns using the builtin, I'll add \"zbad\\\\dir/bogus\" to the list\nof directories to include, which will add another pattern to the set.\n\nThanks,\n-Stolee\n\n"},{"id":"390724","messageId":"c4f3b0d4-c837-bec7-7ad2-2203bb577b17@gmail.com","threadId":"52638","inReplyTo":"6003bbf2-ad16-0686-dc58-2010fe02ce05@gmail.com","subject":"Re: [PATCH v3 09/12] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-29T14:04:13Z","receivedAt":"2020-01-29T14:04:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 8:58 AM, Derrick Stolee wrote:\n> On 1/29/2020 5:03 AM, Jeff King wrote:\n>> On Tue, Jan 28, 2020 at 06:26:40PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> From: Derrick Stolee \n>>>\n>>> In cone mode, the sparse-checkout feature uses hashset containment\n>>> queries to match paths. Make this algorithm respect escaped asterisk\n>>> (*) and backslash (\\) characters.\n>>\n>> Do we also need to worry about other glob metacharacters? E.g., \"?\" or\n>> ranges like \"[A-Z]\"?\n> \n> These are not part of the .gitignore patterns [1].\n> \n> [1] https://git-scm.com/docs/gitignore#_pattern_format\n\nI should read things more carefully. There is also this information in\none of the bullets:\n\n\tAn asterisk \"*\" matches anything except a slash. The character\n\t\"?\" matches any one character except \"/\". The range notation,\n\te.g. [a-zA-Z], can be used to match one of the characters in a range.\n\tSee fnmatch(3) and the FNM_PATHNAME flag for a more detailed\n\tdescription.\n\nSo this series does not attempt to properly work with globs, and I'll\nneed to test those a bit. Certainly they shouldn't work in cone mode,\nso an extra patch to remove those would be simple. Input sanitizing\nwould be interesting, and I'll see what `git ls-tree` would output\nwith paths containing these characters.\n\n-Stolee\n"},{"id":"390725","messageId":"7a85cded-eccf-9f33-4056-08b2923b2861@gmail.com","threadId":"52638","inReplyTo":"20200129103352.GD4218@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-29T14:16:11Z","receivedAt":"2020-01-29T14:16:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 5:33 AM, Jeff King wrote:\n> On Wed, Jan 29, 2020 at 05:17:13AM -0500, Jeff King wrote:\n> \n>>> From: Derrick Stolee \n>>>\n>>> If a user somehow creates a directory with an asterisk (*) or backslash\n>>> (\\), then the \"git sparse-checkout set\" command will struggle to provide\n>>> the correct pattern in the sparse-checkout file. When not in cone mode,\n>>> the provided pattern is written directly into the sparse-checkout file.\n>>> However, in cone mode we expect a list of paths to directories and then\n>>> we convert those into patterns.\n>>\n>> Is this really about cone mode? It seems more like it is about --stdin.\n>> I.e., what are the rules for when the input is a filename and when it is\n>> a pattern? In our earlier discussion, I had assumed that command-line\n>> arguments to \"sparse-checkout set\" were actual filenames, and \"--stdin\"\n>> just read them from stdin.\n>>\n>> But looking at the documentation, they are always called \"patterns\" on\n>> the command-line. Should the \"--stdin\" documentation make it clear that\n>> we are no longer taking patterns, but instead actual filenames?\n>>\n>> Or am I confused, and in non-cone-mode the \"ls-tree | sparse-checkout\"\n>> pipeline is not supposed to work at all? (I.e., they really are always\n>> patterns)?\n> \n> Hmph, sorry, I _was_ just confused. I was reading a copy of the manpage\n> without your final patch, which made things much clearer.\n> \n> So OK, I think the resulting documentation does make things clear. And\n> this is just about cone mode, not --stdin. Please ignore my ramblings in\n> the rest of the replied-to message. But...\n> \n>>> Even more specifically, the goal is to always allow the following from\n>>> the root of a repo:\n>>>\n>>> git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n>>>\n>>> The ls-tree command provides directory names with an unescaped asterisk.\n>>> It also quotes the directories that contain an escaped backslash. We\n>>> must remove these quotes, then keep the escaped backslashes.\n>>>\n>>> However, there is some care needed for the timing of these escapes. The\n>>> in-memory pattern list is used to update the working directory before\n>>> writing the patterns to disk. Thus, we need the command to have the\n>>> unescaped names in the hashsets for the cone comparisons, then escape\n>>> the patterns later.\n>>\n>> OK, this part make sense.\n> \n> You could also demonstrate this even without --stdin with something\n> like:\n> \n> git config core.sparsecheckoutcone true\n> git sparse-checkout set 'foo*bar'\n> \n> which should take that as a literal filename and put the pattern\n> 'foo\\*bar' in the sparse-checkout file. And your tests do cover that.\n> \n> So really there are two separate bugs here, and it might be a little\n> easier to explain the \"timing of these escapes\" thing by doing them\n> separately. I.e., the case above needs escaping and we could demonstrate\n> the bug with a command-line \"set\". And then follow up by fixing the\n> problem with correctly de-quoting --stdin.\n\nI've locally split the commit into two parts. That makes things much\nsimpler to read.\n\n>>> +static char *escaped_pattern(char *pattern)\n>> [...]\n>> Do we need to catch other metacharacters here (using is_glob_special()\n>> perhaps)?\n> \n> After de-confusing myself, I think the individual code comments I wrote\n> still apply though (especially this one).\n\nI've applied the smaller comments and am now investigating the right\nthing to do with other is_glob_special() characters. There is a small\nchance that I can replace any \"c == '*' || c == '\\'\" with is_glob_special(),\nbut we shall see. At the very least, I'll need to expand my tests.\n\nThanks,\n-Stolee\n"},{"id":"390726","messageId":"50a17375-d2ab-239c-8a23-2c8fdd306903@gmail.com","threadId":"52638","inReplyTo":"7a85cded-eccf-9f33-4056-08b2923b2861@gmail.com","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-29T14:39:15Z","receivedAt":"2020-01-29T14:39:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 9:16 AM, Derrick Stolee wrote:\n> On 1/29/2020 5:33 AM, Jeff King wrote:\n>> On Wed, Jan 29, 2020 at 05:17:13AM -0500, Jeff King wrote:\n>>>> +static char *escaped_pattern(char *pattern)\n>>> [...]\n>>> Do we need to catch other metacharacters here (using is_glob_special()\n>>> perhaps)?\n>>\n>> After de-confusing myself, I think the individual code comments I wrote\n>> still apply though (especially this one).\n> \n> I've applied the smaller comments and am now investigating the right\n> thing to do with other is_glob_special() characters. There is a small\n> chance that I can replace any \"c == '*' || c == '\\'\" with is_glob_special(),\n> but we shall see. At the very least, I'll need to expand my tests.\n\nI think I have a handle on these cases, and I've pushed it to my GGG PR.\nI'll let this version settle a bit for more review before updating it\nwith a v4.\n\nThanks,\n-Stolee\n\n"},{"id":"390772","messageId":"20200130072957.GE2189233@coredump.intra.peff.net","threadId":"52638","inReplyTo":"7a85cded-eccf-9f33-4056-08b2923b2861@gmail.com","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-01-30T07:29:57Z","receivedAt":"2020-01-30T07:30:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 29, 2020 at 09:16:11AM -0500, Derrick Stolee wrote:\n\n> I've applied the smaller comments and am now investigating the right\n> thing to do with other is_glob_special() characters. There is a small\n> chance that I can replace any \"c == '*' || c == '\\'\" with is_glob_special(),\n> but we shall see. At the very least, I'll need to expand my tests.\n\nYeah, that's all I'd expect to need. You mentioned earlier about how\nls-tree would output them, but I don't think it would matter. Now that\nyou're using unquote_c_style(), you'll get the literal filenames no\nmatter which way ls-tree decides to quote them (and I don't think it\nwould quote '?', just as it wouldn't '*', because those are not\nsyntactically significant in its output).\n\n-Peff\n"},{"id":"390801","messageId":"35c451a0-f06a-ada1-9614-49dae91e6e4d@gmail.com","threadId":"52638","inReplyTo":"20200130072957.GE2189233@coredump.intra.peff.net","subject":"Re: [PATCH v3 10/12] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T15:01:13Z","receivedAt":"2020-01-30T15:01:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 2:29 AM, Jeff King wrote:\n> On Wed, Jan 29, 2020 at 09:16:11AM -0500, Derrick Stolee wrote:\n> \n>> I've applied the smaller comments and am now investigating the right\n>> thing to do with other is_glob_special() characters. There is a small\n>> chance that I can replace any \"c == '*' || c == '\\'\" with is_glob_special(),\n>> but we shall see. At the very least, I'll need to expand my tests.\n> \n> Yeah, that's all I'd expect to need. You mentioned earlier about how\n> ls-tree would output them, but I don't think it would matter. Now that\n> you're using unquote_c_style(), you'll get the literal filenames no\n> matter which way ls-tree decides to quote them (and I don't think it\n> would quote '?', just as it wouldn't '*', because those are not\n> syntactically significant in its output).\n\nYes, even this case for 'git ls-tree' gets covered in the final\nversion of the test:\n\ntest_expect_success BSLASHPSPEC 'pattern-checks: escaped characters' '\n\tgit clone repo escaped &&\n\tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n\tNEWTREE=$(git -C escaped mktree <<-EOF\n\t$(git -C escaped ls-tree HEAD)\n\t040000 tree $TREEOID\tzbad\\\\dir\n\t040000 tree $TREEOID\tzdoes*exist\n\t040000 tree $TREEOID\tzglob[!a]?\n\tEOF\n\t) &&\n\tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n\tgit -C escaped reset --hard $COMMIT &&\n\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" zglob[!a]? &&\n\tgit -C escaped sparse-checkout init --cone &&\n\tgit -C escaped sparse-checkout set zbad\\\\dir/bogus \"zdoes*not*exist\" \"zdoes*exist\" \"zglob[!a]?\" &&\n\tcat >expect <<-\\EOF &&\n\t/*\n\t!/*/\n\t/zbad\\\\dir/\n\t!/zbad\\\\dir/*/\n\t/zbad\\\\dir/bogus/\n\t/zdoes\\*exist/\n\t/zdoes\\*not\\*exist/\n\t/zglob\\[!a]\\?/\n\tEOF\n\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist zglob[!a]?\" &&\n\tgit -C escaped ls-tree -d --name-only HEAD >list-expect &&\n\tgit -C escaped sparse-checkout set --stdin expect <<-\\EOF &&\n\t/*\n\t!/*/\n\t/deep/\n\t/folder1/\n\t/folder2/\n\t/zbad\\\\dir/\n\t/zdoes\\*exist/\n\t/zglob\\[!a]\\?/\n\tEOF\n\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" zglob[!a]? &&\n\tgit -C escaped sparse-checkout list >list-actual &&\n\ttest_cmp list-expect list-actual\n'\n\nThanks,\n-Stolee\n\n\n"},{"id":"390941","messageId":"1cc825412f0f2b66f7b89ed5ce33bcda937f4ee6.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 01/15] t1091: use check_files to reduce boilerplate","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:01Z","receivedAt":"2020-01-31T20:16:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen testing the sparse-checkout feature, we need to compare the\ncontents of the working-directory against some expected output.\nUsing here-docs was useful in the beginning, but became repetetive\nas the test script grew.\n\nCreate a check_files helper to make the tests simpler and easier\nto extend. It also reduces instances of bad here-doc whitespace.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 117 ++++++-----------------------\n 1 file changed, 22 insertions(+), 95 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ff7f8f7a1f..e058a20ad6 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -12,6 +12,13 @@ list_files() {\n \t(cd \"$1\" && printf '%s\\n' *)\n }\n \n+check_files() {\n+\tlist_files \"$1\" >actual &&\n+\tshift &&\n+\tprintf \"%s\\n\" $@ >expect &&\n+\ttest_cmp expect actual\n+}\n+\n test_expect_success 'setup' '\n \tgit init repo &&\n \t(\n@@ -58,9 +65,7 @@ test_expect_success 'git sparse-checkout init' '\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n-\tlist_files repo >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a\n '\n \n test_expect_success 'git sparse-checkout list after init' '\n@@ -81,13 +86,7 @@ test_expect_success 'init with existing sparse-checkout' '\n \t\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'clone --sparse' '\n@@ -98,9 +97,7 @@ test_expect_success 'clone --sparse' '\n \t\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n-\tlist_files clone >dir &&\n-\techo a >expect &&\n-\ttest_cmp expect dir\n+\tcheck_files clone a\n '\n \n test_expect_success 'set enables config' '\n@@ -127,13 +124,7 @@ test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n@@ -147,13 +138,7 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo \"a folder1 folder2\"\n '\n \n test_expect_success 'cone mode: match patterns' '\n@@ -162,13 +147,7 @@ test_expect_success 'cone mode: match patterns' '\n \tgit -C repo read-tree -mu HEAD 2>err &&\n \ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: warn on bad pattern' '\n@@ -185,14 +164,7 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_path_is_file repo/.git/info/sparse-checkout &&\n \tgit -C repo config --list >config &&\n \ttest_must_fail git config core.sparseCheckout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a deep folder1 folder2\n '\n \n test_expect_success 'cone mode: init and set' '\n@@ -204,24 +176,9 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir &&\n \tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n \ttest_must_be_empty err &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeep\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files repo/deep/deeper1 >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeepest\n-\tEOF\n-\ttest_cmp expect dir &&\n+\tcheck_files repo a deep &&\n+\tcheck_files repo/deep a deeper1 &&\n+\tcheck_files repo/deep/deeper1 a deepest &&\n \tcat >expect <<-EOF &&\n \t\t/*\n \t\t!/*/\n@@ -237,13 +194,7 @@ test_expect_success 'cone mode: init and set' '\n \t\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\tlist_files repo >dir &&\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n@@ -275,13 +226,7 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n \ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n \ttest_cmp repo/.git/info/sparse-checkout expect &&\n-\tlist_files repo/deep >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tdeeper1\n-\t\tdeeper2\n-\tEOF\n-\ttest_cmp dir expect\n+\tcheck_files repo/deep a deeper1 deeper2\n '\n \n test_expect_success 'revert to old sparse-checkout on empty update' '\n@@ -332,12 +277,7 @@ test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \t\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tlist_files repo >dir &&\n-\tcat >expect <<-EOF &&\n-\t\ta\n-\t\tfolder1\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files repo a folder1\n '\n \n test_expect_success 'interaction with submodules' '\n@@ -351,21 +291,8 @@ test_expect_success 'interaction with submodules' '\n \t\tgit sparse-checkout init --cone &&\n \t\tgit sparse-checkout set folder1\n \t) &&\n-\tlist_files super >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tfolder1\n-\t\tmodules\n-\tEOF\n-\ttest_cmp expect dir &&\n-\tlist_files super/modules/child >dir &&\n-\tcat >expect <<-\\EOF &&\n-\t\ta\n-\t\tdeep\n-\t\tfolder1\n-\t\tfolder2\n-\tEOF\n-\ttest_cmp expect dir\n+\tcheck_files super a folder1 modules &&\n+\tcheck_files super/modules/child a deep folder1 folder2\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390942","messageId":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v3.git.1580236003.gitgitgadget@gmail.com","subject":"[PATCH v4 00/15] Harden the sparse-checkout builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:00Z","receivedAt":"2020-01-31T20:16:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series is based on ds/sparse-list-in-cone-mode.\n\nThis series attempts to clean up some rough edges in the sparse-checkout\nfeature, especially around the cone mode.\n\nUnfortunately, after the v2.25.0 release, we noticed an issue with the \"git\nclone --sparse\" option when using a URL instead of a local path. This is\nfixed and properly tested here.\n\nAlso, let's improve Git's response to these more complicated scenarios:\n\n 1. Running \"git sparse-checkout init\" in a worktree would complain because\n the \"info\" dir doesn't exist.\n 2. Tracked paths that include \"*\" and \"\\\" in their filenames.\n 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n as \"**\" anywhere or \"*\" in the wrong place, then we should respond\n appropriately. That is: warn that the patterns are not cone-mode, then\n revert to the old logic.\n\nUpdates in V2:\n\n * Added C-style quoting to the output of \"git sparse-checkout list\" in cone\n mode.\n * Improved documentation.\n * Responded to most style feedback. Hopefully I didn't miss anything.\n * I was lingering on this a little to see if I could also fix the issue\n raised in [1], but I have not figured that one out, yet.\n\nUpdate in V3:\n\n * Input now uses Peff's recommended pattern: unquote C-style strings over\n stdin and otherwise do not un-escape input.\n\n[1] \nhttps://lore.kernel.org/git/062301d5d0bc$c3e17760$4ba46620$@Frontier.com/\n\nThanks, -Stolee\n\nDerrick Stolee (14):\n t1091: use check_files to reduce boilerplate\n t1091: improve here-docs\n sparse-checkout: create leading directories\n clone: fix --sparse option with URLs\n sparse-checkout: cone mode does not recognize \"**\"\n sparse-checkout: detect short patterns\n sparse-checkout: warn on globs in cone patterns\n sparse-checkout: properly match escaped characters\n sparse-checkout: write escaped patterns in cone mode\n sparse-checkout: unquote C-style strings over --stdin\n sparse-checkout: use C-style quotes in 'list' subcommand\n sparse-checkout: escape all glob characters on write\n sparse-checkout: improve docs around 'set' in cone mode\n sparse-checkout: fix cone mode behavior mismatch\n\nJeff King (1):\n sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n\n Documentation/git-sparse-checkout.txt | 19 +-\n builtin/clone.c | 2 +-\n builtin/sparse-checkout.c | 48 +++-\n dir.c | 79 +++++-\n t/t1091-sparse-checkout-builtin.sh | 352 +++++++++++++++-----------\n unpack-trees.c | 2 +-\n 6 files changed, 346 insertions(+), 156 deletions(-)\n\n\nbase-commit: 4fd683b6a35eabd23dd5183da7f654a1e1f00325\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-513%2Fderrickstolee%2Fsparse-harden-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-513/derrickstolee/sparse-harden-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/513\n\nRange-diff vs v3:\n\n 1: 1cc825412f = 1: 1cc825412f t1091: use check_files to reduce boilerplate\n 2: b7a6ad145a = 2: b7a6ad145a t1091: improve here-docs\n 3: 5497ad8778 = 3: 5497ad8778 sparse-checkout: create leading directories\n 4: 4991a51f6d = 4: 4991a51f6d clone: fix --sparse option with URLs\n 5: ae78c3069b = 5: ae78c3069b sparse-checkout: fix documentation typo for core.sparseCheckoutCone\n 6: 2ad4d3e467 = 6: 2ad4d3e467 sparse-checkout: cone mode does not recognize \"**\"\n 7: aace064510 = 7: aace064510 sparse-checkout: detect short patterns\n 8: d2a510a3bb ! 8: 66caabef5f sparse-checkout: warn on incorrect '*' in patterns\n @@ -1,6 +1,6 @@\n Author: Derrick Stolee \n \n - sparse-checkout: warn on incorrect '*' in patterns\n + sparse-checkout: warn on globs in cone patterns\n \n In cone mode, the sparse-checkout commmand will write patterns that\n allow faster pattern matching. This matching only works if the patterns\n @@ -9,10 +9,10 @@\n cone mode matching to fail.\n \n The cone mode patterns may end in \"/*\" but otherwise an un-escaped\n - asterisk is invalid. Add checks to disable cone mode when seeing these\n - values.\n + asterisk or other glob character is invalid. Add checks to disable\n + cone mode when seeing these values.\n \n - A later change will properly handle escaped asterisks.\n + A later change will properly handle escaped globs.\n \n Signed-off-by: Derrick Stolee \n \n @@ -43,16 +43,23 @@\n +\tnext = given->pattern + 2;\n +\n +\twhile (*cur) {\n -+\t\t/* We care about *cur == '*' */\n -+\t\tif (*cur != '*')\n ++\t\t/* Watch for glob characters '*', '\\', '[', '?' */\n ++\t\tif (!is_glob_special(*cur))\n +\t\t\tgoto increment;\n +\n +\t\t/* But only if *prev != '\\\\' */\n +\t\tif (*prev == '\\\\')\n +\t\t\tgoto increment;\n +\n ++\t\t/* But allow the initial '\\' */\n ++\t\tif (*cur == '\\\\' &&\n ++\t\t is_glob_special(*next))\n ++\t\t\tgoto increment;\n ++\n +\t\t/* But a trailing '/' then '*' is fine */\n -+\t\tif (*prev == '/' && *next == 0)\n ++\t\tif (*prev == '/' &&\n ++\t\t *cur == '*' &&\n ++\t\t *next == 0)\n +\t\t\tgoto increment;\n +\n +\t\t/* Not a cone pattern. */\n @@ -94,6 +101,18 @@\n +\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n +'\n +\n ++test_expect_success 'pattern-checks: contained glob characters' '\n ++\tfor c in \"[a]\" \"\\\\\" \"?\" \"*\"\n ++\tdo\n ++\t\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n ++\t\t/*\n ++\t\t!/*/\n ++\t\tsomething$c-else/\n ++\t\tEOF\n ++\t\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n ++\tdone\n ++'\n ++\n +test_expect_success 'pattern-checks: escaped \"*\"' '\n +\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n +\t/*\n 9: 9ea69e9069 ! 9: 4c86d01f0e sparse-checkout: properly match escaped characters\n @@ -23,6 +23,7 @@\n +static char *dup_and_filter_pattern(const char *pattern)\n +{\n +\tchar *set, *read;\n ++\tsize_t count = 0;\n +\tchar *result = xstrdup(pattern);\n +\n +\tset = result;\n @@ -37,11 +38,14 @@\n +\n +\t\tset++;\n +\t\tread++;\n ++\t\tcount++;\n +\t}\n +\t*set = 0;\n +\n -+\tif (*(read - 2) == '/' && *(read - 1) == '*')\n -+\t\t*(read - 2) = 0;\n ++\tif (count > 2 &&\n ++\t *(set - 1) == '*' &&\n ++\t *(set - 2) == '/')\n ++\t\t*(set - 2) = 0;\n +\n +\treturn result;\n +}\n @@ -73,7 +77,7 @@\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n + \tdone\n '\n \n -test_expect_success 'pattern-checks: escaped \"*\"' '\n @@ -96,6 +100,7 @@\n \t!/*/\n -\t/does\\*not\\*exist/\n +\t/zbad\\\\dir/\n ++\t!/zbad\\\\dir/*/\n +\t/zdoes\\*not\\*exist/\n +\t/zdoes\\*exist/\n \tEOF\n 10: e2f9afc70c ! 10: 0b9346f67b sparse-checkout: write escaped patterns in cone mode\n @@ -9,26 +9,12 @@\n However, in cone mode we expect a list of paths to directories and then\n we convert those into patterns.\n \n - Even more specifically, the goal is to always allow the following from\n - the root of a repo:\n -\n - git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n -\n - The ls-tree command provides directory names with an unescaped asterisk.\n - It also quotes the directories that contain an escaped backslash. We\n - must remove these quotes, then keep the escaped backslashes.\n -\n However, there is some care needed for the timing of these escapes. The\n in-memory pattern list is used to update the working directory before\n writing the patterns to disk. Thus, we need the command to have the\n unescaped names in the hashsets for the cone comparisons, then escape\n the patterns later.\n \n - Use unquote_c_style() when parsing lines from stdin. Command-line\n - arguments will be parsed as-is, assuming the user can do the correct\n - level of escaping from their environment to match the exact directory\n - names.\n -\n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n @@ -89,29 +75,6 @@\n \t}\n }\n \n -@@\n - \t\tpl.use_cone_patterns = 1;\n - \n - \t\tif (set_opts.use_stdin) {\n --\t\t\twhile (!strbuf_getline(&line, stdin))\n -+\t\t\tstruct strbuf unquoted = STRBUF_INIT;\n -+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n -+\t\t\t\tif (line.buf[0] == '\"') {\n -+\t\t\t\t\tstrbuf_setlen(&unquoted, 0);\n -+\t\t\t\t\tif (unquote_c_style(&unquoted, line.buf, NULL))\n -+\t\t\t\t\t\tdie(_(\"unable to unquote C-style string '%s'\"),\n -+\t\t\t\t\t\tline.buf);\n -+\n -+\t\t\t\t\tstrbuf_swap(&unquoted, &line);\n -+\t\t\t\t}\n -+\n - \t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n -+\t\t\t}\n -+\n -+\t\t\tstrbuf_release(&unquoted);\n - \t\t} else {\n - \t\t\tfor (i = 0; i < argc; i++) {\n - \t\t\t\tstrbuf_setlen(&line, 0);\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -131,29 +94,18 @@\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n -\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n -+\tgit -C escaped sparse-checkout set zbad\\\\dir \"zdoes*not*exist\" \"zdoes*exist\" &&\n ++\tgit -C escaped sparse-checkout set zbad\\\\dir/bogus \"zdoes*not*exist\" \"zdoes*exist\" &&\n +\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n \t/zbad\\\\dir/\n -+\t/zdoes\\*exist/\n - \t/zdoes\\*not\\*exist/\n -+\tEOF\n -+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n -+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n -+\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n -+\tcat >expect <<-\\EOF &&\n -+\t/*\n -+\t!/*/\n -+\t/deep/\n -+\t/folder1/\n -+\t/folder2/\n -+\t/zbad\\\\dir/\n + \t!/zbad\\\\dir/*/\n +-\t/zdoes\\*not\\*exist/\n ++\t/zbad\\\\dir/bogus/\n \t/zdoes\\*exist/\n ++\t/zdoes\\*not\\*exist/\n \tEOF\n --\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n +\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n -+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n + \tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n - test_done\n -: ---------- > 11: 9f682e6076 sparse-checkout: unquote C-style strings over --stdin\n 11: ec714a4cf0 = 12: e2c6f85617 sparse-checkout: use C-style quotes in 'list' subcommand\n -: ---------- > 13: 54be8e89eb sparse-checkout: escape all glob characters on write\n 12: 1867746d97 = 14: 3dd8f97b3a sparse-checkout: improve docs around 'set' in cone mode\n -: ---------- > 15: 5e9fcce75f sparse-checkout: fix cone mode behavior mismatch\n\n-- \ngitgitgadget\n"},{"id":"390943","messageId":"b7a6ad145af6a5560bb07ed7e36f587f0f60f41f.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 02/15] t1091: improve here-docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:02Z","receivedAt":"2020-01-31T20:16:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nt1091-sparse-checkout-builtin.sh uses here-docs to populate the\nexpected contents of the sparse-checkout file. These do not use\nshell interpolation, so use \"-\\EOF\" instead of \"-EOF\". Also use\nproper tabbing.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 98 +++++++++++++++---------------\n 1 file changed, 49 insertions(+), 49 deletions(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e058a20ad6..e28e1c797f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -46,11 +46,11 @@ test_expect_success 'git sparse-checkout list (empty)' '\n \n test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n-\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n-\t\t/folder1/*\n-\t\t/deep/\n-\t\t**/a\n-\t\t!*bin*\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/folder1/*\n+\t/deep/\n+\t**/a\n+\t!*bin*\n \tEOF\n \tcp repo/.git/info/sparse-checkout expect &&\n \tgit -C repo sparse-checkout list >list &&\n@@ -59,9 +59,9 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \n test_expect_success 'git sparse-checkout init' '\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \ttest_cmp_config -C repo true core.sparsecheckout &&\n@@ -70,9 +70,9 @@ test_expect_success 'git sparse-checkout init' '\n \n test_expect_success 'git sparse-checkout list after init' '\n \tgit -C repo sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual\n '\n@@ -80,10 +80,10 @@ test_expect_success 'git sparse-checkout list after init' '\n test_expect_success 'init with existing sparse-checkout' '\n \techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout init &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1 folder2\n@@ -92,9 +92,9 @@ test_expect_success 'init with existing sparse-checkout' '\n test_expect_success 'clone --sparse' '\n \tgit clone --sparse repo clone &&\n \tgit -C clone sparse-checkout list >actual &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n \tEOF\n \ttest_cmp expect actual &&\n \tcheck_files clone a\n@@ -116,10 +116,10 @@ test_expect_success 'set enables config' '\n \n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t*folder*\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*folder*\n \tEOF\n \tgit -C repo sparse-checkout list >actual &&\n \ttest_cmp expect actual &&\n@@ -128,11 +128,11 @@ test_expect_success 'set sparse-checkout using builtin' '\n '\n \n test_expect_success 'set sparse-checkout using --stdin' '\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n-\t\t/folder2/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n+\t/folder2/\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual &&\n@@ -179,28 +179,28 @@ test_expect_success 'cone mode: init and set' '\n \tcheck_files repo a deep &&\n \tcheck_files repo/deep a deeper1 &&\n \tcheck_files repo/deep/deeper1 a deepest &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n-\t\t!/deep/*/\n-\t\t/deep/deeper1/\n-\t\t!/deep/deeper1/*/\n-\t\t/deep/deeper1/deepest/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t!/deep/*/\n+\t/deep/deeper1/\n+\t!/deep/deeper1/*/\n+\t/deep/deeper1/deepest/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n-\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \ttest_must_be_empty err &&\n \tcheck_files repo a folder1 folder2\n '\n \n test_expect_success 'cone mode: list' '\n-\tcat >expect <<-EOF &&\n-\t\tfolder1\n-\t\tfolder2\n+\tcat >expect <<-\\EOF &&\n+\tfolder1\n+\tfolder2\n \tEOF\n \tgit -C repo sparse-checkout set --stdin actual 2>err &&\n@@ -211,10 +211,10 @@ test_expect_success 'cone mode: list' '\n test_expect_success 'cone mode: set with nested folders' '\n \tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n \ttest_line_count = 0 err &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/deep/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n \tEOF\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n@@ -271,10 +271,10 @@ test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status'\n test_expect_success 'cone mode: set with core.ignoreCase=true' '\n \tgit -C repo sparse-checkout init --cone &&\n \tgit -C repo -c core.ignoreCase=true sparse-checkout set folder1 &&\n-\tcat >expect <<-EOF &&\n-\t\t/*\n-\t\t!/*/\n-\t\t/folder1/\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/\n \tEOF\n \ttest_cmp expect repo/.git/info/sparse-checkout &&\n \tcheck_files repo a folder1\n-- \ngitgitgadget\n\n"},{"id":"390944","messageId":"4991a51f6d5d840eaa3bb830e68f1530c2ee08e4.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 04/15] clone: fix --sparse option with URLs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:04Z","receivedAt":"2020-01-31T20:16:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --sparse option was added to the clone builtin in d89f09c (clone:\nadd --sparse mode, 2019-11-21) and was tested with a local path clone\nin t1091-sparse-checkout-builtin.sh. However, due to a difference in\nhow local paths are handled versus URLs, this mechanism does not work\nwith URLs.\n\nModify the test to use a \"file://\" URL, which would output this error\nbefore the code change:\n\n Cloning into 'clone'...\n fatal: cannot change to 'file://.../repo': No such file or directory\n error: failed to initialize sparse-checkout\n\nThese errors are due to using a \"-C \" option to call 'git -C\n sparse-checkout init' but the URL is being given instead of\nthe target directory.\n\nUpdate that target directory to evaluate this correctly. I have also\nmanually tested that https:// URLs are handled correctly as well.\n\nAcked-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 4348d962c9..2caefc44fb 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1130,7 +1130,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n-\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\tif (option_sparse_checkout && git_sparse_checkout_init(dir))\n \t\treturn 1;\n \n \tremote = remote_get(option_origin);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 43d1f7520c..cf4a595c86 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -90,7 +90,7 @@ test_expect_success 'init with existing sparse-checkout' '\n '\n \n test_expect_success 'clone --sparse' '\n-\tgit clone --sparse repo clone &&\n+\tgit clone --sparse \"file://$(pwd)/repo\" clone &&\n \tgit -C clone sparse-checkout list >actual &&\n \tcat >expect <<-\\EOF &&\n \t/*\n-- \ngitgitgadget\n\n"},{"id":"390945","messageId":"2ad4d3e467a7e2be679de4084e54f426aa242bd6.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 06/15] sparse-checkout: cone mode does not recognize \"**\"","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:06Z","receivedAt":"2020-01-31T20:16:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen core.sparseCheckoutCone is enabled, the 'git sparse-checkout set'\ncommand creates a restricted set of possible patterns that are used\nby a custom algorithm to quickly match those patterns.\n\nIf a user manually edits the sparse-checkout file, then they could\ncreate patterns that do not match these expectations. The cone-mode\nmatching algorithm can return incorrect results. The solution is to\ndetect these incorrect patterns, warn that we do not recognize them,\nand revert to the standard algorithm.\n\nCheck each pattern for the \"**\" substring, and revert to the old\nlogic if seen. While technically a \"//**\" pattern matches\nthe meaning of \"//\", it is not one that would be written by\nthe sparse-checkout builtin in cone mode. Attempting to accept that\npattern change complicates the logic and instead we punt and do\nnot accept any instance of \"**\".\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 7 +++++-\n t/t1091-sparse-checkout-builtin.sh | 34 ++++++++++++++++++++++++++++++\n 2 files changed, 40 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 22d08e61c2..40fed73a94 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,11 +651,16 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n+\tif (strstr(given->pattern, \"**\")) {\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n \t\t\t/* Not a cone pattern. */\n-\t\t\tpl->use_cone_patterns = 0;\n \t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\t\tgoto clear_hashmaps;\n \t\t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex cf4a595c86..e2e45dc7fd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -305,4 +305,38 @@ test_expect_success 'different sparse-checkouts with worktrees' '\n \tcheck_files worktree a deep\n '\n \n+check_read_tree_errors () {\n+\tREPO=$1\n+\tFILES=$2\n+\tERRORS=$3\n+\tgit -C $REPO read-tree -mu HEAD 2>err &&\n+\tif test -z \"$ERRORS\"\n+\tthen\n+\t\ttest_must_be_empty err\n+\telse\n+\t\ttest_i18ngrep \"$ERRORS\" err\n+\tfi &&\n+\tcheck_files $REPO $FILES\n+}\n+\n+test_expect_success 'pattern-checks: /A/**' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/folder1/**\n+\tEOF\n+\tcheck_read_tree_errors repo \"a folder1\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: /A/**/B/' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/**/deepest\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\" &&\n+\tcheck_files repo/deep \"deeper1\" &&\n+\tcheck_files repo/deep/deeper1 \"deepest\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390946","messageId":"aace0645107719f62c611296adc44f8c286d8403.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 07/15] sparse-checkout: detect short patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:07Z","receivedAt":"2020-01-31T20:16:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the shortest pattern the sparse-checkout command will\nwrite into the sparse-checkout file is \"/*\". This is handled carefully\nin add_pattern_to_hashsets(), so warn if any other pattern is this\nshort. This will assist future pattern checks by allowing us to assume\nthere are at least three characters in the pattern.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 3 ++-\n t/t1091-sparse-checkout-builtin.sh | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 40fed73a94..c2e585607e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -651,7 +651,8 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\treturn;\n \t}\n \n-\tif (strstr(given->pattern, \"**\")) {\n+\tif (given->patternlen <= 2 ||\n+\t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e2e45dc7fd..2e57534799 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -339,4 +339,13 @@ test_expect_success 'pattern-checks: /A/**/B/' '\n \tcheck_files repo/deep/deeper1 \"deepest\"\n '\n \n+test_expect_success 'pattern-checks: too short' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390947","messageId":"ae78c3069b08557e3acc544601c71ee932ace19d.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 05/15] sparse-checkout: fix documentation typo for core.sparseCheckoutCone","fromName":"Jeff King via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:05Z","receivedAt":"2020-01-31T20:16:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"From: Jeff King \n\nSigned-off-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 3b341cf0fc..4834fb434d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -106,7 +106,7 @@ The full pattern set allows for arbitrary pattern matches and complicated\n inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n updating the index, where N is the number of patterns and M is the number\n of paths in the index. To combat this performance issue, a more restricted\n-pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+pattern set is allowed when `core.sparseCheckoutCone` is enabled.\n \n The accepted patterns in the cone pattern set are:\n \n-- \ngitgitgadget\n\n"},{"id":"390948","messageId":"66caabef5f998e15cc564e91231590280892191b.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 08/15] sparse-checkout: warn on globs in cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:08Z","receivedAt":"2020-01-31T20:16:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout commmand will write patterns that\nallow faster pattern matching. This matching only works if the patterns\nin the sparse-checkout file are those written by that command. Users\ncan edit the sparse-checkout file and create patterns that cause the\ncone mode matching to fail.\n\nThe cone mode patterns may end in \"/*\" but otherwise an un-escaped\nasterisk or other glob character is invalid. Add checks to disable\ncone mode when seeing these values.\n\nA later change will properly handle escaped globs.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 36 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 39 ++++++++++++++++++++++++++++++\n 2 files changed, 75 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex c2e585607e..71d28331f3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -635,6 +635,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \tstruct pattern_entry *translated;\n \tchar *truncated;\n \tchar *data = NULL;\n+\tconst char *prev, *cur, *next;\n \n \tif (!pl->use_cone_patterns)\n \t\treturn;\n@@ -652,12 +653,47 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t}\n \n \tif (given->patternlen <= 2 ||\n+\t *given->pattern == '*' ||\n \t strstr(given->pattern, \"**\")) {\n \t\t/* Not a cone pattern. */\n \t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n \t\tgoto clear_hashmaps;\n \t}\n \n+\tprev = given->pattern;\n+\tcur = given->pattern + 1;\n+\tnext = given->pattern + 2;\n+\n+\twhile (*cur) {\n+\t\t/* Watch for glob characters '*', '\\', '[', '?' */\n+\t\tif (!is_glob_special(*cur))\n+\t\t\tgoto increment;\n+\n+\t\t/* But only if *prev != '\\\\' */\n+\t\tif (*prev == '\\\\')\n+\t\t\tgoto increment;\n+\n+\t\t/* But allow the initial '\\' */\n+\t\tif (*cur == '\\\\' &&\n+\t\t is_glob_special(*next))\n+\t\t\tgoto increment;\n+\n+\t\t/* But a trailing '/' then '*' is fine */\n+\t\tif (*prev == '/' &&\n+\t\t *cur == '*' &&\n+\t\t *next == 0)\n+\t\t\tgoto increment;\n+\n+\t\t/* Not a cone pattern. */\n+\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\tgoto clear_hashmaps;\n+\n+\tincrement:\n+\t\tprev++;\n+\t\tcur++;\n+\t\tnext++;\n+\t}\n+\n \tif (given->patternlen > 2 &&\n \t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n \t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2e57534799..c732abeacd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -348,4 +348,43 @@ test_expect_success 'pattern-checks: too short' '\n \tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n '\n \n+test_expect_success 'pattern-checks: trailing \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a*\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: starting \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t*eep/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a deep\" \"disabling cone pattern matching\"\n+'\n+\n+test_expect_success 'pattern-checks: contained glob characters' '\n+\tfor c in \"[a]\" \"\\\\\" \"?\" \"*\"\n+\tdo\n+\t\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\tsomething$c-else/\n+\t\tEOF\n+\t\tcheck_read_tree_errors repo \"a\" \"disabling cone pattern matching\"\n+\tdone\n+'\n+\n+test_expect_success 'pattern-checks: escaped \"*\"' '\n+\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/does\\*not\\*exist/\n+\tEOF\n+\tcheck_read_tree_errors repo \"a\" \"\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390949","messageId":"5497ad8778c29ca936206d2ded6773758c236c12.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 03/15] sparse-checkout: create leading directories","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:03Z","receivedAt":"2020-01-31T20:16:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git init' command creates the \".git/info\" directory and fills it\nwith some default files. However, 'git worktree add' does not create\nthe info directory for that worktree. This causes a problem when running\n\"git sparse-checkout init\" inside a worktree. While care was taken to\nallow the sparse-checkout config to be specific to a worktree, this\ninitialization was untested.\n\nSafely create the leading directories for the sparse-checkout file. This\nis the safest thing to do even without worktrees, as a user could delete\ntheir \".git/info\" directory and expect Git to recover safely.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 4 ++++\n t/t1091-sparse-checkout-builtin.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b3bed891cb..3cee8ab46e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -199,6 +199,10 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tint result;\n \n \tsparse_filename = get_sparse_checkout_filename();\n+\n+\tif (safe_create_leading_directories(sparse_filename))\n+\t\tdie(_(\"failed to create directory for sparse-checkout file\"));\n+\n \tfd = hold_lock_file_for_update(&lk, sparse_filename,\n \t\t\t\t LOCK_DIE_ON_ERROR);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e28e1c797f..43d1f7520c 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -295,4 +295,14 @@ test_expect_success 'interaction with submodules' '\n \tcheck_files super/modules/child a deep folder1 folder2\n '\n \n+test_expect_success 'different sparse-checkouts with worktrees' '\n+\tgit -C repo worktree add --detach ../worktree &&\n+\tcheck_files worktree \"a deep folder1 folder2\" &&\n+\tgit -C worktree sparse-checkout init --cone &&\n+\tgit -C repo sparse-checkout set folder1 &&\n+\tgit -C worktree sparse-checkout set deep/deeper1 &&\n+\tcheck_files repo a folder1 &&\n+\tcheck_files worktree a deep\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"390950","messageId":"4c86d01f0e965db0285336a88f2dc34cbcd0c073.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 09/15] sparse-checkout: properly match escaped characters","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:09Z","receivedAt":"2020-01-31T20:16:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn cone mode, the sparse-checkout feature uses hashset containment\nqueries to match paths. Make this algorithm respect escaped asterisk\n(*) and backslash (\\) characters.\n\nCreate dup_and_filter_pattern() method to convert a pattern by\nremoving escape characters and dropping an optional \"/*\" at the end.\nThis method is available in dir.h as we will use it in\nbuiltin/sparse-checkout.c in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 35 +++++++++++++++++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 23 ++++++++++++++++----\n 2 files changed, 51 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 71d28331f3..7ac0920b71 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -630,6 +630,36 @@ int pl_hashmap_cmp(const void *unused_cmp_data,\n \treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n }\n \n+static char *dup_and_filter_pattern(const char *pattern)\n+{\n+\tchar *set, *read;\n+\tsize_t count = 0;\n+\tchar *result = xstrdup(pattern);\n+\n+\tset = result;\n+\tread = result;\n+\n+\twhile (*read) {\n+\t\t/* skip escape characters (once) */\n+\t\tif (*read == '\\\\')\n+\t\t\tread++;\n+\n+\t\t*set = *read;\n+\n+\t\tset++;\n+\t\tread++;\n+\t\tcount++;\n+\t}\n+\t*set = 0;\n+\n+\tif (count > 2 &&\n+\t *(set - 1) == '*' &&\n+\t *(set - 2) == '/')\n+\t\t*(set - 2) = 0;\n+\n+\treturn result;\n+}\n+\n static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n {\n \tstruct pattern_entry *translated;\n@@ -702,8 +732,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \t\t\tgoto clear_hashmaps;\n \t\t}\n \n-\t\ttruncated = xstrdup(given->pattern);\n-\t\ttruncated[given->patternlen - 2] = 0;\n+\t\ttruncated = dup_and_filter_pattern(given->pattern);\n \n \t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n \t\ttranslated->pattern = truncated;\n@@ -737,7 +766,7 @@ static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern\n \n \ttranslated = xmalloc(sizeof(struct pattern_entry));\n \n-\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->pattern = dup_and_filter_pattern(given->pattern);\n \ttranslated->patternlen = given->patternlen;\n \thashmap_entry_init(&translated->ent,\n \t\t\t ignore_case ?\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex c732abeacd..9ea700896d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -378,13 +378,28 @@ test_expect_success 'pattern-checks: contained glob characters' '\n \tdone\n '\n \n-test_expect_success 'pattern-checks: escaped \"*\"' '\n-\tcat >repo/.git/info/sparse-checkout <<-\\EOF &&\n+test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n+\tgit clone repo escaped &&\n+\tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n+\tNEWTREE=$(git -C escaped mktree <<-EOF\n+\t$(git -C escaped ls-tree HEAD)\n+\t040000 tree $TREEOID\tzbad\\\\dir\n+\t040000 tree $TREEOID\tzdoes*exist\n+\tEOF\n+\t) &&\n+\tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n+\tgit -C escaped reset --hard $COMMIT &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout init --cone &&\n+\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n \t/*\n \t!/*/\n-\t/does\\*not\\*exist/\n+\t/zbad\\\\dir/\n+\t!/zbad\\\\dir/*/\n+\t/zdoes\\*not\\*exist/\n+\t/zdoes\\*exist/\n \tEOF\n-\tcheck_read_tree_errors repo \"a\" \"\"\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390951","messageId":"9f682e6076fd23491d51561ff2e9cf506cddbce6.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 11/15] sparse-checkout: unquote C-style strings over --stdin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:11Z","receivedAt":"2020-01-31T20:16:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user somehow creates a directory with an asterisk (*) or backslash\n(\\), then the \"git sparse-checkout set\" command will struggle to provide\nthe correct pattern in the sparse-checkout file. When not in cone mode,\nthe provided pattern is written directly into the sparse-checkout file.\nHowever, in cone mode we expect a list of paths to directories and then\nwe convert those into patterns.\n\nEven more specifically, the goal is to always allow the following from\nthe root of a repo:\n\n git ls-tree --name-only -d HEAD | git sparse-checkout set --stdin\n\nThe ls-tree command provides directory names with an unescaped asterisk.\nIt also quotes the directories that contain an escaped backslash. We\nmust remove these quotes, then keep the escaped backslashes.\n\nUse unquote_c_style() when parsing lines from stdin. Command-line\narguments will be parsed as-is, assuming the user can do the correct\nlevel of escaping from their environment to match the exact directory\nnames.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 15 ++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 14 +++++++++++++-\n 2 files changed, 27 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex cc86b8a014..6083aa10f2 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -442,8 +442,21 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n-\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\tstruct strbuf unquoted = STRBUF_INIT;\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tif (line.buf[0] == '\"') {\n+\t\t\t\t\tstrbuf_reset(&unquoted);\n+\t\t\t\t\tif (unquote_c_style(&unquoted, line.buf, NULL))\n+\t\t\t\t\t\tdie(_(\"unable to unquote C-style string '%s'\"),\n+\t\t\t\t\t\tline.buf);\n+\n+\t\t\t\t\tstrbuf_swap(&unquoted, &line);\n+\t\t\t\t}\n+\n \t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n+\n+\t\t\tstrbuf_release(&unquoted);\n \t\t} else {\n \t\t\tfor (i = 0; i < argc; i++) {\n \t\t\t\tstrbuf_setlen(&line, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex fb8718e64a..a46a310740 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -405,7 +405,19 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/zdoes\\*not\\*exist/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/deep/\n+\t/folder1/\n+\t/folder2/\n+\t/zbad\\\\dir/\n+\t/zdoes\\*exist/\n+\tEOF\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390952","messageId":"0b9346f67b207d9495ea5223bddf7b517291455e.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 10/15] sparse-checkout: write escaped patterns in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:10Z","receivedAt":"2020-01-31T20:16:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user somehow creates a directory with an asterisk (*) or backslash\n(\\), then the \"git sparse-checkout set\" command will struggle to provide\nthe correct pattern in the sparse-checkout file. When not in cone mode,\nthe provided pattern is written directly into the sparse-checkout file.\nHowever, in cone mode we expect a list of paths to directories and then\nwe convert those into patterns.\n\nHowever, there is some care needed for the timing of these escapes. The\nin-memory pattern list is used to update the working directory before\nwriting the patterns to disk. Thus, we need the command to have the\nunescaped names in the hashsets for the cone comparisons, then escape\nthe patterns later.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 23 +++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 10 ++++++++--\n 2 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 3cee8ab46e..cc86b8a014 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -13,6 +13,7 @@\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n #include \"wt-status.h\"\n+#include \"quote.h\"\n \n static const char *empty_base = \"\";\n \n@@ -140,6 +141,22 @@ static int update_working_directory(struct pattern_list *pl)\n \treturn result;\n }\n \n+static char *escaped_pattern(char *pattern)\n+{\n+\tchar *p = pattern;\n+\tstruct strbuf final = STRBUF_INIT;\n+\n+\twhile (*p) {\n+\t\tif (*p == '*' || *p == '\\\\')\n+\t\t\tstrbuf_addch(&final, '\\\\');\n+\n+\t\tstrbuf_addch(&final, *p);\n+\t\tp++;\n+\t}\n+\n+\treturn strbuf_detach(&final, NULL);\n+}\n+\n static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n {\n \tint i;\n@@ -164,10 +181,11 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \n \t\tif (strlen(pattern))\n \t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t\tfree(pattern);\n \t}\n \n \tstring_list_clear(&sl, 0);\n@@ -185,8 +203,9 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstring_list_remove_duplicates(&sl, 0);\n \n \tfor (i = 0; i < sl.nr; i++) {\n-\t\tchar *pattern = sl.items[i].string;\n+\t\tchar *pattern = escaped_pattern(sl.items[i].string);\n \t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t\tfree(pattern);\n \t}\n }\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9ea700896d..fb8718e64a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -309,6 +309,9 @@ check_read_tree_errors () {\n \tREPO=$1\n \tFILES=$2\n \tERRORS=$3\n+\tgit -C $REPO -c core.sparseCheckoutCone=false read-tree -mu HEAD 2>err &&\n+\ttest_must_be_empty err &&\n+\tcheck_files $REPO \"$FILES\" &&\n \tgit -C $REPO read-tree -mu HEAD 2>err &&\n \tif test -z \"$ERRORS\"\n \tthen\n@@ -391,14 +394,17 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tgit -C escaped reset --hard $COMMIT &&\n \tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n \tgit -C escaped sparse-checkout init --cone &&\n-\tcat >escaped/.git/info/sparse-checkout <<-\\EOF &&\n+\tgit -C escaped sparse-checkout set zbad\\\\dir/bogus \"zdoes*not*exist\" \"zdoes*exist\" &&\n+\tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n \t/zbad\\\\dir/\n \t!/zbad\\\\dir/*/\n-\t/zdoes\\*not\\*exist/\n+\t/zbad\\\\dir/bogus/\n \t/zdoes\\*exist/\n+\t/zdoes\\*not\\*exist/\n \tEOF\n+\ttest_cmp expect escaped/.git/info/sparse-checkout &&\n \tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\"\n '\n \n-- \ngitgitgadget\n\n"},{"id":"390953","messageId":"5e9fcce75f37676613f6cb919249ce8afd51c825.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 15/15] sparse-checkout: fix cone mode behavior mismatch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:15Z","receivedAt":"2020-01-31T20:16:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe intention of the special \"cone mode\" in the sparse-checkout\nfeature is to always match the same patterns that are matched by the\nsame sparse-checkout file as when cone mode is disabled.\n\nWhen a file path is given to \"git sparse-checkout set\" in cone mode,\nthen the cone mode improperly matches the file as a recursive path.\nWhen setting the skip-worktree bits, files were not expecting the\nMATCHED_RECURSIVE response, and hence these were left out of the\nmatched cone.\n\nFix this bug by checking for MATCHED_RECURSIVE in addition to MATCHED\nand add a test that prevents regression.\n\nReported-by: Finn Bryant \nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 12 ++++++++++++\n unpack-trees.c | 2 +-\n 2 files changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 37e9304ef3..7d982096fb 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -305,6 +305,18 @@ test_expect_success 'different sparse-checkouts with worktrees' '\n \tcheck_files worktree a deep\n '\n \n+test_expect_success 'set using filename keeps file on-disk' '\n+\tgit -C repo sparse-checkout set a deep &&\n+\tcat >expect <<-\\EOF &&\n+\t/*\n+\t!/*/\n+\t/a/\n+\t/deep/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tcheck_files repo a deep\n+'\n+\n check_read_tree_errors () {\n \tREPO=$1\n \tFILES=$2\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3789a22cf0..78425ce74b 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1416,7 +1416,7 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t\t\t\tname, &dtype, pl, istate);\n \t\tif (ret == UNDECIDED)\n \t\t\tret = default_match;\n-\t\tif (ret == MATCHED)\n+\t\tif (ret == MATCHED || ret == MATCHED_RECURSIVE)\n \t\t\tce->ce_flags &= ~clear_mask;\n \t\tcache++;\n \t\tprogress_nr++;\n-- \ngitgitgadget\n"},{"id":"390954","messageId":"3dd8f97b3a7c301955aba7dd5778549d1b49b0fd.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 14/15] sparse-checkout: improve docs around 'set' in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:14Z","receivedAt":"2020-01-31T20:16:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe existing documentation does not clarify how the 'set' subcommand\nchanges when core.sparseCheckoutCone is enabled. Correct this by\nchanging some language around the \"A/B/C\" example. Also include a\ndescription of the input format matching the output of 'git ls-tree\n--name-only'.\n\nHelped-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 4834fb434d..0914619881 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -50,6 +50,14 @@ To avoid interfering with other worktrees, it first enables the\n +\n When the `--stdin` option is provided, the patterns are read from\n standard in as a newline-delimited list instead of from the arguments.\n++\n+When `core.sparseCheckoutCone` is enabled, the input list is considered a\n+list of directories instead of sparse-checkout patterns. The command writes\n+patterns to the sparse-checkout file to include all files contained in those\n+directories (recursively) as well as files that are siblings of ancestor\n+directories. The input format matches the output of `git ls-tree --name-only`.\n+This includes interpreting pathnames that begin with a double quote (\") as\n+C-style quoted strings.\n \n 'disable'::\n \tDisable the `core.sparseCheckout` config setting, and restore the\n@@ -128,9 +136,12 @@ the following patterns:\n ----------------\n \n This says \"include everything in root, but nothing two levels below root.\"\n-If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n-`A/B` are added as parent patterns. The resulting sparse-checkout file is\n-now\n+\n+When in cone mode, the `git sparse-checkout set` subcommand takes a list of\n+directories instead of a list of sparse-checkout patterns. In this mode,\n+the command `git sparse-checkout set A/B/C` sets the directory `A/B/C` as\n+a recursive pattern, the directories `A` and `A/B` are added as parent\n+patterns. The resulting sparse-checkout file is now\n \n ----------------\n /*\n-- \ngitgitgadget\n\n"},{"id":"390955","messageId":"e2c6f85617f5e92c2fc52d2f869295bb0eefc242.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 12/15] sparse-checkout: use C-style quotes in 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:12Z","receivedAt":"2020-01-31T20:16:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen in cone mode, the 'git sparse-checkout list' subcommand lists\nthe directories included in the sparse cone. When these directories\ncontain odd characters, such as a backslash, then we need to use\nC-style quotes similar to 'git ls-tree'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 6 ++++--\n t/t1091-sparse-checkout-builtin.sh | 7 +++++--\n 2 files changed, 9 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 6083aa10f2..facdb6bda7 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -78,8 +78,10 @@ static int sparse_checkout_list(int argc, const char **argv)\n \n \t\tstring_list_sort(&sl);\n \n-\t\tfor (i = 0; i < sl.nr; i++)\n-\t\t\tprintf(\"%s\\n\", sl.items[i].string);\n+\t\tfor (i = 0; i < sl.nr; i++) {\n+\t\t\tquote_c_style(sl.items[i].string, NULL, stdout, 0);\n+\t\t\tprintf(\"\\n\");\n+\t\t}\n \n \t\treturn 0;\n \t}\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a46a310740..545e8d5ebe 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -406,7 +406,8 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n \tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n-\tgit -C escaped ls-tree -d --name-only HEAD | git -C escaped sparse-checkout set --stdin &&\n+\tgit -C escaped ls-tree -d --name-only HEAD >list-expect &&\n+\tgit -C escaped sparse-checkout set --stdin expect <<-\\EOF &&\n \t/*\n \t!/*/\n@@ -417,7 +418,9 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/zdoes\\*exist/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\"\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout list >list-actual &&\n+\ttest_cmp list-expect list-actual\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"390956","messageId":"54be8e89eb7ebad67ec18bfb871a13726f7c6769.1580501775.git.gitgitgadget@gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"[PATCH v4 13/15] sparse-checkout: escape all glob characters on write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T20:16:13Z","receivedAt":"2020-01-31T20:16:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout patterns allow special globs according to\nfnmatch(3). When writing cone-mode patterns for paths containing\nthese characters, they must be escaped.\n\nUse is_glob_special() to check which characters must be escaped\nthis way, and add a path to the tests that contains all glob\ncharacters at once. Note that ']' is not special, since the\ninitial bracket '[' is escaped.\n\nReported-by: Jeff King \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 13 ++++++++-----\n 2 files changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex facdb6bda7..7aeb384362 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -149,7 +149,7 @@ static char *escaped_pattern(char *pattern)\n \tstruct strbuf final = STRBUF_INIT;\n \n \twhile (*p) {\n-\t\tif (*p == '*' || *p == '\\\\')\n+\t\tif (is_glob_special(*p))\n \t\t\tstrbuf_addch(&final, '\\\\');\n \n \t\tstrbuf_addch(&final, *p);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 545e8d5ebe..37e9304ef3 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -381,20 +381,21 @@ test_expect_success 'pattern-checks: contained glob characters' '\n \tdone\n '\n \n-test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n+test_expect_success BSLASHPSPEC 'pattern-checks: escaped characters' '\n \tgit clone repo escaped &&\n \tTREEOID=$(git -C escaped rev-parse HEAD:folder1) &&\n \tNEWTREE=$(git -C escaped mktree <<-EOF\n \t$(git -C escaped ls-tree HEAD)\n \t040000 tree $TREEOID\tzbad\\\\dir\n \t040000 tree $TREEOID\tzdoes*exist\n+\t040000 tree $TREEOID\tzglob[!a]?\n \tEOF\n \t) &&\n \tCOMMIT=$(git -C escaped commit-tree $NEWTREE -p HEAD) &&\n \tgit -C escaped reset --hard $COMMIT &&\n-\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" zglob[!a]? &&\n \tgit -C escaped sparse-checkout init --cone &&\n-\tgit -C escaped sparse-checkout set zbad\\\\dir/bogus \"zdoes*not*exist\" \"zdoes*exist\" &&\n+\tgit -C escaped sparse-checkout set zbad\\\\dir/bogus \"zdoes*not*exist\" \"zdoes*exist\" \"zglob[!a]?\" &&\n \tcat >expect <<-\\EOF &&\n \t/*\n \t!/*/\n@@ -403,9 +404,10 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/zbad\\\\dir/bogus/\n \t/zdoes\\*exist/\n \t/zdoes\\*not\\*exist/\n+\t/zglob\\[!a]\\?/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist\" &&\n+\tcheck_read_tree_errors escaped \"a zbad\\\\dir zdoes*exist zglob[!a]?\" &&\n \tgit -C escaped ls-tree -d --name-only HEAD >list-expect &&\n \tgit -C escaped sparse-checkout set --stdin expect <<-\\EOF &&\n@@ -416,9 +418,10 @@ test_expect_success BSLASHPSPEC 'pattern-checks: escaped \"*\"' '\n \t/folder2/\n \t/zbad\\\\dir/\n \t/zdoes\\*exist/\n+\t/zglob\\[!a]\\?/\n \tEOF\n \ttest_cmp expect escaped/.git/info/sparse-checkout &&\n-\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" &&\n+\tcheck_files escaped \"a deep folder1 folder2 zbad\\\\dir zdoes*exist\" zglob[!a]? &&\n \tgit -C escaped sparse-checkout list >list-actual &&\n \ttest_cmp list-expect list-actual\n '\n-- \ngitgitgadget\n\n"},{"id":"390958","messageId":"CABPp-BHoEgYXgzxjweWDR2BZPhLdW9wcbWzwo6N+HF2kste3WA@mail.gmail.com","threadId":"52638","inReplyTo":"pull.513.v4.git.1580501775.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/15] Harden the sparse-checkout builtin","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-31T20:36:18Z","receivedAt":"2020-01-31T20:36:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jan 31, 2020 at 12:16 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This series is based on ds/sparse-list-in-cone-mode.\n>\n> This series attempts to clean up some rough edges in the sparse-checkout\n> feature, especially around the cone mode.\n>\n> Unfortunately, after the v2.25.0 release, we noticed an issue with the \"git\n> clone --sparse\" option when using a URL instead of a local path. This is\n> fixed and properly tested here.\n>\n> Also, let's improve Git's response to these more complicated scenarios:\n>\n> 1. Running \"git sparse-checkout init\" in a worktree would complain because\n> the \"info\" dir doesn't exist.\n> 2. Tracked paths that include \"*\" and \"\\\" in their filenames.\n> 3. If a user edits the sparse-checkout file to have non-cone pattern, such\n> as \"**\" anywhere or \"*\" in the wrong place, then we should respond\n> appropriately. That is: warn that the patterns are not cone-mode, then\n> revert to the old logic.\n>\n> Updates in V2:\n>\n> * Added C-style quoting to the output of \"git sparse-checkout list\" in cone\n> mode.\n> * Improved documentation.\n> * Responded to most style feedback. Hopefully I didn't miss anything.\n> * I was lingering on this a little to see if I could also fix the issue\n> raised in [1], but I have not figured that one out, yet.\n>\n> Update in V3:\n>\n> * Input now uses Peff's recommended pattern: unquote C-style strings over\n> stdin and otherwise do not un-escape input.\n\n...and updates in V4 are? (I looked over the range-diff which\ndefinitely helps, but a summary would still be nice.)\n"},{"id":"391041","messageId":"f28beb17-50dc-ff53-46bf-35010d48d23d@gmail.com","threadId":"52638","inReplyTo":"CABPp-BHoEgYXgzxjweWDR2BZPhLdW9wcbWzwo6N+HF2kste3WA@mail.gmail.com","subject":"Re: [PATCH v4 00/15] Harden the sparse-checkout builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-02-03T14:09:54Z","receivedAt":"2020-02-03T14:10:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/31/2020 3:36 PM, Elijah Newren wrote:\n> On Fri, Jan 31, 2020 at 12:16 PM Derrick Stolee via GitGitGadget\n>> Update in V3:\n>>\n>> * Input now uses Peff's recommended pattern: unquote C-style strings over\n>> stdin and otherwise do not un-escape input.\n> \n> ...and updates in V4 are? (I looked over the range-diff which\n> definitely helps, but a summary would still be nice.)\n\nSorry! I definitely should have double-checked before sending.\n\nUpdates in V4:\n\n* Special-character checking now considers all glob characters\n ( '[', '*', '\\\\', '?' ) See Patches 8 and 13.\n\n* Patch 10 is is now split into two (Patches 10 and 11), to properly\n escape patterns and to unquote C-style strings. \n\n* The file/directory path bug reported in [1] is fixed in Patch 15.\n\nThanks,\n-Stolee\n\n[1] https://lore.kernel.org/git/CADSBhNbbO=aq-Oo2MpzDMN2VAX4m6f9Jb-eCtVVX1NfWKE9zJw@mail.gmail.com/\n"},{"id":"391383","messageId":"20200208233247.GD33529@syl.local","threadId":"52638","inReplyTo":"f28beb17-50dc-ff53-46bf-35010d48d23d@gmail.com","subject":"Re: [PATCH v4 00/15] Harden the sparse-checkout builtin","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-08T23:32:47Z","receivedAt":"2020-02-08T23:33:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Derrick,\n\nOn Mon, Feb 03, 2020 at 09:09:54AM -0500, Derrick Stolee wrote:\n> On 1/31/2020 3:36 PM, Elijah Newren wrote:\n> > On Fri, Jan 31, 2020 at 12:16 PM Derrick Stolee via GitGitGadget\n> >> Update in V3:\n> >>\n> >> * Input now uses Peff's recommended pattern: unquote C-style strings over\n> >> stdin and otherwise do not un-escape input.\n> >\n> > ...and updates in V4 are? (I looked over the range-diff which\n> > definitely helps, but a summary would still be nice.)\n>\n> Sorry! I definitely should have double-checked before sending.\n>\n> Updates in V4:\n>\n> * Special-character checking now considers all glob characters\n> ( '[', '*', '\\\\', '?' ) See Patches 8 and 13.\n>\n> * Patch 10 is is now split into two (Patches 10 and 11), to properly\n> escape patterns and to unquote C-style strings.\n>\n> * The file/directory path bug reported in [1] is fixed in Patch 15.\n\nThanks for including these. I haven't been super active in the earlier\nrounds of review on this series, but I gave a thorough look to what you\nhave in v4, and it all looks good to me.\n\nPlease consider this:\n\n Reviewed-by: Taylor Blau \n\n> Thanks,\n> -Stolee\n>\n> [1] https://lore.kernel.org/git/CADSBhNbbO=aq-Oo2MpzDMN2VAX4m6f9Jb-eCtVVX1NfWKE9zJw@mail.gmail.com/\n\nThanks,\nTaylor\n"},{"id":"391408","messageId":"xmqq1rr33c3w.fsf@gitster-ct.c.googlers.com","threadId":"52638","inReplyTo":"20200208233247.GD33529@syl.local","subject":"Re: [PATCH v4 00/15] Harden the sparse-checkout builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-09T17:27:15Z","receivedAt":"2020-02-09T17:27:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Mon, Feb 03, 2020 at 09:09:54AM -0500, Derrick Stolee wrote:\n> ...\n> Thanks for including these. I haven't been super active in the earlier\n> rounds of review on this series, but I gave a thorough look to what you\n> have in v4, and it all looks good to me.\n>\n> Please consider this:\n>\n> Reviewed-by: Taylor Blau \n\nThanks, all.\n"}]}