{"thread":{"id":"51691","subject":"[PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","startedAt":"2019-08-20T15:11:10Z","lastAt":"2019-11-21T22:05:26Z","messageCount":196,"participants":["Derrick Stolee via GitGitGadget","Jeff Hostetler via GitGitGadget","Elijah Newren","Derrick Stolee","Eric Sunshine","Junio C Hamano","Jon Simons","SZEDER Gábor","Phillip Wood"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"380804","messageId":"pull.316.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":null,"subject":"[PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:06Z","receivedAt":"2019-08-20T15:11:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This RFC includes a potential direction to make the sparse-checkout more\nuser-friendly. While there, I also present a way to use a limited set of\npatterns to gain a significant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis RFC attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout add\n/myFolder/*\n^D\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout add\n/myFolder/*\n^D\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout add reads patterns from stdin, one per line, then\n adds them to the sparse-checkout file and refreshes the working\n directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/*\n/A/*\n!/A/*/*\n/A/B/*\n!/A/B/*/*\n/A/B/C/*\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (8):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'add' subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and add in cone mode\n\nJeff Hostetler (1):\n trace2:experiment: clear_ce_flags_1\n\n .gitignore | 1 +\n Documentation/config/core.txt | 7 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 146 ++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/sparse-checkout.c | 389 ++++++++++++++++++++++++++\n cache.h | 8 +-\n config.c | 10 +-\n dir.c | 154 +++++++++-\n dir.h | 27 ++\n environment.c | 2 +-\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 195 +++++++++++++\n unpack-trees.c | 12 +-\n 17 files changed, 976 insertions(+), 15 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 5fa0f5238b0cd46cfe7f6fa76c3f526ea98148d9\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n-- \ngitgitgadget\n"},{"id":"380805","messageId":"c37b5f2c29d42e7c0c727596625d4eb50b6ddb4f.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 1/9] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:06Z","receivedAt":"2019-08-20T15:11:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nFor now, create the basics of the builtin. Includes a single\nsubcommand, \"git sparse-checkout list\", that lists the patterns\ncurrently in the sparse-checkout file. Test that these patterns\nare parsed and written correctly to the output.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na moer restricted pattern set.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 90 +++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 90 +++++++++++++++++++++++++++\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 50 +++++++++++++++\n 8 files changed, 235 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 521d8f4fb4..9411522c4f 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -158,6 +158,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..ca0ca6a12f\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,90 @@\n+git-sparse-checkout(1)\n+=======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of directories\n+given by a list of prefixes.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of directories given by a list of prefixes.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+----------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it resets the skip-worktree bit in the index based on this\n+file. If an entry\n+matches a pattern in this file, skip-worktree will not be set on\n+that entry. Otherwise, skip-worktree will be set.\n+\n+Then it compares the new skip-worktree value with the previous one. If\n+skip-worktree turns from set to unset, it will add the corresponding\n+file back. If it turns from unset to set, that file will be removed.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are in, you can also specify what files are _not_ in, using\n+negate patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+read-tree' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f9255344ae..d1d067ab65 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1126,6 +1126,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex 5cf5df69f7..2b25a80cde 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..6477a6ed9c\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,90 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout [list]\"),\n+\tNULL\n+};\n+\n+struct opts_sparse_checkout {\n+\tconst char *subcommand;\n+\tint read_stdin;\n+} opts;\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_excludes_to_file(FILE *fp, struct exclude_list *el)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < el->nr; i++) {\n+\t\tstruct exclude *x = el->excludes[i];\n+\n+\t\tif (x->flags & EXC_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", x->pattern);\n+\n+\t\tif (x->flags & EXC_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct exclude_list el;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&el, 0, sizeof(el));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"failed to parse sparse-checkout file; it may not exist\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_excludes_to_file(stdout, &el);\n+\tclear_exclude_list(&el);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/git.c b/git.c\nindex c1ee7124ed..dfb680e15d 100644\n--- a/git.c\n+++ b/git.c\n@@ -573,6 +573,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..ba6928c641\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,50 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_line_count = 0 list &&\n+\ttest_i18ngrep \"failed to parse sparse-checkout file; it may not exist\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tgit -C repo sparse-checkout list >list &&\n+\tcat >expect <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\ttest_cmp expect list\n+'\n+\n+test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"380806","messageId":"0f095e85d5bf29346bdc5bf1707bb51eaf2202ae.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 4/9] sparse-checkout: 'add' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:09Z","receivedAt":"2019-08-20T15:11:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout add' subcommand takes a list of patterns\nover stdin and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nNote: if a user adds a negative pattern that would lead to the\nremoval of a non-empty directory, then Git may not delete that\ndirectory (on Windows).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 4 ++++\n builtin/sparse-checkout.c | 32 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n 3 files changed, 55 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 50c53ee60a..6f540a3443 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -34,6 +34,10 @@ COMMANDS\n \tby Git. Add patterns to the sparse-checkout file to\n \trepopulate the working directory.\n \n+'add'::\n+\tAdd a set of patterns to the sparse-checkout file, as given over\n+\tstdin. Updates the working directory to match the new patterns.\n+\n SPARSE CHECKOUT\n ----------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 86d24e6295..ec6134fecc 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list]\"),\n+\tN_(\"git sparse-checkout [init|add|list]\"),\n \tNULL\n };\n \n@@ -166,6 +166,34 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn sc_read_tree();\n }\n \n+static int sparse_checkout_add(int argc, const char **argv)\n+{\n+\tstruct exclude_list el;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tstruct strbuf line = STRBUF_INIT;\n+\n+\tmemset(&el, 0, sizeof(el));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tadd_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n+\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_excludes_to_file(fp, &el);\n+\n+\twhile (!strbuf_getline(&line, stdin)) {\n+\t\tstrbuf_trim(&line);\n+\t\tfprintf(fp, \"%s\\n\", line.buf);\n+\t}\n+\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\tclear_exclude_list(&el);\n+\n+\treturn sc_read_tree();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -187,6 +215,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"add\"))\n+\t\t\treturn sparse_checkout_add(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b7d5f15830..499bd8d6d0 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -100,4 +100,24 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'add to existing sparse-checkout' '\n+\techo \"/folder2/*\" | git -C repo sparse-checkout add &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\t\t/folder1/*\n+\t\t/folder2/*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"380807","messageId":"e6e982e5a6e4517d97a7a404384057110f3a151d.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 2/9] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:07Z","receivedAt":"2019-08-20T15:11:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nUsing 'git read-tree' to clear directories does not work cleanly\non Windows, so manually delete directories that are tracked by Git\nbefore running read-tree.\n\nThe use of running another process for 'git read-tree' is likely\nsuboptimal, but that can be improved in a later change, if valuable.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 7 ++\n builtin/sparse-checkout.c | 106 +++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 40 ++++++++++\n 3 files changed, 152 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex ca0ca6a12f..50c53ee60a 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -26,6 +26,13 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n \n SPARSE CHECKOUT\n ----------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 6477a6ed9c..86d24e6295 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [list]\"),\n+\tN_(\"git sparse-checkout [init|list]\"),\n \tNULL\n };\n \n@@ -64,6 +64,108 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int sc_read_tree(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+static int sc_enable_config(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+static int delete_directory(const struct object_id *oid, struct strbuf *base,\n+\t\tconst char *pathname, unsigned mode, int stage, void *context)\n+{\n+\tstruct strbuf dirname = STRBUF_INIT;\n+\tstruct stat sb;\n+\n+\tstrbuf_addstr(&dirname, the_repository->worktree);\n+\tstrbuf_addch(&dirname, '/');\n+\tstrbuf_addstr(&dirname, pathname);\n+\n+\tif (stat(dirname.buf, &sb) || !(sb.st_mode & S_IFDIR))\n+\t\treturn 0;\n+\n+\tif (remove_dir_recursively(&dirname, 0))\n+\t\twarning(_(\"failed to remove directory '%s'\"),\n+\t\t\tdirname.buf);\n+\n+\tstrbuf_release(&dirname);\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct tree *t;\n+\tstruct object_id oid;\n+\tstruct exclude_list el;\n+\tstatic struct pathspec pathspec;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (sc_enable_config())\n+\t\treturn 1;\n+\n+\tmemset(&el, 0, sizeof(el));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/*\\n\");\n+\tfclose(fp);\n+\n+\t/* remove all directories in the root, if tracked by Git */\n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n+\tt = parse_tree_indirect(&oid);\n+\n+\tparse_pathspec(&pathspec, PATHSPEC_ALL_MAGIC &\n+\t\t\t\t ~(PATHSPEC_FROMTOP | PATHSPEC_LITERAL),\n+\t\t PATHSPEC_PREFER_CWD,\n+\t\t \"\", NULL);\n+\n+\tif (read_tree_recursive(the_repository, t, \"\", 0, 0, &pathspec,\n+\t\t\t\tdelete_directory, NULL))\n+\t\treturn 1;\n+\n+reset_dir:\n+\treturn sc_read_tree();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -83,6 +185,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ba6928c641..35ab84aabd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -47,4 +47,44 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"/folder1/*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\t\t/folder1/*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"380809","messageId":"4ccd36b3966b62934cfbb0ec65e2851b50dfa1e3.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 3/9] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:08Z","receivedAt":"2019-08-20T15:11:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/*\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 3 files changed, 47 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 34011c2940..0fe91d2f04 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex f665b28ccc..d6d49a73ff 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -60,6 +60,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -147,6 +148,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -734,6 +737,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1107,6 +1131,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 35ab84aabd..b7d5f15830 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -87,4 +87,17 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"380808","messageId":"5f332b799f90b64e2c0720fd0aba0e163d41587a.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 5/9] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:10Z","receivedAt":"2019-08-20T15:11:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 26 +++++++---------\n builtin/sparse-checkout.c | 45 ++++++++++++++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++\n 3 files changed, 67 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 6f540a3443..de04b768ae 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -38,6 +38,10 @@ COMMANDS\n \tAdd a set of patterns to the sparse-checkout file, as given over\n \tstdin. Updates the working directory to match the new patterns.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ----------------\n \n@@ -60,6 +64,13 @@ Then it compares the new skip-worktree value with the previous one. If\n skip-worktree turns from set to unset, it will add the corresponding\n file back. If it turns from unset to set, that file will be removed.\n \n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n+Sparse checkout support in 'git read-tree' and similar commands is\n+disabled by default. You need to set `core.sparseCheckout` to `true`\n+in order to have sparse checkout support.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -74,21 +85,6 @@ negate patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-read-tree' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex ec6134fecc..8f97c27ec7 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|add|list]\"),\n+\tN_(\"git sparse-checkout [init|add|list|disable]\"),\n \tNULL\n };\n \n@@ -79,11 +79,24 @@ static int sc_read_tree(void)\n \treturn result;\n }\n \n-static int sc_enable_config(void)\n+static int sc_set_config(int mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n-\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n+\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n+\n+\tswitch (mode) {\n+\tcase 1:\n+\t\targv_array_pushl(&argv, \"true\", NULL);\n+\t\tbreak;\n+\n+\tcase 0:\n+\t\targv_array_pushl(&argv, \"false\", NULL);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tdie(_(\"invalid config mode\"));\n+\t}\n \n \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n \t\terror(_(\"failed to enable core.sparseCheckout\"));\n@@ -125,7 +138,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \n-\tif (sc_enable_config())\n+\tif (sc_set_config(1))\n \t\treturn 1;\n \n \tmemset(&el, 0, sizeof(el));\n@@ -194,6 +207,28 @@ static int sparse_checkout_add(int argc, const char **argv)\n \treturn sc_read_tree();\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (sc_set_config(1))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (sc_read_tree())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn sc_set_config(0);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -217,6 +252,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"add\"))\n \t\t\treturn sparse_checkout_add(argc, argv);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 499bd8d6d0..68ca63a6f6 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -120,4 +120,19 @@ test_expect_success 'add to existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"380810","messageId":"86f12dc77dd94c1b2c6294842d29d8712f79e3d7.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 6/9] trace2:experiment: clear_ce_flags_1","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:10Z","receivedAt":"2019-08-20T15:11:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThe clear_ce_flags_1 method is used by many types of calls to\nunpack_trees(). Add trace2 regions around the method, including\nsome flag information, so we can get granular performance data\nduring experiments.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 62276d4fef..8c3b5e8849 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1397,15 +1397,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct exclude_list *el)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"exp\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tel, 0);\n+\ttrace2_region_leave(\"exp\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"380811","messageId":"568fda2d0395ae82e8b0c43f6e5757d45b50a508.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 9/9] sparse-checkout: init and add in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:13Z","receivedAt":"2019-08-20T15:11:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout [init|add]'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckout=cone'.\n\nWhen running 'git sparse-checkout add' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 134 +++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 35 ++++++++\n 2 files changed, 164 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 77e5235720..0a4e101ddd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,15 +6,22 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|add|list|disable]\"),\n \tNULL\n };\n \n+static const char * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n struct opts_sparse_checkout {\n \tconst char *subcommand;\n \tint read_stdin;\n+\tint cone;\n } opts;\n \n static char *get_sparse_checkout_filename(void)\n@@ -41,6 +48,60 @@ static void write_excludes_to_file(FILE *fp, struct exclude_list *el)\n \t}\n }\n \n+static void write_cone_to_file(FILE *fp, struct exclude_list *el)\n+{\n+\tint i;\n+\tstruct exclude_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_iter_init(&el->parent_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter))) {\n+\t\tchar *pattern = xstrdup(entry->pattern);\n+\t\tchar *converted = pattern;\n+\t\tif (pattern[0] == '/')\n+\t\t\tconverted++;\n+\t\tif (pattern[entry->patternlen - 1] == '/')\n+\t\t\tpattern[entry->patternlen - 1] = 0;\n+\t\tstring_list_insert(&sl, converted);\n+\t\tfree(pattern);\n+\t}\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (!strcmp(pattern, \"\"))\n+\t\t\tfprintf(fp, \"/*\\n!/*/*\\n\");\n+\t\telse\n+\t\t\tfprintf(fp, \"/%s/*\\n!/%s/*/*\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_iter_init(&el->recursive_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter))) {\n+\t\tchar *pattern = xstrdup(entry->pattern);\n+\t\tchar *converted = pattern;\n+\t\tif (pattern[0] == '/')\n+\t\t\tconverted++;\n+\t\tif (pattern[entry->patternlen - 1] == '/')\n+\t\t\tpattern[entry->patternlen - 1] = 0;\n+\t\tstring_list_insert(&sl, converted);\n+\t\tfree(pattern);\n+\t}\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"/%s/*\\n\", pattern);\n+\t}\n+}\n+\n static int sparse_checkout_list(int argc, const char **argv)\n {\n \tstruct exclude_list el;\n@@ -141,8 +202,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tenum sparse_checkout_mode mode;\n \n-\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &opts.cone,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = opts.cone ? SPARSE_CHECKOUT_CONE : SPARSE_CHECKOUT_FULL;\n+\n+\tif (sc_set_config(mode))\n \t\treturn 1;\n \n \tmemset(&el, 0, sizeof(el));\n@@ -183,6 +257,34 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn sc_read_tree();\n }\n \n+static void insert_recursive_pattern(struct exclude_list *el, struct strbuf *path)\n+{\n+\tstruct exclude_entry *e = xmalloc(sizeof(struct exclude_entry));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\thashmap_add(&el->recursive_hashmap, e);\n+\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (!slash)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct exclude_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get(&el->parent_hashmap, e, NULL))\n+\t\t\thashmap_add(&el->parent_hashmap, e);\n+\t}\n+}\n+\n static int sparse_checkout_add(int argc, const char **argv)\n {\n \tstruct exclude_list el;\n@@ -196,11 +298,33 @@ static int sparse_checkout_add(int argc, const char **argv)\n \tadd_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n \n \tfp = fopen(sparse_filename, \"w\");\n-\twrite_excludes_to_file(fp, &el);\n \n-\twhile (!strbuf_getline(&line, stdin)) {\n-\t\tstrbuf_trim(&line);\n-\t\tfprintf(fp, \"%s\\n\", line.buf);\n+\tif (core_sparse_checkout == SPARSE_CHECKOUT_FULL) {\n+\t\twrite_excludes_to_file(fp, &el);\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tstrbuf_trim(&line);\n+\t\t\tfprintf(fp, \"%s\\n\", line.buf);\n+\t\t}\n+\t} else if (core_sparse_checkout == SPARSE_CHECKOUT_CONE) {\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tstrbuf_trim(&line);\n+\n+\t\t\tstrbuf_trim_trailing_dir_sep(&line);\n+\n+\t\t\tif (!line.len)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (line.buf[0] == '/')\n+\t\t\t\tstrbuf_remove(&line, 0, 1);\n+\n+\t\t\tif (!line.len)\n+\t\t\t\tcontinue;\n+\n+\t\t\tinsert_recursive_pattern(&el, &line);\n+\t\t}\n+\n+\t\twrite_cone_to_file(fp, &el);\n \t}\n \n \tfclose(fp);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 60f10864a1..3412bafdff 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -157,4 +157,39 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and add' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=cone\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\techo deep/deeper1/deepest | git -C repo sparse-checkout add &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/*\n+\t\t/deep/*\n+\t\t!/deep/*/*\n+\t\t/deep/deeper1/*\n+\t\t!/deep/deeper1/*/*\n+\t\t/deep/deeper1/deepest/*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout\n+'\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n"},{"id":"380812","messageId":"b99acea4a09a6ed5302e5e622a6b6eb35d2d01b1.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 8/9] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:12Z","receivedAt":"2019-08-20T15:11:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/*\n\t!/$folder/*/*\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n\tcore.sparseCheckout=false: 0.21 s (0.00 s)\n\t core.sparseCheckout=true: 3.75 s (3.50 s)\n\t core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 1 -\n dir.c | 154 +++++++++++++++++++++++++-\n dir.h | 27 +++++\n t/t1091-sparse-checkout-builtin.sh | 8 ++\n 4 files changed, 183 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 463319055b..7ade827370 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -85,7 +85,6 @@ negate patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-\n ## CONE PATTERN SET\n \n The full pattern set allows for arbitrary pattern matches and complicated\ndiff --git a/dir.c b/dir.c\nindex d021c908e5..2c5ff89a72 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -599,6 +599,99 @@ void parse_exclude_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int el_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const void *a, const void *b, const void *key)\n+{\n+\tconst struct exclude_entry *ee1 = a;\n+\tconst struct exclude_entry *ee2 = b;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, ee1->patternlen);\n+}\n+\n+static void add_exclude_to_hashsets(struct exclude_list *el, struct exclude *x)\n+{\n+\tstruct exclude_entry *e;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!el->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (x->patternlen >= 4 &&\n+\t !strcmp(x->pattern + x->patternlen - 4, \"/*/*\")) {\n+\t\tif (!(x->flags & EXC_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tel->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), x->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(x->pattern);\n+\t\ttruncated[x->patternlen - 4] = 0;\n+\n+\t\te = xmalloc(sizeof(struct exclude_entry));\n+\t\te->pattern = truncated;\n+\t\te->patternlen = x->patternlen - 4;\n+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get(&el->recursive_hashmap, e, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"), x->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&el->parent_hashmap, e);\n+\t\thashmap_remove(&el->recursive_hashmap, e, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (x->patternlen >= 2 &&\n+\t !strcmp(x->pattern + x->patternlen - 2, \"/*\")) {\n+\t\tif (x->flags & EXC_FLAG_NEGATIVE) {\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"), x->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\te = xmalloc(sizeof(struct exclude_entry));\n+\n+\t\ttruncated = xstrdup(x->pattern);\n+\t\ttruncated[x->patternlen - 2] = 0;\n+\t\te->pattern = truncated;\n+\t\te->patternlen = x->patternlen - 2;\n+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\t\thashmap_add(&el->recursive_hashmap, e);\n+\n+\t\tif (hashmap_get(&el->parent_hashmap, e, NULL)) {\n+\t\t\t/* we already included this at the parent level */\n+\t\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\t\tx->pattern);\n+\t\t\thashmap_remove(&el->parent_hashmap, e, &data);\n+\t\t\tfree(data);\n+\t\t}\n+\t\treturn;\n+\t}\n+\n+clear_hashmaps:\n+\thashmap_free(&el->parent_hashmap, 1);\n+\thashmap_free(&el->recursive_hashmap, 1);\n+\tel->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct exclude_entry e;\n+\n+\t/* Check straight mapping */\n+\te.pattern = pattern->buf;\n+\te.patternlen = pattern->len;\n+\thashmap_entry_init(&e, memhash(e.pattern, e.patternlen));\n+\treturn !!hashmap_get(map, &e, NULL);\n+}\n+\n void add_exclude(const char *string, const char *base,\n \t\t int baselen, struct exclude_list *el, int srcpos)\n {\n@@ -623,6 +716,8 @@ void add_exclude(const char *string, const char *base,\n \tALLOC_GROW(el->excludes, el->nr + 1, el->alloc);\n \tel->excludes[el->nr++] = x;\n \tx->el = el;\n+\n+\tadd_exclude_to_hashsets(el, x);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -848,6 +943,10 @@ static int add_excludes_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\tel->use_cone_patterns = core_sparse_checkout == SPARSE_CHECKOUT_CONE ? 1 : 0;\n+\thashmap_init(&el->recursive_hashmap, el_hashmap_cmp, NULL, 0);\n+\thashmap_init(&el->parent_hashmap, el_hashmap_cmp, NULL, 0);\n+\n \tel->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1070,18 +1169,61 @@ static struct exclude *last_exclude_matching_from_list(const char *pathname,\n \n /*\n * Scan the list and let the last match determine the fate.\n- * Return 1 for exclude, 0 for include and -1 for undecided.\n+ * Return 0 for exclude, 1 for include and -1 for undecided.\n */\n int is_excluded_from_list(const char *pathname,\n \t\t\t int pathlen, const char *basename, int *dtype,\n \t\t\t struct exclude_list *el, struct index_state *istate)\n {\n \tstruct exclude *exclude;\n-\texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, el, istate);\n-\tif (exclude)\n-\t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n-\treturn -1; /* undecided */\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = 0;\n+\tconst char *slash_pos;\n+\n+\tif (!el->use_cone_patterns) {\n+\t\texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\t\tdtype, el, istate);\n+\n+\t\tif (exclude)\n+\t\t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n+\n+\t\treturn -1; /* undecided */\n+\t}\n+\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = 1;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&el->parent_hashmap, &parent_pathname)) {\n+\t\tresult = 1;\n+\t\tgoto done;\n+\t}\n+\n+\twhile (parent_pathname.len) {\n+\t\tif (hashmap_contains_path(&el->recursive_hashmap,\n+\t\t\t\t\t &parent_pathname)) {\n+\t\t\tresult = -1;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\t\tif (slash_pos == parent_pathname.buf)\n+\t\t\tbreak;\n+\n+\t\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\t}\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct exclude *last_exclude_matching_from_lists(struct dir_struct *dir,\ndiff --git a/dir.h b/dir.h\nindex 680079bbe3..2d3356d1c0 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct exclude {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct exclude_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,25 @@ struct exclude_list {\n \tconst char *src;\n \n \tstruct exclude **excludes;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 8cc377b839..60f10864a1 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -134,6 +134,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"380813","messageId":"19d664a5dada87a9a8dcf18d7548582275593f10.1566313865.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH 7/9] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-08-20T15:11:11Z","receivedAt":"2019-08-20T15:11:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new 'cone' mode for the core.sparseCheckout config\noption, and adjust the parser to set an appropriate enum value.\n\nWhile adjusting the type of this variable, rename it from\ncore_apply_sparse_checkout to core_sparse_checkout. This will\nhelp avoid parallel changes from hitting type issues, and we\ncan guarantee that all uses now consider the enum values instead\nof the int value.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 7 ++--\n Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n builtin/clone.c | 2 +-\n builtin/sparse-checkout.c | 16 +++++----\n cache.h | 8 ++++-\n config.c | 10 +++++-\n environment.c | 2 +-\n t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n unpack-trees.c | 2 +-\n 9 files changed, 98 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 75538d27e7..9b8ab2a6d4 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -591,8 +591,11 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n+\tis disabled. If \"true\", then sparse-checkout is enabled with the full\n+\t.gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n+\ta restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n+\tinformation.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex de04b768ae..463319055b 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -86,6 +86,56 @@ negate patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+## CONE PATTERN SET\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckout` is set to `cone`.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+```\n+/*\n+!/*/*\n+```\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+```\n+/*\n+!/*/*\n+/A/*\n+!/A/*/*\n+/A/B/*\n+!/A/B/*/*\n+/A/B/C/*\n+```\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckout=cone`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex d6d49a73ff..763898ada5 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -747,7 +747,7 @@ static int git_sparse_checkout_init(const char *repo)\n \t * We must apply the setting in the current process\n \t * for the later checkout to use the sparse-checkout file.\n \t */\n-\tcore_apply_sparse_checkout = 1;\n+\tcore_sparse_checkout = SPARSE_CHECKOUT_FULL;\n \n \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n \t\terror(_(\"failed to initialize sparse-checkout\"));\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 8f97c27ec7..77e5235720 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -79,18 +79,22 @@ static int sc_read_tree(void)\n \treturn result;\n }\n \n-static int sc_set_config(int mode)\n+static int sc_set_config(enum sparse_checkout_mode mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n \targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n \n \tswitch (mode) {\n-\tcase 1:\n+\tcase SPARSE_CHECKOUT_FULL:\n \t\targv_array_pushl(&argv, \"true\", NULL);\n \t\tbreak;\n \n-\tcase 0:\n+\tcase SPARSE_CHECKOUT_CONE:\n+\t\targv_array_pushl(&argv, \"cone\", NULL);\n+\t\tbreak;\n+\n+\tcase SPARSE_CHECKOUT_NONE:\n \t\targv_array_pushl(&argv, \"false\", NULL);\n \t\tbreak;\n \n@@ -138,7 +142,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \n-\tif (sc_set_config(1))\n+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n \t\treturn 1;\n \n \tmemset(&el, 0, sizeof(el));\n@@ -212,7 +216,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \n-\tif (sc_set_config(1))\n+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n \t\tdie(_(\"failed to change config\"));\n \n \tsparse_filename = get_sparse_checkout_filename();\n@@ -226,7 +230,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tunlink(sparse_filename);\n \tfree(sparse_filename);\n \n-\treturn sc_set_config(0);\n+\treturn sc_set_config(SPARSE_CHECKOUT_NONE);\n }\n \n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\ndiff --git a/cache.h b/cache.h\nindex b1da1ab08f..4426816ca1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -865,12 +865,18 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+enum sparse_checkout_mode {\n+\tSPARSE_CHECKOUT_NONE = 0,\n+\tSPARSE_CHECKOUT_FULL = 1,\n+\tSPARSE_CHECKOUT_CONE = 2,\n+};\n+enum sparse_checkout_mode core_sparse_checkout;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex 3900e4947b..15b7a20dd9 100644\n--- a/config.c\n+++ b/config.c\n@@ -1360,7 +1360,15 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t}\n \n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n-\t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n+\t\tint result = git_parse_maybe_bool(value);\n+\n+\t\tif (result < 0) {\n+\t\t\tcore_sparse_checkout = SPARSE_CHECKOUT_NONE;\n+\n+\t\t\tif (!strcasecmp(value, \"cone\"))\n+\t\t\t\tcore_sparse_checkout = SPARSE_CHECKOUT_CONE;\n+\t\t} else\n+\t\t\tcore_sparse_checkout = result;\n \t\treturn 0;\n \t}\n \ndiff --git a/environment.c b/environment.c\nindex 89af47cb85..cc12e30bd6 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -68,7 +68,7 @@ enum push_default_type push_default = PUSH_DEFAULT_UNSPECIFIED;\n enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n-int core_apply_sparse_checkout;\n+enum sparse_checkout_mode core_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 68ca63a6f6..8cc377b839 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -120,6 +120,20 @@ test_expect_success 'add to existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --replace-all core.sparseCheckout cone &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8c3b5e8849..289c62305f 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1468,7 +1468,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n \ttrace_performance_enter();\n \tmemset(&el, 0, sizeof(el));\n-\tif (!core_apply_sparse_checkout || !o->update)\n+\tif (!core_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n-- \ngitgitgadget\n\n"},{"id":"380908","messageId":"CABPp-BFcH5hQqujjmc88L3qGx3QAYZ_chH6PXQXyp13ipfV6hQ@mail.gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-21T21:52:38Z","receivedAt":"2019-08-21T21:52:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This RFC includes a potential direction to make the sparse-checkout more\n> user-friendly. While there, I also present a way to use a limited set of\n> patterns to gain a significant performance boost in very large repositories.\n>\n> Sparse-checkout is only documented as a subsection of the read-tree docs\n> [1], which makes the feature hard to discover. Users have trouble navigating\n> the feature, especially at clone time [2], and have even resorted to\n> creating their own helper tools [3].\n\nOoh, intriguing. Count me as another person who has resorted to\nmaking my own helper tool for others to use (specific to our internal\nrepository, though, as it also figures out inter-module dependencies\nto allow specifying only a few modules of interest while still\nchecking out everything needed to build those; but it'd be nice to\nneed less scripting to handle the git-related bits to actually\nsparsify or densify).\n\n> This RFC attempts to solve these problems using a new builtin. Here is a\n> sample workflow to give a feeling for how it can work:\n>\n> In an existing repo:\n>\n> $ git sparse-checkout init\n> $ ls\n> myFile1.txt myFile2.txt\n>\n> $ git sparse-checkout add\n> /myFolder/*\n> ^D\n> $ ls\n> myFile1.txt myFile2.txt myFolder\n> $ ls myFolder\n> a.c a.h\n> $ git sparse-checkout disable\n> $ ls\n> hiddenFolder myFile1.txt myFile2.txt myFolder\n>\n> At clone time:\n>\n> $ git clone --sparse origin repo\n> $ cd repo\n> $ ls\n> myFile1.txt myFile2.txt\n> $ git sparse-checkout add\n> /myFolder/*\n> ^D\n> $ ls\n> myFile1.txt myFile2.txt myFolder\n>\n> Here are some more specific details:\n>\n> * git sparse-checkout init enables core.sparseCheckout and populates the\n> sparse-checkout file with patterns that match only the files at root.\n\nDoes it enable core.sparseCheckout in the current worktree, or for all\nworktrees? Do we require extensions.worktreeConfig to be set to true\nfirst? If we don't require extensions.worktreeConfig to be set to\ntrue, and users add worktrees later, do they encounter negative\nsurprises (immediately or later)?\n\nworktrees in combination with sparseCheckouts were a headache here\nuntil I just forced people to manually first set\nextensions.worktreeConfig to true before using my 'sparsify' script,\nregardless of whether the user was currently using worktrees. That\nfixed the issues, but having to provide a long error message and\nexplanation of why I wanted users to set some special config first was\nslightly annoying.\n\nI wonder if 'git worktree' and maybe even 'git config' should\nthemselves have special handling for core.sparseCheckouts, because it\ncan be a real mess otherwise.\n\n> * git clone learns the --sparse argument to run git sparse-checkout init\n> before the first checkout.\n\nNice.\n\n> * git sparse-checkout add reads patterns from stdin, one per line, then\n> adds them to the sparse-checkout file and refreshes the working\n> directory.\n\nThe default of reading from stdin seems a bit unusual to me, and I\nworry about having to explain that to users. I'd rather the add\ncommand took positional parameters (anything that doesn't start with a\nhyphen) and added those, e.g.\n $ git sparse-checkout add '/myFolder/*' '\nwith the option of the user specifying --stdin.\n\n> * git sparse-checkout disable removes the patterns from the sparse-checkout\n> file, disables core.sparseCheckout, and refills the working directory.\n\nDoes it leave an empty sparse-checkout file around? Also, what if\nusers have several paths defining their sparse pattern, and want to\ntemporarily get a full checkout and then come back -- do they need to\nre-specify all the paths? (Maybe this *is* the route we want to go;\nI'm just trying to mention any possible negative effects we _might_\nrun into so we can consider them. It's not quite as relevant in my\ncase since people specify a few toplevel modules and sparse-checkout\ngets several entries auto-generated for them.)\n\nAlso, I'm particularly worried that a user with multiple worktrees,\nboth sparse, could run 'git sparse-checkout disable' in one and then\nfind that when they return to the other worktree they get a variety of\nnasty surprises (e.g. accidental staging or committing of the deletion\nof a huge number of files, random weird errors, or gradual and weird\nde-sparsifying as various git commands are run). This, of course, can\nbe averted by making sure core.sparseCheckout is set on a per-worktree\nbasis, but that seems to be something people only do after running\ninto problems several times unless some kind of tooling enforces it.\n\n> * git sparse-checkout list lists the contents of the sparse-checkout file.\n>\n>\n>\n> The documentation for the sparse-checkout feature can now live primarily\n> with the git-sparse-checkout documentation.\n\nYaay!\n\n> Cone Mode\n> =========\n>\n> What really got me interested in this area is a performance problem. If we\n> have N patterns in the sparse-checkout file and M entries in the index, then\n> we can perform up to O(N * M) pattern checks in clear_ce_flags(). This\n> quadratic growth is not sustainable in a repo with 1,000+ patterns and\n> 1,000,000+ index entries.\n\nThis has worried me for a while, even if it hasn't yet caused us\nissues in practice.\n\n> To solve this problem, I propose a new, more restrictive mode to\n> sparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\n> matches at a directory level. This can then use hashsets for fast\n> performance -- O(M) instead of O(N*M). My hashset implementation is based on\n> the virtual filesystem hook in the VFS for Git custom code [4].\n\nSweet!\n\n> In cone mode, a user specifies a list of folders which the user wants every\n> file inside. In addition, the cone adds all blobs that are siblings of the\n> folders in the directory path to that folder. This makes the directories\n> look \"hydrated\" as a user drills down to those recursively-closed folders.\n> These directories are called \"parent\" folders, as a file matches them only\n> if the file's immediate parent is that directory.\n>\n> When building a prototype of this feature, I used a separate file to contain\n> the list of recursively-closed folders and built the hashsets dynamically\n> based on that file. In this implementation, I tried to maximize the amount\n> of backwards-compatibility by storing all data in the sparse-checkout file\n> using patterns recognized by earlier Git versions.\n>\n> For example, if we add A/B/C as a recursive folder, then we add the\n> following patterns to the sparse-checkout file:\n>\n> /*\n> !/*/*\n> /A/*\n> !/A/*/*\n> /A/B/*\n> !/A/B/*/*\n> /A/B/C/*\n>\n> The alternating positive/negative patterns say \"include everything in this\n> folder, but exclude everything another level deeper\". The final pattern has\n> no matching negation, so is a recursively closed pattern.\n\nOh, um, would there be any option for fast but without grabbing\nsibling and parent files of requested directories? And could users\nstill request individual files (not with regex or pathspec, but fully\nspecifying the path) and still get the fast mode?\n\nBasically, our sparse usage is exclusively specifying leading\ndirectories or full pathnames of individual files, but we really want\nthe repo to feel smaller and make sure people notice at a glance. We\nhave a huge 'modules/' directory, and want people to be able to get\njust 15 of the 500 or so subdirectories that would appear in that\ndirectory with a non-sparse checkout. And similarly we want to be\nable to grab just one or two files from a directory of many files.\n\n> Note that I have some basic warnings to try and check that the\n> sparse-checkout file doesn't match what would be written by a cone-mode add.\n> In such a case, Git writes a warning to stderr and continues with the old\n> pattern matching algorithm. These checks are currently very barebones, and\n> would need to be updated with more robust checks for things like regex\n> characters in the middle of the pattern. As review moves forward (and if we\n> don't change the data storage) then we could spend more time on this.\n\nInstead of trying to validate the sparse-checkout file everytime,\nperhaps we want to change core.sparseCheckout from a boolean to a\ntri-state or something where it specifies how to parse the\nsparse-checkout file? Or maybe when special directive (some form of\ncomment-looking line) appears at the top of sparse-checkout then we\nuse the hashsets speedup while disallowing general regexes and\npathspecs other than leading directories and full pathnames?\n\nI'm not sure if that makes sense or not; just throwing it out there as an idea.\n\n> Derrick Stolee (8):\n> sparse-checkout: create builtin with 'list' subcommand\n> sparse-checkout: create 'init' subcommand\n> clone: add --sparse mode\n> sparse-checkout: 'add' subcommand\n> sparse-checkout: create 'disable' subcommand\n> sparse-checkout: add 'cone' mode\n> sparse-checkout: use hashmaps for cone patterns\n> sparse-checkout: init and add in cone mode\n>\n> Jeff Hostetler (1):\n> trace2:experiment: clear_ce_flags_1\n\nI'll try to get some time to look over these patches in the next few days.\n"},{"id":"380927","messageId":"08bd295b-ec5a-3128-42cf-acb24971d679@gmail.com","threadId":"51691","inReplyTo":"CABPp-BFcH5hQqujjmc88L3qGx3QAYZ_chH6PXQXyp13ipfV6hQ@mail.gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-22T13:10:48Z","receivedAt":"2019-08-22T13:10:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/21/2019 5:52 PM, Elijah Newren wrote:\n> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> This RFC includes a potential direction to make the sparse-checkout more\n>> user-friendly. While there, I also present a way to use a limited set of\n>> patterns to gain a significant performance boost in very large repositories.\n>>\n>> Sparse-checkout is only documented as a subsection of the read-tree docs\n>> [1], which makes the feature hard to discover. Users have trouble navigating\n>> the feature, especially at clone time [2], and have even resorted to\n>> creating their own helper tools [3].\n> \n> Ooh, intriguing. Count me as another person who has resorted to\n> making my own helper tool for others to use (specific to our internal\n> repository, though, as it also figures out inter-module dependencies\n> to allow specifying only a few modules of interest while still\n> checking out everything needed to build those; but it'd be nice to\n> need less scripting to handle the git-related bits to actually\n> sparsify or densify).\n> \n>> This RFC attempts to solve these problems using a new builtin. Here is a\n>> sample workflow to give a feeling for how it can work:\n>>\n>> In an existing repo:\n>>\n>> $ git sparse-checkout init\n>> $ ls\n>> myFile1.txt myFile2.txt\n>>\n>> $ git sparse-checkout add\n>> /myFolder/*\n>> ^D\n>> $ ls\n>> myFile1.txt myFile2.txt myFolder\n>> $ ls myFolder\n>> a.c a.h\n>> $ git sparse-checkout disable\n>> $ ls\n>> hiddenFolder myFile1.txt myFile2.txt myFolder\n>>\n>> At clone time:\n>>\n>> $ git clone --sparse origin repo\n>> $ cd repo\n>> $ ls\n>> myFile1.txt myFile2.txt\n>> $ git sparse-checkout add\n>> /myFolder/*\n>> ^D\n>> $ ls\n>> myFile1.txt myFile2.txt myFolder\n>>\n>> Here are some more specific details:\n>>\n>> * git sparse-checkout init enables core.sparseCheckout and populates the\n>> sparse-checkout file with patterns that match only the files at root.\n> \n> Does it enable core.sparseCheckout in the current worktree, or for all\n> worktrees? Do we require extensions.worktreeConfig to be set to true\n> first? If we don't require extensions.worktreeConfig to be set to\n> true, and users add worktrees later, do they encounter negative\n> surprises (immediately or later)?\n\nThis is an interesting scenario that I had not considered. Thanks!\n\nMy guess is that we should set `extensions.worktreeConfig=true` to\navoid surprises. I'll need to play with this to discover the answers\nto these questions:\n\n1. Where does the worktree look for the sparse-checkout file? Does\n each worktree have its own sparse-checkout file? Should it?\n\n2. If I have `extensions.worktreeConfig=true` and `core.sparseCheckout=true`\n in the current worktree and run `git worktree add`, does the new worktree\n have `core.sparseCheckout=true`? Can we `git clone --sparse` and then\n start building sparse worktrees seamlessly?\n \n> worktrees in combination with sparseCheckouts were a headache here\n> until I just forced people to manually first set\n> extensions.worktreeConfig to true before using my 'sparsify' script,\n> regardless of whether the user was currently using worktrees. That\n> fixed the issues, but having to provide a long error message and\n> explanation of why I wanted users to set some special config first was\n> slightly annoying.\n> \n> I wonder if 'git worktree' and maybe even 'git config' should\n> themselves have special handling for core.sparseCheckouts, because it\n> can be a real mess otherwise.\n> \n>> * git clone learns the --sparse argument to run git sparse-checkout init\n>> before the first checkout.\n> \n> Nice.\n> \n>> * git sparse-checkout add reads patterns from stdin, one per line, then\n>> adds them to the sparse-checkout file and refreshes the working\n>> directory.\n> \n> The default of reading from stdin seems a bit unusual to me, and I\n> worry about having to explain that to users. I'd rather the add\n> command took positional parameters (anything that doesn't start with a\n> hyphen) and added those, e.g.\n> $ git sparse-checkout add '/myFolder/*' '\n> with the option of the user specifying --stdin.\n\nI had the same thought, and likely that's where we should go with the\nbuiltin. For our needs, the input over stdin is more important for\ntesting, so I built it first. I will adjust the CLI here to take a set\nof paths over the arguments unless --stdin is given.\n\n>> * git sparse-checkout disable removes the patterns from the sparse-checkout\n>> file, disables core.sparseCheckout, and refills the working directory.\n> \n> Does it leave an empty sparse-checkout file around? Also, what if\n> users have several paths defining their sparse pattern, and want to\n> temporarily get a full checkout and then come back -- do they need to\n> re-specify all the paths? (Maybe this *is* the route we want to go;\n> I'm just trying to mention any possible negative effects we _might_\n> run into so we can consider them. It's not quite as relevant in my\n> case since people specify a few toplevel modules and sparse-checkout\n> gets several entries auto-generated for them.)\n\nIn this case, there is an intermediate step (that follows the existing\nadvice) to modify the sparse-checkout file to contain only \"/*\\n\" then\nrun read-tree to fill the working directory before disabling the config\nsetting.\n\nPerhaps \"disable\" is the wrong word to use, as it makes you think that\nthere should be an \"enable\" that can quickly toggle between the two\nmodes. Maybe instead it should be \"git sparse-checkout reset [empty|full]\"\nwhere you could 'reset' the sparse-checkout to one of two initial\nstates:\n\n1. empty: only files at root are included.\n2. full: all files are included.\n\nIn each case, we would obliterate the existing sparse-checkout entries,\nbut hopefully that behavior is more clear from the command names.\n\n> Also, I'm particularly worried that a user with multiple worktrees,\n> both sparse, could run 'git sparse-checkout disable' in one and then\n> find that when they return to the other worktree they get a variety of\n> nasty surprises (e.g. accidental staging or committing of the deletion\n> of a huge number of files, random weird errors, or gradual and weird\n> de-sparsifying as various git commands are run). This, of course, can\n> be averted by making sure core.sparseCheckout is set on a per-worktree\n> basis, but that seems to be something people only do after running\n> into problems several times unless some kind of tooling enforces it.\n> \n>> * git sparse-checkout list lists the contents of the sparse-checkout file.\n>>\n>>\n>>\n>> The documentation for the sparse-checkout feature can now live primarily\n>> with the git-sparse-checkout documentation.\n> \n> Yaay!\n> \n>> Cone Mode\n>> =========\n>>\n>> What really got me interested in this area is a performance problem. If we\n>> have N patterns in the sparse-checkout file and M entries in the index, then\n>> we can perform up to O(N * M) pattern checks in clear_ce_flags(). This\n>> quadratic growth is not sustainable in a repo with 1,000+ patterns and\n>> 1,000,000+ index entries.\n> \n> This has worried me for a while, even if it hasn't yet caused us\n> issues in practice.\n> \n>> To solve this problem, I propose a new, more restrictive mode to\n>> sparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\n>> matches at a directory level. This can then use hashsets for fast\n>> performance -- O(M) instead of O(N*M). My hashset implementation is based on\n>> the virtual filesystem hook in the VFS for Git custom code [4].\n> \n> Sweet!\n> \n>> In cone mode, a user specifies a list of folders which the user wants every\n>> file inside. In addition, the cone adds all blobs that are siblings of the\n>> folders in the directory path to that folder. This makes the directories\n>> look \"hydrated\" as a user drills down to those recursively-closed folders.\n>> These directories are called \"parent\" folders, as a file matches them only\n>> if the file's immediate parent is that directory.\n>>\n>> When building a prototype of this feature, I used a separate file to contain\n>> the list of recursively-closed folders and built the hashsets dynamically\n>> based on that file. In this implementation, I tried to maximize the amount\n>> of backwards-compatibility by storing all data in the sparse-checkout file\n>> using patterns recognized by earlier Git versions.\n>>\n>> For example, if we add A/B/C as a recursive folder, then we add the\n>> following patterns to the sparse-checkout file:\n>>\n>> /*\n>> !/*/*\n>> /A/*\n>> !/A/*/*\n>> /A/B/*\n>> !/A/B/*/*\n>> /A/B/C/*\n>>\n>> The alternating positive/negative patterns say \"include everything in this\n>> folder, but exclude everything another level deeper\". The final pattern has\n>> no matching negation, so is a recursively closed pattern.\n> \n> Oh, um, would there be any option for fast but without grabbing\n> sibling and parent files of requested directories? And could users\n> still request individual files (not with regex or pathspec, but fully\n> specifying the path) and still get the fast mode?\n\nExact files could probably be included and still be fast. It requires an\nextra hash check per entry, but that's a small price to pay I think.\n\nWith the sibling files, this is something I believe to be user-friendly:\nas a user drills down into the folder they included recursively, there may\nbe helpful files along the way, like documentation, project files, etc.\n\nHere is my philosophical position here: a repo can take advantage of the\nsparse-checkout feature if it is properly componetized. Those component\nboundaries are likely at folder boundaries. Any file that exists in a parent\nfolder for two components is likely important to _both_ components. If\na file is large and is not needed by both components, it should be placed\ndeeper in the tree, so it can be avoided.\n\nWith that philosophy in mind, I designed this to help users fall into the\n\"pit of success\" when their repo is in a good shape AND to motivate users\nwith repos in non-optimal shapes to reorganize.\n\nThe thought I had about exact file names is similar: if there is a large\nlist of files in a folder where I only need a subset, then how do I know\nif a new file is added that I need? It will not show up in the directory\nwithout updating the sparse-checkout. A user would discover this need by\nsomething going wrong when they are not interacting with version control:\na build.\n\nThis is particularly important with the root directory. We need things\nlike .gitignore, .gitattributes, README, LICENSE, etc. to be populated\nby default. If there are too many files at root to reasonably work with\nthe repo, then the repo should be reorganized using folders.\n\n> Basically, our sparse usage is exclusively specifying leading\n> directories or full pathnames of individual files, but we really want\n> the repo to feel smaller and make sure people notice at a glance. We\n> have a huge 'modules/' directory, and want people to be able to get\n> just 15 of the 500 or so subdirectories that would appear in that\n> directory with a non-sparse checkout. And similarly we want to be\n> able to grab just one or two files from a directory of many files.\n\nYour modules/ example seems to work with the feature as designed, as\nyou want a set of folders one level deeper. Grabbing one or two files\nfrom a directory is a direction we can go with the feature, but I will\ncontinue to believe that should be a rare occurrence compared to including\na folder recursively.\n\n>> Note that I have some basic warnings to try and check that the\n>> sparse-checkout file doesn't match what would be written by a cone-mode add.\n>> In such a case, Git writes a warning to stderr and continues with the old\n>> pattern matching algorithm. These checks are currently very barebones, and\n>> would need to be updated with more robust checks for things like regex\n>> characters in the middle of the pattern. As review moves forward (and if we\n>> don't change the data storage) then we could spend more time on this.\n> \n> Instead of trying to validate the sparse-checkout file everytime,\n> perhaps we want to change core.sparseCheckout from a boolean to a\n> tri-state or something where it specifies how to parse the\n> sparse-checkout file? Or maybe when special directive (some form of\n> comment-looking line) appears at the top of sparse-checkout then we\n> use the hashsets speedup while disallowing general regexes and\n> pathspecs other than leading directories and full pathnames?\n\nIn this series, I turn `core.sparseCheckout` into a tri-state, and only\ntry to validate the sparse-checkout when `core.sparseCheckout=cone`.\nThis avoids spending time on the validation when someone is content using\nthe existing feature.\n\nThe _intent_ of using the sparse-checkout file and no extra data structure\nwas to let other clients (or an older client) read the sparse-checkout data\nand result in the same working directory. One thing I realized after\nsubmitting is that the tri-state config variable will cause old clients\nto error on parsing the non-boolean value. Instead, in v2 I will introduce\na new boolean config variable \"core.sparseCheckoutCone\" that will do the\nsame thing as the current series when `core.sparseCheckout=cone` and will\nfix this compat scenario.\n\n> I'll try to get some time to look over these patches in the next few days.\n\nI look forward to your feedback! I also have some feedback to respond to\nfrom my team [1], but I'm waiting to make sure the community likes the\noverall idea before jumping into code style and method organization\ndetails.\n\nThanks,\n-Stolee\n\n[1] https://github.com/microsoft/git/pull/180\n\n"},{"id":"380935","messageId":"67dc33a6-3995-bd3e-f1be-db337c04c7b2@gmail.com","threadId":"51691","inReplyTo":"08bd295b-ec5a-3128-42cf-acb24971d679@gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-22T14:25:21Z","receivedAt":"2019-08-22T14:25:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/22/2019 9:10 AM, Derrick Stolee wrote:\n> On 8/21/2019 5:52 PM, Elijah Newren wrote:\n>> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n>>> For example, if we add A/B/C as a recursive folder, then we add the\n>>> following patterns to the sparse-checkout file:\n>>>\n>>> /*\n>>> !/*/*\n>>> /A/*\n>>> !/A/*/*\n>>> /A/B/*\n>>> !/A/B/*/*\n>>> /A/B/C/*\n>>>\n>>> The alternating positive/negative patterns say \"include everything in this\n>>> folder, but exclude everything another level deeper\". The final pattern has\n>>> no matching negation, so is a recursively closed pattern.\n>>\n>> Oh, um, would there be any option for fast but without grabbing\n>> sibling and parent files of requested directories? And could users\n>> still request individual files (not with regex or pathspec, but fully\n>> specifying the path) and still get the fast mode?\n> \n> Exact files could probably be included and still be fast. It requires an\n> extra hash check per entry, but that's a small price to pay I think.\n\nQuick side note on this point about exact file names and the REAL reason\nfor the parent paths that I had forgotten until just now.\n\nThe following comment exists in unpack-trees.c, clear_ce_flags_dir():\n\n\t/*\n\t * TODO: check el, if there are no patterns that may conflict\n\t * with ret (iow, we know in advance the incl/excl\n\t * decision for the entire directory), clear flag here without\n\t * calling clear_ce_flags_1(). That function will call\n\t * the expensive is_excluded_from_list() on every entry.\n\t */\n\nWhile I haven't implemented it yet in this RFC, this TODO can actually\nhappen with the current set of cone patterns:\n\n1. If we hit a directory that is not in a parent or recursive path,\n then all paths it contains must have their skipworktree bits set.\n We can avoid computing hashes for them.\n\n2. If we hit a directory that is in a recursive path, then all paths\n it contains must have skipworktree bits off. We can avoid computing\n hashes for them.\n\nWhen we have a million index entries, these hash computations are not\ninsignificant!\n\nWith that in mind, I think there is a _performance_ reason to include\nthe parent folders in addition to _user experience_ reason. If we\nwere to add the complexity of exact file matches, then we would also\nwant to add the parent folders leading to that file so we can still\ndo the logic above.\n\nThanks,\n-Stolee\n"},{"id":"381031","messageId":"CABPp-BFT9A5n=_bx5LsjCvbogqwSjiwgr5amcjgbU1iAk4KLJg@mail.gmail.com","threadId":"51691","inReplyTo":"c37b5f2c29d42e7c0c727596625d4eb50b6ddb4f.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/9] sparse-checkout: create builtin with 'list' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T22:30:24Z","receivedAt":"2019-08-23T22:53:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:13 AM Derrick Stolee via GitGitGadget\n wrote:\n\n> The documentation provided is adapted from the \"git read-tree\"\n> documentation with a few edits for clarity in the new context.\n> Extra sections are added to hint toward a future change to\n> a moer restricted pattern set.\n\ns/moer/more/\n\n\n> +\"Sparse checkout\" allows populating the working directory sparsely.\n> +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n> +Git whether a file in the working directory is worth looking at. If\n> +the skip-worktree bit is set, then the file is ignored in the working\n> +directory. Git will not populate the contents of those files, which\n> +makes a sparse checkout helpful when working in a repository with many\n> +files, but only a few are important to the current user.\n> +\n> +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n> +skip-worktree reference bitmap. When Git updates the working\n> +directory, it resets the skip-worktree bit in the index based on this\n> +file. If an entry\n> +matches a pattern in this file, skip-worktree will not be set on\n> +that entry. Otherwise, skip-worktree will be set.\n> +\n> +Then it compares the new skip-worktree value with the previous one. If\n> +skip-worktree turns from set to unset, it will add the corresponding\n> +file back. If it turns from unset to set, that file will be removed.\n\nI had to read this twice for it to make sense. Not sure I have a real\ngood suggestion here, the name \"skip-worktree\" instead of e.g\n\"wanted-in-worktree\" just naturally leads us into sentences with one\nnegation automatically and we sometimes have to add more.\n\nMaybe just replace the last two paragraphs with:\n\nThe `$GIT_DIR/info/sparse-checkout` file is used to define the\nskip-worktree reference bitmap. When Git updates the working\ndirectory, it updates the skip-worktree bits in the index based on this\nfile and removes or restores files in the working copy to match.\n\n> +\n> +## FULL PATTERN SET\n> +\n> +By default, the sparse-checkout file uses the same syntax as `.gitignore`\n> +files.\n> +\n> +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n> +files are in, you can also specify what files are _not_ in, using\n> +negate patterns. For example, to remove the file `unwanted`:\n\ns/in/included/?\n\n> +Another tricky thing is fully repopulating the working directory when you\n> +no longer want sparse checkout. You cannot just disable \"sparse\n> +checkout\" because skip-worktree bits are still in the index and your working\n> +directory is still sparsely populated. You should re-populate the working\n> +directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> +follows:\n> +\n> +----------------\n> +/*\n> +----------------\n\nCan we just get rid of this part of the documentation, since there\nwill be a sparse-checkout command to disable/undo/reset? However, it\ncould be useful to mention cases when disabling/undoing/resetting a\nsparse-checkout won't work, if there are some. For example, with the\nprevious read-tree implementation, you could not undo the sparse\ncheckout if the index had any unstaged entries, and you couldn't undo\nit if there were any files present that corresponding to the sparse\npatterns (for fear they'd be overwritten -- however, every once in a\nwhile someone tried to desparsify, it failed e.g. due to the disk\nbecoming full, and then after freeing up space there were zillions of\nfiles that exactly matched what de-sparsifying would have put there\nbut the command wanted the user to manually delete them first.)\n\n> +\n> +Then you can disable sparse checkout. Sparse checkout support in 'git\n> +read-tree' and similar commands is disabled by default. You need to\n> +set `core.sparseCheckout` to `true` in order to have sparse checkout\n> +support.\n\n...and get rid of this paragraph because I'd expect git\nsparse-checkout to come with a subcommand (init/add/whatever) to set\nthis for the user? Unless maybe you want to add some words about why\nthe command sets core.sparseCheckout...and related workspace related\nstuff as we talked about elsewhere.\n\n> +test_expect_success 'git sparse-checkout list (empty)' '\n> + git -C repo sparse-checkout list >list 2>err &&\n> + test_line_count = 0 list &&\n> + test_i18ngrep \"failed to parse sparse-checkout file; it may not exist\" err\n\nIs that the error we want, rather than something like \"This worktree\nis not sparse (no sparse-checkout file exists and core.sparseCheckout\nis false\"?\n\n> +'\n> +\n> +test_expect_success 'git sparse-checkout list (populated)' '\n> + test_when_finished rm -f repo/.git/info/sparse-checkout &&\n> + cat >repo/.git/info/sparse-checkout <<-EOF &&\n> + /folder1/*\n> + /deep/\n> + **/a\n> + !*bin*\n> + EOF\n> + git -C repo sparse-checkout list >list &&\n> + cat >expect <<-EOF &&\n> + /folder1/*\n> + /deep/\n> + **/a\n> + !*bin*\n> + EOF\n> + test_cmp expect list\n\nI have a `./sparsify --stats` that reports\n You are now in a sparse checkout with only 3499 of the 53625 files.\nor\n You are not in a sparse checkout.\n\nand I have a --info option that reports both the stats and the list of\nsparse paths similar to this sparse-checkout list command. The stats\naren't important, I guess, but seem nice for the user. I don't know\nif you want to include anything like that in this or another command,\nbut just thought I'd mention it.\n"},{"id":"381032","messageId":"CABPp-BEmwSwg4tgJg6nVG8a3Hpn_g-=ZjApZF4EiJO+qVgu4uw@mail.gmail.com","threadId":"51691","inReplyTo":"e6e982e5a6e4517d97a7a404384057110f3a151d.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/9] sparse-checkout: create 'init' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T23:02:57Z","receivedAt":"2019-08-23T23:03:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:13 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Getting started with a sparse-checkout file can be daunting. Help\n> users start their sparse enlistment using 'git sparse-checkout init'.\n> This will set 'core.sparseCheckout=true' in their config, write\n> an initial set of patterns to the sparse-checkout file, and update\n> their working directory.\n>\n> Using 'git read-tree' to clear directories does not work cleanly\n> on Windows, so manually delete directories that are tracked by Git\n> before running read-tree.\n\nIs that a bug in read-tree that needs to be fixed?\n\n> The use of running another process for 'git read-tree' is likely\n> suboptimal, but that can be improved in a later change, if valuable.\n\nI think it's valuable. The bigger problem may be that \"git read-tree\n-mu HEAD\" may turn out to be insufficient for our needs; see below....\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 7 ++\n> builtin/sparse-checkout.c | 106 +++++++++++++++++++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 40 ++++++++++\n> 3 files changed, 152 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index ca0ca6a12f..50c53ee60a 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -26,6 +26,13 @@ COMMANDS\n> 'list'::\n> Provide a list of the contents in the sparse-checkout file.\n>\n> +'init'::\n> + Enable the `core.sparseCheckout` setting. If the\n> + sparse-checkout file does not exist, then populate it with\n> + patterns that match every file in the root directory and\n> + no other directories, then will remove all directories tracked\n> + by Git. Add patterns to the sparse-checkout file to\n> + repopulate the working directory.\n>\n> SPARSE CHECKOUT\n> ----------------\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 6477a6ed9c..86d24e6295 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> - N_(\"git sparse-checkout [list]\"),\n> + N_(\"git sparse-checkout [init|list]\"),\n> NULL\n> };\n>\n> @@ -64,6 +64,108 @@ static int sparse_checkout_list(int argc, const char **argv)\n> return 0;\n> }\n>\n> +static int sc_read_tree(void)\n> +{\n> + struct argv_array argv = ARGV_ARRAY_INIT;\n> + int result = 0;\n> + argv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n> +\n> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> + error(_(\"failed to update index with new sparse-checkout paths\"));\n> + result = 1;\n> + }\n\n`git read-tree -m -u HEAD` will fail if the index has any higher stage\nentries in it, even if those higher stage entries correspond to files\nwhich are included in the sparseness patterns and thus would not need\nan update. It might be nice if we can find a way to provide a better\nerror message, and/or implement the read-tree -m -u HEAD internally in\na way that will allow us to not fail if the conflicted files are\nincluded in the sparse set.\n\n> +\n> + argv_array_clear(&argv);\n> + return result;\n> +}\n> +\n> +static int sc_enable_config(void)\n> +{\n> + struct argv_array argv = ARGV_ARRAY_INIT;\n> + int result = 0;\n> + argv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n\nWhy --add? That seems really odd to me.\n\nThis should also have \"--worktree\". And this function should either\nset extensions.worktreeConfig to true or die if it isn't already set;\nnot sure which. There's some UI and documentation stuff to figure out\nhere...\n\n> +\n> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> + error(_(\"failed to enable core.sparseCheckout\"));\n> + result = 1;\n> + }\n> +\n> + argv_array_clear(&argv);\n> + return result;\n> +}\n> +\n> +static int delete_directory(const struct object_id *oid, struct strbuf *base,\n> + const char *pathname, unsigned mode, int stage, void *context)\n> +{\n> + struct strbuf dirname = STRBUF_INIT;\n> + struct stat sb;\n> +\n> + strbuf_addstr(&dirname, the_repository->worktree);\n> + strbuf_addch(&dirname, '/');\n> + strbuf_addstr(&dirname, pathname);\n> +\n> + if (stat(dirname.buf, &sb) || !(sb.st_mode & S_IFDIR))\n> + return 0;\n> +\n> + if (remove_dir_recursively(&dirname, 0))\n\nflags = 0 implies not REMOVE_DIR_EMPTY_ONLY. I'm not familiar with\nremove_dir_recursively(), but won't this delete everything...including\nuntracked files? If so, that sounds like a bug.\n\n> + warning(_(\"failed to remove directory '%s'\"),\n> + dirname.buf);\n> +\n> + strbuf_release(&dirname);\n> + return 0;\n> +}\n> +\n> +static int sparse_checkout_init(int argc, const char **argv)\n> +{\n> + struct tree *t;\n> + struct object_id oid;\n> + struct exclude_list el;\n> + static struct pathspec pathspec;\n> + char *sparse_filename;\n> + FILE *fp;\n> + int res;\n> +\n> + if (sc_enable_config())\n> + return 1;\n> +\n> + memset(&el, 0, sizeof(el));\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + res = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n\nBut 'el' isn't used again? Why are we getting the list of files from\nsparse_filename then?\n\n> +\n> + /* If we already have a sparse-checkout file, use it. */\n> + if (res >= 0) {\n> + free(sparse_filename);\n> + goto reset_dir;\n> + }\n> +\n> + /* initial mode: all blobs at root */\n> + fp = fopen(sparse_filename, \"w\");\n> + free(sparse_filename);\n> + fprintf(fp, \"/*\\n!/*/*\\n\");\n> + fclose(fp);\n\nMakes sense.\n\n> +\n> + /* remove all directories in the root, if tracked by Git */\n> + if (get_oid(\"HEAD\", &oid)) {\n> + /* assume we are in a fresh repo */\n> + return 0;\n> + }\n> +\n> + t = parse_tree_indirect(&oid);\n> +\n> + parse_pathspec(&pathspec, PATHSPEC_ALL_MAGIC &\n> + ~(PATHSPEC_FROMTOP | PATHSPEC_LITERAL),\n> + PATHSPEC_PREFER_CWD,\n> + \"\", NULL);\n> +\n> + if (read_tree_recursive(the_repository, t, \"\", 0, 0, &pathspec,\n> + delete_directory, NULL))\n> + return 1;\n\nSince this is only needed on Windows, as per your commit message,\nshould it be #ifdef'd? Or is this actually a bug that should be fixed\nin \"git read-tree -mu HEAD\"?\n\n> +\n> +reset_dir:\n> + return sc_read_tree();\n> +}\n> +\n\nThe rest looks fine.\n"},{"id":"381033","messageId":"CABPp-BGyDwgTRx3A4OPJT=5rMM-onkPtg9m58M=rDMHiJQxQ6Q@mail.gmail.com","threadId":"51691","inReplyTo":"4ccd36b3966b62934cfbb0ec65e2851b50dfa1e3.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/9] clone: add --sparse mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T23:17:06Z","receivedAt":"2019-08-23T23:17:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When someone wants to clone a large repository, but plans to work\n> using a sparse-checkout file, they either need to do a full\n> checkout first and then reduce the patterns they included, or\n> clone with --no-checkout, set up their patterns, and then run\n> a checkout manually. This requires knowing a lot about the repo\n> shape and how sparse-checkout works.\n>\n> Add a new '--sparse' option to 'git clone' that initializes the\n> sparse-checkout file to include the following patterns:\n>\n> /*\n> !/*/*\n>\n> These patterns include every file in the root directory, but\n> no directories. This allows a repo to include files like a\n> README or a bootstrapping script to grow enlistments from that\n> point.\n\nNice.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-clone.txt | 8 +++++++-\n> builtin/clone.c | 27 +++++++++++++++++++++++++++\n> t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n> 3 files changed, 47 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n> index 34011c2940..0fe91d2f04 100644\n> --- a/Documentation/git-clone.txt\n> +++ b/Documentation/git-clone.txt\n> @@ -15,7 +15,7 @@ SYNOPSIS\n> [--dissociate] [--separate-git-dir ]\n> [--depth ] [--[no-]single-branch] [--no-tags]\n> [--recurse-submodules[=]] [--[no-]shallow-submodules]\n> - [--[no-]remote-submodules] [--jobs ] [--] \n> + [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n> []\n>\n> DESCRIPTION\n> @@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n> used, neither remote-tracking branches nor the related\n> configuration variables are created.\n>\n> +--sparse::\n> + Initialize the sparse-checkout file so the working\n> + directory starts with only the files in the root\n> + of the repository. The sparse-checkout file can be\n> + modified to grow the working directory as needed.\n> +\n> --mirror::\n> Set up a mirror of the source repository. This implies `--bare`.\n> Compared to `--bare`, `--mirror` not only maps local branches of the\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index f665b28ccc..d6d49a73ff 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -60,6 +60,7 @@ static const char *real_git_dir;\n> static char *option_upload_pack = \"git-upload-pack\";\n> static int option_verbosity;\n> static int option_progress = -1;\n> +static int option_sparse_checkout;\n> static enum transport_family family;\n> static struct string_list option_config = STRING_LIST_INIT_NODUP;\n> static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n> @@ -147,6 +148,8 @@ static struct option builtin_clone_options[] = {\n> OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> OPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n> N_(\"any cloned submodules will use their remote-tracking branch\")),\n> + OPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n> + N_(\"initialize sparse-checkout file to include only files at root\")),\n> OPT_END()\n> };\n>\n> @@ -734,6 +737,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n> }\n> }\n>\n> +static int git_sparse_checkout_init(const char *repo)\n> +{\n> + struct argv_array argv = ARGV_ARRAY_INIT;\n> + int result = 0;\n> + argv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n> +\n> + /*\n> + * We must apply the setting in the current process\n> + * for the later checkout to use the sparse-checkout file.\n> + */\n> + core_apply_sparse_checkout = 1;\n> +\n> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> + error(_(\"failed to initialize sparse-checkout\"));\n> + result = 1;\n> + }\n\nSigh...so much forking of additional processes. I'd really rather\nthat we were reducing how much of this we are doing in the codebase\ninstead of adding more. Every fork makes following stuff in a\ndebugger harder.\n\n> +\n> + argv_array_clear(&argv);\n> + return result;\n> +}\n> +\n> static int checkout(int submodule_progress)\n> {\n> struct object_id oid;\n> @@ -1107,6 +1131,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> if (option_required_reference.nr || option_optional_reference.nr)\n> setup_reference();\n>\n> + if (option_sparse_checkout && git_sparse_checkout_init(repo))\n> + return 1;\n> +\n> remote = remote_get(option_origin);\n>\n> strbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 35ab84aabd..b7d5f15830 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -87,4 +87,17 @@ test_expect_success 'init with existing sparse-checkout' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'clone --sparse' '\n> + git clone --sparse repo clone &&\n> + git -C clone sparse-checkout list >actual &&\n> + cat >expect <<-EOF &&\n> + /*\n> + !/*/*\n> + EOF\n> + test_cmp expect actual &&\n> + ls clone >dir &&\n> + echo a >expect &&\n> + test_cmp expect dir\n\nChecking that a toplevel entry is present, but not checking that an\nentry from a subdir is missing as expected?\n"},{"id":"381034","messageId":"CABPp-BEhc00g3CwKg13wDrtFHM-sw9tSy75UEbTvugKkdwTyeA@mail.gmail.com","threadId":"51691","inReplyTo":"0f095e85d5bf29346bdc5bf1707bb51eaf2202ae.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/9] sparse-checkout: 'add' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T23:30:56Z","receivedAt":"2019-08-23T23:31:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The 'git sparse-checkout add' subcommand takes a list of patterns\n> over stdin and writes them to the sparse-checkout file. Then, it\n> updates the working directory using 'git read-tree -mu HEAD'.\n\nAs mentioned in response to the cover letter, I'd rather see it take\npatterns as positional arguments (though requiring a '--' argument\nbefore any patterns that start with a hyphen). It could also take\n--stdin to read from stdin.\n\n> Note: if a user adds a negative pattern that would lead to the\n> removal of a non-empty directory, then Git may not delete that\n> directory (on Windows).\n\nThis sounds like you're re-iterating a bug mentioned earlier, but if\nsomeone in the future comes and reads this comment it might sound like\nyou're saying git can avoid clearing a directory for optimization or\nother reasons. (And, of course, it'd be nice to figure out why this\nbug exists.)\n\nAnother question this brings up, though, is that you worked around\nthis bug in 'init' so why would you not also do so for 'add'? Seems\ninconsistent to me.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 4 ++++\n> builtin/sparse-checkout.c | 32 ++++++++++++++++++++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n> 3 files changed, 55 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index 50c53ee60a..6f540a3443 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -34,6 +34,10 @@ COMMANDS\n> by Git. Add patterns to the sparse-checkout file to\n> repopulate the working directory.\n>\n> +'add'::\n> + Add a set of patterns to the sparse-checkout file, as given over\n> + stdin. Updates the working directory to match the new patterns.\n> +\n> SPARSE CHECKOUT\n> ----------------\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 86d24e6295..ec6134fecc 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> - N_(\"git sparse-checkout [init|list]\"),\n> + N_(\"git sparse-checkout [init|add|list]\"),\n> NULL\n> };\n>\n> @@ -166,6 +166,34 @@ static int sparse_checkout_init(int argc, const char **argv)\n> return sc_read_tree();\n> }\n>\n> +static int sparse_checkout_add(int argc, const char **argv)\n> +{\n> + struct exclude_list el;\n> + char *sparse_filename;\n> + FILE *fp;\n> + struct strbuf line = STRBUF_INIT;\n> +\n> + memset(&el, 0, sizeof(el));\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n\nel is an exclude_list and we call add_excludes_..., but it's actually\nan *include* list. This is going to cause errors at some point, and\nwill cause lots of headaches.\n\n> +\n> + fp = fopen(sparse_filename, \"w\");\n> + write_excludes_to_file(fp, &el);\n> +\n> + while (!strbuf_getline(&line, stdin)) {\n> + strbuf_trim(&line);\n> + fprintf(fp, \"%s\\n\", line.buf);\n> + }\n\nShould we first check whether these excludes are already in the\nsparse-checkout file?\n\n> + fclose(fp);\n> + free(sparse_filename);\n> +\n> + clear_exclude_list(&el);\n> +\n> + return sc_read_tree();\n\nWhat if someone calls 'git sparse-checkout add' without first calling\n'git sparse-checkout init'? As far as I can tell, core.sparseCheckout\nwill be unset (i.e. treated as false), meaning that this operation\nwill do some work, but result in no changes and a report of success.\nAfter users try to figure out why it won't work, they eventually run\n'git sparse-checkout init', which will delete all the entries they\npreviously added with the 'add' subcommand.\n\nWhat should happen instead?\n\n> +}\n> +\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> {\n> static struct option builtin_sparse_checkout_options[] = {\n> @@ -187,6 +215,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> return sparse_checkout_list(argc, argv);\n> if (!strcmp(argv[0], \"init\"))\n> return sparse_checkout_init(argc, argv);\n> + if (!strcmp(argv[0], \"add\"))\n> + return sparse_checkout_add(argc, argv);\n> }\n>\n> usage_with_options(builtin_sparse_checkout_usage,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index b7d5f15830..499bd8d6d0 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -100,4 +100,24 @@ test_expect_success 'clone --sparse' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'add to existing sparse-checkout' '\n> + echo \"/folder2/*\" | git -C repo sparse-checkout add &&\n\nI've always been using '/folder2/' in sparse-checkout, without the\ntrailing asterisk. That seems more friendly for cone mode too. Are\nthere benefits to keeping the trailing asterisk?\n\n> + cat >expect <<-EOF &&\n> + /*\n> + !/*/*\n> + /folder1/*\n> + /folder2/*\n> + EOF\n> + git -C repo sparse-checkout list >actual &&\n> + test_cmp expect actual &&\n> + test_cmp expect repo/.git/info/sparse-checkout &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_done\n> \\ No newline at end of file\n> --\n> gitgitgadget\n>\n"},{"id":"381036","messageId":"CABPp-BHon=QgZXPjwyZspq0wW3rFLZqx1HZgE8eMq6DuTCc-0g@mail.gmail.com","threadId":"51691","inReplyTo":"5f332b799f90b64e2c0720fd0aba0e163d41587a.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/9] sparse-checkout: create 'disable' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-23T23:50:10Z","receivedAt":"2019-08-23T23:50:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:14 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The instructions for disabling a sparse-checkout to a full\n> working directory are complicated and non-intuitive. Add a\n> subcommand, 'git sparse-checkout disable', to perform those\n> steps for the user.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 26 +++++++---------\n> builtin/sparse-checkout.c | 45 ++++++++++++++++++++++++---\n> t/t1091-sparse-checkout-builtin.sh | 15 +++++++++\n> 3 files changed, 67 insertions(+), 19 deletions(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index 6f540a3443..de04b768ae 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -38,6 +38,10 @@ COMMANDS\n> Add a set of patterns to the sparse-checkout file, as given over\n> stdin. Updates the working directory to match the new patterns.\n>\n> +'disable'::\n> + Remove the sparse-checkout file, set `core.sparseCheckout` to\n> + `false`, and restore the working directory to include all files.\n> +\n> SPARSE CHECKOUT\n> ----------------\n>\n> @@ -60,6 +64,13 @@ Then it compares the new skip-worktree value with the previous one. If\n> skip-worktree turns from set to unset, it will add the corresponding\n> file back. If it turns from unset to set, that file will be removed.\n>\n> +To repopulate the working directory with all files, use the\n> +`git sparse-checkout disable` command.\n\nMake sense.\n\n> +\n> +Sparse checkout support in 'git read-tree' and similar commands is\n> +disabled by default. You need to set `core.sparseCheckout` to `true`\n> +in order to have sparse checkout support.\n\nBut why add this paragraph? read-tree is plumbing that we'd rather\nnot force on users, so I'd rather not mention it. And I thought we\nwere setting core.sparseCheckout for the user.\n\n> -Another tricky thing is fully repopulating the working directory when you\n> -no longer want sparse checkout. You cannot just disable \"sparse\n> -checkout\" because skip-worktree bits are still in the index and your working\n> -directory is still sparsely populated. You should re-populate the working\n> -directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> -follows:\n> -\n> -----------------\n> -/*\n> -----------------\n> -\n> -Then you can disable sparse checkout. Sparse checkout support in 'git\n> -read-tree' and similar commands is disabled by default. You need to\n> -set `core.sparseCheckout` to `true` in order to have sparse checkout\n> -support.\n\nWahoo!\n\n\n> -static int sc_enable_config(void)\n> +static int sc_set_config(int mode)\n> {\n> struct argv_array argv = ARGV_ARRAY_INIT;\n> int result = 0;\n> - argv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n> + argv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n\nRemove \"--add\" and add \"--worktree\".\n\n> +\n> + switch (mode) {\n> + case 1:\n> + argv_array_pushl(&argv, \"true\", NULL);\n> + break;\n> +\n> + case 0:\n> + argv_array_pushl(&argv, \"false\", NULL);\n> + break;\n> +\n> + default:\n> + die(_(\"invalid config mode\"));\n> + }\n>\n> if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n\nCan't we use git_config_set instead of run_command_v_opt?\n\n> error(_(\"failed to enable core.sparseCheckout\"));\n> @@ -125,7 +138,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n> FILE *fp;\n> int res;\n>\n> - if (sc_enable_config())\n> + if (sc_set_config(1))\n\nRandom idea: The code would be more self-documenting, if you just made\nsc_set_config take a string and you passed either \"true\" or \"false\".\nThen you could also dispense with the switch statement. Yes, you can\nmake typos, but there are only going to be 2-3 places in the code that\ncall it, right?\n\n> return 1;\n>\n> memset(&el, 0, sizeof(el));\n> @@ -194,6 +207,28 @@ static int sparse_checkout_add(int argc, const char **argv)\n> return sc_read_tree();\n> }\n>\n> +static int sparse_checkout_disable(int argc, const char **argv)\n> +{\n> + char *sparse_filename;\n> + FILE *fp;\n> +\n> + if (sc_set_config(1))\n> + die(_(\"failed to change config\"));\n\nOoh, preventative setting. This'll probably save some grief for\npeople who get themselves in a weird state and start mucking around\nmanually; very nice.\n\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + fp = fopen(sparse_filename, \"w\");\n> + fprintf(fp, \"/*\\n\");\n> + fclose(fp);\n> +\n> + if (sc_read_tree())\n> + die(_(\"error while refreshing working directory\"));\n\nThis can fail e.g. if there are conflicts in the index. Luckily, if\nthey fix them up and then re-run \"git sparse-checkout disable\" then it\ncan work. But you've already changed the\n$GIT_DIR/info/sparse-checkout file to a state that won't match what\nthey actually have checked out. Will that cause any problems? (I\ncan't think of any immediately, but I am curious.)\n\nAlso, should or can we give a better error message, such as\ninstructing them to try again after they've fixed any problems\nreported by the sc_read_tree() operation?\n\n> +\n> + unlink(sparse_filename);\n> + free(sparse_filename);\n> +\n> + return sc_set_config(0);\n> +}\n> +\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> {\n> static struct option builtin_sparse_checkout_options[] = {\n> @@ -217,6 +252,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> return sparse_checkout_init(argc, argv);\n> if (!strcmp(argv[0], \"add\"))\n> return sparse_checkout_add(argc, argv);\n> + if (!strcmp(argv[0], \"disable\"))\n> + return sparse_checkout_disable(argc, argv);\n> }\n>\n> usage_with_options(builtin_sparse_checkout_usage,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 499bd8d6d0..68ca63a6f6 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -120,4 +120,19 @@ test_expect_success 'add to existing sparse-checkout' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'sparse-checkout disable' '\n> + git -C repo sparse-checkout disable &&\n> + test_path_is_missing repo/.git/info/sparse-checkout &&\n> + git -C repo config --list >config &&\n\nShould this command also have a --worktree flag (just for consistency,\neven if we know there will only be one worktree so it doesn't matter\nin practice)? There are probably similar cases I overlooked on the\nprevious patches.\n\n> + test_i18ngrep \"core.sparsecheckout=false\" config &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deep\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_done\n> \\ No newline at end of file\n> --\n> gitgitgadget\n>\n"},{"id":"381037","messageId":"CABPp-BGJv1QJ0zeDx33My9X+GDQC1F-s8W=L1Qt-__YPj=ObCg@mail.gmail.com","threadId":"51691","inReplyTo":"86f12dc77dd94c1b2c6294842d29d8712f79e3d7.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] trace2:experiment: clear_ce_flags_1","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-24T00:08:24Z","receivedAt":"2019-08-24T00:08:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Jeff Hostetler via GitGitGadget\n wrote:\n>\n> From: Jeff Hostetler \n\nCan the commit summary be turned into English?\n\n> The clear_ce_flags_1 method is used by many types of calls to\n> unpack_trees(). Add trace2 regions around the method, including\n> some flag information, so we can get granular performance data\n> during experiments.\n\nIt might be nice to have some words in the cover letter about why this\npatch is included in this series instead of being a separate\nsubmission. I'm not familiar with the trace2 stuff yet; this looks\nprobably useful, but the commit message makes it sound like something\ngeneral rather than specific to this series.\n\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Derrick Stolee \n> ---\n\n"},{"id":"381038","messageId":"CABPp-BF_RSQfju7ObKCpkDssGadUKB7NcrtMytveioeZX5rd9w@mail.gmail.com","threadId":"51691","inReplyTo":"19d664a5dada87a9a8dcf18d7548582275593f10.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/9] sparse-checkout: add 'cone' mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-24T00:31:28Z","receivedAt":"2019-08-24T00:31:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:13 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The sparse-checkout feature can have quadratic performance as\n> the number of patterns and number of entries in the index grow.\n> If there are 1,000 patterns and 1,000,000 entries, this time can\n> be very significant.\n>\n> Create a new 'cone' mode for the core.sparseCheckout config\n> option, and adjust the parser to set an appropriate enum value.\n>\n> While adjusting the type of this variable, rename it from\n> core_apply_sparse_checkout to core_sparse_checkout. This will\n> help avoid parallel changes from hitting type issues, and we\n> can guarantee that all uses now consider the enum values instead\n> of the int value.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/core.txt | 7 ++--\n> Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n> builtin/clone.c | 2 +-\n> builtin/sparse-checkout.c | 16 +++++----\n> cache.h | 8 ++++-\n> config.c | 10 +++++-\n> environment.c | 2 +-\n> t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n> unpack-trees.c | 2 +-\n> 9 files changed, 98 insertions(+), 13 deletions(-)\n>\n> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\n> index 75538d27e7..9b8ab2a6d4 100644\n> --- a/Documentation/config/core.txt\n> +++ b/Documentation/config/core.txt\n> @@ -591,8 +591,11 @@ core.multiPackIndex::\n> multi-pack-index design document].\n>\n> core.sparseCheckout::\n> - Enable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> - linkgit:git-read-tree[1] for more information.\n> + Enable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n> + is disabled. If \"true\", then sparse-checkout is enabled with the full\n> + .gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n> + a restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n> + information.\n>\n> core.abbrev::\n> Set the length object names are abbreviated to. If\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index de04b768ae..463319055b 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -86,6 +86,56 @@ negate patterns. For example, to remove the file `unwanted`:\n> ----------------\n>\n>\n> +## CONE PATTERN SET\n> +\n> +The full pattern set allows for arbitrary pattern matches and complicated\n> +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n> +updating the index, where N is the number of patterns and M is the number\n> +of paths in the index. To combat this performance issue, a more restricted\n> +pattern set is allowed when `core.spareCheckout` is set to `cone`.\n> +\n> +The accepted patterns in the cone pattern set are:\n> +\n> +1. *Recursive:* All paths inside a directory are included.\n> +\n> +2. *Parent:* All files immediately inside a directory are included.\n> +\n> +In addition to the above two patterns, we also expect that all files in the\n> +root directory are included. If a recursive pattern is added, then all\n> +leading directories are added as parent patterns.\n> +\n> +By default, when running `git sparse-checkout init`, the root directory is\n> +added as a parent pattern. At this point, the sparse-checkout file contains\n> +the following patterns:\n> +\n> +```\n> +/*\n> +!/*/*\n> +```\n> +\n> +This says \"include everything in root, but nothing two levels below root.\"\n\n...but nothing at the level below root...?\n\n> +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n> +`A/B` are added as parent patterns. The resulting sparse-checkout file is\n> +now\n> +\n> +```\n> +/*\n> +!/*/*\n> +/A/*\n> +!/A/*/*\n> +/A/B/*\n> +!/A/B/*/*\n> +/A/B/C/*\n> +```\n\nCan we dispense with the trailing asterisks (other than on the first\nline for the root level)? This reads a lot cleaner to me:\n\n```\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n```\n\nWe could also dispense with the trailing '/' on the inclusion lines\nfrom this version, but I'm not sure that helps.\n\n> +\n> +Here, order matters, so the negative patterns are overridden by the positive\n> +patterns that appear lower in the file.\n> +\n> +If `core.sparseCheckout=cone`, then Git will parse the sparse-checkout file\n> +expecting patterns of these types. Git will warn if the patterns do not match.\n> +If the patterns do match the expected format, then Git will use faster hash-\n> +based algorithms to compute inclusion in the sparse-checkout.\n> +\n> SEE ALSO\n> --------\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index d6d49a73ff..763898ada5 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -747,7 +747,7 @@ static int git_sparse_checkout_init(const char *repo)\n> * We must apply the setting in the current process\n> * for the later checkout to use the sparse-checkout file.\n> */\n> - core_apply_sparse_checkout = 1;\n> + core_sparse_checkout = SPARSE_CHECKOUT_FULL;\n>\n> if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> error(_(\"failed to initialize sparse-checkout\"));\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 8f97c27ec7..77e5235720 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -79,18 +79,22 @@ static int sc_read_tree(void)\n> return result;\n> }\n>\n> -static int sc_set_config(int mode)\n> +static int sc_set_config(enum sparse_checkout_mode mode)\n> {\n> struct argv_array argv = ARGV_ARRAY_INIT;\n> int result = 0;\n> argv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n>\n> switch (mode) {\n> - case 1:\n> + case SPARSE_CHECKOUT_FULL:\n> argv_array_pushl(&argv, \"true\", NULL);\n> break;\n>\n> - case 0:\n> + case SPARSE_CHECKOUT_CONE:\n> + argv_array_pushl(&argv, \"cone\", NULL);\n> + break;\n> +\n> + case SPARSE_CHECKOUT_NONE:\n> argv_array_pushl(&argv, \"false\", NULL);\n> break;\n>\n> @@ -138,7 +142,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n> FILE *fp;\n> int res;\n>\n> - if (sc_set_config(1))\n> + if (sc_set_config(SPARSE_CHECKOUT_FULL))\n\nGoing back to my comment on the previous patch, perhaps\nSPARSE_CHECKOUT_FULL could be the string \"true\", so you can avoid the\nswitch statement?\n\n> return 1;\n>\n> memset(&el, 0, sizeof(el));\n> @@ -212,7 +216,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> char *sparse_filename;\n> FILE *fp;\n>\n> - if (sc_set_config(1))\n> + if (sc_set_config(SPARSE_CHECKOUT_FULL))\n> die(_(\"failed to change config\"));\n>\n> sparse_filename = get_sparse_checkout_filename();\n> @@ -226,7 +230,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> unlink(sparse_filename);\n> free(sparse_filename);\n>\n> - return sc_set_config(0);\n> + return sc_set_config(SPARSE_CHECKOUT_NONE);\n> }\n>\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> diff --git a/cache.h b/cache.h\n> index b1da1ab08f..4426816ca1 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -865,12 +865,18 @@ extern char *git_replace_ref_base;\n>\n> extern int fsync_object_files;\n> extern int core_preload_index;\n> -extern int core_apply_sparse_checkout;\n> extern int precomposed_unicode;\n> extern int protect_hfs;\n> extern int protect_ntfs;\n> extern const char *core_fsmonitor;\n>\n> +enum sparse_checkout_mode {\n> + SPARSE_CHECKOUT_NONE = 0,\n> + SPARSE_CHECKOUT_FULL = 1,\n> + SPARSE_CHECKOUT_CONE = 2,\n> +};\n> +enum sparse_checkout_mode core_sparse_checkout;\n> +\n> /*\n> * Include broken refs in all ref iterations, which will\n> * generally choke dangerous operations rather than letting\n\nWait, you're not changing the add command? So the cone mode just\n\n> diff --git a/config.c b/config.c\n> index 3900e4947b..15b7a20dd9 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1360,7 +1360,15 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> }\n>\n> if (!strcmp(var, \"core.sparsecheckout\")) {\n> - core_apply_sparse_checkout = git_config_bool(var, value);\n> + int result = git_parse_maybe_bool(value);\n> +\n> + if (result < 0) {\n> + core_sparse_checkout = SPARSE_CHECKOUT_NONE;\n> +\n> + if (!strcasecmp(value, \"cone\"))\n> + core_sparse_checkout = SPARSE_CHECKOUT_CONE;\n> + } else\n> + core_sparse_checkout = result;\n> return 0;\n> }\n>\n> diff --git a/environment.c b/environment.c\n> index 89af47cb85..cc12e30bd6 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -68,7 +68,7 @@ enum push_default_type push_default = PUSH_DEFAULT_UNSPECIFIED;\n> enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n> char *notes_ref_name;\n> int grafts_replace_parents = 1;\n> -int core_apply_sparse_checkout;\n> +enum sparse_checkout_mode core_sparse_checkout;\n> int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> unsigned long pack_size_limit_cfg;\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 68ca63a6f6..8cc377b839 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -120,6 +120,20 @@ test_expect_success 'add to existing sparse-checkout' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'cone mode: match patterns' '\n> + git -C repo config --replace-all core.sparseCheckout cone &&\n\n--replace-all? This makes me wonder if you were actually doing the\n--add in the previous patchsets on purpose. I'm so confused.\n\n> + rm -rf repo/a repo/folder1 repo/folder2 &&\n> + git -C repo read-tree -mu HEAD &&\n> + git -C repo reset --hard &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_expect_success 'sparse-checkout disable' '\n> git -C repo sparse-checkout disable &&\n> test_path_is_missing repo/.git/info/sparse-checkout &&\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 8c3b5e8849..289c62305f 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1468,7 +1468,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>\n> trace_performance_enter();\n> memset(&el, 0, sizeof(el));\n> - if (!core_apply_sparse_checkout || !o->update)\n> + if (!core_sparse_checkout || !o->update)\n> o->skip_sparse_checkout = 1;\n> if (!o->skip_sparse_checkout) {\n> char *sparse = git_pathdup(\"info/sparse-checkout\");\n> --\n> gitgitgadget\n\nWait...I didn't see anything checking the value of \"cone\" and using\nit, it only has an ability to set it. What's the point? Or is that\ngoing to come in a later patch? (If it does, should the commit\nmessage mention that?)\n"},{"id":"381041","messageId":"CABPp-BFMtO=7UGVZPbqh3tthSetvz5F=W3S=RsryPSuchmZeZw@mail.gmail.com","threadId":"51691","inReplyTo":"b99acea4a09a6ed5302e5e622a6b6eb35d2d01b1.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/9] sparse-checkout: use hashmaps for cone patterns","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-24T04:56:52Z","receivedAt":"2019-08-24T04:57:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The parent and recursive patterns allowed by the \"cone mode\"\n> option in sparse-checkout are restrictive enough that we\n> can avoid using the regex parsing. Everything is based on\n> prefix matches, so we can use hashsets to store the prefixes\n> from the sparse-checkout file. When checking a path, we can\n> strip path entries from the path and check the hashset for\n> an exact match.\n>\n> As a test, I created a cone-mode sparse-checkout file for the\n> Linux repository that actually includes every file. This was\n> constructed by taking every folder in the Linux repo and creating\n> the pattern pairs here:\n>\n> /$folder/*\n> !/$folder/*/*\n>\n> This resulted in a sparse-checkout file sith 8,296 patterns.\n> Running 'git read-tree -mu HEAD' on this file had the following\n> performance:\n>\n> core.sparseCheckout=false: 0.21 s (0.00 s)\n> core.sparseCheckout=true: 3.75 s (3.50 s)\n> core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nNice!\n\n> The times in parentheses above correspond to the time spent\n> in the first clear_ce_flags() call, according to the trace2\n> performance traces.\n>\n> While this example is contrived, it demonstrates how these\n> patterns can slow the sparse-checkout feature.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 1 -\n> dir.c | 154 +++++++++++++++++++++++++-\n> dir.h | 27 +++++\n> t/t1091-sparse-checkout-builtin.sh | 8 ++\n> 4 files changed, 183 insertions(+), 7 deletions(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index 463319055b..7ade827370 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -85,7 +85,6 @@ negate patterns. For example, to remove the file `unwanted`:\n> !unwanted\n> ----------------\n>\n> -\n> ## CONE PATTERN SET\n>\n> The full pattern set allows for arbitrary pattern matches and complicated\n> diff --git a/dir.c b/dir.c\n> index d021c908e5..2c5ff89a72 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -599,6 +599,99 @@ void parse_exclude_pattern(const char **pattern,\n> *patternlen = len;\n> }\n>\n> +static int el_hashmap_cmp(const void *unused_cmp_data,\n> + const void *a, const void *b, const void *key)\n> +{\n> + const struct exclude_entry *ee1 = a;\n> + const struct exclude_entry *ee2 = b;\n> +\n> + return strncmp(ee1->pattern, ee2->pattern, ee1->patternlen);\n> +}\n> +\n> +static void add_exclude_to_hashsets(struct exclude_list *el, struct exclude *x)\n\nIs this for something that is logically an \"include\" or an \"exclude\"?\nThe earlier use of \"exclude\" for \"include\"s now how me totally confused\nand makes me know I'll have to read the code really carefully and may\nstill get messed up.\n\nI don't know if there's a better way, but re-using .gitignore internals\nfor sparse-checkout stuff leads to this inverted switcheroo and makes\nthings hard to follow. Is it too late or too hard to easily rename the\n\"exclude\" stuff used by gitignore to something else more neutral?\nSigh...\n\n> +{\n> + struct exclude_entry *e;\n> + char *truncated;\n> + char *data = NULL;\n> +\n> + if (!el->use_cone_patterns)\n> + return;\n> +\n> + if (x->patternlen >= 4 &&\n> + !strcmp(x->pattern + x->patternlen - 4, \"/*/*\")) {\n> + if (!(x->flags & EXC_FLAG_NEGATIVE)) {\n> + /* Not a cone pattern. */\n> + el->use_cone_patterns = 0;\n> + warning(_(\"unrecognized pattern: '%s'\"), x->pattern);\n> + goto clear_hashmaps;\n> + }\n> +\n> + truncated = xstrdup(x->pattern);\n> + truncated[x->patternlen - 4] = 0;\n> +\n> + e = xmalloc(sizeof(struct exclude_entry));\n> + e->pattern = truncated;\n> + e->patternlen = x->patternlen - 4;\n> + hashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n> +\n> + if (!hashmap_get(&el->recursive_hashmap, e, NULL)) {\n> + /* We did not see the \"parent\" included */\n> + warning(_(\"unrecognized negative pattern: '%s'\"), x->pattern);\n> + free(truncated);\n> + goto clear_hashmaps;\n> + }\n> +\n> + hashmap_add(&el->parent_hashmap, e);\n> + hashmap_remove(&el->recursive_hashmap, e, &data);\n> + free(data);\n> + return;\n> + }\n> +\n> + if (x->patternlen >= 2 &&\n> + !strcmp(x->pattern + x->patternlen - 2, \"/*\")) {\n> + if (x->flags & EXC_FLAG_NEGATIVE) {\n> + warning(_(\"unrecognized negative pattern: '%s'\"), x->pattern);\n> + goto clear_hashmaps;\n> + }\n> +\n> + e = xmalloc(sizeof(struct exclude_entry));\n> +\n> + truncated = xstrdup(x->pattern);\n> + truncated[x->patternlen - 2] = 0;\n> + e->pattern = truncated;\n> + e->patternlen = x->patternlen - 2;\n> + hashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n> +\n> + hashmap_add(&el->recursive_hashmap, e);\n> +\n> + if (hashmap_get(&el->parent_hashmap, e, NULL)) {\n> + /* we already included this at the parent level */\n> + warning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n> + x->pattern);\n> + hashmap_remove(&el->parent_hashmap, e, &data);\n> + free(data);\n> + }\n> + return;\n> + }\n> +\n> +clear_hashmaps:\n> + hashmap_free(&el->parent_hashmap, 1);\n> + hashmap_free(&el->recursive_hashmap, 1);\n> + el->use_cone_patterns = 0;\n> +}\n> +\n> +static int hashmap_contains_path(struct hashmap *map,\n> + struct strbuf *pattern)\n> +{\n> + struct exclude_entry e;\n> +\n> + /* Check straight mapping */\n> + e.pattern = pattern->buf;\n> + e.patternlen = pattern->len;\n> + hashmap_entry_init(&e, memhash(e.pattern, e.patternlen));\n> + return !!hashmap_get(map, &e, NULL);\n> +}\n> +\n> void add_exclude(const char *string, const char *base,\n> int baselen, struct exclude_list *el, int srcpos)\n> {\n> @@ -623,6 +716,8 @@ void add_exclude(const char *string, const char *base,\n> ALLOC_GROW(el->excludes, el->nr + 1, el->alloc);\n> el->excludes[el->nr++] = x;\n> x->el = el;\n> +\n> + add_exclude_to_hashsets(el, x);\n> }\n>\n> static int read_skip_worktree_file_from_index(const struct index_state *istate,\n> @@ -848,6 +943,10 @@ static int add_excludes_from_buffer(char *buf, size_t size,\n> int i, lineno = 1;\n> char *entry;\n>\n> + el->use_cone_patterns = core_sparse_checkout == SPARSE_CHECKOUT_CONE ? 1 : 0;\n> + hashmap_init(&el->recursive_hashmap, el_hashmap_cmp, NULL, 0);\n> + hashmap_init(&el->parent_hashmap, el_hashmap_cmp, NULL, 0);\n> +\n> el->filebuf = buf;\n>\n> if (skip_utf8_bom(&buf, size))\n> @@ -1070,18 +1169,61 @@ static struct exclude *last_exclude_matching_from_list(const char *pathname,\n>\n> /*\n> * Scan the list and let the last match determine the fate.\n> - * Return 1 for exclude, 0 for include and -1 for undecided.\n> + * Return 0 for exclude, 1 for include and -1 for undecided.\n\nUm...this doesn't make me feel any better about whether we're going to\nrun into bugs about \"include\" vs. \"exclude\".\n\n> */\n> int is_excluded_from_list(const char *pathname,\n> int pathlen, const char *basename, int *dtype,\n> struct exclude_list *el, struct index_state *istate)\n> {\n> struct exclude *exclude;\n> - exclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n> - dtype, el, istate);\n> - if (exclude)\n> - return exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n> - return -1; /* undecided */\n> + struct strbuf parent_pathname = STRBUF_INIT;\n> + int result = 0;\n> + const char *slash_pos;\n> +\n> + if (!el->use_cone_patterns) {\n> + exclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n> + dtype, el, istate);\n> +\n> + if (exclude)\n> + return exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n> +\n> + return -1; /* undecided */\n> + }\n> +\n> + strbuf_addch(&parent_pathname, '/');\n> + strbuf_add(&parent_pathname, pathname, pathlen);\n> + slash_pos = strrchr(parent_pathname.buf, '/');\n> +\n> + if (slash_pos == parent_pathname.buf) {\n> + /* include every file in root */\n> + result = 1;\n> + goto done;\n> + }\n> +\n> + strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n> +\n> + if (hashmap_contains_path(&el->parent_hashmap, &parent_pathname)) {\n> + result = 1;\n> + goto done;\n> + }\n> +\n> + while (parent_pathname.len) {\n> + if (hashmap_contains_path(&el->recursive_hashmap,\n> + &parent_pathname)) {\n> + result = -1;\n> + goto done;\n> + }\n> +\n> + slash_pos = strrchr(parent_pathname.buf, '/');\n> + if (slash_pos == parent_pathname.buf)\n> + break;\n> +\n> + strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n> + }\n> +\n> +done:\n> + strbuf_release(&parent_pathname);\n> + return result;\n> }\n>\n> static struct exclude *last_exclude_matching_from_lists(struct dir_struct *dir,\n> diff --git a/dir.h b/dir.h\n> index 680079bbe3..2d3356d1c0 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -4,6 +4,7 @@\n> /* See Documentation/technical/api-directory-listing.txt */\n>\n> #include \"cache.h\"\n> +#include \"hashmap.h\"\n> #include \"strbuf.h\"\n>\n> struct dir_entry {\n> @@ -37,6 +38,13 @@ struct exclude {\n> int srcpos;\n> };\n>\n> +/* used for hashmaps for cone patterns */\n> +struct exclude_entry {\n> + struct hashmap_entry ent;\n> + char *pattern;\n> + size_t patternlen;\n> +};\n> +\n> /*\n> * Each excludes file will be parsed into a fresh exclude_list which\n> * is appended to the relevant exclude_list_group (either EXC_DIRS or\n> @@ -55,6 +63,25 @@ struct exclude_list {\n> const char *src;\n>\n> struct exclude **excludes;\n> +\n> + /*\n> + * While scanning the excludes, we attempt to match the patterns\n> + * with a more restricted set that allows us to use hashsets for\n> + * matching logic, which is faster than the linear lookup in the\n> + * excludes array above. If non-zero, that check succeeded.\n> + */\n> + unsigned use_cone_patterns;\n> +\n> + /*\n> + * Stores paths where everything starting with those paths\n> + * is included.\n> + */\n> + struct hashmap recursive_hashmap;\n> +\n> + /*\n> + * Used to check single-level parents of blobs.\n> + */\n> + struct hashmap parent_hashmap;\n> };\n>\n> /*\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 8cc377b839..60f10864a1 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -134,6 +134,14 @@ test_expect_success 'cone mode: match patterns' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'cone mode: warn on bad pattern' '\n> + test_when_finished mv sparse-checkout repo/.git/info &&\n> + cp repo/.git/info/sparse-checkout . &&\n> + echo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n> + git -C repo read-tree -mu HEAD 2>err &&\n> + test_i18ngrep \"unrecognized negative pattern\" err\n> +'\n> +\n> test_expect_success 'sparse-checkout disable' '\n> git -C repo sparse-checkout disable &&\n> test_path_is_missing repo/.git/info/sparse-checkout &&\n> --\n> gitgitgadget\n\nSo, uh, I saw the exclude vs. include thing, started scanning to see\nif it was going to get better, saw the next include/exclude thing that\ntriggered a bell, and I more or less just scrolled quickly to the\nbottom checking if anything might catch my eye. So I didn't look at\nthis patch very closely at all. I decided to just start playing with\nthe series instead...and as far as that goes, this patch is probably\nfine because my testing seemed to work.\n"},{"id":"381042","messageId":"CABPp-BH13XbNR3MQKE7cHO5e=pMY7kLtGhkX1SQg_o9it=uUug@mail.gmail.com","threadId":"51691","inReplyTo":"568fda2d0395ae82e8b0c43f6e5757d45b50a508.1566313865.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 9/9] sparse-checkout: init and add in cone mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-24T05:07:55Z","receivedAt":"2019-08-24T05:08:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n wrote:\n>\n\nPerhaps \"sparse-checkout: modify 'init' and 'add' for cone mode\" for\nthe summary?\n\n> From: Derrick Stolee \n>\n> To make the cone pattern set easy to use, update the behavior of\n> 'git sparse-checkout [init|add]'.\n\nMaybe switch these phrases around (starting with \"Update\", ending with \"use.\")\n\n> Add '--cone' flag to 'git sparse-checkout init' to set the config\n> option 'core.sparseCheckout=cone'.\n>\n> When running 'git sparse-checkout add' in cone mode, a user only\n> needs to supply a list of recursive folder matches. Git will\n> automatically add the necessary parent matches for the leading\n> directories.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/sparse-checkout.c | 134 +++++++++++++++++++++++++++--\n> t/t1091-sparse-checkout-builtin.sh | 35 ++++++++\n> 2 files changed, 164 insertions(+), 5 deletions(-)\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 77e5235720..0a4e101ddd 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -6,15 +6,22 @@\n> #include \"repository.h\"\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n> +#include \"string-list.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> N_(\"git sparse-checkout [init|add|list|disable]\"),\n> NULL\n> };\n>\n> +static const char * const builtin_sparse_checkout_init_usage[] = {\n> + N_(\"git sparse-checkout init [--cone]\"),\n> + NULL\n> +};\n> +\n> struct opts_sparse_checkout {\n> const char *subcommand;\n> int read_stdin;\n> + int cone;\n> } opts;\n>\n> static char *get_sparse_checkout_filename(void)\n> @@ -41,6 +48,60 @@ static void write_excludes_to_file(FILE *fp, struct exclude_list *el)\n> }\n> }\n>\n> +static void write_cone_to_file(FILE *fp, struct exclude_list *el)\n> +{\n> + int i;\n> + struct exclude_entry *entry;\n> + struct hashmap_iter iter;\n> + struct string_list sl = STRING_LIST_INIT_DUP;\n> +\n> + hashmap_iter_init(&el->parent_hashmap, &iter);\n> + while ((entry = hashmap_iter_next(&iter))) {\n> + char *pattern = xstrdup(entry->pattern);\n> + char *converted = pattern;\n> + if (pattern[0] == '/')\n> + converted++;\n> + if (pattern[entry->patternlen - 1] == '/')\n> + pattern[entry->patternlen - 1] = 0;\n> + string_list_insert(&sl, converted);\n> + free(pattern);\n> + }\n> +\n> + string_list_sort(&sl);\n\nI was worried that if someone had a directory named '(parenthetical)'\nor '%looks_like_a_comment' that they wanted to include in cone mode,\nthat since '*' sorts after '(' and '%' that entries would appear in\nsparse-checkout in the wrong order. But you are comparing on the\ndirectory name, not the full line that will be written to the\nsparse-checkout file (i.e. without the trailing '/*'), so my worries\nwere unfounded and there's no actual problem here.\n\n> + string_list_remove_duplicates(&sl, 0);\n> +\n> + for (i = 0; i < sl.nr; i++) {\n> + char *pattern = sl.items[i].string;\n> +\n> + if (!strcmp(pattern, \"\"))\n> + fprintf(fp, \"/*\\n!/*/*\\n\");\n> + else\n> + fprintf(fp, \"/%s/*\\n!/%s/*/*\\n\", pattern, pattern);\n> + }\n> +\n> + string_list_clear(&sl, 0);\n> +\n> + hashmap_iter_init(&el->recursive_hashmap, &iter);\n> + while ((entry = hashmap_iter_next(&iter))) {\n> + char *pattern = xstrdup(entry->pattern);\n> + char *converted = pattern;\n> + if (pattern[0] == '/')\n> + converted++;\n> + if (pattern[entry->patternlen - 1] == '/')\n> + pattern[entry->patternlen - 1] = 0;\n> + string_list_insert(&sl, converted);\n> + free(pattern);\n> + }\n> +\n> + string_list_sort(&sl);\n> + string_list_remove_duplicates(&sl, 0);\n> +\n> + for (i = 0; i < sl.nr; i++) {\n> + char *pattern = sl.items[i].string;\n> + fprintf(fp, \"/%s/*\\n\", pattern);\n> + }\n> +}\n> +\n> static int sparse_checkout_list(int argc, const char **argv)\n\nShould the list mode in cone mode be modified to just show the\ndirectories the user added? It seems a little weird to show the\ninternal details of the implementation (all the parent directories and\nnegated entries and whatnot). That's also not in a form that users\ncan pass along to future `sparse-checkout add` invocations.\n\nIn fact, maybe the implementation should be changed? Perhaps we\nshould write out a .git/info/sparse-checkout-cone file instead with\njust the wanted directories (with a trailing slash to permit future\naddition of individual paths without bringing parents along)? It'd be\neasier on the parsing, and older git doesn't work with cone mode\nanyway:\n\n$ /usr/bin/git status\nfatal: bad numeric config value 'cone' for 'core.sparsecheckout': invalid unit\n\nOr, if we need to make old git work reasonably with cone mode, we\ncould write *both* .git/info/sparse-checkout-cone and\n.git/info/sparse-checkout, but the latter begins with a comment\nsomething akin to:\n\n# DO NOT EDIT; auto-generated by 'git sparse-checkout' command and\nonly used by older git clients; see .git/info/sparse-checkout-clone\nfor active settings\n\n> {\n> struct exclude_list el;\n> @@ -141,8 +202,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n> char *sparse_filename;\n> FILE *fp;\n> int res;\n> + enum sparse_checkout_mode mode;\n>\n> - if (sc_set_config(SPARSE_CHECKOUT_FULL))\n> + static struct option builtin_sparse_checkout_init_options[] = {\n> + OPT_BOOL(0, \"cone\", &opts.cone,\n> + N_(\"initialize the sparse-checkout in cone mode\")),\n> + OPT_END(),\n> + };\n> +\n> + argc = parse_options(argc, argv, NULL,\n> + builtin_sparse_checkout_init_options,\n> + builtin_sparse_checkout_init_usage, 0);\n> +\n> + mode = opts.cone ? SPARSE_CHECKOUT_CONE : SPARSE_CHECKOUT_FULL;\n> +\n> + if (sc_set_config(mode))\n> return 1;\n>\n> memset(&el, 0, sizeof(el));\n> @@ -183,6 +257,34 @@ static int sparse_checkout_init(int argc, const char **argv)\n> return sc_read_tree();\n> }\n>\n> +static void insert_recursive_pattern(struct exclude_list *el, struct strbuf *path)\n> +{\n> + struct exclude_entry *e = xmalloc(sizeof(struct exclude_entry));\n> + e->patternlen = path->len;\n> + e->pattern = strbuf_detach(path, NULL);\n> + hashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n> +\n> + hashmap_add(&el->recursive_hashmap, e);\n> +\n> + while (e->patternlen) {\n> + char *slash = strrchr(e->pattern, '/');\n> + char *oldpattern = e->pattern;\n> + size_t newlen;\n> +\n> + if (!slash)\n> + break;\n> +\n> + newlen = slash - e->pattern;\n> + e = xmalloc(sizeof(struct exclude_entry));\n> + e->patternlen = newlen;\n> + e->pattern = xstrndup(oldpattern, newlen);\n> + hashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n> +\n> + if (!hashmap_get(&el->parent_hashmap, e, NULL))\n> + hashmap_add(&el->parent_hashmap, e);\n> + }\n> +}\n> +\n> static int sparse_checkout_add(int argc, const char **argv)\n> {\n> struct exclude_list el;\n> @@ -196,11 +298,33 @@ static int sparse_checkout_add(int argc, const char **argv)\n> add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n>\n> fp = fopen(sparse_filename, \"w\");\n> - write_excludes_to_file(fp, &el);\n>\n> - while (!strbuf_getline(&line, stdin)) {\n> - strbuf_trim(&line);\n> - fprintf(fp, \"%s\\n\", line.buf);\n> + if (core_sparse_checkout == SPARSE_CHECKOUT_FULL) {\n> + write_excludes_to_file(fp, &el);\n> +\n> + while (!strbuf_getline(&line, stdin)) {\n> + strbuf_trim(&line);\n> + fprintf(fp, \"%s\\n\", line.buf);\n> + }\n> + } else if (core_sparse_checkout == SPARSE_CHECKOUT_CONE) {\n> + while (!strbuf_getline(&line, stdin)) {\n> + strbuf_trim(&line);\n> +\n> + strbuf_trim_trailing_dir_sep(&line);\n> +\n> + if (!line.len)\n> + continue;\n> +\n> + if (line.buf[0] == '/')\n> + strbuf_remove(&line, 0, 1);\n> +\n> + if (!line.len)\n> + continue;\n> +\n> + insert_recursive_pattern(&el, &line);\n> + }\n> +\n> + write_cone_to_file(fp, &el);\n> }\n\nNo sanity checking that the user provided directory names rather than\nglobs or regexes? Won't that silently do nothing, suggesting we need\nto warn users? Or are we just assuming that any 'glob' or 'regex'\ncharacters they provide are actually meant as literal characters in a\npathname?\n\nAlso, what if they try to add an exclusion rule for a directory to\nundo their add? e.g.\n echo \"!some/subdir/\" | git sparse-checkout add\n?\n\nIs there a way to undo an add without undoing everything? (Perhaps if\n'git sparse-checkout list' showed just the added directories, it'd be\neasy for users to take that output, delete one they don't want, and\nthen deal with undo and redoing. Is that how we should handle it?)\n\n>\n> fclose(fp);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 60f10864a1..3412bafdff 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -157,4 +157,39 @@ test_expect_success 'sparse-checkout disable' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'cone mode: init and add' '\n> + git -C repo sparse-checkout init --cone &&\n> + git -C repo config --list >config &&\n> + test_i18ngrep \"core.sparsecheckout=cone\" config &&\n> + ls repo >dir &&\n> + echo a >expect &&\n> + test_cmp expect dir &&\n> + echo deep/deeper1/deepest | git -C repo sparse-checkout add &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deep\n> + EOF\n> + ls repo/deep >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deeper1\n> + EOF\n> + ls repo/deep/deeper1 >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deepest\n> + EOF\n> + test_cmp expect dir &&\n> + cat >expect <<-EOF &&\n> + /*\n> + !/*/*\n> + /deep/*\n> + !/deep/*/*\n> + /deep/deeper1/*\n> + !/deep/deeper1/*/*\n> + /deep/deeper1/deepest/*\n> + EOF\n> + test_cmp expect repo/.git/info/sparse-checkout\n> +'\n> test_done\n> \\ No newline at end of file\n> --\n> gitgitgadget\n"},{"id":"381043","messageId":"CABPp-BG8GtdaogbCCZqPXYkhMAqseyAXX0ppjUbg_1hVShGG=Q@mail.gmail.com","threadId":"51691","inReplyTo":"08bd295b-ec5a-3128-42cf-acb24971d679@gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-24T05:40:11Z","receivedAt":"2019-08-24T05:43:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Aug 22, 2019 at 6:10 AM Derrick Stolee wrote:\n>\n> On 8/21/2019 5:52 PM, Elijah Newren wrote:\n> > On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> > wrote:\n\n\n> >> Here are some more specific details:\n> >>\n> >> * git sparse-checkout init enables core.sparseCheckout and populates the\n> >> sparse-checkout file with patterns that match only the files at root.\n> >\n> > Does it enable core.sparseCheckout in the current worktree, or for all\n> > worktrees? Do we require extensions.worktreeConfig to be set to true\n> > first? If we don't require extensions.worktreeConfig to be set to\n> > true, and users add worktrees later, do they encounter negative\n> > surprises (immediately or later)?\n>\n> This is an interesting scenario that I had not considered. Thanks!\n>\n> My guess is that we should set `extensions.worktreeConfig=true` to\n> avoid surprises. I'll need to play with this to discover the answers\n> to these questions:\n>\n> 1. Where does the worktree look for the sparse-checkout file? Does\n> each worktree have its own sparse-checkout file? Should it?\n\nFor the main/first/primary worktree: .git/info/sparse-checkout\nFor all other worktrees: .git/worktrees/$WORKTREE/info/sparse-checkout\n\nSo, yes, each has its own, and from my viewpoint, absolutely yes that\nis what we want.\n\n> 2. If I have `extensions.worktreeConfig=true` and `core.sparseCheckout=true`\n> in the current worktree and run `git worktree add`, does the new worktree\n> have `core.sparseCheckout=true`? Can we `git clone --sparse` and then\n> start building sparse worktrees seamlessly?\n\nMy $0.02: I think `git worktree add` should not only adopt the setting\nof core.sparseCheckout from the current worktree, but it should also\nadopt the $GIT_DIR/info/sparse-checkout file too. Granted, users can\nchange it to something else, but much like a new shell starts up with\nthe same current working directory as its parent shell, I think it'd\nbe most obvious for people to have a worktree that looked similar to\nthe one they launched it from.\n\n\n> > The default of reading from stdin seems a bit unusual to me, and I\n> > worry about having to explain that to users. I'd rather the add\n> > command took positional parameters (anything that doesn't start with a\n> > hyphen) and added those, e.g.\n> > $ git sparse-checkout add '/myFolder/*' '\n> > with the option of the user specifying --stdin.\n>\n> I had the same thought, and likely that's where we should go with the\n> builtin. For our needs, the input over stdin is more important for\n> testing, so I built it first. I will adjust the CLI here to take a set\n> of paths over the arguments unless --stdin is given.\n>\n> >> * git sparse-checkout disable removes the patterns from the sparse-checkout\n> >> file, disables core.sparseCheckout, and refills the working directory.\n> >\n> > Does it leave an empty sparse-checkout file around? Also, what if\n> > users have several paths defining their sparse pattern, and want to\n> > temporarily get a full checkout and then come back -- do they need to\n> > re-specify all the paths? (Maybe this *is* the route we want to go;\n> > I'm just trying to mention any possible negative effects we _might_\n> > run into so we can consider them. It's not quite as relevant in my\n> > case since people specify a few toplevel modules and sparse-checkout\n> > gets several entries auto-generated for them.)\n>\n> In this case, there is an intermediate step (that follows the existing\n> advice) to modify the sparse-checkout file to contain only \"/*\\n\" then\n> run read-tree to fill the working directory before disabling the config\n> setting.\n>\n> Perhaps \"disable\" is the wrong word to use, as it makes you think that\n> there should be an \"enable\" that can quickly toggle between the two\n> modes. Maybe instead it should be \"git sparse-checkout reset [empty|full]\"\n> where you could 'reset' the sparse-checkout to one of two initial\n> states:\n>\n> 1. empty: only files at root are included.\n> 2. full: all files are included.\n>\n> In each case, we would obliterate the existing sparse-checkout entries,\n> but hopefully that behavior is more clear from the command names.\n\nWill \"reset\" be seen as slightly less obvious wording that needs to be\nexplained to users? If so, maybe have \"undo\" and \"empty\" verbs? (Of\ncourse, \"init\" already empties, both when starting from full or when\nwe have some kind of sparse checkout.) I dunno, just some ideas.\n\n> >> In cone mode, a user specifies a list of folders which the user wants every\n> >> file inside. In addition, the cone adds all blobs that are siblings of the\n> >> folders in the directory path to that folder. This makes the directories\n> >> look \"hydrated\" as a user drills down to those recursively-closed folders.\n> >> These directories are called \"parent\" folders, as a file matches them only\n> >> if the file's immediate parent is that directory.\n> >>\n> >> When building a prototype of this feature, I used a separate file to contain\n> >> the list of recursively-closed folders and built the hashsets dynamically\n> >> based on that file. In this implementation, I tried to maximize the amount\n> >> of backwards-compatibility by storing all data in the sparse-checkout file\n> >> using patterns recognized by earlier Git versions.\n> >>\n> >> For example, if we add A/B/C as a recursive folder, then we add the\n> >> following patterns to the sparse-checkout file:\n> >>\n> >> /*\n> >> !/*/*\n> >> /A/*\n> >> !/A/*/*\n> >> /A/B/*\n> >> !/A/B/*/*\n> >> /A/B/C/*\n> >>\n> >> The alternating positive/negative patterns say \"include everything in this\n> >> folder, but exclude everything another level deeper\". The final pattern has\n> >> no matching negation, so is a recursively closed pattern.\n> >\n> > Oh, um, would there be any option for fast but without grabbing\n> > sibling and parent files of requested directories? And could users\n> > still request individual files (not with regex or pathspec, but fully\n> > specifying the path) and still get the fast mode?\n>\n> Exact files could probably be included and still be fast. It requires an\n> extra hash check per entry, but that's a small price to pay I think.\n>\n> With the sibling files, this is something I believe to be user-friendly:\n> as a user drills down into the folder they included recursively, there may\n> be helpful files along the way, like documentation, project files, etc.\n>\n> Here is my philosophical position here: a repo can take advantage of the\n> sparse-checkout feature if it is properly componetized. Those component\n> boundaries are likely at folder boundaries. Any file that exists in a parent\n> folder for two components is likely important to _both_ components. If\n> a file is large and is not needed by both components, it should be placed\n> deeper in the tree, so it can be avoided.\n>\n> With that philosophy in mind, I designed this to help users fall into the\n> \"pit of success\" when their repo is in a good shape AND to motivate users\n> with repos in non-optimal shapes to reorganize.\n>\n> The thought I had about exact file names is similar: if there is a large\n> list of files in a folder where I only need a subset, then how do I know\n> if a new file is added that I need? It will not show up in the directory\n> without updating the sparse-checkout. A user would discover this need by\n> something going wrong when they are not interacting with version control:\n> a build.\n>\n> This is particularly important with the root directory. We need things\n> like .gitignore, .gitattributes, README, LICENSE, etc. to be populated\n> by default. If there are too many files at root to reasonably work with\n> the repo, then the repo should be reorganized using folders.\n>\n> > Basically, our sparse usage is exclusively specifying leading\n> > directories or full pathnames of individual files, but we really want\n> > the repo to feel smaller and make sure people notice at a glance. We\n> > have a huge 'modules/' directory, and want people to be able to get\n> > just 15 of the 500 or so subdirectories that would appear in that\n> > directory with a non-sparse checkout. And similarly we want to be\n> > able to grab just one or two files from a directory of many files.\n>\n> Your modules/ example seems to work with the feature as designed, as\n> you want a set of folders one level deeper. Grabbing one or two files\n> from a directory is a direction we can go with the feature, but I will\n> continue to believe that should be a rare occurrence compared to including\n> a folder recursively.\n\nOh, you're right, I was misunderstanding what it'd do. This does look\nlike it's really close to what we're using, and most of the\ndifferences are probably worth some slightly reshuffling of paths in\nthe repo. Now that I've played with it some, it seems really awesome.\n\nBeing able to grab one or two files from a directory without grabbing\nan entire directory and its parents I think would probably still be\nuseful, but I do agree that it'd be a rare occurrence.\n\n> >> Note that I have some basic warnings to try and check that the\n> >> sparse-checkout file doesn't match what would be written by a cone-mode add.\n> >> In such a case, Git writes a warning to stderr and continues with the old\n> >> pattern matching algorithm. These checks are currently very barebones, and\n> >> would need to be updated with more robust checks for things like regex\n> >> characters in the middle of the pattern. As review moves forward (and if we\n> >> don't change the data storage) then we could spend more time on this.\n> >\n> > Instead of trying to validate the sparse-checkout file everytime,\n> > perhaps we want to change core.sparseCheckout from a boolean to a\n> > tri-state or something where it specifies how to parse the\n> > sparse-checkout file? Or maybe when special directive (some form of\n> > comment-looking line) appears at the top of sparse-checkout then we\n> > use the hashsets speedup while disallowing general regexes and\n> > pathspecs other than leading directories and full pathnames?\n>\n> In this series, I turn `core.sparseCheckout` into a tri-state, and only\n> try to validate the sparse-checkout when `core.sparseCheckout=cone`.\n> This avoids spending time on the validation when someone is content using\n> the existing feature.\n>\n> The _intent_ of using the sparse-checkout file and no extra data structure\n> was to let other clients (or an older client) read the sparse-checkout data\n> and result in the same working directory. One thing I realized after\n> submitting is that the tri-state config variable will cause old clients\n> to error on parsing the non-boolean value. Instead, in v2 I will introduce\n> a new boolean config variable \"core.sparseCheckoutCone\" that will do the\n> same thing as the current series when `core.sparseCheckout=cone` and will\n> fix this compat scenario.\n\nOnce we are forced to use yet another config variable, we may as well\nuse yet another config file ($GITDIR/info/sparse-checkout-cone or\nsomething; or maybe a less specific name with greater future\ncompatibility via some version marking in it).\n\nOne thing I noticed twice while using this series was that when I had\nan existing sparse checkout it was easy to get into a weird state\nwhere things were messed up, I think due to the fact that\n\"sparse-checkout init [--cone]\" prefers to honor any pre-existing\n$GITDIR/info/sparse-checkout file. Once my config file was very much\nnot cone-compatible, and another time it was empty and caused\nread-tree to error out with something like \"there'd be nothing left!\".\nI manually twiddled with core.sparseCheckout and the sparse-checkout\nfile and 'git read-tree -mu HEAD' to get it fixed, but I'd rather\navoid others running into such problems. Sorry I didn't take good\nnotes on it; I was just trying to get a good feel for this series.\n\n> > I'll try to get some time to look over these patches in the next few days.\n>\n> I look forward to your feedback! I also have some feedback to respond to\n> from my team [1], but I'm waiting to make sure the community likes the\n> overall idea before jumping into code style and method organization\n> details.\n\nI think this idea is great; I'm a big fan right now. I'm excited to\nsee how this will pan out.\n"},{"id":"381171","messageId":"c919513a-f41f-2ce8-60ed-e0b0733c0c7f@gmail.com","threadId":"51691","inReplyTo":"CABPp-BG8GtdaogbCCZqPXYkhMAqseyAXX0ppjUbg_1hVShGG=Q@mail.gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-26T13:29:38Z","receivedAt":"2019-08-26T13:29:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/24/2019 1:40 AM, Elijah Newren wrote:\n> On Thu, Aug 22, 2019 at 6:10 AM Derrick Stolee wrote:\n>>\n>> On 8/21/2019 5:52 PM, Elijah Newren wrote:\n>>> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n>>> wrote:\n> \n> \n>>>> Here are some more specific details:\n>>>>\n>>>> * git sparse-checkout init enables core.sparseCheckout and populates the\n>>>> sparse-checkout file with patterns that match only the files at root.\n>>>\n>>> Does it enable core.sparseCheckout in the current worktree, or for all\n>>> worktrees? Do we require extensions.worktreeConfig to be set to true\n>>> first? If we don't require extensions.worktreeConfig to be set to\n>>> true, and users add worktrees later, do they encounter negative\n>>> surprises (immediately or later)?\n>>\n>> This is an interesting scenario that I had not considered. Thanks!\n>>\n>> My guess is that we should set `extensions.worktreeConfig=true` to\n>> avoid surprises. I'll need to play with this to discover the answers\n>> to these questions:\n>>\n>> 1. Where does the worktree look for the sparse-checkout file? Does\n>> each worktree have its own sparse-checkout file? Should it?\n> \n> For the main/first/primary worktree: .git/info/sparse-checkout\n> For all other worktrees: .git/worktrees/$WORKTREE/info/sparse-checkout\n> \n> So, yes, each has its own, and from my viewpoint, absolutely yes that\n> is what we want.\n\nThanks for the info! I will definitely consider this in the next version,\nand include a test to verify the interaction.\n\n>> 2. If I have `extensions.worktreeConfig=true` and `core.sparseCheckout=true`\n>> in the current worktree and run `git worktree add`, does the new worktree\n>> have `core.sparseCheckout=true`? Can we `git clone --sparse` and then\n>> start building sparse worktrees seamlessly?\n> \n> My $0.02: I think `git worktree add` should not only adopt the setting\n> of core.sparseCheckout from the current worktree, but it should also\n> adopt the $GIT_DIR/info/sparse-checkout file too. Granted, users can\n> change it to something else, but much like a new shell starts up with\n> the same current working directory as its parent shell, I think it'd\n> be most obvious for people to have a worktree that looked similar to\n> the one they launched it from.\n\nThis seems natural to me: I'm adding a new worktree and expect the\nsettings to match my current worktree. If we later want to extend\n`git worktree add` to include an `--empty-cone` option (that creates\nthe worktree as if it was created by `git clone --sparse-cone`) we\ncould do that independently.\n\n> \n>>> The default of reading from stdin seems a bit unusual to me, and I\n>>> worry about having to explain that to users. I'd rather the add\n>>> command took positional parameters (anything that doesn't start with a\n>>> hyphen) and added those, e.g.\n>>> $ git sparse-checkout add '/myFolder/*' '\n>>> with the option of the user specifying --stdin.\n>>\n>> I had the same thought, and likely that's where we should go with the\n>> builtin. For our needs, the input over stdin is more important for\n>> testing, so I built it first. I will adjust the CLI here to take a set\n>> of paths over the arguments unless --stdin is given.\n>>\n>>>> * git sparse-checkout disable removes the patterns from the sparse-checkout\n>>>> file, disables core.sparseCheckout, and refills the working directory.\n>>>\n>>> Does it leave an empty sparse-checkout file around? Also, what if\n>>> users have several paths defining their sparse pattern, and want to\n>>> temporarily get a full checkout and then come back -- do they need to\n>>> re-specify all the paths? (Maybe this *is* the route we want to go;\n>>> I'm just trying to mention any possible negative effects we _might_\n>>> run into so we can consider them. It's not quite as relevant in my\n>>> case since people specify a few toplevel modules and sparse-checkout\n>>> gets several entries auto-generated for them.)\n>>\n>> In this case, there is an intermediate step (that follows the existing\n>> advice) to modify the sparse-checkout file to contain only \"/*\\n\" then\n>> run read-tree to fill the working directory before disabling the config\n>> setting.\n>>\n>> Perhaps \"disable\" is the wrong word to use, as it makes you think that\n>> there should be an \"enable\" that can quickly toggle between the two\n>> modes. Maybe instead it should be \"git sparse-checkout reset [empty|full]\"\n>> where you could 'reset' the sparse-checkout to one of two initial\n>> states:\n>>\n>> 1. empty: only files at root are included.\n>> 2. full: all files are included.\n>>\n>> In each case, we would obliterate the existing sparse-checkout entries,\n>> but hopefully that behavior is more clear from the command names.\n> \n> Will \"reset\" be seen as slightly less obvious wording that needs to be\n> explained to users? If so, maybe have \"undo\" and \"empty\" verbs? (Of\n> course, \"init\" already empties, both when starting from full or when\n> we have some kind of sparse checkout.) I dunno, just some ideas.\n\nThanks for pointing out that my word choice could be improved. I'll\nconsider several options.\n\n>>>> In cone mode, a user specifies a list of folders which the user wants every\n>>>> file inside. In addition, the cone adds all blobs that are siblings of the\n>>>> folders in the directory path to that folder. This makes the directories\n>>>> look \"hydrated\" as a user drills down to those recursively-closed folders.\n>>>> These directories are called \"parent\" folders, as a file matches them only\n>>>> if the file's immediate parent is that directory.\n>>>>\n>>>> When building a prototype of this feature, I used a separate file to contain\n>>>> the list of recursively-closed folders and built the hashsets dynamically\n>>>> based on that file. In this implementation, I tried to maximize the amount\n>>>> of backwards-compatibility by storing all data in the sparse-checkout file\n>>>> using patterns recognized by earlier Git versions.\n>>>>\n>>>> For example, if we add A/B/C as a recursive folder, then we add the\n>>>> following patterns to the sparse-checkout file:\n>>>>\n>>>> /*\n>>>> !/*/*\n>>>> /A/*\n>>>> !/A/*/*\n>>>> /A/B/*\n>>>> !/A/B/*/*\n>>>> /A/B/C/*\n>>>>\n>>>> The alternating positive/negative patterns say \"include everything in this\n>>>> folder, but exclude everything another level deeper\". The final pattern has\n>>>> no matching negation, so is a recursively closed pattern.\n>>>\n>>> Oh, um, would there be any option for fast but without grabbing\n>>> sibling and parent files of requested directories? And could users\n>>> still request individual files (not with regex or pathspec, but fully\n>>> specifying the path) and still get the fast mode?\n>>\n>> Exact files could probably be included and still be fast. It requires an\n>> extra hash check per entry, but that's a small price to pay I think.\n>>\n>> With the sibling files, this is something I believe to be user-friendly:\n>> as a user drills down into the folder they included recursively, there may\n>> be helpful files along the way, like documentation, project files, etc.\n>>\n>> Here is my philosophical position here: a repo can take advantage of the\n>> sparse-checkout feature if it is properly componetized. Those component\n>> boundaries are likely at folder boundaries. Any file that exists in a parent\n>> folder for two components is likely important to _both_ components. If\n>> a file is large and is not needed by both components, it should be placed\n>> deeper in the tree, so it can be avoided.\n>>\n>> With that philosophy in mind, I designed this to help users fall into the\n>> \"pit of success\" when their repo is in a good shape AND to motivate users\n>> with repos in non-optimal shapes to reorganize.\n>>\n>> The thought I had about exact file names is similar: if there is a large\n>> list of files in a folder where I only need a subset, then how do I know\n>> if a new file is added that I need? It will not show up in the directory\n>> without updating the sparse-checkout. A user would discover this need by\n>> something going wrong when they are not interacting with version control:\n>> a build.\n>>\n>> This is particularly important with the root directory. We need things\n>> like .gitignore, .gitattributes, README, LICENSE, etc. to be populated\n>> by default. If there are too many files at root to reasonably work with\n>> the repo, then the repo should be reorganized using folders.\n>>\n>>> Basically, our sparse usage is exclusively specifying leading\n>>> directories or full pathnames of individual files, but we really want\n>>> the repo to feel smaller and make sure people notice at a glance. We\n>>> have a huge 'modules/' directory, and want people to be able to get\n>>> just 15 of the 500 or so subdirectories that would appear in that\n>>> directory with a non-sparse checkout. And similarly we want to be\n>>> able to grab just one or two files from a directory of many files.\n>>\n>> Your modules/ example seems to work with the feature as designed, as\n>> you want a set of folders one level deeper. Grabbing one or two files\n>> from a directory is a direction we can go with the feature, but I will\n>> continue to believe that should be a rare occurrence compared to including\n>> a folder recursively.\n> \n> Oh, you're right, I was misunderstanding what it'd do. This does look\n> like it's really close to what we're using, and most of the\n> differences are probably worth some slightly reshuffling of paths in\n> the repo. Now that I've played with it some, it seems really awesome.\n> \n> Being able to grab one or two files from a directory without grabbing\n> an entire directory and its parents I think would probably still be\n> useful, but I do agree that it'd be a rare occurrence.\n\nI think we can leave the file-by-file addition for later, but may need\nto make certain design decisions in this initial version to avoid issues\nwith adding that feature in the future. (Perhaps the recursive-folder input\nshould have \"/\" at the end, to clearly state these are folders, not files.)\n\n>>>> Note that I have some basic warnings to try and check that the\n>>>> sparse-checkout file doesn't match what would be written by a cone-mode add.\n>>>> In such a case, Git writes a warning to stderr and continues with the old\n>>>> pattern matching algorithm. These checks are currently very barebones, and\n>>>> would need to be updated with more robust checks for things like regex\n>>>> characters in the middle of the pattern. As review moves forward (and if we\n>>>> don't change the data storage) then we could spend more time on this.\n>>>\n>>> Instead of trying to validate the sparse-checkout file everytime,\n>>> perhaps we want to change core.sparseCheckout from a boolean to a\n>>> tri-state or something where it specifies how to parse the\n>>> sparse-checkout file? Or maybe when special directive (some form of\n>>> comment-looking line) appears at the top of sparse-checkout then we\n>>> use the hashsets speedup while disallowing general regexes and\n>>> pathspecs other than leading directories and full pathnames?\n>>\n>> In this series, I turn `core.sparseCheckout` into a tri-state, and only\n>> try to validate the sparse-checkout when `core.sparseCheckout=cone`.\n>> This avoids spending time on the validation when someone is content using\n>> the existing feature.\n>>\n>> The _intent_ of using the sparse-checkout file and no extra data structure\n>> was to let other clients (or an older client) read the sparse-checkout data\n>> and result in the same working directory. One thing I realized after\n>> submitting is that the tri-state config variable will cause old clients\n>> to error on parsing the non-boolean value. Instead, in v2 I will introduce\n>> a new boolean config variable \"core.sparseCheckoutCone\" that will do the\n>> same thing as the current series when `core.sparseCheckout=cone` and will\n>> fix this compat scenario.\n> \n> Once we are forced to use yet another config variable, we may as well\n> use yet another config file ($GITDIR/info/sparse-checkout-cone or\n> something; or maybe a less specific name with greater future\n> compatibility via some version marking in it).\n\nI'm hesitant to include a second \"source of truth,\" as that can cause\nissues when users modify the sparse-checkout file directly. Since the\nexisting way to interact with the sparse-checkout is to directly edit\nthe file, I want to be as careful as possible around users who modify\nthat themselves. The caveat is that if they specify \"cone\" mode then\nthey will get warnings and worse performance if they modify it outside\nthe limited patterns we allow.\n\n> One thing I noticed twice while using this series was that when I had\n> an existing sparse checkout it was easy to get into a weird state\n> where things were messed up, I think due to the fact that\n> \"sparse-checkout init [--cone]\" prefers to honor any pre-existing\n> $GITDIR/info/sparse-checkout file. Once my config file was very much\n> not cone-compatible, and another time it was empty and caused\n> read-tree to error out with something like \"there'd be nothing left!\".\n> I manually twiddled with core.sparseCheckout and the sparse-checkout\n> file and 'git read-tree -mu HEAD' to get it fixed, but I'd rather\n> avoid others running into such problems. Sorry I didn't take good\n> notes on it; I was just trying to get a good feel for this series.\n\nThanks for this interesting use case! I think an empty file should be\nupdated with the root files, since Git does not think that is a valid\nstate. The current series must only check for existence, not content.\n \n>>> I'll try to get some time to look over these patches in the next few days.\n>>\n>> I look forward to your feedback! I also have some feedback to respond to\n>> from my team [1], but I'm waiting to make sure the community likes the\n>> overall idea before jumping into code style and method organization\n>> details.\n> \n> I think this idea is great; I'm a big fan right now. I'm excited to\n> see how this will pan out.\n\nThanks! I'll be taking a close look at your patch-by-patch feedback\nthis week and hope to have a non-RFC v2 soon.\n\n-Stolee\n\n"},{"id":"381225","messageId":"CABPp-BFGcuf6jnLndi_s_LGA9M4FAb1_GGOYo=Bxky5pv5CWug@mail.gmail.com","threadId":"51691","inReplyTo":"c919513a-f41f-2ce8-60ed-e0b0733c0c7f@gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-08-26T18:16:19Z","receivedAt":"2019-08-26T18:16:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Aug 26, 2019 at 6:29 AM Derrick Stolee wrote:\n>\n> On 8/24/2019 1:40 AM, Elijah Newren wrote:\n> > On Thu, Aug 22, 2019 at 6:10 AM Derrick Stolee wrote:\n> >>\n> >> On 8/21/2019 5:52 PM, Elijah Newren wrote:\n> >>> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> >>> wrote:\n> >\n\n> >> In this series, I turn `core.sparseCheckout` into a tri-state, and only\n> >> try to validate the sparse-checkout when `core.sparseCheckout=cone`.\n> >> This avoids spending time on the validation when someone is content using\n> >> the existing feature.\n> >>\n> >> The _intent_ of using the sparse-checkout file and no extra data structure\n> >> was to let other clients (or an older client) read the sparse-checkout data\n> >> and result in the same working directory. One thing I realized after\n> >> submitting is that the tri-state config variable will cause old clients\n> >> to error on parsing the non-boolean value. Instead, in v2 I will introduce\n> >> a new boolean config variable \"core.sparseCheckoutCone\" that will do the\n> >> same thing as the current series when `core.sparseCheckout=cone` and will\n> >> fix this compat scenario.\n> >\n> > Once we are forced to use yet another config variable, we may as well\n> > use yet another config file ($GITDIR/info/sparse-checkout-cone or\n> > something; or maybe a less specific name with greater future\n> > compatibility via some version marking in it).\n>\n> I'm hesitant to include a second \"source of truth,\" as that can cause\n> issues when users modify the sparse-checkout file directly. Since the\n> existing way to interact with the sparse-checkout is to directly edit\n> the file, I want to be as careful as possible around users who modify\n> that themselves. The caveat is that if they specify \"cone\" mode then\n> they will get warnings and worse performance if they modify it outside\n> the limited patterns we allow.\n\nWait...does that mean you allow mixing and matching both regular style\nsparse-checkout declarations with cone-mode style declarations within\nthe same file? Are the non-cone mode entries ignored? Does it\nfallback to non-cone mode for all entries? Or does that mean you\nallow checking out both old and new styles of filesets, where you\noptimize the cone-mode style declarations with your hashsets, and have\nthe remaining ones fall back to the old O(N*M) matching? (I think it\ndoes the last of those, right?)\n\nIf you support both, it sounds like you naturally support doing cone\nmode with allowing people to also grab a handful of additional files\nwithout the rest of their directories or parents. It's just that\nfolks who want to do that will ask for a way to turn off any warnings\nyou spew, and if you turn the warnings off, then people who meant to\nget cone behavior but mistyped stuff might complain about no warnings.\nHmm....\n\n(Just trying to think things through out loud; I don't necessarily\nknow what's good or bad here, just voicing what I think might happen.)\n"},{"id":"381239","messageId":"3ca3a292-5fb4-4937-1293-979297ef34b7@gmail.com","threadId":"51691","inReplyTo":"CABPp-BFGcuf6jnLndi_s_LGA9M4FAb1_GGOYo=Bxky5pv5CWug@mail.gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-08-26T19:16:59Z","receivedAt":"2019-08-26T19:17:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2019 2:16 PM, Elijah Newren wrote:\n> On Mon, Aug 26, 2019 at 6:29 AM Derrick Stolee wrote:\n>>\n>> On 8/24/2019 1:40 AM, Elijah Newren wrote:\n>>> On Thu, Aug 22, 2019 at 6:10 AM Derrick Stolee wrote:\n>>>>\n>>>> On 8/21/2019 5:52 PM, Elijah Newren wrote:\n>>>>> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n>>>>> wrote:\n>>>\n> \n>>>> In this series, I turn `core.sparseCheckout` into a tri-state, and only\n>>>> try to validate the sparse-checkout when `core.sparseCheckout=cone`.\n>>>> This avoids spending time on the validation when someone is content using\n>>>> the existing feature.\n>>>>\n>>>> The _intent_ of using the sparse-checkout file and no extra data structure\n>>>> was to let other clients (or an older client) read the sparse-checkout data\n>>>> and result in the same working directory. One thing I realized after\n>>>> submitting is that the tri-state config variable will cause old clients\n>>>> to error on parsing the non-boolean value. Instead, in v2 I will introduce\n>>>> a new boolean config variable \"core.sparseCheckoutCone\" that will do the\n>>>> same thing as the current series when `core.sparseCheckout=cone` and will\n>>>> fix this compat scenario.\n>>>\n>>> Once we are forced to use yet another config variable, we may as well\n>>> use yet another config file ($GITDIR/info/sparse-checkout-cone or\n>>> something; or maybe a less specific name with greater future\n>>> compatibility via some version marking in it).\n>>\n>> I'm hesitant to include a second \"source of truth,\" as that can cause\n>> issues when users modify the sparse-checkout file directly. Since the\n>> existing way to interact with the sparse-checkout is to directly edit\n>> the file, I want to be as careful as possible around users who modify\n>> that themselves. The caveat is that if they specify \"cone\" mode then\n>> they will get warnings and worse performance if they modify it outside\n>> the limited patterns we allow.\n> \n> Wait...does that mean you allow mixing and matching both regular style\n> sparse-checkout declarations with cone-mode style declarations within\n> the same file? Are the non-cone mode entries ignored? Does it\n> fallback to non-cone mode for all entries? Or does that mean you\n> allow checking out both old and new styles of filesets, where you\n> optimize the cone-mode style declarations with your hashsets, and have\n> the remaining ones fall back to the old O(N*M) matching? (I think it\n> does the last of those, right?)\n> \n> If you support both, it sounds like you naturally support doing cone\n> mode with allowing people to also grab a handful of additional files\n> without the rest of their directories or parents. It's just that\n> folks who want to do that will ask for a way to turn off any warnings\n> you spew, and if you turn the warnings off, then people who meant to\n> get cone behavior but mistyped stuff might complain about no warnings.\n> Hmm....\n> \n> (Just trying to think things through out loud; I don't necessarily\n> know what's good or bad here, just voicing what I think might happen.)\n\nThe way I built the current series is that we honor what is in the\nsparse-checkout as historically allowed. Always.\n\nIf a user modifies the sparse-checkout to have patterns that don't match\nthose that are added in cone mode, then Git warns the user this is the\ncase then reverts to the old pattern-by-pattern logic. This is to have\nthe Git client always match what another Git client would expect. (This\ncould be JGit or an older version of Git.) A user could always disable\ncone mode to remove the warning and keep their sparse-checkout in its\ncurrent state.\n\nNote: I have not made the \"non-code-mode pattern\" checks very robust.\nFor instance, I don't check the middle characters for wildcards. This\nneeds to happen at write time, too. The plan is to make these more\nrobust in future versions of the series.\n\nThanks,\n-Stolee\n"},{"id":"381663","messageId":"CAPig+cRJ8cCT4zPJmUmS6mvAhXM1QBZ4LUTtf3Std_45cWR5hA@mail.gmail.com","threadId":"51691","inReplyTo":"CABPp-BG8GtdaogbCCZqPXYkhMAqseyAXX0ppjUbg_1hVShGG=Q@mail.gmail.com","subject":"Re: [PATCH 0/9] [RFC] New sparse-checkout builtin and \"cone\" mode","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-09-02T17:55:45Z","receivedAt":"2019-09-02T17:55:59Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Aug 24, 2019 at 1:40 AM Elijah Newren wrote:\n> My $0.02: I think `git worktree add` should not only adopt the setting\n> of core.sparseCheckout from the current worktree, but it should also\n> adopt the $GIT_DIR/info/sparse-checkout file too.\n\nAs another example in favor of imbuing \"git worktree add\" with\nfirst-class support for this feature (via command-line option and/or\ninheriting existing settings), the commit message of ef2a0ac9a0\n(worktree: add: introduce --checkout option, 2016-03-29) specifically\nsites sparse checkout as the motivation for adding --no-checkout to\n\"git worktree add\".\n"},{"id":"382162","messageId":"7949295b-cda3-fe86-967d-4c070dbd2d5c@gmail.com","threadId":"51691","inReplyTo":"CABPp-BEmwSwg4tgJg6nVG8a3Hpn_g-=ZjApZF4EiJO+qVgu4uw@mail.gmail.com","subject":"Re: [PATCH 2/9] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-11T14:27:38Z","receivedAt":"2019-09-11T14:27:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/23/2019 7:02 PM, Elijah Newren wrote:\n> On Tue, Aug 20, 2019 at 8:13 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> Getting started with a sparse-checkout file can be daunting. Help\n>> users start their sparse enlistment using 'git sparse-checkout init'.\n>> This will set 'core.sparseCheckout=true' in their config, write\n>> an initial set of patterns to the sparse-checkout file, and update\n>> their working directory.\n>>\n>> Using 'git read-tree' to clear directories does not work cleanly\n>> on Windows, so manually delete directories that are tracked by Git\n>> before running read-tree.\n> \n> Is that a bug in read-tree that needs to be fixed?\n\nJust to follow up on this: it turns out that this is NOT a bug in\nread-tree, but rather a side-effect of our custom \"core.gvfs\" config\nsetting. In the virtualized world, we didn't want Git to hard-delete\na folder just because we marked everything sparse.\n\nBy removing that option from my environment, the deletions work as\nexpected.\n\nThanks,\n-Stolee\n\n"},{"id":"382182","messageId":"ab98c292-a1b1-ff47-2e8e-f74b7ade2b94@gmail.com","threadId":"51691","inReplyTo":"CABPp-BEmwSwg4tgJg6nVG8a3Hpn_g-=ZjApZF4EiJO+qVgu4uw@mail.gmail.com","subject":"Re: [PATCH 2/9] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-11T20:28:45Z","receivedAt":"2019-09-11T20:28:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/23/2019 7:02 PM, Elijah Newren wrote:\n> On Tue, Aug 20, 2019 at 8:13 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> +static int sc_read_tree(void)\n>> +{\n>> + struct argv_array argv = ARGV_ARRAY_INIT;\n>> + int result = 0;\n>> + argv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n>> +\n>> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n>> + error(_(\"failed to update index with new sparse-checkout paths\"));\n>> + result = 1;\n>> + }\n> \n> `git read-tree -m -u HEAD` will fail if the index has any higher stage\n> entries in it, even if those higher stage entries correspond to files\n> which are included in the sparseness patterns and thus would not need\n> an update. It might be nice if we can find a way to provide a better\n> error message, and/or implement the read-tree -m -u HEAD internally in\n> a way that will allow us to not fail if the conflicted files are\n> included in the sparse set.\n\nI agree that this is not the _best_ thing to do, but it does mimic the\ncurrent recommendation for a user interacting with sparse-checkout.\n\nI'll rename this helper to something like \"update_working_directory()\"\nso it can be swapped with a different implementation later, after we\nwork out those usability kinks.\n\nThe other thing that is needed here: allow reverting the sparse-checkout\nsettings if this fails. I'll isolate that to a new commit so we can\nexamine that behavior carefully.\n\n> \n>> +\n>> + argv_array_clear(&argv);\n>> + return result;\n>> +}\n>> +\n>> +static int sc_enable_config(void)\n>> +{\n>> + struct argv_array argv = ARGV_ARRAY_INIT;\n>> + int result = 0;\n>> + argv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n> > Why --add? That seems really odd to me.\n\nYeah, that's a mistake. Good find.\n\n> \n> This should also have \"--worktree\". And this function should either\n> set extensions.worktreeConfig to true or die if it isn't already set;\n> not sure which. There's some UI and documentation stuff to figure out\n> here...\n\nI was planning to switch my `git config` subcommand to use in-process\nmethods, but I'm struggling to find a way to ensure we follow the\n`--worktree` option. It likely would work if extensions.worktreeConfig\nwas enabled when the process starts, but adding it in-process likely\ncauses a problem.\n\n> \n>> +\n>> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n>> + error(_(\"failed to enable core.sparseCheckout\"));\n>> + result = 1;\n>> + }\n>> +\n>> + argv_array_clear(&argv);\n>> + return result;\n>> +}\n>> +\n>> +static int delete_directory(const struct object_id *oid, struct strbuf *base,\n>> + const char *pathname, unsigned mode, int stage, void *context)\n>> +{\n>> + struct strbuf dirname = STRBUF_INIT;\n>> + struct stat sb;\n>> +\n>> + strbuf_addstr(&dirname, the_repository->worktree);\n>> + strbuf_addch(&dirname, '/');\n>> + strbuf_addstr(&dirname, pathname);\n>> +\n>> + if (stat(dirname.buf, &sb) || !(sb.st_mode & S_IFDIR))\n>> + return 0;\n>> +\n>> + if (remove_dir_recursively(&dirname, 0))\n> \n> flags = 0 implies not REMOVE_DIR_EMPTY_ONLY. I'm not familiar with\n> remove_dir_recursively(), but won't this delete everything...including\n> untracked files? If so, that sounds like a bug.\nThis whole thing isn't needed any more, since read-tree does the right\nthing.\n\n> \n>> + warning(_(\"failed to remove directory '%s'\"),\n>> + dirname.buf);\n>> +\n>> + strbuf_release(&dirname);\n>> + return 0;\n>> +}\n>> +\n>> +static int sparse_checkout_init(int argc, const char **argv)\n>> +{\n>> + struct tree *t;\n>> + struct object_id oid;\n>> + struct exclude_list el;\n>> + static struct pathspec pathspec;\n>> + char *sparse_filename;\n>> + FILE *fp;\n>> + int res;\n>> +\n>> + if (sc_enable_config())\n>> + return 1;\n>> +\n>> + memset(&el, 0, sizeof(el));\n>> +\n>> + sparse_filename = get_sparse_checkout_filename();\n>> + res = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n> \n> But 'el' isn't used again? Why are we getting the list of files from\n> sparse_filename then?\n\nThis is the only way I could think to check that the sparse-checkout file parses well without just doing the file open myself. Maybe we only need to check if the file exists (and is not empty).\n\n>> +\n>> + /* If we already have a sparse-checkout file, use it. */\n>> + if (res >= 0) {\n>> + free(sparse_filename);\n>> + goto reset_dir;\n>> + }\n>> +\n>> + /* initial mode: all blobs at root */\n>> + fp = fopen(sparse_filename, \"w\");\n>> + free(sparse_filename);\n>> + fprintf(fp, \"/*\\n!/*/*\\n\");\n>> + fclose(fp);\n> \n> Makes sense.\n> \n>> +\n>> + /* remove all directories in the root, if tracked by Git */\n>> + if (get_oid(\"HEAD\", &oid)) {\n>> + /* assume we are in a fresh repo */\n>> + return 0;\n>> + }\n>> +\n>> + t = parse_tree_indirect(&oid);\n>> +\n>> + parse_pathspec(&pathspec, PATHSPEC_ALL_MAGIC &\n>> + ~(PATHSPEC_FROMTOP | PATHSPEC_LITERAL),\n>> + PATHSPEC_PREFER_CWD,\n>> + \"\", NULL);\n>> +\n>> + if (read_tree_recursive(the_repository, t, \"\", 0, 0, &pathspec,\n>> + delete_directory, NULL))\n>> + return 1;\n> \n> Since this is only needed on Windows, as per your commit message,\n> should it be #ifdef'd? Or is this actually a bug that should be fixed\n> in \"git read-tree -mu HEAD\"?\n\n(this will not be needed, but thanks!)\n \n>> +\n>> +reset_dir:\n>> + return sc_read_tree();\n>> +}\n>> +\n> \n> The rest looks fine.\n\nThanks,\n-Stolee\n\n"},{"id":"382550","messageId":"52514598-0c7c-4cc9-187b-8d40c4f5172f@gmail.com","threadId":"51691","inReplyTo":"CABPp-BGyDwgTRx3A4OPJT=5rMM-onkPtg9m58M=rDMHiJQxQ6Q@mail.gmail.com","subject":"Re: [PATCH 3/9] clone: add --sparse mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-18T13:51:46Z","receivedAt":"2019-09-18T13:51:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/23/2019 7:17 PM, Elijah Newren wrote:\n> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When someone wants to clone a large repository, but plans to work\n>> using a sparse-checkout file, they either need to do a full\n>> checkout first and then reduce the patterns they included, or\n>> clone with --no-checkout, set up their patterns, and then run\n>> a checkout manually. This requires knowing a lot about the repo\n>> shape and how sparse-checkout works.\n>>\n>> Add a new '--sparse' option to 'git clone' that initializes the\n>> sparse-checkout file to include the following patterns:\n>>\n>> /*\n>> !/*/*\n>>\n>> These patterns include every file in the root directory, but\n>> no directories. This allows a repo to include files like a\n>> README or a bootstrapping script to grow enlistments from that\n>> point.\n> \n> Nice.\n> \n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-clone.txt | 8 +++++++-\n>> builtin/clone.c | 27 +++++++++++++++++++++++++++\n>> t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n>> 3 files changed, 47 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n>> index 34011c2940..0fe91d2f04 100644\n>> --- a/Documentation/git-clone.txt\n>> +++ b/Documentation/git-clone.txt\n>> @@ -15,7 +15,7 @@ SYNOPSIS\n>> [--dissociate] [--separate-git-dir ]\n>> [--depth ] [--[no-]single-branch] [--no-tags]\n>> [--recurse-submodules[=]] [--[no-]shallow-submodules]\n>> - [--[no-]remote-submodules] [--jobs ] [--] \n>> + [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n>> []\n>>\n>> DESCRIPTION\n>> @@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n>> used, neither remote-tracking branches nor the related\n>> configuration variables are created.\n>>\n>> +--sparse::\n>> + Initialize the sparse-checkout file so the working\n>> + directory starts with only the files in the root\n>> + of the repository. The sparse-checkout file can be\n>> + modified to grow the working directory as needed.\n>> +\n>> --mirror::\n>> Set up a mirror of the source repository. This implies `--bare`.\n>> Compared to `--bare`, `--mirror` not only maps local branches of the\n>> diff --git a/builtin/clone.c b/builtin/clone.c\n>> index f665b28ccc..d6d49a73ff 100644\n>> --- a/builtin/clone.c\n>> +++ b/builtin/clone.c\n>> @@ -60,6 +60,7 @@ static const char *real_git_dir;\n>> static char *option_upload_pack = \"git-upload-pack\";\n>> static int option_verbosity;\n>> static int option_progress = -1;\n>> +static int option_sparse_checkout;\n>> static enum transport_family family;\n>> static struct string_list option_config = STRING_LIST_INIT_NODUP;\n>> static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n>> @@ -147,6 +148,8 @@ static struct option builtin_clone_options[] = {\n>> OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n>> OPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n>> N_(\"any cloned submodules will use their remote-tracking branch\")),\n>> + OPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n>> + N_(\"initialize sparse-checkout file to include only files at root\")),\n>> OPT_END()\n>> };\n>>\n>> @@ -734,6 +737,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n>> }\n>> }\n>>\n>> +static int git_sparse_checkout_init(const char *repo)\n>> +{\n>> + struct argv_array argv = ARGV_ARRAY_INIT;\n>> + int result = 0;\n>> + argv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n>> +\n>> + /*\n>> + * We must apply the setting in the current process\n>> + * for the later checkout to use the sparse-checkout file.\n>> + */\n>> + core_apply_sparse_checkout = 1;\n>> +\n>> + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n>> + error(_(\"failed to initialize sparse-checkout\"));\n>> + result = 1;\n>> + }\n> \n> Sigh...so much forking of additional processes. I'd really rather\n> that we were reducing how much of this we are doing in the codebase\n> instead of adding more. Every fork makes following stuff in a\n> debugger harder.\n\nAt the moment, this is the simplest way to do this interaction. The\ninit subcommand is doing multiple things, and we can consider moving\nthis to be a library method instead of builtin-specific code later.\n\nThis is not a huge performance hit, as \"clone\" is called only once\nper repo.\n\n>> +\n>> + argv_array_clear(&argv);\n>> + return result;\n>> +}\n>> +\n>> static int checkout(int submodule_progress)\n>> {\n>> struct object_id oid;\n>> @@ -1107,6 +1131,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n>> if (option_required_reference.nr || option_optional_reference.nr)\n>> setup_reference();\n>>\n>> + if (option_sparse_checkout && git_sparse_checkout_init(repo))\n>> + return 1;\n>> +\n>> remote = remote_get(option_origin);\n>>\n>> strbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index 35ab84aabd..b7d5f15830 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -87,4 +87,17 @@ test_expect_success 'init with existing sparse-checkout' '\n>> test_cmp expect dir\n>> '\n>>\n>> +test_expect_success 'clone --sparse' '\n>> + git clone --sparse repo clone &&\n>> + git -C clone sparse-checkout list >actual &&\n>> + cat >expect <<-EOF &&\n>> + /*\n>> + !/*/*\n>> + EOF\n>> + test_cmp expect actual &&\n>> + ls clone >dir &&\n>> + echo a >expect &&\n>> + test_cmp expect dir\n> \n> Checking that a toplevel entry is present, but not checking that an\n> entry from a subdir is missing as expected?\n\nThis test is checking that the file \"a\" is the _only_ entry in the root\nof the repo. The directories \"folder1\" and \"folder2\" are not present, since\nwe are comparing the ls output to \"expect\".\n\nThanks,\n-Stolee\n"},{"id":"382551","messageId":"52c5b7cc-eab1-a74d-2abb-be8f125df9bf@gmail.com","threadId":"51691","inReplyTo":"CABPp-BEhc00g3CwKg13wDrtFHM-sw9tSy75UEbTvugKkdwTyeA@mail.gmail.com","subject":"Re: [PATCH 4/9] sparse-checkout: 'add' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-18T13:55:27Z","receivedAt":"2019-09-18T13:55:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/23/2019 7:30 PM, Elijah Newren wrote:\n> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The 'git sparse-checkout add' subcommand takes a list of patterns\n>> over stdin and writes them to the sparse-checkout file. Then, it\n>> updates the working directory using 'git read-tree -mu HEAD'.\n> \n> As mentioned in response to the cover letter, I'd rather see it take\n> patterns as positional arguments (though requiring a '--' argument\n> before any patterns that start with a hyphen). It could also take\n> --stdin to read from stdin.\n> \n>> Note: if a user adds a negative pattern that would lead to the\n>> removal of a non-empty directory, then Git may not delete that\n>> directory (on Windows).\n> \n> This sounds like you're re-iterating a bug mentioned earlier, but if\n> someone in the future comes and reads this comment it might sound like\n> you're saying git can avoid clearing a directory for optimization or\n> other reasons. (And, of course, it'd be nice to figure out why this\n> bug exists.)\n> \n> Another question this brings up, though, is that you worked around\n> this bug in 'init' so why would you not also do so for 'add'? Seems\n> inconsistent to me.\n> \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-sparse-checkout.txt | 4 ++++\n>> builtin/sparse-checkout.c | 32 ++++++++++++++++++++++++++-\n>> t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n>> 3 files changed, 55 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n>> index 50c53ee60a..6f540a3443 100644\n>> --- a/Documentation/git-sparse-checkout.txt\n>> +++ b/Documentation/git-sparse-checkout.txt\n>> @@ -34,6 +34,10 @@ COMMANDS\n>> by Git. Add patterns to the sparse-checkout file to\n>> repopulate the working directory.\n>>\n>> +'add'::\n>> + Add a set of patterns to the sparse-checkout file, as given over\n>> + stdin. Updates the working directory to match the new patterns.\n>> +\n>> SPARSE CHECKOUT\n>> ----------------\n>>\n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index 86d24e6295..ec6134fecc 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -8,7 +8,7 @@\n>> #include \"strbuf.h\"\n>>\n>> static char const * const builtin_sparse_checkout_usage[] = {\n>> - N_(\"git sparse-checkout [init|list]\"),\n>> + N_(\"git sparse-checkout [init|add|list]\"),\n>> NULL\n>> };\n>>\n>> @@ -166,6 +166,34 @@ static int sparse_checkout_init(int argc, const char **argv)\n>> return sc_read_tree();\n>> }\n>>\n>> +static int sparse_checkout_add(int argc, const char **argv)\n>> +{\n>> + struct exclude_list el;\n>> + char *sparse_filename;\n>> + FILE *fp;\n>> + struct strbuf line = STRBUF_INIT;\n>> +\n>> + memset(&el, 0, sizeof(el));\n>> +\n>> + sparse_filename = get_sparse_checkout_filename();\n>> + add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n> \n> el is an exclude_list and we call add_excludes_..., but it's actually\n> an *include* list. This is going to cause errors at some point, and\n> will cause lots of headaches.\n> \n>> +\n>> + fp = fopen(sparse_filename, \"w\");\n>> + write_excludes_to_file(fp, &el);\n>> +\n>> + while (!strbuf_getline(&line, stdin)) {\n>> + strbuf_trim(&line);\n>> + fprintf(fp, \"%s\\n\", line.buf);\n>> + }\n> \n> Should we first check whether these excludes are already in the\n> sparse-checkout file?\n> \n>> + fclose(fp);\n>> + free(sparse_filename);\n>> +\n>> + clear_exclude_list(&el);\n>> +\n>> + return sc_read_tree();\n> \n> What if someone calls 'git sparse-checkout add' without first calling\n> 'git sparse-checkout init'? As far as I can tell, core.sparseCheckout\n> will be unset (i.e. treated as false), meaning that this operation\n> will do some work, but result in no changes and a report of success.\n> After users try to figure out why it won't work, they eventually run\n> 'git sparse-checkout init', which will delete all the entries they\n> previously added with the 'add' subcommand.\n> \n> What should happen instead?\n\nIf someone runs 'git sparse-checkout init' after an 'add', the\nsparse-checkout file has contents, so the init does not overwrite\nthose.\n\n(In the update I'm working on, 'init' doesn't delete folders, so\nit will only remove the files that do not match the patterns.)\n\n>> +}\n>> +\n>> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n>> {\n>> static struct option builtin_sparse_checkout_options[] = {\n>> @@ -187,6 +215,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n>> return sparse_checkout_list(argc, argv);\n>> if (!strcmp(argv[0], \"init\"))\n>> return sparse_checkout_init(argc, argv);\n>> + if (!strcmp(argv[0], \"add\"))\n>> + return sparse_checkout_add(argc, argv);\n>> }\n>>\n>> usage_with_options(builtin_sparse_checkout_usage,\n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index b7d5f15830..499bd8d6d0 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -100,4 +100,24 @@ test_expect_success 'clone --sparse' '\n>> test_cmp expect dir\n>> '\n>>\n>> +test_expect_success 'add to existing sparse-checkout' '\n>> + echo \"/folder2/*\" | git -C repo sparse-checkout add &&\n> \n> I've always been using '/folder2/' in sparse-checkout, without the\n> trailing asterisk. That seems more friendly for cone mode too. Are\n> there benefits to keeping the trailing asterisk?\n\nI think I've been seeing issues with pattern matching on Windows without\nthe trailing asterisk. I'm currently double-checking to make sure this\nis important or not.\n \n>> + cat >expect <<-EOF &&\n>> + /*\n>> + !/*/*\n>> + /folder1/*\n>> + /folder2/*\n>> + EOF\n>> + git -C repo sparse-checkout list >actual &&\n>> + test_cmp expect actual &&\n>> + test_cmp expect repo/.git/info/sparse-checkout &&\n>> + ls repo >dir &&\n>> + cat >expect <<-EOF &&\n>> + a\n>> + folder1\n>> + folder2\n>> + EOF\n>> + test_cmp expect dir\n>> +'\n>> +\n>> test_done\n>> \\ No newline at end of file\n\nI'm trying to fix this newline issue everywhere.\n\nThanks,\n-Stolee\n"},{"id":"382554","messageId":"CABPp-BGcPkbLfDQDBMrG+5PK9o67kq62c1TPiEwLpwZQXcZTCg@mail.gmail.com","threadId":"51691","inReplyTo":"52c5b7cc-eab1-a74d-2abb-be8f125df9bf@gmail.com","subject":"Re: [PATCH 4/9] sparse-checkout: 'add' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-09-18T14:56:26Z","receivedAt":"2019-09-18T14:56:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Sep 18, 2019 at 6:55 AM Derrick Stolee wrote:\n>\n> On 8/23/2019 7:30 PM, Elijah Newren wrote:\n> > On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n> > wrote:\n> >>\n...\n> >> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> >> index b7d5f15830..499bd8d6d0 100755\n> >> --- a/t/t1091-sparse-checkout-builtin.sh\n> >> +++ b/t/t1091-sparse-checkout-builtin.sh\n> >> @@ -100,4 +100,24 @@ test_expect_success 'clone --sparse' '\n> >> test_cmp expect dir\n> >> '\n> >>\n> >> +test_expect_success 'add to existing sparse-checkout' '\n> >> + echo \"/folder2/*\" | git -C repo sparse-checkout add &&\n> >\n> > I've always been using '/folder2/' in sparse-checkout, without the\n> > trailing asterisk. That seems more friendly for cone mode too. Are\n> > there benefits to keeping the trailing asterisk?\n>\n> I think I've been seeing issues with pattern matching on Windows without\n> the trailing asterisk. I'm currently double-checking to make sure this\n> is important or not.\n\nCan you try with the en/clean-nested-with-ignored topic in pu to see\nif that fixes those issues?\n"},{"id":"382561","messageId":"d0bb29ef-6702-06b6-2369-a49817a0faa1@gmail.com","threadId":"51691","inReplyTo":"CABPp-BGcPkbLfDQDBMrG+5PK9o67kq62c1TPiEwLpwZQXcZTCg@mail.gmail.com","subject":"Re: [PATCH 4/9] sparse-checkout: 'add' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-18T17:23:49Z","receivedAt":"2019-09-18T17:23:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/18/2019 10:56 AM, Elijah Newren wrote:\n> On Wed, Sep 18, 2019 at 6:55 AM Derrick Stolee wrote:\n>>\n>> On 8/23/2019 7:30 PM, Elijah Newren wrote:\n>>> On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget\n>>> wrote:\n>>>>\n> ...\n>>>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>>>> index b7d5f15830..499bd8d6d0 100755\n>>>> --- a/t/t1091-sparse-checkout-builtin.sh\n>>>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>>>> @@ -100,4 +100,24 @@ test_expect_success 'clone --sparse' '\n>>>> test_cmp expect dir\n>>>> '\n>>>>\n>>>> +test_expect_success 'add to existing sparse-checkout' '\n>>>> + echo \"/folder2/*\" | git -C repo sparse-checkout add &&\n>>>\n>>> I've always been using '/folder2/' in sparse-checkout, without the\n>>> trailing asterisk. That seems more friendly for cone mode too. Are\n>>> there benefits to keeping the trailing asterisk?\n>>\n>> I think I've been seeing issues with pattern matching on Windows without\n>> the trailing asterisk. I'm currently double-checking to make sure this\n>> is important or not.\n> \n> Can you try with the en/clean-nested-with-ignored topic in pu to see\n> if that fixes those issues?\n\nMerging with that branch was very difficult. There is a lot of unshared\nhistory between our branches.\n\nInstead, I tried once more to dig into the strange issue on Windows, and\nit appears it is an issue with how the Git for Windows SDK modifies shell\narguments with a \"/\".\n\nWhen I ran `git sparse-checkout set \"/folder1/*\"` it worked.\n\nWhen I run `git sparse-checkout set \"/folder1/\"`, the SDK completes that\nargument to \"C:/git-sdk-64/folder1/\" on my machine (something more\ncomplicated on the build machine). It's not actually a bug in the Git\ncode, but something in the build and test environment.\n\nI can get around it by testing the builtin without using these cone-like\npatterns. When using `git sparse-checkout set folder1 folder2` in cone\nmode, Git does the right thing.\n\nSorry for the noise here.\n\n-Stolee\n\n"},{"id":"382583","messageId":"dbaf3de88e036fb23d7760582030726f32581a4d.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 01/11] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:10Z","receivedAt":"2019-09-19T14:43:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nThe `$GIT_DIR/info/sparse-checkout` file defines the skip-\nworktree reference bitmap. When Git updates the working\ndirectory, it updates the skip-worktree bits in the index\nbased on this file and removes or restores files in the\nworking copy to match.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na more restricted pattern set.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 90 +++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 86 +++++++++++++++++++++++++\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 51 +++++++++++++++\n 8 files changed, 232 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 4470d7cfc0..5ccc3d00dd 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -156,6 +156,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..cdef451642\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,90 @@\n+git-sparse-checkout(1)\n+=======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of directories\n+given by a list of prefixes.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of directories given by a list of prefixes.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+----------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it resets the skip-worktree bit in the index based on this\n+file. If an entry\n+matches a pattern in this file, skip-worktree will not be set on\n+that entry. Otherwise, skip-worktree will be set.\n+\n+Then it compares the new skip-worktree value with the previous one. If\n+skip-worktree turns from set to unset, it will add the corresponding\n+file back. If it turns from unset to set, that file will be removed.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are included, you can also specify what files are _not_ included,\n+using negative patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+read-tree' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f58bf14c7b..f3322b75dd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1121,6 +1121,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex ec7e0954c4..d517068faa 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -223,6 +223,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..eed9625a05\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,86 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout [list]\"),\n+\tNULL\n+};\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pl->nr; i++) {\n+\t\tstruct path_pattern *p = pl->patterns[i];\n+\n+\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", p->pattern);\n+\n+\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_patterns_to_file(stdout, &pl);\n+\tclear_pattern_list(&pl);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/git.c b/git.c\nindex c2eec470c9..e775fbad42 100644\n--- a/git.c\n+++ b/git.c\n@@ -576,6 +576,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..46e7b2dded\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,51 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_line_count = 0 list &&\n+\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tgit -C repo sparse-checkout list >list &&\n+\tcat >expect <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\ttest_cmp expect list\n+'\n+\n+test_done\n+\n-- \ngitgitgadget\n\n"},{"id":"382584","messageId":"412211f5dd6d4d995f258403bf377bc0cb6332b4.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 02/11] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:11Z","receivedAt":"2019-09-19T14:43:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nUsing 'git read-tree' to clear directories does not work cleanly\non Windows, so manually delete directories that are tracked by Git\nbefore running read-tree.\n\nThe use of running another process for 'git read-tree' is likely\nsuboptimal, but that can be improved in a later change, if valuable.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 7 +++\n builtin/sparse-checkout.c | 69 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++++\n 3 files changed, 116 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex cdef451642..9707ef93b1 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -26,6 +26,13 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n \n SPARSE CHECKOUT\n ----------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex eed9625a05..895479970d 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [list]\"),\n+\tN_(\"git sparse-checkout [init|list]\"),\n \tNULL\n };\n \n@@ -59,6 +59,71 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int update_working_directory(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+static int sc_enable_config(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n+\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n+\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n+\t\treturn 1;\n+\t}\n+\n+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (sc_enable_config())\n+\t\treturn 1;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\tfclose(fp);\n+\n+reset_dir:\n+\treturn update_working_directory();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -79,6 +144,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 46e7b2dded..a6c6b336c9 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -47,5 +47,46 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382585","messageId":"fef41b794a9886664616ce5e5c7902a82a474c2d.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 03/11] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:12Z","receivedAt":"2019-09-19T14:43:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nDuring the 'git sparse-checkout init' call, we must first look\nto see if HEAD is valid, or else we will fail while trying to\nupdate the working directory. The first checkout will actually\nupdate the working directory correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n builtin/sparse-checkout.c | 6 ++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 4 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 5fc97f14de..03299a8adb 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a693e6ca44..16f4e8b6fd 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -58,6 +58,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -145,6 +146,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -723,6 +726,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1096,6 +1120,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 895479970d..656e6ebdd5 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -99,6 +99,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tstruct object_id oid;\n \n \tif (sc_enable_config())\n \t\treturn 1;\n@@ -120,6 +121,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \tfclose(fp);\n \n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n reset_dir:\n \treturn update_working_directory();\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a6c6b336c9..26b4ce9acd 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -88,5 +88,18 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382586","messageId":"pull.316.v2.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.git.gitgitgadget@gmail.com","subject":"[PATCH v2 00/11] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:09Z","receivedAt":"2019-09-19T14:43:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series makes the sparse-checkout feature more user-friendly. While\nthere, I also present a way to use a limited set of patterns to gain a\nsignificant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis series attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout set reads patterns from the arguments, or with\n --stdin reads patterns from stdin one per line, then writes them to the\n sparse-checkout file and refreshes the working directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\nUpdates in v2, relative to the RFC:\n\n * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n adding 'add' and/or 'remove' subcommands later.\n \n \n * 'set' reads from the arguments by default. '--stdin' option is available.\n \n \n * A new performance-oriented commit is added at the end.\n \n \n * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n pattern.\n \n \n * References to a \"bug\" (that was really a strange GVFS interaction in\n microsoft/git) around deleting outside the cone are removed.\n \n \n\nThings to leave for future patches:\n\n 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n worktree-specific file.\n \n \n 2. More robustness around detecting non-cone patterns with wildcards in the\n middle of the line.\n \n \n 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n super-valuable, as it only starts changing behavior when someone calls\n 'git sparse-checkout set', but may be interesting.\n \n \n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (10):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'set' subcommand\n sparse-checkout: add '--stdin' option to set subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and set in cone mode\n unpack-trees: hash less in cone mode\n\nJeff Hostetler (1):\n trace2: add region in clear_ce_flags\n\n .gitignore | 1 +\n Documentation/config/core.txt | 7 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 148 ++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/sparse-checkout.c | 395 ++++++++++++++++++++++++++\n cache.h | 4 +-\n config.c | 5 +\n dir.c | 173 ++++++++++-\n dir.h | 31 ++\n environment.c | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 231 +++++++++++++++\n unpack-trees.c | 48 ++--\n 17 files changed, 1055 insertions(+), 29 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 468ce99b77a0efaf1ace4c31a7b0a7d036fd9ca1\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n\nRange-diff vs v1:\n\n 1: c37b5f2c29 ! 1: dbaf3de88e sparse-checkout: create builtin with 'list' subcommand\n @@ -15,16 +15,18 @@\n builtin will be the preferred mechanism for manipulating the\n sparse-checkout file and syncing the working directory.\n \n - For now, create the basics of the builtin. Includes a single\n - subcommand, \"git sparse-checkout list\", that lists the patterns\n - currently in the sparse-checkout file. Test that these patterns\n - are parsed and written correctly to the output.\n + The `$GIT_DIR/info/sparse-checkout` file defines the skip-\n + worktree reference bitmap. When Git updates the working\n + directory, it updates the skip-worktree bits in the index\n + based on this file and removes or restores files in the\n + working copy to match.\n \n The documentation provided is adapted from the \"git read-tree\"\n documentation with a few edits for clarity in the new context.\n Extra sections are added to hint toward a future change to\n - a moer restricted pattern set.\n + a more restricted pattern set.\n \n + Helped-by: Elijah Newren \n Signed-off-by: Derrick Stolee \n \n diff --git a/.gitignore b/.gitignore\n @@ -114,8 +116,8 @@\n +files.\n +\n +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n -+files are in, you can also specify what files are _not_ in, using\n -+negate patterns. For example, to remove the file `unwanted`:\n ++files are included, you can also specify what files are _not_ included,\n ++using negative patterns. For example, to remove the file `unwanted`:\n +\n +----------------\n +/*\n @@ -191,29 +193,24 @@\n +\tNULL\n +};\n +\n -+struct opts_sparse_checkout {\n -+\tconst char *subcommand;\n -+\tint read_stdin;\n -+} opts;\n -+\n +static char *get_sparse_checkout_filename(void)\n +{\n +\treturn git_pathdup(\"info/sparse-checkout\");\n +}\n +\n -+static void write_excludes_to_file(FILE *fp, struct exclude_list *el)\n ++static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n +{\n +\tint i;\n +\n -+\tfor (i = 0; i < el->nr; i++) {\n -+\t\tstruct exclude *x = el->excludes[i];\n ++\tfor (i = 0; i < pl->nr; i++) {\n ++\t\tstruct path_pattern *p = pl->patterns[i];\n +\n -+\t\tif (x->flags & EXC_FLAG_NEGATIVE)\n ++\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n +\t\t\tfprintf(fp, \"!\");\n +\n -+\t\tfprintf(fp, \"%s\", x->pattern);\n ++\t\tfprintf(fp, \"%s\", p->pattern);\n +\n -+\t\tif (x->flags & EXC_FLAG_MUSTBEDIR)\n ++\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n +\t\t\tfprintf(fp, \"/\");\n +\n +\t\tfprintf(fp, \"\\n\");\n @@ -222,23 +219,23 @@\n +\n +static int sparse_checkout_list(int argc, const char **argv)\n +{\n -+\tstruct exclude_list el;\n ++\tstruct pattern_list pl;\n +\tchar *sparse_filename;\n +\tint res;\n +\n -+\tmemset(&el, 0, sizeof(el));\n ++\tmemset(&pl, 0, sizeof(pl));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n -+\tres = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n ++\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n +\tfree(sparse_filename);\n +\n +\tif (res < 0) {\n -+\t\twarning(_(\"failed to parse sparse-checkout file; it may not exist\"));\n ++\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n +\t\treturn 0;\n +\t}\n +\n -+\twrite_excludes_to_file(stdout, &el);\n -+\tclear_exclude_list(&el);\n ++\twrite_patterns_to_file(stdout, &pl);\n ++\tclear_pattern_list(&pl);\n +\n +\treturn 0;\n +}\n @@ -253,12 +250,13 @@\n +\t\tusage_with_options(builtin_sparse_checkout_usage,\n +\t\t\t\t builtin_sparse_checkout_options);\n +\n -+\tgit_config(git_default_config, NULL);\n +\targc = parse_options(argc, argv, prefix,\n +\t\t\t builtin_sparse_checkout_options,\n +\t\t\t builtin_sparse_checkout_usage,\n +\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n +\n ++\tgit_config(git_default_config, NULL);\n ++\n +\tif (argc > 0) {\n +\t\tif (!strcmp(argv[0], \"list\"))\n +\t\t\treturn sparse_checkout_list(argc, argv);\n @@ -313,7 +311,7 @@\n +test_expect_success 'git sparse-checkout list (empty)' '\n +\tgit -C repo sparse-checkout list >list 2>err &&\n +\ttest_line_count = 0 list &&\n -+\ttest_i18ngrep \"failed to parse sparse-checkout file; it may not exist\" err\n ++\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n +'\n +\n +test_expect_success 'git sparse-checkout list (populated)' '\n @@ -335,4 +333,4 @@\n +'\n +\n +test_done\n - \\ No newline at end of file\n ++\n 2: e6e982e5a6 ! 2: 412211f5dd sparse-checkout: create 'init' subcommand\n @@ -51,7 +51,7 @@\n \treturn 0;\n }\n \n -+static int sc_read_tree(void)\n ++static int update_working_directory(void)\n +{\n +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n +\tint result = 0;\n @@ -69,45 +69,25 @@\n +static int sc_enable_config(void)\n +{\n +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n -+\tint result = 0;\n -+\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n +\n -+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n -+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n -+\t\tresult = 1;\n ++\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n ++\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n ++\t\treturn 1;\n +\t}\n +\n -+\targv_array_clear(&argv);\n -+\treturn result;\n -+}\n -+\n -+static int delete_directory(const struct object_id *oid, struct strbuf *base,\n -+\t\tconst char *pathname, unsigned mode, int stage, void *context)\n -+{\n -+\tstruct strbuf dirname = STRBUF_INIT;\n -+\tstruct stat sb;\n -+\n -+\tstrbuf_addstr(&dirname, the_repository->worktree);\n -+\tstrbuf_addch(&dirname, '/');\n -+\tstrbuf_addstr(&dirname, pathname);\n -+\n -+\tif (stat(dirname.buf, &sb) || !(sb.st_mode & S_IFDIR))\n -+\t\treturn 0;\n ++\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n +\n -+\tif (remove_dir_recursively(&dirname, 0))\n -+\t\twarning(_(\"failed to remove directory '%s'\"),\n -+\t\t\tdirname.buf);\n ++\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n ++\t\terror(_(\"failed to enable core.sparseCheckout\"));\n ++\t\treturn 1;\n ++\t}\n +\n -+\tstrbuf_release(&dirname);\n +\treturn 0;\n +}\n +\n +static int sparse_checkout_init(int argc, const char **argv)\n +{\n -+\tstruct tree *t;\n -+\tstruct object_id oid;\n -+\tstruct exclude_list el;\n -+\tstatic struct pathspec pathspec;\n ++\tstruct pattern_list pl;\n +\tchar *sparse_filename;\n +\tFILE *fp;\n +\tint res;\n @@ -115,10 +95,10 @@\n +\tif (sc_enable_config())\n +\t\treturn 1;\n +\n -+\tmemset(&el, 0, sizeof(el));\n ++\tmemset(&pl, 0, sizeof(pl));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n -+\tres = add_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n ++\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n +\n +\t/* If we already have a sparse-checkout file, use it. */\n +\tif (res >= 0) {\n @@ -129,28 +109,11 @@\n +\t/* initial mode: all blobs at root */\n +\tfp = fopen(sparse_filename, \"w\");\n +\tfree(sparse_filename);\n -+\tfprintf(fp, \"/*\\n!/*/*\\n\");\n ++\tfprintf(fp, \"/*\\n!/*/\\n\");\n +\tfclose(fp);\n +\n -+\t/* remove all directories in the root, if tracked by Git */\n -+\tif (get_oid(\"HEAD\", &oid)) {\n -+\t\t/* assume we are in a fresh repo */\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tt = parse_tree_indirect(&oid);\n -+\n -+\tparse_pathspec(&pathspec, PATHSPEC_ALL_MAGIC &\n -+\t\t\t\t ~(PATHSPEC_FROMTOP | PATHSPEC_LITERAL),\n -+\t\t PATHSPEC_PREFER_CWD,\n -+\t\t \"\", NULL);\n -+\n -+\tif (read_tree_recursive(the_repository, t, \"\", 0, 0, &pathspec,\n -+\t\t\t\tdelete_directory, NULL))\n -+\t\treturn 1;\n -+\n +reset_dir:\n -+\treturn sc_read_tree();\n ++\treturn update_working_directory();\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n @@ -177,7 +140,7 @@\n +\tgit -C repo sparse-checkout init &&\n +\tcat >expect <<-EOF &&\n +\t\t/*\n -+\t\t!/*/*\n ++\t\t!/*/\n +\tEOF\n +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n +\tgit -C repo config --list >config &&\n @@ -191,27 +154,28 @@\n +\tgit -C repo sparse-checkout list >actual &&\n +\tcat >expect <<-EOF &&\n +\t\t/*\n -+\t\t!/*/*\n ++\t\t!/*/\n +\tEOF\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'init with existing sparse-checkout' '\n -+\techo \"/folder1/*\" >> repo/.git/info/sparse-checkout &&\n ++\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n +\tgit -C repo sparse-checkout init &&\n +\tcat >expect <<-EOF &&\n +\t\t/*\n -+\t\t!/*/*\n -+\t\t/folder1/*\n ++\t\t!/*/\n ++\t\t*folder*\n +\tEOF\n +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n +\tls repo >dir &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n +\t\tfolder1\n ++\t\tfolder2\n +\tEOF\n +\ttest_cmp expect dir\n +'\n +\n test_done\n - \\ No newline at end of file\n + \n 3: 4ccd36b396 ! 3: fef41b794a clone: add --sparse mode\n @@ -13,13 +13,18 @@\n sparse-checkout file to include the following patterns:\n \n /*\n - !/*/*\n + !/*/\n \n These patterns include every file in the root directory, but\n no directories. This allows a repo to include files like a\n README or a bootstrapping script to grow enlistments from that\n point.\n \n + During the 'git sparse-checkout init' call, we must first look\n + to see if HEAD is valid, or else we will fail while trying to\n + update the working directory. The first checkout will actually\n + update the working directory correctly.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\n @@ -107,6 +112,30 @@\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\n \n + diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n + --- a/builtin/sparse-checkout.c\n + +++ b/builtin/sparse-checkout.c\n +@@\n + \tchar *sparse_filename;\n + \tFILE *fp;\n + \tint res;\n ++\tstruct object_id oid;\n + \n + \tif (sc_enable_config())\n + \t\treturn 1;\n +@@\n + \tfprintf(fp, \"/*\\n!/*/\\n\");\n + \tfclose(fp);\n + \n ++\tif (get_oid(\"HEAD\", &oid)) {\n ++\t\t/* assume we are in a fresh repo */\n ++\t\treturn 0;\n ++\t}\n ++\n + reset_dir:\n + \treturn update_working_directory();\n + }\n +\n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@ -119,7 +148,7 @@\n +\tgit -C clone sparse-checkout list >actual &&\n +\tcat >expect <<-EOF &&\n +\t\t/*\n -+\t\t!/*/*\n ++\t\t!/*/\n +\tEOF\n +\ttest_cmp expect actual &&\n +\tls clone >dir &&\n @@ -128,4 +157,4 @@\n +'\n +\n test_done\n - \\ No newline at end of file\n + \n 4: 0f095e85d5 ! 4: 9a78f9ea0f sparse-checkout: 'add' subcommand\n @@ -1,14 +1,15 @@\n Author: Derrick Stolee \n \n - sparse-checkout: 'add' subcommand\n + sparse-checkout: 'set' subcommand\n \n - The 'git sparse-checkout add' subcommand takes a list of patterns\n - over stdin and writes them to the sparse-checkout file. Then, it\n + The 'git sparse-checkout set' subcommand takes a list of patterns\n + as arguments and writes them to the sparse-checkout file. Then, it\n updates the working directory using 'git read-tree -mu HEAD'.\n \n - Note: if a user adds a negative pattern that would lead to the\n - removal of a non-empty directory, then Git may not delete that\n - directory (on Windows).\n + The 'set' subcommand will replace the entire contents of the\n + sparse-checkout file. The write_patterns_and_update() method is\n + extracted from cmd_sparse_checkout() to make it easier to implement\n + 'add' and/or 'remove' subcommands in the future.\n \n Signed-off-by: Derrick Stolee \n \n @@ -19,9 +20,10 @@\n \tby Git. Add patterns to the sparse-checkout file to\n \trepopulate the working directory.\n \n -+'add'::\n -+\tAdd a set of patterns to the sparse-checkout file, as given over\n -+\tstdin. Updates the working directory to match the new patterns.\n ++'set'::\n ++\tWrite a set of patterns to the sparse-checkout file, as given as\n ++\ta list of arguments following the 'set' subcommand. Update the\n ++\tworking directory to match the new patterns.\n +\n SPARSE CHECKOUT\n ----------------\n @@ -35,40 +37,39 @@\n \n static char const * const builtin_sparse_checkout_usage[] = {\n -\tN_(\"git sparse-checkout [init|list]\"),\n -+\tN_(\"git sparse-checkout [init|add|list]\"),\n ++\tN_(\"git sparse-checkout [init|list|set] \"),\n \tNULL\n };\n \n @@\n - \treturn sc_read_tree();\n + \treturn update_working_directory();\n }\n \n -+static int sparse_checkout_add(int argc, const char **argv)\n ++static int write_patterns_and_update(struct pattern_list *pl)\n +{\n -+\tstruct exclude_list el;\n +\tchar *sparse_filename;\n +\tFILE *fp;\n -+\tstruct strbuf line = STRBUF_INIT;\n -+\n -+\tmemset(&el, 0, sizeof(el));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n -+\tadd_excludes_from_file_to_list(sparse_filename, \"\", 0, &el, NULL);\n -+\n +\tfp = fopen(sparse_filename, \"w\");\n -+\twrite_excludes_to_file(fp, &el);\n -+\n -+\twhile (!strbuf_getline(&line, stdin)) {\n -+\t\tstrbuf_trim(&line);\n -+\t\tfprintf(fp, \"%s\\n\", line.buf);\n -+\t}\n -+\n ++\twrite_patterns_to_file(fp, pl);\n +\tfclose(fp);\n +\tfree(sparse_filename);\n +\n -+\tclear_exclude_list(&el);\n ++\tclear_pattern_list(pl);\n ++\treturn update_working_directory();\n ++}\n +\n -+\treturn sc_read_tree();\n ++static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n ++{\n ++\tint i;\n ++\tstruct pattern_list pl;\n ++\tmemset(&pl, 0, sizeof(pl));\n ++\n ++\tfor (i = 1; i < argc; i++)\n ++\t\tadd_pattern(argv[i], NULL, 0, &pl, 0);\n ++\n ++\treturn write_patterns_and_update(&pl);\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n @@ -78,8 +79,8 @@\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n -+\t\tif (!strcmp(argv[0], \"add\"))\n -+\t\t\treturn sparse_checkout_add(argc, argv);\n ++\t\tif (!strcmp(argv[0], \"set\"))\n ++\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\n @@ -91,13 +92,12 @@\n \ttest_cmp expect dir\n '\n \n -+test_expect_success 'add to existing sparse-checkout' '\n -+\techo \"/folder2/*\" | git -C repo sparse-checkout add &&\n ++test_expect_success 'set sparse-checkout using builtin' '\n ++\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n +\tcat >expect <<-EOF &&\n +\t\t/*\n -+\t\t!/*/*\n -+\t\t/folder1/*\n -+\t\t/folder2/*\n ++\t\t!/*/\n ++\t\t*folder*\n +\tEOF\n +\tgit -C repo sparse-checkout list >actual &&\n +\ttest_cmp expect actual &&\n @@ -112,4 +112,4 @@\n +'\n +\n test_done\n - \\ No newline at end of file\n + \n -: ---------- > 5: 21a0165be7 sparse-checkout: add '--stdin' option to set subcommand\n 5: 5f332b799f ! 6: b62b76013f sparse-checkout: create 'disable' subcommand\n @@ -13,8 +13,8 @@\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n - \tAdd a set of patterns to the sparse-checkout file, as given over\n - \tstdin. Updates the working directory to match the new patterns.\n + \ta list of arguments following the 'set' subcommand. Update the\n + \tworking directory to match the new patterns.\n \n +'disable'::\n +\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n @@ -30,7 +30,7 @@\n +To repopulate the working directory with all files, use the\n +`git sparse-checkout disable` command.\n +\n -+Sparse checkout support in 'git read-tree' and similar commands is\n ++Sparse checkout support in 'git checkout' and similar commands is\n +disabled by default. You need to set `core.sparseCheckout` to `true`\n +in order to have sparse checkout support.\n +\n @@ -67,8 +67,8 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n --\tN_(\"git sparse-checkout [init|add|list]\"),\n -+\tN_(\"git sparse-checkout [init|add|list|disable]\"),\n +-\tN_(\"git sparse-checkout [init|list|set] \"),\n ++\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n \tNULL\n };\n \n @@ -80,36 +80,32 @@\n +static int sc_set_config(int mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n - \tint result = 0;\n --\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", \"true\", NULL);\n -+\targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n + \n +@@\n + \t\treturn 1;\n + \t}\n + \n +-\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n ++\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n +\n -+\tswitch (mode) {\n -+\tcase 1:\n ++\tif (mode)\n +\t\targv_array_pushl(&argv, \"true\", NULL);\n -+\t\tbreak;\n -+\n -+\tcase 0:\n ++\telse\n +\t\targv_array_pushl(&argv, \"false\", NULL);\n -+\t\tbreak;\n -+\n -+\tdefault:\n -+\t\tdie(_(\"invalid config mode\"));\n -+\t}\n \n \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n \t\terror(_(\"failed to enable core.sparseCheckout\"));\n @@\n - \tFILE *fp;\n \tint res;\n + \tstruct object_id oid;\n \n -\tif (sc_enable_config())\n +\tif (sc_set_config(1))\n \t\treturn 1;\n \n - \tmemset(&el, 0, sizeof(el));\n + \tmemset(&pl, 0, sizeof(pl));\n @@\n - \treturn sc_read_tree();\n + \treturn write_patterns_and_update(&pl);\n }\n \n +static int sparse_checkout_disable(int argc, const char **argv)\n @@ -125,7 +121,7 @@\n +\tfprintf(fp, \"/*\\n\");\n +\tfclose(fp);\n +\n -+\tif (sc_read_tree())\n ++\tif (update_working_directory())\n +\t\tdie(_(\"error while refreshing working directory\"));\n +\n +\tunlink(sparse_filename);\n @@ -139,8 +135,8 @@\n \tstatic struct option builtin_sparse_checkout_options[] = {\n @@\n \t\t\treturn sparse_checkout_init(argc, argv);\n - \t\tif (!strcmp(argv[0], \"add\"))\n - \t\t\treturn sparse_checkout_add(argc, argv);\n + \t\tif (!strcmp(argv[0], \"set\"))\n + \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n +\t\tif (!strcmp(argv[0], \"disable\"))\n +\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n @@ -170,4 +166,4 @@\n +'\n +\n test_done\n - \\ No newline at end of file\n + \n 6: 86f12dc77d ! 7: 25642f8df2 trace2:experiment: clear_ce_flags_1\n @@ -1,11 +1,16 @@\n Author: Jeff Hostetler \n \n - trace2:experiment: clear_ce_flags_1\n + trace2: add region in clear_ce_flags\n \n - The clear_ce_flags_1 method is used by many types of calls to\n - unpack_trees(). Add trace2 regions around the method, including\n - some flag information, so we can get granular performance data\n - during experiments.\n + When Git updates the working directory with the sparse-checkout\n + feature enabled, the unpack_trees() method calls clear_ce_flags()\n + to update the skip-wortree bits on the cache entries. This\n + check can be expensive, depending on the patterns used.\n +\n + Add trace2 regions around the method, including some flag\n + information, so we can get granular performance data during\n + experiments. This data will be used to measure improvements\n + to the pattern-matching algorithms for sparse-checkout.\n \n Signed-off-by: Jeff Hostetler \n Signed-off-by: Derrick Stolee \n @@ -14,7 +19,7 @@\n --- a/unpack-trees.c\n +++ b/unpack-trees.c\n @@\n - \t\t\t struct exclude_list *el)\n + \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n +\tchar label[100];\n @@ -25,14 +30,14 @@\n -\treturn clear_ce_flags_1(istate,\n +\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n +\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n -+\ttrace2_region_enter(\"exp\", label, the_repository);\n ++\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n +\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n - \t\t\t\tel, 0);\n -+\ttrace2_region_leave(\"exp\", label, the_repository);\n + \t\t\t\tpl, 0);\n ++\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n +\n +\treturn rval;\n }\n 7: 19d664a5da ! 8: 84511255d1 sparse-checkout: add 'cone' mode\n @@ -7,14 +7,14 @@\n If there are 1,000 patterns and 1,000,000 entries, this time can\n be very significant.\n \n - Create a new 'cone' mode for the core.sparseCheckout config\n - option, and adjust the parser to set an appropriate enum value.\n + Create a new Boolean config option, core.sparseCheckoutCone, to\n + indicate that we expect the sparse-checkout file to contain a\n + more limited set of patterns. This is a separate config setting\n + from core.sparseCheckout to avoid breaking older clients by\n + introcuding a tri-state option.\n \n - While adjusting the type of this variable, rename it from\n - core_apply_sparse_checkout to core_sparse_checkout. This will\n - help avoid parallel changes from hitting type issues, and we\n - can guarantee that all uses now consider the enum values instead\n - of the int value.\n + The config option does nothing right now, but will be expanded\n + upon in a later commit.\n \n Signed-off-by: Derrick Stolee \n \n @@ -49,7 +49,7 @@\n +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n +updating the index, where N is the number of patterns and M is the number\n +of paths in the index. To combat this performance issue, a more restricted\n -+pattern set is allowed when `core.spareCheckout` is set to `cone`.\n ++pattern set is allowed when `core.spareCheckoutCone` is enabled.\n +\n +The accepted patterns in the cone pattern set are:\n +\n @@ -67,7 +67,7 @@\n +\n +```\n +/*\n -+!/*/*\n ++!/*/\n +```\n +\n +This says \"include everything in root, but nothing two levels below root.\"\n @@ -77,18 +77,18 @@\n +\n +```\n +/*\n -+!/*/*\n -+/A/*\n -+!/A/*/*\n -+/A/B/*\n -+!/A/B/*/*\n -+/A/B/C/*\n ++!/*/\n ++/A/\n ++!/A/*/\n ++/A/B/\n ++!/A/B/*/\n ++/A/B/C/\n +```\n +\n +Here, order matters, so the negative patterns are overridden by the positive\n +patterns that appear lower in the file.\n +\n -+If `core.sparseCheckout=cone`, then Git will parse the sparse-checkout file\n ++If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n +expecting patterns of these types. Git will warn if the patterns do not match.\n +If the patterns do match the expected format, then Git will use faster hash-\n +based algorithms to compute inclusion in the sparse-checkout.\n @@ -97,76 +97,6 @@\n --------\n \n \n - diff --git a/builtin/clone.c b/builtin/clone.c\n - --- a/builtin/clone.c\n - +++ b/builtin/clone.c\n -@@\n - \t * We must apply the setting in the current process\n - \t * for the later checkout to use the sparse-checkout file.\n - \t */\n --\tcore_apply_sparse_checkout = 1;\n -+\tcore_sparse_checkout = SPARSE_CHECKOUT_FULL;\n - \n - \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n - \t\terror(_(\"failed to initialize sparse-checkout\"));\n -\n - diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n - --- a/builtin/sparse-checkout.c\n - +++ b/builtin/sparse-checkout.c\n -@@\n - \treturn result;\n - }\n - \n --static int sc_set_config(int mode)\n -+static int sc_set_config(enum sparse_checkout_mode mode)\n - {\n - \tstruct argv_array argv = ARGV_ARRAY_INIT;\n - \tint result = 0;\n - \targv_array_pushl(&argv, \"config\", \"--add\", \"core.sparseCheckout\", NULL);\n - \n - \tswitch (mode) {\n --\tcase 1:\n -+\tcase SPARSE_CHECKOUT_FULL:\n - \t\targv_array_pushl(&argv, \"true\", NULL);\n - \t\tbreak;\n - \n --\tcase 0:\n -+\tcase SPARSE_CHECKOUT_CONE:\n -+\t\targv_array_pushl(&argv, \"cone\", NULL);\n -+\t\tbreak;\n -+\n -+\tcase SPARSE_CHECKOUT_NONE:\n - \t\targv_array_pushl(&argv, \"false\", NULL);\n - \t\tbreak;\n - \n -@@\n - \tFILE *fp;\n - \tint res;\n - \n --\tif (sc_set_config(1))\n -+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n - \t\treturn 1;\n - \n - \tmemset(&el, 0, sizeof(el));\n -@@\n - \tchar *sparse_filename;\n - \tFILE *fp;\n - \n --\tif (sc_set_config(1))\n -+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n - \t\tdie(_(\"failed to change config\"));\n - \n - \tsparse_filename = get_sparse_checkout_filename();\n -@@\n - \tunlink(sparse_filename);\n - \tfree(sparse_filename);\n - \n --\treturn sc_set_config(0);\n -+\treturn sc_set_config(SPARSE_CHECKOUT_NONE);\n - }\n - \n - int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n -\n diff --git a/cache.h b/cache.h\n --- a/cache.h\n +++ b/cache.h\n @@ -180,12 +110,8 @@\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n -+enum sparse_checkout_mode {\n -+\tSPARSE_CHECKOUT_NONE = 0,\n -+\tSPARSE_CHECKOUT_FULL = 1,\n -+\tSPARSE_CHECKOUT_CONE = 2,\n -+};\n -+enum sparse_checkout_mode core_sparse_checkout;\n ++int core_apply_sparse_checkout;\n ++int core_sparse_checkout_cone;\n +\n /*\n * Include broken refs in all ref iterations, which will\n @@ -195,32 +121,26 @@\n --- a/config.c\n +++ b/config.c\n @@\n + \t\treturn 0;\n \t}\n \n - \tif (!strcmp(var, \"core.sparsecheckout\")) {\n --\t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n -+\t\tint result = git_parse_maybe_bool(value);\n ++\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n ++\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n ++\t\treturn 0;\n ++\t}\n +\n -+\t\tif (result < 0) {\n -+\t\t\tcore_sparse_checkout = SPARSE_CHECKOUT_NONE;\n -+\n -+\t\t\tif (!strcasecmp(value, \"cone\"))\n -+\t\t\t\tcore_sparse_checkout = SPARSE_CHECKOUT_CONE;\n -+\t\t} else\n -+\t\t\tcore_sparse_checkout = result;\n + \tif (!strcmp(var, \"core.precomposeunicode\")) {\n + \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\n - \t}\n - \n \n diff --git a/environment.c b/environment.c\n --- a/environment.c\n +++ b/environment.c\n @@\n - enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n --int core_apply_sparse_checkout;\n -+enum sparse_checkout_mode core_sparse_checkout;\n + int core_apply_sparse_checkout;\n ++int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\n @@ -233,7 +153,7 @@\n '\n \n +test_expect_success 'cone mode: match patterns' '\n -+\tgit -C repo config --replace-all core.sparseCheckout cone &&\n ++\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n +\trm -rf repo/a repo/folder1 repo/folder2 &&\n +\tgit -C repo read-tree -mu HEAD &&\n +\tgit -C repo reset --hard &&\n @@ -249,16 +169,3 @@\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n -\n - diff --git a/unpack-trees.c b/unpack-trees.c\n - --- a/unpack-trees.c\n - +++ b/unpack-trees.c\n -@@\n - \n - \ttrace_performance_enter();\n - \tmemset(&el, 0, sizeof(el));\n --\tif (!core_apply_sparse_checkout || !o->update)\n -+\tif (!core_sparse_checkout || !o->update)\n - \t\to->skip_sparse_checkout = 1;\n - \tif (!o->skip_sparse_checkout) {\n - \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n 8: b99acea4a0 < -: ---------- sparse-checkout: use hashmaps for cone patterns\n 9: 568fda2d03 < -: ---------- sparse-checkout: init and add in cone mode\n -: ---------- > 9: 95a3285bc6 sparse-checkout: use hashmaps for cone patterns\n -: ---------- > 10: 995c5b8e2b sparse-checkout: init and set in cone mode\n -: ---------- > 11: 1d4321488e unpack-trees: hash less in cone mode\n\n-- \ngitgitgadget\n"},{"id":"382587","messageId":"21a0165be73dccf25c2c83a37d506ec061fd1d07.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 05/11] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:13Z","receivedAt":"2019-09-19T14:43:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nand places them in the sparse-checkout file. Then, it updates the\nworking directory to match those patterns. For a large list of\npatterns, the command-line call can get very cumbersome.\n\nAdd a '--stdin' option to instead read patterns over standard in.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n 2 files changed, 53 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 13333fba6a..f726fcd6b8 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -145,14 +145,45 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static char const * const builtin_sparse_checkout_set_usage[] = {\n+\tN_(\"git sparse-checkout set [--stdin|]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_set_opts {\n+\tint use_stdin;\n+} set_opts;\n+\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct pattern_list pl;\n+\n+\tstatic struct option builtin_sparse_checkout_set_options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n+\t\t\t N_(\"read patterns from standard in\")),\n+\t\tOPT_END(),\n+\t};\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n-\tfor (i = 1; i < argc; i++)\n-\t\tadd_pattern(argv[i], NULL, 0, &pl, 0);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_set_options,\n+\t\t\t builtin_sparse_checkout_set_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (set_opts.use_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tsize_t len;\n+\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n+\t}\n \n \treturn write_patterns_and_update(&pl);\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex f21ea61494..02ba9ec314 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -120,5 +120,25 @@ test_expect_success 'set sparse-checkout using builtin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using --stdin' '\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/folder1/\n+\t\t/folder2/\n+\tEOF\n+\tgit -C repo sparse-checkout set --stdin actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382588","messageId":"9a78f9ea0fe8d1988654f52a86a01031607621fe.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 04/11] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:13Z","receivedAt":"2019-09-19T14:43:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nas arguments and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nThe 'set' subcommand will replace the entire contents of the\nsparse-checkout file. The write_patterns_and_update() method is\nextracted from cmd_sparse_checkout() to make it easier to implement\n'add' and/or 'remove' subcommands in the future.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 5 +++++\n builtin/sparse-checkout.c | 31 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 19 ++++++++++++++++\n 3 files changed, 54 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 9707ef93b1..87813e5797 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -34,6 +34,11 @@ COMMANDS\n \tby Git. Add patterns to the sparse-checkout file to\n \trepopulate the working directory.\n \n+'set'::\n+\tWrite a set of patterns to the sparse-checkout file, as given as\n+\ta list of arguments following the 'set' subcommand. Update the\n+\tworking directory to match the new patterns.\n+\n SPARSE CHECKOUT\n ----------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 656e6ebdd5..13333fba6a 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list]\"),\n+\tN_(\"git sparse-checkout [init|list|set] \"),\n \tNULL\n };\n \n@@ -130,6 +130,33 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_patterns_to_file(fp, pl);\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\tclear_pattern_list(pl);\n+\treturn update_working_directory();\n+}\n+\n+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n+{\n+\tint i;\n+\tstruct pattern_list pl;\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tfor (i = 1; i < argc; i++)\n+\t\tadd_pattern(argv[i], NULL, 0, &pl, 0);\n+\n+\treturn write_patterns_and_update(&pl);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -152,6 +179,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"set\"))\n+\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 26b4ce9acd..f21ea61494 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -101,5 +101,24 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using builtin' '\n+\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382589","messageId":"25642f8df28825cce61812a24cbd87bf7cb2025f.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 07/11] trace2: add region in clear_ce_flags","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:15Z","receivedAt":"2019-09-19T14:43:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nWhen Git updates the working directory with the sparse-checkout\nfeature enabled, the unpack_trees() method calls clear_ce_flags()\nto update the skip-wortree bits on the cache entries. This\ncheck can be expensive, depending on the patterns used.\n\nAdd trace2 regions around the method, including some flag\ninformation, so we can get granular performance data during\nexperiments. This data will be used to measure improvements\nto the pattern-matching algorithms for sparse-checkout.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex cd548f4fa2..26be8f3569 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1404,15 +1404,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tpl, 0);\n+\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"382590","messageId":"84511255d1f28e1bdcec3de6096d2d9ac2a9f483.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 08/11] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:16Z","receivedAt":"2019-09-19T14:43:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new Boolean config option, core.sparseCheckoutCone, to\nindicate that we expect the sparse-checkout file to contain a\nmore limited set of patterns. This is a separate config setting\nfrom core.sparseCheckout to avoid breaking older clients by\nintrocuding a tri-state option.\n\nThe config option does nothing right now, but will be expanded\nupon in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 7 ++--\n Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n cache.h | 4 ++-\n config.c | 5 +++\n environment.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n 6 files changed, 78 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 75538d27e7..9b8ab2a6d4 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -591,8 +591,11 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n+\tis disabled. If \"true\", then sparse-checkout is enabled with the full\n+\t.gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n+\ta restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n+\tinformation.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex da95b28b1c..757326618d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -87,6 +87,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+## CONE PATTERN SET\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+```\n+/*\n+!/*/\n+```\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+```\n+/*\n+!/*/\n+/A/\n+!/A/*/\n+/A/B/\n+!/A/B/*/\n+/A/B/C/\n+```\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/cache.h b/cache.h\nindex cf5d70c196..8e8ea67efa 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -911,12 +911,14 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex 296a6d9cc4..f65c74f5b7 100644\n--- a/config.c\n+++ b/config.c\n@@ -1329,6 +1329,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n+\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 89af47cb85..670d92bcc0 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -69,6 +69,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 22fa032d6d..9b089c98c4 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -140,6 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"382591","messageId":"95a3285bc6021daa236d98d7e1bbdc5c45fc73b0.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 09/11] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:16Z","receivedAt":"2019-09-19T14:43:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/\n\t!/$folder/*/\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n\tcore.sparseCheckout=false: 0.21 s (0.00 s)\n\t core.sparseCheckout=true: 3.75 s (3.50 s)\n\t core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 173 +++++++++++++++++++++++++++--\n dir.h | 27 +++++\n t/t1091-sparse-checkout-builtin.sh | 11 +-\n 3 files changed, 202 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 34972abdaf..4fc57187e9 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -599,6 +599,109 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const void *a, const void *b, const void *key)\n+{\n+\tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n+\tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\n+\n+\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n+\t\t\t ? ee1->patternlen\n+\t\t\t : ee2->patternlen;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n+}\n+\n+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n+{\n+\tstruct pattern_entry *translated;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!pl->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (!strcmp(given->pattern, \"/*\"))\n+\t\treturn;\n+\n+\tif (given->patternlen > 2 &&\n+\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n+\t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tpl->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(given->pattern);\n+\t\ttruncated[given->patternlen - 2] = 0;\n+\n+\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\t\ttranslated->pattern = truncated;\n+\t\ttranslated->patternlen = given->patternlen - 2;\n+\t\thashmap_entry_init(translated,\n+\t\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\t\tif (!hashmap_get(&pl->recursive_hashmap, translated, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\t\tgiven->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tfree(translated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&pl->parent_hashmap, translated);\n+\t\thashmap_remove(&pl->recursive_hashmap, translated, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE) {\n+\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\tgiven->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n+\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\n+\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->patternlen = given->patternlen;\n+\thashmap_entry_init(translated,\n+\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, translated);\n+\n+\tif (hashmap_get(&pl->parent_hashmap, translated, NULL)) {\n+\t\t/* we already included this at the parent level */\n+\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\tgiven->pattern);\n+\t\thashmap_remove(&pl->parent_hashmap, translated, &data);\n+\t\tfree(data);\n+\t\tfree(translated);\n+\t}\n+\n+\treturn;\n+\n+clear_hashmaps:\n+\twarning(_(\"disabling cone pattern matching\"));\n+\thashmap_free(&pl->parent_hashmap, 1);\n+\thashmap_free(&pl->recursive_hashmap, 1);\n+\tpl->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct pattern_entry p;\n+\n+\t/* Check straight mapping */\n+\tp.pattern = pattern->buf;\n+\tp.patternlen = pattern->len;\n+\thashmap_entry_init(&p, memhash(p.pattern, p.patternlen));\n+\treturn !!hashmap_get(map, &p, NULL);\n+}\n+\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n {\n@@ -623,6 +726,8 @@ void add_pattern(const char *string, const char *base,\n \tALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);\n \tpl->patterns[pl->nr++] = pattern;\n \tpattern->pl = pl;\n+\n+\tadd_pattern_to_hashsets(pl, pattern);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -848,6 +953,10 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\tpl->use_cone_patterns = core_sparse_checkout_cone;\n+\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n \tpl->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1084,16 +1193,64 @@ enum pattern_match_result path_matches_pattern_list(\n \t\t\t\tstruct index_state *istate)\n {\n \tstruct path_pattern *pattern;\n-\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, pl, istate);\n-\tif (pattern) {\n-\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n-\t\t\treturn NOT_MATCHED;\n-\t\telse\n-\t\t\treturn MATCHED;\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = NOT_MATCHED;\n+\tconst char *slash_pos;\n+\n+\tif (!pl->use_cone_patterns) {\n+\t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\tdtype, pl, istate);\n+\t\tif (pattern) {\n+\t\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\t\treturn NOT_MATCHED;\n+\t\t\telse\n+\t\t\t\treturn MATCHED;\n+\t\t}\n+\n+\t\treturn UNDECIDED;\n \t}\n \n-\treturn UNDECIDED;\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\n+\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t\t&parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\twhile (parent_pathname.len) {\n+\t\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t\t &parent_pathname)) {\n+\t\t\tresult = UNDECIDED;\n+\t\t\tgoto done;\n+\t\t}\n+\n+\t\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\t\tif (slash_pos == parent_pathname.buf)\n+\t\t\tbreak;\n+\n+\t\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\t}\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct path_pattern *last_matching_pattern_from_lists(\ndiff --git a/dir.h b/dir.h\nindex 608696c958..bbd5bd1cc9 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct path_pattern {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct pattern_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,25 @@ struct pattern_list {\n \tconst char *src;\n \n \tstruct path_pattern **patterns;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9b089c98c4..f726205d21 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -143,7 +143,8 @@ test_expect_success 'set sparse-checkout using --stdin' '\n test_expect_success 'cone mode: match patterns' '\n \tgit -C repo config --worktree core.sparseCheckoutCone true &&\n \trm -rf repo/a repo/folder1 repo/folder2 &&\n-\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n \tls repo >dir &&\n \tcat >expect <<-EOF &&\n@@ -154,6 +155,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info/ &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"382592","messageId":"b62b76013f98e57089c9b0c03209e28757977dd7.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 06/11] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:14Z","receivedAt":"2019-09-19T14:43:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 26 ++++++++-----------\n builtin/sparse-checkout.c | 37 ++++++++++++++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++\n 3 files changed, 59 insertions(+), 19 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 87813e5797..da95b28b1c 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -39,6 +39,10 @@ COMMANDS\n \ta list of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ----------------\n \n@@ -61,6 +65,13 @@ Then it compares the new skip-worktree value with the previous one. If\n skip-worktree turns from set to unset, it will add the corresponding\n file back. If it turns from unset to set, that file will be removed.\n \n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n+Sparse checkout support in 'git checkout' and similar commands is\n+disabled by default. You need to set `core.sparseCheckout` to `true`\n+in order to have sparse checkout support.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -75,21 +86,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-read-tree' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex f726fcd6b8..f858f0b1b5 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list|set] \"),\n+\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n \tNULL\n };\n \n@@ -74,7 +74,7 @@ static int update_working_directory(void)\n \treturn result;\n }\n \n-static int sc_enable_config(void)\n+static int sc_set_config(int mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n@@ -83,7 +83,12 @@ static int sc_enable_config(void)\n \t\treturn 1;\n \t}\n \n-\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n+\n+\tif (mode)\n+\t\targv_array_pushl(&argv, \"true\", NULL);\n+\telse\n+\t\targv_array_pushl(&argv, \"false\", NULL);\n \n \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n \t\terror(_(\"failed to enable core.sparseCheckout\"));\n@@ -101,7 +106,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tint res;\n \tstruct object_id oid;\n \n-\tif (sc_enable_config())\n+\tif (sc_set_config(1))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -188,6 +193,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \treturn write_patterns_and_update(&pl);\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (sc_set_config(1))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (update_working_directory())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn sc_set_config(0);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -212,6 +239,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"set\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 02ba9ec314..22fa032d6d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -140,5 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382593","messageId":"1d4321488ef4edbd4b19a8e26b329d0b54755bf4.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 11/11] unpack-trees: hash less in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:18Z","receivedAt":"2019-09-19T14:43:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature in \"cone mode\" can use the fact that\nthe recursive patterns are \"connected\" to the root via parent\npatterns to decide if a directory is entirely contained in the\nsparse-checkout or entirely removed.\n\nIn these cases, we can skip hashing the paths within those\ndirectories and simply set the skipworktree bit to the correct\nvalue.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 4 ++--\n dir.h | 1 +\n unpack-trees.c | 38 +++++++++++++++++++++++---------------\n 3 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 298a4539ec..35fd60d487 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1215,7 +1215,7 @@ enum pattern_match_result path_matches_pattern_list(\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t\t&parent_pathname)) {\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n@@ -1237,7 +1237,7 @@ enum pattern_match_result path_matches_pattern_list(\n \twhile (parent_pathname.len) {\n \t\tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t\t &parent_pathname)) {\n-\t\t\tresult = UNDECIDED;\n+\t\t\tresult = MATCHED_RECURSIVE;\n \t\t\tgoto done;\n \t\t}\n \ndiff --git a/dir.h b/dir.h\nindex 7c76a2d55e..5f410eedbb 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -261,6 +261,7 @@ enum pattern_match_result {\n \tUNDECIDED = -1,\n \tNOT_MATCHED = 0,\n \tMATCHED = 1,\n+\tMATCHED_RECURSIVE = 2,\n };\n \n /*\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 26be8f3569..43acc0ffd6 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1280,15 +1280,17 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n \tint rc;\n-\tenum pattern_match_result ret;\n-\tret = path_matches_pattern_list(prefix->buf, prefix->len,\n-\t\t\t\t\tbasename, &dtype, pl, istate);\n+\tenum pattern_match_result ret, orig_ret;\n+\torig_ret = path_matches_pattern_list(prefix->buf, prefix->len,\n+\t\t\t\t\t basename, &dtype, pl, istate);\n \n \tstrbuf_addch(prefix, '/');\n \n \t/* If undecided, use matching result of parent dir in defval */\n-\tif (ret == UNDECIDED)\n+\tif (orig_ret == UNDECIDED)\n \t\tret = default_match;\n+\telse\n+\t\tret = orig_ret;\n \n \tfor (cache_end = cache; cache_end != cache + nr; cache_end++) {\n \t\tstruct cache_entry *ce = *cache_end;\n@@ -1296,17 +1298,23 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\tbreak;\n \t}\n \n-\t/*\n-\t * TODO: check pl, if there are no patterns that may conflict\n-\t * with ret (iow, we know in advance the incl/excl\n-\t * decision for the entire directory), clear flag here without\n-\t * calling clear_ce_flags_1(). That function will call\n-\t * the expensive path_matches_pattern_list() on every entry.\n-\t */\n-\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t prefix,\n-\t\t\t select_mask, clear_mask,\n-\t\t\t pl, ret);\n+\tif (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {\n+\t\tstruct cache_entry **ce = cache;\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\n+\t\twhile (ce < cache_end) {\n+\t\t\t(*ce)->ce_flags &= ~clear_mask;\n+\t\t\tce++;\n+\t\t}\n+\t} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\t} else {\n+\t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t prefix,\n+\t\t\t\t select_mask, clear_mask,\n+\t\t\t\t pl, ret);\n+\t}\n+\n \tstrbuf_setlen(prefix, prefix->len - 1);\n \treturn rc;\n }\n-- \ngitgitgadget\n"},{"id":"382594","messageId":"995c5b8e2b4b4ba36c09165de1bc4719819c3ce4.1568904188.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 10/11] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-09-19T14:43:17Z","receivedAt":"2019-09-19T14:43:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout [init|set]'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckoutCone=true'.\n\nWhen running 'git sparse-checkout set' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nWhen testing 'git sparse-checkout set' in cone mode, check the\nerror stream to ensure we do not see any errors. Specifically,\nwe want to avoid the warning that the patterns do not match\nthe cone-mode patterns.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 171 +++++++++++++++++++++++++++--\n dir.c | 4 +-\n dir.h | 3 +\n t/t1091-sparse-checkout-builtin.sh | 49 +++++++++\n 4 files changed, 213 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex f858f0b1b5..111cbc96d9 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,6 +6,7 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n@@ -74,9 +75,14 @@ static int update_working_directory(void)\n \treturn result;\n }\n \n+#define SPARSE_CHECKOUT_NONE 0\n+#define SPARSE_CHECKOUT_FULL 1\n+#define SPARSE_CHECKOUT_CONE 2\n+\n static int sc_set_config(int mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n \n \tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n \t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n@@ -95,9 +101,31 @@ static int sc_set_config(int mode)\n \t\treturn 1;\n \t}\n \n+\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n+\t\t\t \"core.sparseCheckoutCone\", NULL);\n+\n+\tif (mode == SPARSE_CHECKOUT_CONE)\n+\t\targv_array_push(&cone_argv, \"true\");\n+\telse\n+\t\targv_array_push(&cone_argv, \"false\");\n+\n+\tif (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckoutCone\"));\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\n \n+static char const * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_init_opts {\n+\tint cone_mode;\n+} init_opts;\n+\n static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -105,8 +133,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n+\tint mode;\n+\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n \n-\tif (sc_set_config(1))\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = init_opts.cone_mode ? SPARSE_CHECKOUT_CONE : SPARSE_CHECKOUT_FULL;\n+\n+\tif (sc_set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -135,6 +176,72 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n+{\n+\tstruct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, e);\n+\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (!slash)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct pattern_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get(&pl->parent_hashmap, e, NULL))\n+\t\t\thashmap_add(&pl->parent_hashmap, e);\n+\t}\n+}\n+\n+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\tstruct pattern_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_iter_init(&pl->parent_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter)))\n+\t\tstring_list_insert(&sl, entry->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (strlen(pattern))\n+\t\t\tfprintf(fp, \"/%s/\\n!/%s/*/\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_iter_init(&pl->recursive_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter)))\n+\t\tstring_list_insert(&sl, entry->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"/%s/\\n\", pattern);\n+\t}\n+}\n+\n static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n@@ -142,7 +249,12 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n-\twrite_patterns_to_file(fp, pl);\n+\n+\tif (core_sparse_checkout_cone)\n+\t\twrite_cone_to_file(fp, pl);\n+\telse\n+\t\twrite_patterns_to_file(fp, pl);\n+\n \tfclose(fp);\n \tfree(sparse_filename);\n \n@@ -150,6 +262,24 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n+{\n+\tstrbuf_trim(line);\n+\n+\tstrbuf_trim_trailing_dir_sep(line);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tif (line->buf[0] == '/')\n+\t\tstrbuf_remove(line, 0, 1);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tinsert_recursive_pattern(pl, line);\n+}\n+\n static char const * const builtin_sparse_checkout_set_usage[] = {\n \tN_(\"git sparse-checkout set [--stdin|]\"),\n \tNULL\n@@ -177,17 +307,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n-\tif (set_opts.use_stdin) {\n+\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (!strbuf_getline(&line, stdin)) {\n-\t\t\tsize_t len;\n-\t\t\tchar *buf = strbuf_detach(&line, &len);\n-\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n+\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n+\t\tif (set_opts.use_stdin) {\n+\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++) {\n+\t\t\t\tstrbuf_setlen(&line, 0);\n+\t\t\t\tstrbuf_addstr(&line, argv[i]);\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n \t\t}\n \t} else {\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n+\t\tif (set_opts.use_stdin) {\n+\t\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n+\t\t}\n \t}\n \n \treturn write_patterns_and_update(&pl);\n@@ -198,7 +345,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \n-\tif (sc_set_config(1))\n+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n \t\tdie(_(\"failed to change config\"));\n \n \tsparse_filename = get_sparse_checkout_filename();\n@@ -212,7 +359,7 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tunlink(sparse_filename);\n \tfree(sparse_filename);\n \n-\treturn sc_set_config(0);\n+\treturn sc_set_config(SPARSE_CHECKOUT_NONE);\n }\n \n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\ndiff --git a/dir.c b/dir.c\nindex 4fc57187e9..298a4539ec 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -599,8 +599,8 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n-static int pl_hashmap_cmp(const void *unused_cmp_data,\n-\t\t\t const void *a, const void *b, const void *key)\n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const void *a, const void *b, const void *key)\n {\n \tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n \tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\ndiff --git a/dir.h b/dir.h\nindex bbd5bd1cc9..7c76a2d55e 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -296,6 +296,9 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const void *a, const void *b, const void *key);\n+\n struct pattern_list *add_pattern_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src);\n int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex f726205d21..b6eb02c69a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -178,5 +178,54 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and set' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n+\ttest_line_count = 0 err &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\t\t!/deep/*/\n+\t\t/deep/deeper1/\n+\t\t!/deep/deeper1/*/\n+\t\t/deep/deeper1/deepest/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\tls repo >dir &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n \n-- \ngitgitgadget\n\n"},{"id":"382612","messageId":"6b461ad3-164d-46ff-4a68-99f8e6562a72@gmail.com","threadId":"51691","inReplyTo":"95a3285bc6021daa236d98d7e1bbdc5c45fc73b0.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/11] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-19T20:59:23Z","receivedAt":"2019-09-19T20:59:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n> @@ -848,6 +953,10 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n> \tint i, lineno = 1;\n> \tchar *entry;\n> \n> +\tpl->use_cone_patterns = core_sparse_checkout_cone;\n> +\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n> +\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n> +\n\nJust a head's-up to anyone looking at this series: this is not the\nright place to set use_cone_patterns (without passing a flag or\nsomething). This same path is called from the .gitignore machinery,\nso if you have a non-cone pattern in your .gitignore you will start\nseeing warnings with core.sparseCheckoutCone=true.\n\nI figured it out only via integration tests with our C# layer. In\nv2 I'll fix this and add a test to make sure it stays fixed.\n\nOtherwise, everything is working as expected.\n\n-Stolee\n\n"},{"id":"382651","messageId":"7d87fe4b-160c-34c2-db6d-4a56fd919755@gmail.com","threadId":"51691","inReplyTo":"6b461ad3-164d-46ff-4a68-99f8e6562a72@gmail.com","subject":"Re: [PATCH v2 09/11] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-09-20T14:37:00Z","receivedAt":"2019-09-20T14:37:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/19/2019 4:59 PM, Derrick Stolee wrote:\n> On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n>> @@ -848,6 +953,10 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n>> \tint i, lineno = 1;\n>> \tchar *entry;\n>> \n>> +\tpl->use_cone_patterns = core_sparse_checkout_cone;\n>> +\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n>> +\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n>> +\n> \n> Just a head's-up to anyone looking at this series: this is not the\n> right place to set use_cone_patterns (without passing a flag or\n> something). This same path is called from the .gitignore machinery,\n> so if you have a non-cone pattern in your .gitignore you will start\n> seeing warnings with core.sparseCheckoutCone=true.\n> \n> I figured it out only via integration tests with our C# layer. In\n> v2 I'll fix this and add a test to make sure it stays fixed.\n\nHere is the code fix. I will have a test to check this in v3.\n\n-->8--\n\nFrom 73b100d11d11bf8f045c2e116390120819dcb800 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Fri, 20 Sep 2019 08:55:06 -0400\nSubject: [PATCH v2] fixup! sparse-checkout: use hashmaps for cone patterns\n\n---\n dir.c | 1 -\n unpack-trees.c | 1 +\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 35fd60d487..248a418379 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -953,7 +953,6 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n-\tpl->use_cone_patterns = core_sparse_checkout_cone;\n \thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 43acc0ffd6..b5cf591c38 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1487,6 +1487,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n \t\t\to->skip_sparse_checkout = 1;\n \t\telse\n-- \n2.23.0.vfs.1.1.19.gce6e76d\n\n\n"},{"id":"383206","messageId":"47186f02-164d-fa41-f65c-05a197e4a046@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/11] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-01T13:40:08Z","receivedAt":"2019-10-01T13:40:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n> This series makes the sparse-checkout feature more user-friendly. While\n> there, I also present a way to use a limited set of patterns to gain a\n> significant performance boost in very large repositories.\n> \n> Sparse-checkout is only documented as a subsection of the read-tree docs\n> [1], which makes the feature hard to discover. Users have trouble navigating\n> the feature, especially at clone time [2], and have even resorted to\n> creating their own helper tools [3].\n> \n> This series attempts to solve these problems using a new builtin.\n\nI haven't heard anything about this series since Elijah's careful\nreview of the RFC. There are definitely areas where this can be\nmade more robust, but I'd like to save those for a follow-up series.\n\nJunio: I know you didn't track this in the recent \"what's cooking\"\nlist, and I don't expect you to take it until I re-roll v3 to\ninclude the .gitignore interaction I already pointed out.\n\nThanks,\n-Stolee\n"},{"id":"383218","messageId":"CABPp-BHGNzLNzOW6nSi_ComA0DrRnJukqoemG8z_qS6e22uBUQ@mail.gmail.com","threadId":"51691","inReplyTo":"47186f02-164d-fa41-f65c-05a197e4a046@gmail.com","subject":"Re: [PATCH v2 00/11] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-01T16:54:31Z","receivedAt":"2019-10-01T16:54:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 1, 2019 at 9:48 AM Derrick Stolee wrote:\n>\n> On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n> > This series makes the sparse-checkout feature more user-friendly. While\n> > there, I also present a way to use a limited set of patterns to gain a\n> > significant performance boost in very large repositories.\n> >\n> > Sparse-checkout is only documented as a subsection of the read-tree docs\n> > [1], which makes the feature hard to discover. Users have trouble navigating\n> > the feature, especially at clone time [2], and have even resorted to\n> > creating their own helper tools [3].\n> >\n> > This series attempts to solve these problems using a new builtin.\n>\n> I haven't heard anything about this series since Elijah's careful\n> review of the RFC. There are definitely areas where this can be\n> made more robust, but I'd like to save those for a follow-up series.\n>\n> Junio: I know you didn't track this in the recent \"what's cooking\"\n> list, and I don't expect you to take it until I re-roll v3 to\n> include the .gitignore interaction I already pointed out.\n\nOh, sorry, I missed this. By the way, is there any reason I wasn't\ncc'ed on this round after reviewing the RFC?\n"},{"id":"383223","messageId":"eb1bac92-2c65-b329-46ee-bcb64cb9ac0c@gmail.com","threadId":"51691","inReplyTo":"CABPp-BHGNzLNzOW6nSi_ComA0DrRnJukqoemG8z_qS6e22uBUQ@mail.gmail.com","subject":"Re: [PATCH v2 00/11] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-01T18:15:52Z","receivedAt":"2019-10-01T18:15:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/1/2019 12:54 PM, Elijah Newren wrote:\n> On Tue, Oct 1, 2019 at 9:48 AM Derrick Stolee wrote:\n>>\n>> On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n>>> This series makes the sparse-checkout feature more user-friendly. While\n>>> there, I also present a way to use a limited set of patterns to gain a\n>>> significant performance boost in very large repositories.\n>>>\n>>> Sparse-checkout is only documented as a subsection of the read-tree docs\n>>> [1], which makes the feature hard to discover. Users have trouble navigating\n>>> the feature, especially at clone time [2], and have even resorted to\n>>> creating their own helper tools [3].\n>>>\n>>> This series attempts to solve these problems using a new builtin.\n>>\n>> I haven't heard anything about this series since Elijah's careful\n>> review of the RFC. There are definitely areas where this can be\n>> made more robust, but I'd like to save those for a follow-up series.\n>>\n>> Junio: I know you didn't track this in the recent \"what's cooking\"\n>> list, and I don't expect you to take it until I re-roll v3 to\n>> include the .gitignore interaction I already pointed out.\n> \n> Oh, sorry, I missed this. By the way, is there any reason I wasn't\n> cc'ed on this round after reviewing the RFC?\n\nSorry, I forgot to modify my GitGitGadget cover letter to include you\nas a CC. Totally my oversight, not on purpose.\n\nThanks,\n-Stolee\n"},{"id":"383369","messageId":"xmqqftk9bh1e.fsf@gitster-ct.c.googlers.com","threadId":"51691","inReplyTo":"47186f02-164d-fa41-f65c-05a197e4a046@gmail.com","subject":"Re: [PATCH v2 00/11] New sparse-checkout builtin and \"cone\" mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-10-03T22:28:13Z","receivedAt":"2019-10-03T22:28:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 9/19/2019 10:43 AM, Derrick Stolee via GitGitGadget wrote:\n>> This series makes the sparse-checkout feature more user-friendly. While\n>> there, I also present a way to use a limited set of patterns to gain a\n>> significant performance boost in very large repositories.\n>> \n>> Sparse-checkout is only documented as a subsection of the read-tree docs\n>> [1], which makes the feature hard to discover. Users have trouble navigating\n>> the feature, especially at clone time [2], and have even resorted to\n>> creating their own helper tools [3].\n>> \n>> This series attempts to solve these problems using a new builtin.\n>\n> I haven't heard anything about this series since Elijah's careful\n> review of the RFC. There are definitely areas where this can be\n> made more robust, but I'd like to save those for a follow-up series.\n>\n> Junio: I know you didn't track this in the recent \"what's cooking\"\n> list, and I don't expect you to take it until I re-roll v3 to\n> include the .gitignore interaction I already pointed out.\n\nI have made a mental note that says \"expecting v3, a\nreroll. cf. <7d87fe4b-160c-34c2-db6d-4a56fd919755@gmail.com>\"; there\nis no existing entry to hang it below in the \"what's cooking\"\nreport, though X-<.\n\n\n"},{"id":"383473","messageId":"CABPp-BEb-C=M5FQ7DhXViSooyNeify5v_5v+n6QbqQZm64jMdg@mail.gmail.com","threadId":"51691","inReplyTo":"dbaf3de88e036fb23d7760582030726f32581a4d.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/11] sparse-checkout: create builtin with 'list' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-05T19:22:13Z","receivedAt":"2019-10-05T19:22:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 1:45 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The sparse-checkout feature is mostly hidden to users, as its\n> only documentation is supplementary information in the docs for\n> 'git read-tree'. In addition, users need to know how to edit the\n> .git/info/sparse-checkout file with the right patterns, then run\n> the appropriate 'git read-tree -mu HEAD' command. Keeping the\n> working directory in sync with the sparse-checkout file requires\n> care.\n>\n> Begin an effort to make the sparse-checkout feature a porcelain\n> feature by creating a new 'git sparse-checkout' builtin. This\n> builtin will be the preferred mechanism for manipulating the\n> sparse-checkout file and syncing the working directory.\n\nSounds good.\n\n> The `$GIT_DIR/info/sparse-checkout` file defines the skip-\n> worktree reference bitmap. When Git updates the working\n> directory, it updates the skip-worktree bits in the index\n> based on this file and removes or restores files in the\n> working copy to match.\n\nDoes this paragraph make sense in the commit message? It's not\nexplaining anything new or changing with your patch, just pre-existing\nbehavior, but you don't seem to reference or expound on it.\n\n> The documentation provided is adapted from the \"git read-tree\"\n> documentation with a few edits for clarity in the new context.\n> Extra sections are added to hint toward a future change to\n> a more restricted pattern set.\n\nI think it needs a few more adaptations, as noted below...\n\n> +SPARSE CHECKOUT\n> +----------------\n> +\n> +\"Sparse checkout\" allows populating the working directory sparsely.\n> +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n> +Git whether a file in the working directory is worth looking at. If\n> +the skip-worktree bit is set, then the file is ignored in the working\n> +directory. Git will not populate the contents of those files, which\n> +makes a sparse checkout helpful when working in a repository with many\n> +files, but only a few are important to the current user.\n> +\n> +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n> +skip-worktree reference bitmap. When Git updates the working\n> +directory, it resets the skip-worktree bit in the index based on this\n> +file. If an entry\n> +matches a pattern in this file, skip-worktree will not be set on\n> +that entry. Otherwise, skip-worktree will be set.\n> +\n> +Then it compares the new skip-worktree value with the previous one. If\n> +skip-worktree turns from set to unset, it will add the corresponding\n> +file back. If it turns from unset to set, that file will be removed.\n\nI know this was just copied from elsewhere, but I still have the same\nproblem I mentioned last time with these paragraphs: the double\nnegations just make it confusing to follow. I'd prefer e.g. replacing\nthe last two paragraphs above with the following (which I think you\ndid take but accidentally placed in the commit message instead of\nusing it to replace these confusing paragraphs?):\n\nThe `$GIT_DIR/info/sparse-checkout` file is used to define the\nskip-worktree reference bitmap. When Git updates the working\ndirectory, it updates the skip-worktree bits in the index based on this\nfile and removes or restores files in the working copy to match.\n\nIt doesn't have to be this precise wording, but something like it\nwhich is way easier to follow than those two paragraphs you were\ncopying.\n\n> +Another tricky thing is fully repopulating the working directory when you\n> +no longer want sparse checkout. You cannot just disable \"sparse\n> +checkout\" because skip-worktree bits are still in the index and your working\n> +directory is still sparsely populated. You should re-populate the working\n> +directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> +follows:\n> +\n> +----------------\n> +/*\n> +----------------\n> +\n> +Then you can disable sparse checkout.\n\nI would comment on this section, but it appears you remove this\nsection later in your series when you add 'sparse-checkout disable',\nwhich addresses my concern.\n\n> Sparse checkout support in 'git\n> +read-tree' and similar commands is disabled by default. You need to\n> +set `core.sparseCheckout` to `true` in order to have sparse checkout\n> +support.\n\nI see you change `git read-tree` to `git checkout` later in the\nseries, which is good. However, you keep the second sentence which\nseems unhelpful. Why have a 'git sparse-checkout init' command if the\nuser still has to manually set `core.sparseCheckout`? Also, if we're\ngoing to mention that setting, we should mention\nextensions.worktreeConfig at the same time. Not sure whether it'd be\nbetter to drop the second sentence or restructure it to let the user\nknow that it depends on the core.sparseCheckout setting which the init\ncommand runs, but something should probably be done.\n\n\nThe rest of the patch looks good.\n"},{"id":"383474","messageId":"CABPp-BFV7TA0qwZCQpHCqx9N+JifyRyuBQ-pZ_oGfe-NOgyh7A@mail.gmail.com","threadId":"51691","inReplyTo":"412211f5dd6d4d995f258403bf377bc0cb6332b4.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/11] sparse-checkout: create 'init' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-05T19:34:34Z","receivedAt":"2019-10-05T19:34:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 3:06 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Getting started with a sparse-checkout file can be daunting. Help\n> users start their sparse enlistment using 'git sparse-checkout init'.\n> This will set 'core.sparseCheckout=true' in their config, write\n> an initial set of patterns to the sparse-checkout file, and update\n> their working directory.\n\n...and ensure extensions.worktreeConfig is set to true.\n\n> Using 'git read-tree' to clear directories does not work cleanly\n> on Windows, so manually delete directories that are tracked by Git\n> before running read-tree.\n\nI thought you said you fixed this? It appears to no longer be part of\nthe patch, so I'm guessing you just forgot to remove this comment from\nthe commit message?\n\n> The use of running another process for 'git read-tree' is likely\n> suboptimal, but that can be improved in a later change, if valuable.\n\nI think it would also be worth mentioning that not only is a\nsubprocess suboptimal, but the behavior of `git read-tree -mu HEAD` is\nitself suboptimal for a sparse-checkout. (We either need more error\nchecking e.g. when the user is in the middle of a rebase or merge or\ncherry-pick and have conflicted entries with a more focused error\nmessage for the user, or we need a command that won't abort if the\nconflicts aren't in the paths we're trying to remove from or bring\nback to the working tree.)\n\n\nPatch looks good to me, assuming the caveats of using `git read-tree\n-mu HEAD` are better documented -- and hopefully addressed at some\npoint. You addressed all my other feedback on this patch from the RFC\nseries.\n"},{"id":"383475","messageId":"CABPp-BHaV+XX63Y46b8wHk0rHzdAbfDQDZAQe3pHcA63WerigA@mail.gmail.com","threadId":"51691","inReplyTo":"fef41b794a9886664616ce5e5c7902a82a474c2d.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/11] clone: add --sparse mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-05T19:40:46Z","receivedAt":"2019-10-05T19:40:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 3:06 PM Derrick Stolee via GitGitGadget\n wrote:\n\n> During the 'git sparse-checkout init' call, we must first look\n> to see if HEAD is valid, or else we will fail while trying to\n> update the working directory. The first checkout will actually\n> update the working directory correctly.\n\nThis is new since the RFC series, but I'm not sure I understand. Is\nthe issue you're fixing here that a 'git init somerepo' would hit this\ncodepath and print funny errors because HEAD doesn't exist yet and\nthus the whole `git read-tree -mu HEAD` stuff can't work? Or that\nwhen the remote has HEAD pointing at a bad commit that you get error\nmessages different than expected?\n\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 895479970d..656e6ebdd5 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -99,6 +99,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n> char *sparse_filename;\n> FILE *fp;\n> int res;\n> + struct object_id oid;\n>\n> if (sc_enable_config())\n> return 1;\n> @@ -120,6 +121,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n> fprintf(fp, \"/*\\n!/*/\\n\");\n> fclose(fp);\n>\n> + if (get_oid(\"HEAD\", &oid)) {\n> + /* assume we are in a fresh repo */\n> + return 0;\n> + }\n> +\n> reset_dir:\n> return update_working_directory();\n> }\n"},{"id":"383498","messageId":"CABPp-BHB0UM1G2tHQ7igRX7Sn1O9ujW6mq_e-UGanGMONHxiag@mail.gmail.com","threadId":"51691","inReplyTo":"9a78f9ea0fe8d1988654f52a86a01031607621fe.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/11] sparse-checkout: 'set' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-05T22:44:10Z","receivedAt":"2019-10-05T22:44:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 3:07 PM Derrick Stolee via GitGitGadget\n wrote:\n> +static int write_patterns_and_update(struct pattern_list *pl)\n> +{\n> + char *sparse_filename;\n> + FILE *fp;\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + fp = fopen(sparse_filename, \"w\");\n> + write_patterns_to_file(fp, pl);\n> + fclose(fp);\n> + free(sparse_filename);\n> +\n> + clear_pattern_list(pl);\n\nIt seems slightly odd that pl is passed in but cleared in this\nfunction rather than in the caller that created pl. Should this be\nmoved to the caller, or, alternatively, a comment added to explain\nthis side-effect for future callers of the function?\n\nThe rest of the patch looked good to me.\n"},{"id":"383506","messageId":"CABPp-BGcZT2-m=D_X4d7zE9pzuWTKCnmURruahdQndpp=+-_WA@mail.gmail.com","threadId":"51691","inReplyTo":"CABPp-BHB0UM1G2tHQ7igRX7Sn1O9ujW6mq_e-UGanGMONHxiag@mail.gmail.com","subject":"Re: [PATCH v2 04/11] sparse-checkout: 'set' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-06T00:30:06Z","receivedAt":"2019-10-06T00:30:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, Oct 5, 2019 at 3:44 PM Elijah Newren wrote:\n>\n> On Thu, Sep 19, 2019 at 3:07 PM Derrick Stolee via GitGitGadget\n> wrote:\n> > +static int write_patterns_and_update(struct pattern_list *pl)\n> > +{\n> > + char *sparse_filename;\n> > + FILE *fp;\n> > +\n> > + sparse_filename = get_sparse_checkout_filename();\n> > + fp = fopen(sparse_filename, \"w\");\n> > + write_patterns_to_file(fp, pl);\n> > + fclose(fp);\n> > + free(sparse_filename);\n> > +\n> > + clear_pattern_list(pl);\n>\n> It seems slightly odd that pl is passed in but cleared in this\n> function rather than in the caller that created pl. Should this be\n> moved to the caller, or, alternatively, a comment added to explain\n> this side-effect for future callers of the function?\n>\n> The rest of the patch looked good to me.\n\nActually, thought of something else. What if the user calls 'git\nsparse-checkout set ...' without first calling 'git sparse-checkout\ninit'? Should that report an error to the user, a suggestion to\nfollow it up with 'sparse-checkout init', or should it just call\nsc_set_config() behind the scenes and allow bypassing the init\nsubcommand?\n"},{"id":"383510","messageId":"CABPp-BHYCCD+Vx5fq35jH82eHc1-P53Lz_aGNpHJNcx9kg2K-A@mail.gmail.com","threadId":"51691","inReplyTo":"b62b76013f98e57089c9b0c03209e28757977dd7.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/11] sparse-checkout: create 'disable' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-06T04:10:29Z","receivedAt":"2019-10-06T04:10:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 1:46 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The instructions for disabling a sparse-checkout to a full\n> working directory are complicated and non-intuitive. Add a\n> subcommand, 'git sparse-checkout disable', to perform those\n> steps for the user.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 26 ++++++++-----------\n> builtin/sparse-checkout.c | 37 ++++++++++++++++++++++++---\n> t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++\n> 3 files changed, 59 insertions(+), 19 deletions(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index 87813e5797..da95b28b1c 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -39,6 +39,10 @@ COMMANDS\n> a list of arguments following the 'set' subcommand. Update the\n> working directory to match the new patterns.\n>\n> +'disable'::\n> + Remove the sparse-checkout file, set `core.sparseCheckout` to\n> + `false`, and restore the working directory to include all files.\n\nGood, so 'init' (and maybe 'set'?) will set core.sparseCheckout, and\ndisable will unset it, so the user doesn't have to worry about it...\n\n> +\n> SPARSE CHECKOUT\n> ----------------\n>\n> @@ -61,6 +65,13 @@ Then it compares the new skip-worktree value with the previous one. If\n> skip-worktree turns from set to unset, it will add the corresponding\n> file back. If it turns from unset to set, that file will be removed.\n>\n> +To repopulate the working directory with all files, use the\n> +`git sparse-checkout disable` command.\n\nGood.\n\n> +Sparse checkout support in 'git checkout' and similar commands is\n> +disabled by default. You need to set `core.sparseCheckout` to `true`\n> +in order to have sparse checkout support.\n\nAren't we having the user use 'git sparse-checkout init' to do that?\nWhy guide them to the core.sparseCheckout option? And why mention it\nwithout extensions.worktreeConfig?\n\n> +\n> ## FULL PATTERN SET\n>\n> By default, the sparse-checkout file uses the same syntax as `.gitignore`\n> @@ -75,21 +86,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n> !unwanted\n> ----------------\n>\n> -Another tricky thing is fully repopulating the working directory when you\n> -no longer want sparse checkout. You cannot just disable \"sparse\n> -checkout\" because skip-worktree bits are still in the index and your working\n> -directory is still sparsely populated. You should re-populate the working\n> -directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> -follows:\n> -\n> -----------------\n> -/*\n> -----------------\n\nYaay, glad to see this removed.\n\n> -Then you can disable sparse checkout. Sparse checkout support in 'git\n> -read-tree' and similar commands is disabled by default. You need to\n> -set `core.sparseCheckout` to `true` in order to have sparse checkout\n> -support.\n>\n> SEE ALSO\n> --------\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index f726fcd6b8..f858f0b1b5 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> - N_(\"git sparse-checkout [init|list|set] \"),\n> + N_(\"git sparse-checkout [init|list|set|disable] \"),\n> NULL\n> };\n>\n> @@ -74,7 +74,7 @@ static int update_working_directory(void)\n> return result;\n> }\n>\n> -static int sc_enable_config(void)\n> +static int sc_set_config(int mode)\n\nNice to see this change from the RFC round; do we want to use an enum\ninstead of an int, or is the int good enough? (No strong opinion\nhere, just asking.)\n\n> {\n> struct argv_array argv = ARGV_ARRAY_INIT;\n>\n> @@ -83,7 +83,12 @@ static int sc_enable_config(void)\n> return 1;\n> }\n>\n> - argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n> + argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n> +\n> + if (mode)\n> + argv_array_pushl(&argv, \"true\", NULL);\n> + else\n> + argv_array_pushl(&argv, \"false\", NULL);\n>\n> if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> error(_(\"failed to enable core.sparseCheckout\"));\n> @@ -101,7 +106,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n> int res;\n> struct object_id oid;\n>\n> - if (sc_enable_config())\n> + if (sc_set_config(1))\n> return 1;\n>\n> memset(&pl, 0, sizeof(pl));\n> @@ -188,6 +193,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> return write_patterns_and_update(&pl);\n> }\n>\n> +static int sparse_checkout_disable(int argc, const char **argv)\n> +{\n> + char *sparse_filename;\n> + FILE *fp;\n> +\n> + if (sc_set_config(1))\n> + die(_(\"failed to change config\"));\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + fp = fopen(sparse_filename, \"w\");\n> + fprintf(fp, \"/*\\n\");\n> + fclose(fp);\n> +\n> + if (update_working_directory())\n> + die(_(\"error while refreshing working directory\"));\n> +\n> + unlink(sparse_filename);\n> + free(sparse_filename);\n> +\n> + return sc_set_config(0);\n> +}\n\nSo we update the .git/info/sparse-checkout file first (or the\nworktree-specific equivalent), then call update_working_directory()\nwhich can fail -- in particular if the user calls it when they have\nany conflicted files. But then the sparse-checkout file has already\nbeen emptied, so it did make some changes, just not all the changes\nthe user would expect, leaving them in an intermediate state with an\nerror message that doesn't explain how to recover. Would it be worth\nchecking for this case, and telling the user to fix up conflicts then\nre-run the disable command? Would it make more sense to just replace\nthe 'read-tree -mu HEAD' with something that doesn't error out in such\na case? Or is this just a shortcoming of an experimental feature that\nwe'll get to later? (I'm okay with the last of those, since we also\nstill need to address defaults of several other commands when sparse\ncheckouts are active[1].)\n\n[1] https://public-inbox.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n\n> +\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> {\n> static struct option builtin_sparse_checkout_options[] = {\n> @@ -212,6 +239,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> return sparse_checkout_init(argc, argv);\n> if (!strcmp(argv[0], \"set\"))\n> return sparse_checkout_set(argc, argv, prefix);\n> + if (!strcmp(argv[0], \"disable\"))\n> + return sparse_checkout_disable(argc, argv);\n> }\n>\n> usage_with_options(builtin_sparse_checkout_usage,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 02ba9ec314..22fa032d6d 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -140,5 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'sparse-checkout disable' '\n> + git -C repo sparse-checkout disable &&\n> + test_path_is_missing repo/.git/info/sparse-checkout &&\n> + git -C repo config --list >config &&\n> + test_i18ngrep \"core.sparsecheckout=false\" config &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deep\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_done\n\nThe rest of the patch looks good.\n"},{"id":"383511","messageId":"CABPp-BFmUOoQJ6nepb9AHGcjRU+-mHvFH8TB2pP17e1C0djxmw@mail.gmail.com","threadId":"51691","inReplyTo":"25642f8df28825cce61812a24cbd87bf7cb2025f.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/11] trace2: add region in clear_ce_flags","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-06T04:13:54Z","receivedAt":"2019-10-06T04:14:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 10:15 AM Jeff Hostetler via GitGitGadget\n wrote:\n>\n> From: Jeff Hostetler \n>\n> When Git updates the working directory with the sparse-checkout\n> feature enabled, the unpack_trees() method calls clear_ce_flags()\n> to update the skip-wortree bits on the cache entries. This\n> check can be expensive, depending on the patterns used.\n>\n> Add trace2 regions around the method, including some flag\n> information, so we can get granular performance data during\n> experiments. This data will be used to measure improvements\n> to the pattern-matching algorithms for sparse-checkout.\n>\n> Signed-off-by: Jeff Hostetler \n> Signed-off-by: Derrick Stolee \n> ---\n> unpack-trees.c | 10 +++++++++-\n> 1 file changed, 9 insertions(+), 1 deletion(-)\n>\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index cd548f4fa2..26be8f3569 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1404,15 +1404,23 @@ static int clear_ce_flags(struct index_state *istate,\n> struct pattern_list *pl)\n> {\n> static struct strbuf prefix = STRBUF_INIT;\n> + char label[100];\n> + int rval;\n>\n> strbuf_reset(&prefix);\n>\n> - return clear_ce_flags_1(istate,\n> + xsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n> + (unsigned long)select_mask, (unsigned long)clear_mask);\n> + trace2_region_enter(\"unpack_trees\", label, the_repository);\n> + rval = clear_ce_flags_1(istate,\n> istate->cache,\n> istate->cache_nr,\n> &prefix,\n> select_mask, clear_mask,\n> pl, 0);\n> + trace2_region_leave(\"unpack_trees\", label, the_repository);\n> +\n> + return rval;\n> }\n>\n> /*\n> --\n> gitgitgadget\n\nThanks for the updates to the commit message, and the tweaks from\n\"exp\" to \"unpack_trees\" in the patch. I still don't know trace2, but\nit's much clearer how this relates to the series now.\n"},{"id":"383512","messageId":"CABPp-BE9HrMyskiMhQ7VxeMvZX_CCurUM_20M6md2UXZG13XEA@mail.gmail.com","threadId":"51691","inReplyTo":"84511255d1f28e1bdcec3de6096d2d9ac2a9f483.1568904188.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/11] sparse-checkout: add 'cone' mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-06T04:22:32Z","receivedAt":"2019-10-06T04:22:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Sep 19, 2019 at 1:45 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The sparse-checkout feature can have quadratic performance as\n> the number of patterns and number of entries in the index grow.\n> If there are 1,000 patterns and 1,000,000 entries, this time can\n> be very significant.\n>\n> Create a new Boolean config option, core.sparseCheckoutCone, to\n> indicate that we expect the sparse-checkout file to contain a\n> more limited set of patterns. This is a separate config setting\n> from core.sparseCheckout to avoid breaking older clients by\n> introcuding a tri-state option.\n\ns/introcuding/introducing/\n\n> The config option does nothing right now, but will be expanded\n> upon in a later commit.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/core.txt | 7 ++--\n> Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n> cache.h | 4 ++-\n> config.c | 5 +++\n> environment.c | 1 +\n> t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n> 6 files changed, 78 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\n> index 75538d27e7..9b8ab2a6d4 100644\n> --- a/Documentation/config/core.txt\n> +++ b/Documentation/config/core.txt\n> @@ -591,8 +591,11 @@ core.multiPackIndex::\n> multi-pack-index design document].\n>\n> core.sparseCheckout::\n> - Enable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> - linkgit:git-read-tree[1] for more information.\n> + Enable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n> + is disabled. If \"true\", then sparse-checkout is enabled with the full\n> + .gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n> + a restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n> + information.\n\nThis isn't consistent with the commit message that suggests it's a new\noption rather than a new possible value for an old option.\n\n> core.abbrev::\n> Set the length object names are abbreviated to. If\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index da95b28b1c..757326618d 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -87,6 +87,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n> ----------------\n>\n>\n> +## CONE PATTERN SET\n> +\n> +The full pattern set allows for arbitrary pattern matches and complicated\n> +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n> +updating the index, where N is the number of patterns and M is the number\n> +of paths in the index. To combat this performance issue, a more restricted\n> +pattern set is allowed when `core.spareCheckoutCone` is enabled.\n> +\n> +The accepted patterns in the cone pattern set are:\n> +\n> +1. *Recursive:* All paths inside a directory are included.\n> +\n> +2. *Parent:* All files immediately inside a directory are included.\n> +\n> +In addition to the above two patterns, we also expect that all files in the\n> +root directory are included. If a recursive pattern is added, then all\n> +leading directories are added as parent patterns.\n> +\n> +By default, when running `git sparse-checkout init`, the root directory is\n> +added as a parent pattern. At this point, the sparse-checkout file contains\n> +the following patterns:\n> +\n> +```\n> +/*\n> +!/*/\n> +```\n> +\n> +This says \"include everything in root, but nothing two levels below root.\"\n> +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n> +`A/B` are added as parent patterns. The resulting sparse-checkout file is\n> +now\n> +\n> +```\n> +/*\n> +!/*/\n> +/A/\n> +!/A/*/\n> +/A/B/\n> +!/A/B/*/\n> +/A/B/C/\n> +```\n> +\n> +Here, order matters, so the negative patterns are overridden by the positive\n> +patterns that appear lower in the file.\n> +\n> +If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n> +expecting patterns of these types. Git will warn if the patterns do not match.\n> +If the patterns do match the expected format, then Git will use faster hash-\n> +based algorithms to compute inclusion in the sparse-checkout.\n> +\n> SEE ALSO\n> --------\n>\n> diff --git a/cache.h b/cache.h\n> index cf5d70c196..8e8ea67efa 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -911,12 +911,14 @@ extern char *git_replace_ref_base;\n>\n> extern int fsync_object_files;\n> extern int core_preload_index;\n> -extern int core_apply_sparse_checkout;\n> extern int precomposed_unicode;\n> extern int protect_hfs;\n> extern int protect_ntfs;\n> extern const char *core_fsmonitor;\n>\n> +int core_apply_sparse_checkout;\n> +int core_sparse_checkout_cone;\n> +\n> /*\n> * Include broken refs in all ref iterations, which will\n> * generally choke dangerous operations rather than letting\n> diff --git a/config.c b/config.c\n> index 296a6d9cc4..f65c74f5b7 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1329,6 +1329,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n> return 0;\n> }\n>\n> + if (!strcmp(var, \"core.sparsecheckoutcone\")) {\n> + core_sparse_checkout_cone = git_config_bool(var, value);\n> + return 0;\n> + }\n> +\n> if (!strcmp(var, \"core.precomposeunicode\")) {\n> precomposed_unicode = git_config_bool(var, value);\n> return 0;\n> diff --git a/environment.c b/environment.c\n> index 89af47cb85..670d92bcc0 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -69,6 +69,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n> char *notes_ref_name;\n> int grafts_replace_parents = 1;\n> int core_apply_sparse_checkout;\n> +int core_sparse_checkout_cone;\n> int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> unsigned long pack_size_limit_cfg;\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 22fa032d6d..9b089c98c4 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -140,6 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'cone mode: match patterns' '\n> + git -C repo config --worktree core.sparseCheckoutCone true &&\n> + rm -rf repo/a repo/folder1 repo/folder2 &&\n> + git -C repo read-tree -mu HEAD &&\n> + git -C repo reset --hard &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_expect_success 'sparse-checkout disable' '\n> git -C repo sparse-checkout disable &&\n> test_path_is_missing repo/.git/info/sparse-checkout &&\n> --\n> gitgitgadget\n\nWhat if core.sparseCheckoutCone is true but core.sparseCheckout is\nfalse? Is that an error case we warn the user about, or do we make\nsense of it somehow?\n"},{"id":"383578","messageId":"c57c71d3-ddd6-464d-a442-713707ee94f6@gmail.com","threadId":"51691","inReplyTo":"CABPp-BHaV+XX63Y46b8wHk0rHzdAbfDQDZAQe3pHcA63WerigA@mail.gmail.com","subject":"Re: [PATCH v2 03/11] clone: add --sparse mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-07T13:56:30Z","receivedAt":"2019-10-07T13:56:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2019 3:40 PM, Elijah Newren wrote:\n> On Thu, Sep 19, 2019 at 3:06 PM Derrick Stolee via GitGitGadget\n> wrote:\n> \n>> During the 'git sparse-checkout init' call, we must first look\n>> to see if HEAD is valid, or else we will fail while trying to\n>> update the working directory. The first checkout will actually\n>> update the working directory correctly.\n> \n> This is new since the RFC series, but I'm not sure I understand. Is\n> the issue you're fixing here that a 'git init somerepo' would hit this\n> codepath and print funny errors because HEAD doesn't exist yet and\n> thus the whole `git read-tree -mu HEAD` stuff can't work? Or that\n> when the remote has HEAD pointing at a bad commit that you get error\n> messages different than expected?\n\nAt the point where `git clone --sparse` calls `git sparse-checkout init`,\nthere is no HEAD. We need to initialize the sparse-checkout before the\nclone operation populates the working directory and creates the HEAD\nref. For that reason, `git read-tree -mu HEAD` wouldn't work. But that's\nfine, since there is nothing to do. The index update will happen later.\n\n> \n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index 895479970d..656e6ebdd5 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -99,6 +99,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n>> char *sparse_filename;\n>> FILE *fp;\n>> int res;\n>> + struct object_id oid;\n>>\n>> if (sc_enable_config())\n>> return 1;\n>> @@ -120,6 +121,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n>> fprintf(fp, \"/*\\n!/*/\\n\");\n>> fclose(fp);\n>>\n>> + if (get_oid(\"HEAD\", &oid)) {\n>> + /* assume we are in a fresh repo */\n>> + return 0;\n>> + }\n>> +\n>> reset_dir:\n>> return update_working_directory();\n>> }\n\n"},{"id":"383599","messageId":"0c3e34ab-1eee-fe29-f584-d21047fd1457@gmail.com","threadId":"51691","inReplyTo":"CABPp-BGcZT2-m=D_X4d7zE9pzuWTKCnmURruahdQndpp=+-_WA@mail.gmail.com","subject":"Re: [PATCH v2 04/11] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-07T18:26:12Z","receivedAt":"2019-10-07T18:26:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2019 8:30 PM, Elijah Newren wrote:\n> On Sat, Oct 5, 2019 at 3:44 PM Elijah Newren wrote:\n>>\n>> On Thu, Sep 19, 2019 at 3:07 PM Derrick Stolee via GitGitGadget\n>> wrote:\n>>> +static int write_patterns_and_update(struct pattern_list *pl)\n>>> +{\n>>> + char *sparse_filename;\n>>> + FILE *fp;\n>>> +\n>>> + sparse_filename = get_sparse_checkout_filename();\n>>> + fp = fopen(sparse_filename, \"w\");\n>>> + write_patterns_to_file(fp, pl);\n>>> + fclose(fp);\n>>> + free(sparse_filename);\n>>> +\n>>> + clear_pattern_list(pl);\n>>\n>> It seems slightly odd that pl is passed in but cleared in this\n>> function rather than in the caller that created pl. Should this be\n>> moved to the caller, or, alternatively, a comment added to explain\n>> this side-effect for future callers of the function?\n>>\n>> The rest of the patch looked good to me.\n> \n> Actually, thought of something else. What if the user calls 'git\n> sparse-checkout set ...' without first calling 'git sparse-checkout\n> init'? Should that report an error to the user, a suggestion to\n> follow it up with 'sparse-checkout init', or should it just call\n> sc_set_config() behind the scenes and allow bypassing the init\n> subcommand?\n\nMaybe a warning would suffice. I still think the workflow of the\nfollowing is most correct, and not difficult to recommend:\n\n* \"git sparse-checkout init [--cone]\" -OR- \"git clone --sparse\"\n* git sparse-checkout set [stuff]\n* git sparse-checkout disable\n\nThanks,\n-Stolee\n"},{"id":"383601","messageId":"b1444dab-24e5-6e4d-bea8-37abc433b546@gmail.com","threadId":"51691","inReplyTo":"CABPp-BHYCCD+Vx5fq35jH82eHc1-P53Lz_aGNpHJNcx9kg2K-A@mail.gmail.com","subject":"Re: [PATCH v2 06/11] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-07T19:12:38Z","receivedAt":"2019-10-07T19:12:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/6/2019 12:10 AM, Elijah Newren wrote:\n> On Thu, Sep 19, 2019 at 1:46 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The instructions for disabling a sparse-checkout to a full\n>> working directory are complicated and non-intuitive. Add a\n>> subcommand, 'git sparse-checkout disable', to perform those\n>> steps for the user.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-sparse-checkout.txt | 26 ++++++++-----------\n>> builtin/sparse-checkout.c | 37 ++++++++++++++++++++++++---\n>> t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++\n>> 3 files changed, 59 insertions(+), 19 deletions(-)\n>>\n>> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n>> index 87813e5797..da95b28b1c 100644\n>> --- a/Documentation/git-sparse-checkout.txt\n>> +++ b/Documentation/git-sparse-checkout.txt\n>> @@ -39,6 +39,10 @@ COMMANDS\n>> a list of arguments following the 'set' subcommand. Update the\n>> working directory to match the new patterns.\n>>\n>> +'disable'::\n>> + Remove the sparse-checkout file, set `core.sparseCheckout` to\n>> + `false`, and restore the working directory to include all files.\n> \n> Good, so 'init' (and maybe 'set'?) will set core.sparseCheckout, and\n> disable will unset it, so the user doesn't have to worry about it...\n> \n>> +\n>> SPARSE CHECKOUT\n>> ----------------\n>>\n>> @@ -61,6 +65,13 @@ Then it compares the new skip-worktree value with the previous one. If\n>> skip-worktree turns from set to unset, it will add the corresponding\n>> file back. If it turns from unset to set, that file will be removed.\n>>\n>> +To repopulate the working directory with all files, use the\n>> +`git sparse-checkout disable` command.\n> \n> Good.\n> \n>> +Sparse checkout support in 'git checkout' and similar commands is\n>> +disabled by default. You need to set `core.sparseCheckout` to `true`\n>> +in order to have sparse checkout support.\n> \n> Aren't we having the user use 'git sparse-checkout init' to do that?\n> Why guide them to the core.sparseCheckout option? And why mention it\n> without extensions.worktreeConfig?\n\nI'll add a paragraph above the \"To repopulate...\" to describe using 'init'\nand 'set' instead of relying on the old phrasing.\n\n>> +\n>> ## FULL PATTERN SET\n>>\n>> By default, the sparse-checkout file uses the same syntax as `.gitignore`\n>> @@ -75,21 +86,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n>> !unwanted\n>> ----------------\n>>\n>> -Another tricky thing is fully repopulating the working directory when you\n>> -no longer want sparse checkout. You cannot just disable \"sparse\n>> -checkout\" because skip-worktree bits are still in the index and your working\n>> -directory is still sparsely populated. You should re-populate the working\n>> -directory with the `$GIT_DIR/info/sparse-checkout` file content as\n>> -follows:\n>> -\n>> -----------------\n>> -/*\n>> -----------------\n> \n> Yaay, glad to see this removed.\n> \n>> -Then you can disable sparse checkout. Sparse checkout support in 'git\n>> -read-tree' and similar commands is disabled by default. You need to\n>> -set `core.sparseCheckout` to `true` in order to have sparse checkout\n>> -support.\n>>\n>> SEE ALSO\n>> --------\n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index f726fcd6b8..f858f0b1b5 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -8,7 +8,7 @@\n>> #include \"strbuf.h\"\n>>\n>> static char const * const builtin_sparse_checkout_usage[] = {\n>> - N_(\"git sparse-checkout [init|list|set] \"),\n>> + N_(\"git sparse-checkout [init|list|set|disable] \"),\n>> NULL\n>> };\n>>\n>> @@ -74,7 +74,7 @@ static int update_working_directory(void)\n>> return result;\n>> }\n>>\n>> -static int sc_enable_config(void)\n>> +static int sc_set_config(int mode)\n> \n> Nice to see this change from the RFC round; do we want to use an enum\n> instead of an int, or is the int good enough? (No strong opinion\n> here, just asking.)\n\nI'll use an enum in v3.\n\n>> {\n>> struct argv_array argv = ARGV_ARRAY_INIT;\n>>\n>> @@ -83,7 +83,12 @@ static int sc_enable_config(void)\n>> return 1;\n>> }\n>>\n>> - argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n>> + argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n>> +\n>> + if (mode)\n>> + argv_array_pushl(&argv, \"true\", NULL);\n>> + else\n>> + argv_array_pushl(&argv, \"false\", NULL);\n>>\n>> if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n>> error(_(\"failed to enable core.sparseCheckout\"));\n>> @@ -101,7 +106,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n>> int res;\n>> struct object_id oid;\n>>\n>> - if (sc_enable_config())\n>> + if (sc_set_config(1))\n>> return 1;\n>>\n>> memset(&pl, 0, sizeof(pl));\n>> @@ -188,6 +193,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n>> return write_patterns_and_update(&pl);\n>> }\n>>\n>> +static int sparse_checkout_disable(int argc, const char **argv)\n>> +{\n>> + char *sparse_filename;\n>> + FILE *fp;\n>> +\n>> + if (sc_set_config(1))\n>> + die(_(\"failed to change config\"));\n>> +\n>> + sparse_filename = get_sparse_checkout_filename();\n>> + fp = fopen(sparse_filename, \"w\");\n>> + fprintf(fp, \"/*\\n\");\n>> + fclose(fp);\n>> +\n>> + if (update_working_directory())\n>> + die(_(\"error while refreshing working directory\"));\n>> +\n>> + unlink(sparse_filename);\n>> + free(sparse_filename);\n>> +\n>> + return sc_set_config(0);\n>> +}\n> \n> So we update the .git/info/sparse-checkout file first (or the\n> worktree-specific equivalent), then call update_working_directory()\n> which can fail -- in particular if the user calls it when they have\n> any conflicted files. But then the sparse-checkout file has already\n> been emptied, so it did make some changes, just not all the changes\n> the user would expect, leaving them in an intermediate state with an\n> error message that doesn't explain how to recover. Would it be worth\n> checking for this case, and telling the user to fix up conflicts then\n> re-run the disable command? Would it make more sense to just replace\n> the 'read-tree -mu HEAD' with something that doesn't error out in such\n> a case? Or is this just a shortcoming of an experimental feature that\n> we'll get to later? (I'm okay with the last of those, since we also\n> still need to address defaults of several other commands when sparse\n> checkouts are active[1].)\n\nI think there are multiple edge cases that make the sparse-checkout\nfeature worthy of the \"experimental\" descriptor. To be explicit about\na case where update_working_directory() would fail when the sparse-checkout\nfile only contains \"/*\", the only case I can think of is when a user has\nwritten a file outside the current sparse set but HEAD thinks that path\nshould be a folder (or vice-versa).\n\nWe will definitely want to make the feature more robust to these corner\ncases, but that will take time. For now, let's get a framework that is\nfunctional for 99% of cases.\n\nAnd this must be said: none of these changes are permanently damaging.\nIf a user gets in a strange state due to these corner cases, they are\nno worse off than they would be trying to follow the existing directions.\n\nAnd in v3, I'll add some new commits that help these kinds of cases\nduring the 'set' operation by not writing to the sparse-checkout file\nuntil the working directory update has succeeded.\n\n> [1] https://public-inbox.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n> \n>> +\n>> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n>> {\n>> static struct option builtin_sparse_checkout_options[] = {\n>> @@ -212,6 +239,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n>> return sparse_checkout_init(argc, argv);\n>> if (!strcmp(argv[0], \"set\"))\n>> return sparse_checkout_set(argc, argv, prefix);\n>> + if (!strcmp(argv[0], \"disable\"))\n>> + return sparse_checkout_disable(argc, argv);\n>> }\n>>\n>> usage_with_options(builtin_sparse_checkout_usage,\n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index 02ba9ec314..22fa032d6d 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -140,5 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n>> test_cmp expect dir\n>> '\n>>\n>> +test_expect_success 'sparse-checkout disable' '\n>> + git -C repo sparse-checkout disable &&\n>> + test_path_is_missing repo/.git/info/sparse-checkout &&\n>> + git -C repo config --list >config &&\n>> + test_i18ngrep \"core.sparsecheckout=false\" config &&\n>> + ls repo >dir &&\n>> + cat >expect <<-EOF &&\n>> + a\n>> + deep\n>> + folder1\n>> + folder2\n>> + EOF\n>> + test_cmp expect dir\n>> +'\n>> +\n>> test_done\n> \n> The rest of the patch looks good.\n> \n\n"},{"id":"383602","messageId":"991a2cee-fa1e-547c-c885-a4aaf663d9b6@gmail.com","threadId":"51691","inReplyTo":"CABPp-BE9HrMyskiMhQ7VxeMvZX_CCurUM_20M6md2UXZG13XEA@mail.gmail.com","subject":"Re: [PATCH v2 08/11] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-07T19:15:27Z","receivedAt":"2019-10-07T19:15:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/6/2019 12:22 AM, Elijah Newren wrote:\n> On Thu, Sep 19, 2019 at 1:45 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The sparse-checkout feature can have quadratic performance as\n>> the number of patterns and number of entries in the index grow.\n>> If there are 1,000 patterns and 1,000,000 entries, this time can\n>> be very significant.\n>>\n>> Create a new Boolean config option, core.sparseCheckoutCone, to\n>> indicate that we expect the sparse-checkout file to contain a\n>> more limited set of patterns. This is a separate config setting\n>> from core.sparseCheckout to avoid breaking older clients by\n>> introcuding a tri-state option.\n> \n> s/introcuding/introducing/\n> \n>> The config option does nothing right now, but will be expanded\n>> upon in a later commit.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/config/core.txt | 7 ++--\n>> Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n>> cache.h | 4 ++-\n>> config.c | 5 +++\n>> environment.c | 1 +\n>> t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n>> 6 files changed, 78 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\n>> index 75538d27e7..9b8ab2a6d4 100644\n>> --- a/Documentation/config/core.txt\n>> +++ b/Documentation/config/core.txt\n>> @@ -591,8 +591,11 @@ core.multiPackIndex::\n>> multi-pack-index design document].\n>>\n>> core.sparseCheckout::\n>> - Enable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n>> - linkgit:git-read-tree[1] for more information.\n>> + Enable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n>> + is disabled. If \"true\", then sparse-checkout is enabled with the full\n>> + .gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n>> + a restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n>> + information.\n> \n> This isn't consistent with the commit message that suggests it's a new\n> option rather than a new possible value for an old option.\n\nThanks for the catch. I forgot to update the docs. Fixed for v3.\n\n>> core.abbrev::\n>> Set the length object names are abbreviated to. If\n>> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n>> index da95b28b1c..757326618d 100644\n>> --- a/Documentation/git-sparse-checkout.txt\n>> +++ b/Documentation/git-sparse-checkout.txt\n>> @@ -87,6 +87,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n>> ----------------\n>>\n>>\n>> +## CONE PATTERN SET\n>> +\n>> +The full pattern set allows for arbitrary pattern matches and complicated\n>> +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n>> +updating the index, where N is the number of patterns and M is the number\n>> +of paths in the index. To combat this performance issue, a more restricted\n>> +pattern set is allowed when `core.spareCheckoutCone` is enabled.\n>> +\n>> +The accepted patterns in the cone pattern set are:\n>> +\n>> +1. *Recursive:* All paths inside a directory are included.\n>> +\n>> +2. *Parent:* All files immediately inside a directory are included.\n>> +\n>> +In addition to the above two patterns, we also expect that all files in the\n>> +root directory are included. If a recursive pattern is added, then all\n>> +leading directories are added as parent patterns.\n>> +\n>> +By default, when running `git sparse-checkout init`, the root directory is\n>> +added as a parent pattern. At this point, the sparse-checkout file contains\n>> +the following patterns:\n>> +\n>> +```\n>> +/*\n>> +!/*/\n>> +```\n>> +\n>> +This says \"include everything in root, but nothing two levels below root.\"\n>> +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n>> +`A/B` are added as parent patterns. The resulting sparse-checkout file is\n>> +now\n>> +\n>> +```\n>> +/*\n>> +!/*/\n>> +/A/\n>> +!/A/*/\n>> +/A/B/\n>> +!/A/B/*/\n>> +/A/B/C/\n>> +```\n>> +\n>> +Here, order matters, so the negative patterns are overridden by the positive\n>> +patterns that appear lower in the file.\n>> +\n>> +If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n>> +expecting patterns of these types. Git will warn if the patterns do not match.\n>> +If the patterns do match the expected format, then Git will use faster hash-\n>> +based algorithms to compute inclusion in the sparse-checkout.\n>> +\n>> SEE ALSO\n>> --------\n>>\n>> diff --git a/cache.h b/cache.h\n>> index cf5d70c196..8e8ea67efa 100644\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -911,12 +911,14 @@ extern char *git_replace_ref_base;\n>>\n>> extern int fsync_object_files;\n>> extern int core_preload_index;\n>> -extern int core_apply_sparse_checkout;\n>> extern int precomposed_unicode;\n>> extern int protect_hfs;\n>> extern int protect_ntfs;\n>> extern const char *core_fsmonitor;\n>>\n>> +int core_apply_sparse_checkout;\n>> +int core_sparse_checkout_cone;\n>> +\n>> /*\n>> * Include broken refs in all ref iterations, which will\n>> * generally choke dangerous operations rather than letting\n>> diff --git a/config.c b/config.c\n>> index 296a6d9cc4..f65c74f5b7 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -1329,6 +1329,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n>> return 0;\n>> }\n>>\n>> + if (!strcmp(var, \"core.sparsecheckoutcone\")) {\n>> + core_sparse_checkout_cone = git_config_bool(var, value);\n>> + return 0;\n>> + }\n>> +\n>> if (!strcmp(var, \"core.precomposeunicode\")) {\n>> precomposed_unicode = git_config_bool(var, value);\n>> return 0;\n>> diff --git a/environment.c b/environment.c\n>> index 89af47cb85..670d92bcc0 100644\n>> --- a/environment.c\n>> +++ b/environment.c\n>> @@ -69,6 +69,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n>> char *notes_ref_name;\n>> int grafts_replace_parents = 1;\n>> int core_apply_sparse_checkout;\n>> +int core_sparse_checkout_cone;\n>> int merge_log_config = -1;\n>> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n>> unsigned long pack_size_limit_cfg;\n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index 22fa032d6d..9b089c98c4 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -140,6 +140,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n>> test_cmp expect dir\n>> '\n>>\n>> +test_expect_success 'cone mode: match patterns' '\n>> + git -C repo config --worktree core.sparseCheckoutCone true &&\n>> + rm -rf repo/a repo/folder1 repo/folder2 &&\n>> + git -C repo read-tree -mu HEAD &&\n>> + git -C repo reset --hard &&\n>> + ls repo >dir &&\n>> + cat >expect <<-EOF &&\n>> + a\n>> + folder1\n>> + folder2\n>> + EOF\n>> + test_cmp expect dir\n>> +'\n>> +\n>> test_expect_success 'sparse-checkout disable' '\n>> git -C repo sparse-checkout disable &&\n>> test_path_is_missing repo/.git/info/sparse-checkout &&\n>> --\n>> gitgitgadget\n> \n> What if core.sparseCheckoutCone is true but core.sparseCheckout is\n> false? Is that an error case we warn the user about, or do we make\n> sense of it somehow?\n\n`core.sparseCheckoutCone` means \"when you are doing sparse-checkout things,\ntry to use the fast logic assuming you have the cone patterns.\" It does not\nenable the sparse-checkout on its own. It only changes run-time performance.\n\nThis is important for compatibility with other clients that will only look\nat `core.sparseCheckout`.\n\nThanks,\n-Stolee\n\n"},{"id":"383607","messageId":"30a0db68cd141bab5a9fb63e26e767579a42249c.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:27Z","receivedAt":"2019-10-07T20:08:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na more restricted pattern set.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 85 ++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 86 +++++++++++++++++++++++++++\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 50 ++++++++++++++++\n 8 files changed, 226 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 4470d7cfc0..5ccc3d00dd 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -156,6 +156,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..81676b1d33\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,85 @@\n+git-sparse-checkout(1)\n+=======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of directories\n+given by a list of prefixes.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of directories given by a list of prefixes.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+----------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it updates the skip-worktree bits in the index based\n+ont this file. The files matching the patterns in the file will\n+appear in the working directory, and the rest will not.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are included, you can also specify what files are _not_ included,\n+using negative patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+checkout' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f58bf14c7b..f3322b75dd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1121,6 +1121,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex ec7e0954c4..d517068faa 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -223,6 +223,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..eed9625a05\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,86 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout [list]\"),\n+\tNULL\n+};\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pl->nr; i++) {\n+\t\tstruct path_pattern *p = pl->patterns[i];\n+\n+\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", p->pattern);\n+\n+\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_patterns_to_file(stdout, &pl);\n+\tclear_pattern_list(&pl);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/git.c b/git.c\nindex c2eec470c9..e775fbad42 100644\n--- a/git.c\n+++ b/git.c\n@@ -576,6 +576,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..a9b04b1a88\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,50 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_line_count = 0 list &&\n+\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tgit -C repo sparse-checkout list >list &&\n+\tcat >expect <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\ttest_cmp expect list\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"383608","messageId":"c8587a1fb0e335f9cbafdbb44b20d02a1e57886b.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 03/17] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:29Z","receivedAt":"2019-10-07T20:08:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nDuring the 'git sparse-checkout init' call, we must first look\nto see if HEAD is valid, since 'git clone' does not have a valid\nHEAD. The first checkout will create the HEAD ref and update the\nworking directory correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n builtin/sparse-checkout.c | 6 ++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 4 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 5fc97f14de..03299a8adb 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex a693e6ca44..16f4e8b6fd 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -58,6 +58,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -145,6 +146,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -723,6 +726,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1096,6 +1120,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 64b2bb2b8c..3ecb7ac2e7 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tstruct object_id oid;\n \n \tif (sc_set_config(MODE_FULL))\n \t\treturn 1;\n@@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \tfclose(fp);\n \n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n reset_dir:\n \treturn update_working_directory();\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex c70085a759..d4c145a3af 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -88,4 +88,17 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383609","messageId":"6ce1d60b38a249f8d9f85d1b1a1ed556aef2469e.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 04/17] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:30Z","receivedAt":"2019-10-07T20:08:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nas arguments and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nThe 'set' subcommand will replace the entire contents of the\nsparse-checkout file. The write_patterns_and_update() method is\nextracted from cmd_sparse_checkout() to make it easier to implement\n'add' and/or 'remove' subcommands in the future.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 5 ++++\n builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 19 +++++++++++++++\n 3 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex e095c4a98b..f4bd951550 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -39,6 +39,11 @@ and sets the `core.sparseCheckout` setting in the worktree-specific config\n file. This prevents the sparse-checkout feature from interfering with other\n worktrees.\n \n+'set'::\n+\tWrite a set of patterns to the sparse-checkout file, as given as\n+\ta list of arguments following the 'set' subcommand. Update the\n+\tworking directory to match the new patterns.\n+\n SPARSE CHECKOUT\n ----------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 3ecb7ac2e7..52d4f832f3 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list]\"),\n+\tN_(\"git sparse-checkout [init|list|set] \"),\n \tNULL\n };\n \n@@ -140,6 +140,37 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_patterns_to_file(fp, pl);\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\treturn update_working_directory();\n+}\n+\n+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *empty_base = \"\";\n+\tint i;\n+\tstruct pattern_list pl;\n+\tint result;\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tfor (i = 1; i < argc; i++)\n+\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\n+\tresult = write_patterns_and_update(&pl);\n+\n+\tclear_pattern_list(&pl);\n+\treturn result;\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -162,6 +193,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"set\"))\n+\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex d4c145a3af..19e8673c6b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -101,4 +101,23 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using builtin' '\n+\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383610","messageId":"pull.316.v3.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:26Z","receivedAt":"2019-10-07T20:08:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series makes the sparse-checkout feature more user-friendly. While\nthere, I also present a way to use a limited set of patterns to gain a\nsignificant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis series attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout set reads patterns from the arguments, or with\n --stdin reads patterns from stdin one per line, then writes them to the\n sparse-checkout file and refreshes the working directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\nUpdates in v2, relative to the RFC:\n\n * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n adding 'add' and/or 'remove' subcommands later.\n \n \n * 'set' reads from the arguments by default. '--stdin' option is available.\n \n \n * A new performance-oriented commit is added at the end.\n \n \n * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n pattern.\n \n \n * References to a \"bug\" (that was really a strange GVFS interaction in\n microsoft/git) around deleting outside the cone are removed.\n \n \n\nUpdates in v3:\n\n * The bad interaction with \"cone mode\" and .gitignore files is fixed. A\n test is added in the last patch.\n \n \n * Several patches are added that make the feature more robust. One\n sanitizes user input, another few add progress indicators, and another\n more prevent users from getting in bad states due to working directory\n changes or concurrent processes.\n \n \n * Updated several docs and commit messages according to feedback. Thanks,\n Elijah!\n \n \n\nThings to leave for future patches:\n\n 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n worktree-specific file.\n \n \n 2. More robustness around detecting non-cone patterns with wildcards in the\n middle of the line.\n \n \n 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n super-valuable, as it only starts changing behavior when someone calls\n 'git sparse-checkout set', but may be interesting.\n \n \n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (16):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'set' subcommand\n sparse-checkout: add '--stdin' option to set subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and set in cone mode\n unpack-trees: hash less in cone mode\n unpack-trees: add progress to clear_ce_flags()\n read-tree: show progress by default\n sparse-checkout: sanitize for nested folders\n sparse-checkout: update working directory in-process\n sparse-checkout: write using lockfile\n sparse-checkout: cone mode should not interact with .gitignore\n\nJeff Hostetler (1):\n trace2: add region in clear_ce_flags\n\n .gitignore | 1 +\n Documentation/config/core.txt | 10 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 149 ++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/read-tree.c | 3 +-\n builtin/sparse-checkout.c | 485 ++++++++++++++++++++++++++\n cache.h | 6 +-\n config.c | 5 +\n dir.c | 202 ++++++++++-\n dir.h | 34 ++\n environment.c | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 279 +++++++++++++++\n unpack-trees.c | 110 ++++--\n unpack-trees.h | 3 +-\n 19 files changed, 1280 insertions(+), 48 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 468ce99b77a0efaf1ace4c31a7b0a7d036fd9ca1\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n\nRange-diff vs v2:\n\n 1: dbaf3de88e ! 1: 30a0db68cd sparse-checkout: create builtin with 'list' subcommand\n @@ -15,12 +15,6 @@\n builtin will be the preferred mechanism for manipulating the\n sparse-checkout file and syncing the working directory.\n \n - The `$GIT_DIR/info/sparse-checkout` file defines the skip-\n - worktree reference bitmap. When Git updates the working\n - directory, it updates the skip-worktree bits in the index\n - based on this file and removes or restores files in the\n - working copy to match.\n -\n The documentation provided is adapted from the \"git read-tree\"\n documentation with a few edits for clarity in the new context.\n Extra sections are added to hint toward a future change to\n @@ -101,14 +95,9 @@\n +\n +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n +skip-worktree reference bitmap. When Git updates the working\n -+directory, it resets the skip-worktree bit in the index based on this\n -+file. If an entry\n -+matches a pattern in this file, skip-worktree will not be set on\n -+that entry. Otherwise, skip-worktree will be set.\n -+\n -+Then it compares the new skip-worktree value with the previous one. If\n -+skip-worktree turns from set to unset, it will add the corresponding\n -+file back. If it turns from unset to set, that file will be removed.\n ++directory, it updates the skip-worktree bits in the index based\n ++ont this file. The files matching the patterns in the file will\n ++appear in the working directory, and the rest will not.\n +\n +## FULL PATTERN SET\n +\n @@ -136,7 +125,7 @@\n +----------------\n +\n +Then you can disable sparse checkout. Sparse checkout support in 'git\n -+read-tree' and similar commands is disabled by default. You need to\n ++checkout' and similar commands is disabled by default. You need to\n +set `core.sparseCheckout` to `true` in order to have sparse checkout\n +support.\n +\n @@ -333,4 +322,3 @@\n +'\n +\n +test_done\n -+\n 2: 412211f5dd ! 2: 08bb6fb7f3 sparse-checkout: create 'init' subcommand\n @@ -8,12 +8,12 @@\n an initial set of patterns to the sparse-checkout file, and update\n their working directory.\n \n - Using 'git read-tree' to clear directories does not work cleanly\n - on Windows, so manually delete directories that are tracked by Git\n - before running read-tree.\n + Make sure to use the `extensions.worktreeConfig` setting and write\n + the sparse checkout config to the worktree-specific config file.\n + This avoids confusing interactions with other worktrees.\n \n - The use of running another process for 'git read-tree' is likely\n - suboptimal, but that can be improved in a later change, if valuable.\n + The use of running another process for 'git read-tree' is sub-\n + optimal. This will be removed in a later change.\n \n Signed-off-by: Derrick Stolee \n \n @@ -31,6 +31,11 @@\n +\tno other directories, then will remove all directories tracked\n +\tby Git. Add patterns to the sparse-checkout file to\n +\trepopulate the working directory.\n +++\n ++The init subcommand also enables the 'extensions.worktreeConfig' setting\n ++and sets the `core.sparseCheckout` setting in the worktree-specific config\n ++file. This prevents the sparse-checkout feature from interfering with other\n ++worktrees.\n \n SPARSE CHECKOUT\n ----------------\n @@ -66,7 +71,12 @@\n +\treturn result;\n +}\n +\n -+static int sc_enable_config(void)\n ++enum sparse_checkout_mode {\n ++\tMODE_NONE = 0,\n ++\tMODE_FULL = 1,\n ++};\n ++\n ++static int sc_set_config(enum sparse_checkout_mode mode)\n +{\n +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n +\n @@ -75,7 +85,12 @@\n +\t\treturn 1;\n +\t}\n +\n -+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n ++\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n ++\n ++\tif (mode)\n ++\t\targv_array_pushl(&argv, \"true\", NULL);\n ++\telse\n ++\t\targv_array_pushl(&argv, \"false\", NULL);\n +\n +\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n +\t\terror(_(\"failed to enable core.sparseCheckout\"));\n @@ -92,7 +107,7 @@\n +\tFILE *fp;\n +\tint res;\n +\n -+\tif (sc_enable_config())\n ++\tif (sc_set_config(MODE_FULL))\n +\t\treturn 1;\n +\n +\tmemset(&pl, 0, sizeof(pl));\n @@ -178,4 +193,3 @@\n +'\n +\n test_done\n - \n 3: fef41b794a ! 3: c8587a1fb0 clone: add --sparse mode\n @@ -21,9 +21,9 @@\n point.\n \n During the 'git sparse-checkout init' call, we must first look\n - to see if HEAD is valid, or else we will fail while trying to\n - update the working directory. The first checkout will actually\n - update the working directory correctly.\n + to see if HEAD is valid, since 'git clone' does not have a valid\n + HEAD. The first checkout will create the HEAD ref and update the\n + working directory correctly.\n \n Signed-off-by: Derrick Stolee \n \n @@ -121,7 +121,7 @@\n \tint res;\n +\tstruct object_id oid;\n \n - \tif (sc_enable_config())\n + \tif (sc_set_config(MODE_FULL))\n \t\treturn 1;\n @@\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n @@ -157,4 +157,3 @@\n +'\n +\n test_done\n - \n 4: 9a78f9ea0f ! 4: 6ce1d60b38 sparse-checkout: 'set' subcommand\n @@ -17,8 +17,8 @@\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n - \tby Git. Add patterns to the sparse-checkout file to\n - \trepopulate the working directory.\n + file. This prevents the sparse-checkout feature from interfering with other\n + worktrees.\n \n +'set'::\n +\tWrite a set of patterns to the sparse-checkout file, as given as\n @@ -56,20 +56,24 @@\n +\tfclose(fp);\n +\tfree(sparse_filename);\n +\n -+\tclear_pattern_list(pl);\n +\treturn update_working_directory();\n +}\n +\n +static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n +{\n ++\tstatic const char *empty_base = \"\";\n +\tint i;\n +\tstruct pattern_list pl;\n ++\tint result;\n +\tmemset(&pl, 0, sizeof(pl));\n +\n +\tfor (i = 1; i < argc; i++)\n -+\t\tadd_pattern(argv[i], NULL, 0, &pl, 0);\n ++\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\n -+\treturn write_patterns_and_update(&pl);\n ++\tresult = write_patterns_and_update(&pl);\n ++\n ++\tclear_pattern_list(&pl);\n ++\treturn result;\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n @@ -112,4 +116,3 @@\n +'\n +\n test_done\n - \n 5: 21a0165be7 ! 5: 0b1ed06bc8 sparse-checkout: add '--stdin' option to set subcommand\n @@ -14,6 +14,18 @@\n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n +@@\n + \tchar *sparse_filename;\n + \tFILE *fp;\n + \n ++\tif (!core_apply_sparse_checkout) {\n ++\t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n ++\t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n ++\t}\n ++\n + \tsparse_filename = get_sparse_checkout_filename();\n + \tfp = fopen(sparse_filename, \"w\");\n + \twrite_patterns_to_file(fp, pl);\n @@\n \treturn update_working_directory();\n }\n @@ -29,8 +41,10 @@\n +\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n + \tstatic const char *empty_base = \"\";\n \tint i;\n \tstruct pattern_list pl;\n + \tint result;\n +\n +\tstatic struct option builtin_sparse_checkout_set_options[] = {\n +\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n @@ -41,7 +55,7 @@\n \tmemset(&pl, 0, sizeof(pl));\n \n -\tfor (i = 1; i < argc; i++)\n --\t\tadd_pattern(argv[i], NULL, 0, &pl, 0);\n +-\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\targc = parse_options(argc, argv, prefix,\n +\t\t\t builtin_sparse_checkout_set_options,\n +\t\t\t builtin_sparse_checkout_set_usage,\n @@ -53,15 +67,15 @@\n +\t\twhile (!strbuf_getline(&line, stdin)) {\n +\t\t\tsize_t len;\n +\t\t\tchar *buf = strbuf_detach(&line, &len);\n -+\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n ++\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n +\t\t}\n +\t} else {\n +\t\tfor (i = 0; i < argc; i++)\n -+\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n ++\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\t}\n \n - \treturn write_patterns_and_update(&pl);\n - }\n + \tresult = write_patterns_and_update(&pl);\n + \n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -70,6 +84,20 @@\n \ttest_cmp expect dir\n '\n \n ++test_expect_success 'warn if core.sparseCheckout is disabled' '\n ++\ttest_when_finished git -C repo config --worktree core.sparseCheckout true &&\n ++\tgit -C repo config --worktree core.sparseCheckout false &&\n ++\tgit -C repo sparse-checkout set folder1 2>err &&\n ++\ttest_i18ngrep \"core.sparseCheckout is disabled\" err\n ++'\n ++\n + test_expect_success 'set sparse-checkout using builtin' '\n + \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n + \tcat >expect <<-EOF &&\n +@@\n + \ttest_cmp expect dir\n + '\n + \n +test_expect_success 'set sparse-checkout using --stdin' '\n +\tcat >expect <<-EOF &&\n +\t\t/*\n @@ -91,4 +119,3 @@\n +'\n +\n test_done\n - \n 6: b62b76013f ! 6: 22b9bd21f4 sparse-checkout: create 'disable' subcommand\n @@ -24,15 +24,16 @@\n ----------------\n \n @@\n - skip-worktree turns from set to unset, it will add the corresponding\n - file back. If it turns from unset to set, that file will be removed.\n + ont this file. The files matching the patterns in the file will\n + appear in the working directory, and the rest will not.\n \n ++To enable the sparse-checkout feature, run `git sparse-checkout init` to\n ++initialize a simple sparse-checkout file and enable the `core.sparseCheckout`\n ++config setting. Then, run `git sparse-checkout set` to modify the patterns in\n ++the sparse-checkout file.\n ++\n +To repopulate the working directory with all files, use the\n +`git sparse-checkout disable` command.\n -+\n -+Sparse checkout support in 'git checkout' and similar commands is\n -+disabled by default. You need to set `core.sparseCheckout` to `true`\n -+in order to have sparse checkout support.\n +\n ## FULL PATTERN SET\n \n @@ -53,7 +54,7 @@\n -----------------\n -\n -Then you can disable sparse checkout. Sparse checkout support in 'git\n --read-tree' and similar commands is disabled by default. You need to\n +-checkout' and similar commands is disabled by default. You need to\n -set `core.sparseCheckout` to `true` in order to have sparse checkout\n -support.\n \n @@ -76,44 +77,12 @@\n \treturn result;\n }\n \n --static int sc_enable_config(void)\n -+static int sc_set_config(int mode)\n - {\n - \tstruct argv_array argv = ARGV_ARRAY_INIT;\n - \n -@@\n - \t\treturn 1;\n - \t}\n - \n --\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", \"true\", NULL);\n -+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n -+\n -+\tif (mode)\n -+\t\targv_array_pushl(&argv, \"true\", NULL);\n -+\telse\n -+\t\targv_array_pushl(&argv, \"false\", NULL);\n - \n - \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n - \t\terror(_(\"failed to enable core.sparseCheckout\"));\n -@@\n - \tint res;\n - \tstruct object_id oid;\n - \n --\tif (sc_enable_config())\n -+\tif (sc_set_config(1))\n - \t\treturn 1;\n - \n - \tmemset(&pl, 0, sizeof(pl));\n -@@\n - \treturn write_patterns_and_update(&pl);\n - }\n - \n +static int sparse_checkout_disable(int argc, const char **argv)\n +{\n +\tchar *sparse_filename;\n +\tFILE *fp;\n +\n -+\tif (sc_set_config(1))\n ++\tif (sc_set_config(MODE_FULL))\n +\t\tdie(_(\"failed to change config\"));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n @@ -127,7 +96,7 @@\n +\tunlink(sparse_filename);\n +\tfree(sparse_filename);\n +\n -+\treturn sc_set_config(0);\n ++\treturn sc_set_config(MODE_NONE);\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n @@ -166,4 +135,3 @@\n +'\n +\n test_done\n - \n 7: 25642f8df2 = 7: 2c53ea13d0 trace2: add region in clear_ce_flags\n 8: 84511255d1 ! 8: a66ec1affc sparse-checkout: add 'cone' mode\n @@ -11,7 +11,7 @@\n indicate that we expect the sparse-checkout file to contain a\n more limited set of patterns. This is a separate config setting\n from core.sparseCheckout to avoid breaking older clients by\n - introcuding a tri-state option.\n + introducing a tri-state option.\n \n The config option does nothing right now, but will be expanded\n upon in a later commit.\n @@ -27,11 +27,14 @@\n core.sparseCheckout::\n -\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n -\tlinkgit:git-read-tree[1] for more information.\n -+\tEnable \"sparse checkout\" feature. If \"false\", then sparse-checkout\n -+\tis disabled. If \"true\", then sparse-checkout is enabled with the full\n -+\t.gitignore pattern set. If \"cone\", then sparse-checkout is enabled with\n -+\ta restricted pattern set. See linkgit:git-sparse-checkout[1] for more\n -+\tinformation.\n ++\tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\n ++\tfor more information.\n ++\n ++core.sparseCheckoutCone::\n ++\tEnables the \"cone mode\" of the sparse checkout feature. When the\n ++\tsparse-checkout file contains a limited set of patterns, then this\n ++\tmode provides significant performance advantages. See\n ++\tlinkgit:git-sparse-checkout[1] for more information.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\n 9: 95a3285bc6 ! 9: 431933bec6 sparse-checkout: use hashmaps for cone patterns\n @@ -64,8 +64,17 @@\n +\tif (!pl->use_cone_patterns)\n +\t\treturn;\n +\n -+\tif (!strcmp(given->pattern, \"/*\"))\n ++\tif (given->flags & PATTERN_FLAG_NEGATIVE &&\n ++\t given->flags & PATTERN_FLAG_MUSTBEDIR &&\n ++\t !strcmp(given->pattern, \"/*\")) {\n ++\t\tpl->full_cone = 0;\n +\t\treturn;\n ++\t}\n ++\n ++\tif (!given->flags && !strcmp(given->pattern, \"/*\")) {\n ++\t\tpl->full_cone = 1;\n ++\t\treturn;\n ++\t}\n +\n +\tif (given->patternlen > 2 &&\n +\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n @@ -144,6 +153,33 @@\n +\thashmap_entry_init(&p, memhash(p.pattern, p.patternlen));\n +\treturn !!hashmap_get(map, &p, NULL);\n +}\n ++\n ++int hashmap_contains_parent(struct hashmap *map,\n ++\t\t\t const char *path,\n ++\t\t\t struct strbuf *buffer)\n ++{\n ++\tchar *slash_pos;\n ++\n ++\tstrbuf_setlen(buffer, 0);\n ++\n ++\tif (path[0] != '/')\n ++\t\tstrbuf_addch(buffer, '/');\n ++\n ++\tstrbuf_addstr(buffer, path);\n ++\n ++\tslash_pos = strrchr(buffer->buf, '/');\n ++\n ++\twhile (slash_pos > buffer->buf) {\n ++\t\tstrbuf_setlen(buffer, slash_pos - buffer->buf);\n ++\n ++\t\tif (hashmap_contains_path(map, buffer))\n ++\t\t\treturn 1;\n ++\n ++\t\tslash_pos = strrchr(buffer->buf, '/');\n ++\t}\n ++\n ++\treturn 0;\n ++}\n +\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n @@ -161,7 +197,6 @@\n \tint i, lineno = 1;\n \tchar *entry;\n \n -+\tpl->use_cone_patterns = core_sparse_checkout_cone;\n +\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n +\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n +\n @@ -194,14 +229,16 @@\n +\t\t}\n +\n +\t\treturn UNDECIDED;\n - \t}\n - \n --\treturn UNDECIDED;\n ++\t}\n ++\n ++\tif (pl->full_cone)\n ++\t\treturn MATCHED;\n ++\n +\tstrbuf_addch(&parent_pathname, '/');\n +\tstrbuf_add(&parent_pathname, pathname, pathlen);\n +\n +\tif (hashmap_contains_path(&pl->recursive_hashmap,\n -+\t\t\t\t\t&parent_pathname)) {\n ++\t\t\t\t &parent_pathname)) {\n +\t\tresult = MATCHED;\n +\t\tgoto done;\n +\t}\n @@ -212,8 +249,9 @@\n +\t\t/* include every file in root */\n +\t\tresult = MATCHED;\n +\t\tgoto done;\n -+\t}\n -+\n + \t}\n + \n +-\treturn UNDECIDED;\n +\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n +\n +\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n @@ -221,19 +259,10 @@\n +\t\tgoto done;\n +\t}\n +\n -+\twhile (parent_pathname.len) {\n -+\t\tif (hashmap_contains_path(&pl->recursive_hashmap,\n -+\t\t\t\t\t &parent_pathname)) {\n -+\t\t\tresult = UNDECIDED;\n -+\t\t\tgoto done;\n -+\t\t}\n -+\n -+\t\tslash_pos = strrchr(parent_pathname.buf, '/');\n -+\t\tif (slash_pos == parent_pathname.buf)\n -+\t\t\tbreak;\n -+\n -+\t\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n -+\t}\n ++\tif (hashmap_contains_parent(&pl->recursive_hashmap,\n ++\t\t\t\t pathname,\n ++\t\t\t\t &parent_pathname))\n ++\t\tresult = MATCHED;\n +\n +done:\n +\tstrbuf_release(&parent_pathname);\n @@ -279,6 +308,7 @@\n +\t * excludes array above. If non-zero, that check succeeded.\n +\t */\n +\tunsigned use_cone_patterns;\n ++\tunsigned full_cone;\n +\n +\t/*\n +\t * Stores paths where everything starting with those paths\n @@ -293,6 +323,16 @@\n };\n \n /*\n +@@\n + \t\tstruct index_state *istate,\n + \t\tconst char *name, int *dtype);\n + \n ++int hashmap_contains_parent(struct hashmap *map,\n ++\t\t\t const char *path,\n ++\t\t\t struct strbuf *buffer);\n + struct pattern_list *add_pattern_list(struct dir_struct *dir,\n + \t\t\t\t int group_type, const char *src);\n + int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -322,3 +362,15 @@\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n +\n + diff --git a/unpack-trees.c b/unpack-trees.c\n + --- a/unpack-trees.c\n + +++ b/unpack-trees.c\n +@@\n + \t\to->skip_sparse_checkout = 1;\n + \tif (!o->skip_sparse_checkout) {\n + \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n ++\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n + \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n + \t\t\to->skip_sparse_checkout = 1;\n + \t\telse\n 10: 995c5b8e2b ! 10: 69bd707e96 sparse-checkout: init and set in cone mode\n @@ -32,14 +32,13 @@\n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n @@\n - \treturn result;\n - }\n + enum sparse_checkout_mode {\n + \tMODE_NONE = 0,\n + \tMODE_FULL = 1,\n ++\tMODE_CONE = 2,\n + };\n \n -+#define SPARSE_CHECKOUT_NONE 0\n -+#define SPARSE_CHECKOUT_FULL 1\n -+#define SPARSE_CHECKOUT_CONE 2\n -+\n - static int sc_set_config(int mode)\n + static int sc_set_config(enum sparse_checkout_mode mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n +\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n @@ -53,7 +52,7 @@\n +\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n +\t\t\t \"core.sparseCheckoutCone\", NULL);\n +\n -+\tif (mode == SPARSE_CHECKOUT_CONE)\n ++\tif (mode == MODE_CONE)\n +\t\targv_array_push(&cone_argv, \"true\");\n +\telse\n +\t\targv_array_push(&cone_argv, \"false\");\n @@ -83,19 +82,19 @@\n \tint res;\n \tstruct object_id oid;\n +\tint mode;\n -+\n + \n +-\tif (sc_set_config(MODE_FULL))\n +\tstatic struct option builtin_sparse_checkout_init_options[] = {\n +\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n +\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n +\t\tOPT_END(),\n +\t};\n - \n --\tif (sc_set_config(1))\n ++\n +\targc = parse_options(argc, argv, NULL,\n +\t\t\t builtin_sparse_checkout_init_options,\n +\t\t\t builtin_sparse_checkout_init_usage, 0);\n +\n -+\tmode = init_opts.cone_mode ? SPARSE_CHECKOUT_CONE : SPARSE_CHECKOUT_FULL;\n ++\tmode = init_opts.cone_mode ? MODE_CONE : MODE_FULL;\n +\n +\tif (sc_set_config(mode))\n \t\treturn 1;\n @@ -119,7 +118,7 @@\n +\t\tchar *oldpattern = e->pattern;\n +\t\tsize_t newlen;\n +\n -+\t\tif (!slash)\n ++\t\tif (slash == e->pattern)\n +\t\t\tbreak;\n +\n +\t\tnewlen = slash - e->pattern;\n @@ -153,7 +152,7 @@\n +\t\tchar *pattern = sl.items[i].string;\n +\n +\t\tif (strlen(pattern))\n -+\t\t\tfprintf(fp, \"/%s/\\n!/%s/*/\\n\", pattern, pattern);\n ++\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n +\t}\n +\n +\tstring_list_clear(&sl, 0);\n @@ -167,7 +166,7 @@\n +\n +\tfor (i = 0; i < sl.nr; i++) {\n +\t\tchar *pattern = sl.items[i].string;\n -+\t\tfprintf(fp, \"/%s/\\n\", pattern);\n ++\t\tfprintf(fp, \"%s/\\n\", pattern);\n +\t}\n +}\n +\n @@ -188,7 +187,6 @@\n \tfclose(fp);\n \tfree(sparse_filename);\n \n -@@\n \treturn update_working_directory();\n }\n \n @@ -201,11 +199,8 @@\n +\tif (!line->len)\n +\t\treturn;\n +\n -+\tif (line->buf[0] == '/')\n -+\t\tstrbuf_remove(line, 0, 1);\n -+\n -+\tif (!line->len)\n -+\t\treturn;\n ++\tif (line->buf[0] != '/')\n ++\t\tstrbuf_insert(line, 0, \"/\", 1);\n +\n +\tinsert_recursive_pattern(pl, line);\n +}\n @@ -224,7 +219,7 @@\n -\t\twhile (!strbuf_getline(&line, stdin)) {\n -\t\t\tsize_t len;\n -\t\t\tchar *buf = strbuf_detach(&line, &len);\n --\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n +-\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n +\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n +\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n +\n @@ -240,40 +235,22 @@\n \t\t}\n \t} else {\n -\t\tfor (i = 0; i < argc; i++)\n --\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n +-\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\t\tif (set_opts.use_stdin) {\n +\t\t\tstruct strbuf line = STRBUF_INIT;\n +\n +\t\t\twhile (!strbuf_getline(&line, stdin)) {\n +\t\t\t\tsize_t len;\n +\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n -+\t\t\t\tadd_pattern(buf, buf, len, &pl, 0);\n ++\t\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n +\t\t\t}\n +\t\t} else {\n +\t\t\tfor (i = 0; i < argc; i++)\n -+\t\t\t\tadd_pattern(argv[i], argv[i], strlen(argv[i]), &pl, 0);\n ++\t\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\t\t}\n \t}\n \n - \treturn write_patterns_and_update(&pl);\n -@@\n - \tchar *sparse_filename;\n - \tFILE *fp;\n - \n --\tif (sc_set_config(1))\n -+\tif (sc_set_config(SPARSE_CHECKOUT_FULL))\n - \t\tdie(_(\"failed to change config\"));\n - \n - \tsparse_filename = get_sparse_checkout_filename();\n -@@\n - \tunlink(sparse_filename);\n - \tfree(sparse_filename);\n - \n --\treturn sc_set_config(0);\n -+\treturn sc_set_config(SPARSE_CHECKOUT_NONE);\n - }\n - \n - int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n + \tresult = write_patterns_and_update(&pl);\n \n diff --git a/dir.c b/dir.c\n --- a/dir.c\n @@ -299,10 +276,9 @@\n \n +int pl_hashmap_cmp(const void *unused_cmp_data,\n +\t\t const void *a, const void *b, const void *key);\n -+\n - struct pattern_list *add_pattern_list(struct dir_struct *dir,\n - \t\t\t\t int group_type, const char *src);\n - int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\n + int hashmap_contains_parent(struct hashmap *map,\n + \t\t\t const char *path,\n + \t\t\t struct strbuf *buffer);\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -361,4 +337,3 @@\n +'\n +\n test_done\n - \n 11: 1d4321488e ! 11: e06349fcec unpack-trees: hash less in cone mode\n @@ -19,21 +19,21 @@\n @@\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n - \t\t\t\t\t&parent_pathname)) {\n + \t\t\t\t &parent_pathname)) {\n -\t\tresult = MATCHED;\n +\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n @@\n - \twhile (parent_pathname.len) {\n - \t\tif (hashmap_contains_path(&pl->recursive_hashmap,\n - \t\t\t\t\t &parent_pathname)) {\n --\t\t\tresult = UNDECIDED;\n -+\t\t\tresult = MATCHED_RECURSIVE;\n - \t\t\tgoto done;\n - \t\t}\n + \tif (hashmap_contains_parent(&pl->recursive_hashmap,\n + \t\t\t\t pathname,\n + \t\t\t\t &parent_pathname))\n +-\t\tresult = MATCHED;\n ++\t\tresult = MATCHED_RECURSIVE;\n \n + done:\n + \tstrbuf_release(&parent_pathname);\n \n diff --git a/dir.h b/dir.h\n --- a/dir.h\n -: ---------- > 12: 3ef32084f5 unpack-trees: add progress to clear_ce_flags()\n -: ---------- > 13: 3a677f32b6 read-tree: show progress by default\n -: ---------- > 14: 56444a5498 sparse-checkout: sanitize for nested folders\n -: ---------- > 15: a6f17e9a77 sparse-checkout: update working directory in-process\n -: ---------- > 16: 8927494b8c sparse-checkout: write using lockfile\n -: ---------- > 17: 7f377c1407 sparse-checkout: cone mode should not interact with .gitignore\n\n-- \ngitgitgadget\n"},{"id":"383611","messageId":"22b9bd21f44cc219b7b8bc0665a5e3a17a5542c3.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 06/17] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:32Z","receivedAt":"2019-10-07T20:08:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n 3 files changed, 52 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex f4bd951550..b12bf385ae 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -44,6 +44,10 @@ worktrees.\n \ta list of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ----------------\n \n@@ -61,6 +65,14 @@ directory, it updates the skip-worktree bits in the index based\n ont this file. The files matching the patterns in the file will\n appear in the working directory, and the rest will not.\n \n+To enable the sparse-checkout feature, run `git sparse-checkout init` to\n+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`\n+config setting. Then, run `git sparse-checkout set` to modify the patterns in\n+the sparse-checkout file.\n+\n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -75,21 +87,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-checkout' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 68f3d8433e..ab02acc125 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list|set] \"),\n+\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n \tNULL\n };\n \n@@ -207,6 +207,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \treturn result;\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (sc_set_config(MODE_FULL))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (update_working_directory())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn sc_set_config(MODE_NONE);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -231,6 +253,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"set\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2a0137fde3..52d24c66ba 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -147,4 +147,19 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383612","messageId":"08bb6fb7f31b92395df4db10bf47e9a9c01257f4.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:28Z","receivedAt":"2019-10-07T20:08:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nMake sure to use the `extensions.worktreeConfig` setting and write\nthe sparse checkout config to the worktree-specific config file.\nThis avoids confusing interactions with other worktrees.\n\nThe use of running another process for 'git read-tree' is sub-\noptimal. This will be removed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 12 ++++\n builtin/sparse-checkout.c | 79 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++\n 3 files changed, 131 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 81676b1d33..e095c4a98b 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -26,6 +26,18 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n++\n+The init subcommand also enables the 'extensions.worktreeConfig' setting\n+and sets the `core.sparseCheckout` setting in the worktree-specific config\n+file. This prevents the sparse-checkout feature from interfering with other\n+worktrees.\n \n SPARSE CHECKOUT\n ----------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex eed9625a05..64b2bb2b8c 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [list]\"),\n+\tN_(\"git sparse-checkout [init|list]\"),\n \tNULL\n };\n \n@@ -59,6 +59,81 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int update_working_directory(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+enum sparse_checkout_mode {\n+\tMODE_NONE = 0,\n+\tMODE_FULL = 1,\n+};\n+\n+static int sc_set_config(enum sparse_checkout_mode mode)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n+\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n+\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n+\t\treturn 1;\n+\t}\n+\n+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n+\n+\tif (mode)\n+\t\targv_array_pushl(&argv, \"true\", NULL);\n+\telse\n+\t\targv_array_pushl(&argv, \"false\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (sc_set_config(MODE_FULL))\n+\t\treturn 1;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\tfclose(fp);\n+\n+reset_dir:\n+\treturn update_working_directory();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -79,6 +154,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a9b04b1a88..c70085a759 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -47,4 +47,45 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383613","messageId":"0b1ed06bc8c27650086b10b587df943d3053b773.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:31Z","receivedAt":"2019-10-07T20:08:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nand places them in the sparse-checkout file. Then, it updates the\nworking directory to match those patterns. For a large list of\npatterns, the command-line call can get very cumbersome.\n\nAdd a '--stdin' option to instead read patterns over standard in.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 40 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 27 ++++++++++++++++++++\n 2 files changed, 65 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 52d4f832f3..68f3d8433e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -145,6 +145,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tchar *sparse_filename;\n \tFILE *fp;\n \n+\tif (!core_apply_sparse_checkout) {\n+\t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n+\t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n+\t}\n+\n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n \twrite_patterns_to_file(fp, pl);\n@@ -154,16 +159,47 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static char const * const builtin_sparse_checkout_set_usage[] = {\n+\tN_(\"git sparse-checkout set [--stdin|]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_set_opts {\n+\tint use_stdin;\n+} set_opts;\n+\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tstatic const char *empty_base = \"\";\n \tint i;\n \tstruct pattern_list pl;\n \tint result;\n+\n+\tstatic struct option builtin_sparse_checkout_set_options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n+\t\t\t N_(\"read patterns from standard in\")),\n+\t\tOPT_END(),\n+\t};\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n-\tfor (i = 1; i < argc; i++)\n-\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_set_options,\n+\t\t\t builtin_sparse_checkout_set_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (set_opts.use_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tsize_t len;\n+\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t}\n \n \tresult = write_patterns_and_update(&pl);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 19e8673c6b..2a0137fde3 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -101,6 +101,13 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'warn if core.sparseCheckout is disabled' '\n+\ttest_when_finished git -C repo config --worktree core.sparseCheckout true &&\n+\tgit -C repo config --worktree core.sparseCheckout false &&\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\ttest_i18ngrep \"core.sparseCheckout is disabled\" err\n+'\n+\n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n \tcat >expect <<-EOF &&\n@@ -120,4 +127,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using --stdin' '\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/folder1/\n+\t\t/folder2/\n+\tEOF\n+\tgit -C repo sparse-checkout set --stdin actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383614","messageId":"431933bec6f05660ad90100eb946a1946c03a09a.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 09/17] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:36Z","receivedAt":"2019-10-07T20:08:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/\n\t!/$folder/*/\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n\tcore.sparseCheckout=false: 0.21 s (0.00 s)\n\t core.sparseCheckout=true: 3.75 s (3.50 s)\n\t core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 202 +++++++++++++++++++++++++++--\n dir.h | 31 +++++\n t/t1091-sparse-checkout-builtin.sh | 11 +-\n unpack-trees.c | 1 +\n 4 files changed, 236 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 34972abdaf..7ba4bc044e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -599,6 +599,145 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const void *a, const void *b, const void *key)\n+{\n+\tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n+\tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\n+\n+\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n+\t\t\t ? ee1->patternlen\n+\t\t\t : ee2->patternlen;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n+}\n+\n+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n+{\n+\tstruct pattern_entry *translated;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!pl->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE &&\n+\t given->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 0;\n+\t\treturn;\n+\t}\n+\n+\tif (!given->flags && !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 1;\n+\t\treturn;\n+\t}\n+\n+\tif (given->patternlen > 2 &&\n+\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n+\t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tpl->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(given->pattern);\n+\t\ttruncated[given->patternlen - 2] = 0;\n+\n+\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\t\ttranslated->pattern = truncated;\n+\t\ttranslated->patternlen = given->patternlen - 2;\n+\t\thashmap_entry_init(translated,\n+\t\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\t\tif (!hashmap_get(&pl->recursive_hashmap, translated, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\t\tgiven->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tfree(translated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&pl->parent_hashmap, translated);\n+\t\thashmap_remove(&pl->recursive_hashmap, translated, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE) {\n+\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\tgiven->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n+\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\n+\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->patternlen = given->patternlen;\n+\thashmap_entry_init(translated,\n+\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, translated);\n+\n+\tif (hashmap_get(&pl->parent_hashmap, translated, NULL)) {\n+\t\t/* we already included this at the parent level */\n+\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\tgiven->pattern);\n+\t\thashmap_remove(&pl->parent_hashmap, translated, &data);\n+\t\tfree(data);\n+\t\tfree(translated);\n+\t}\n+\n+\treturn;\n+\n+clear_hashmaps:\n+\twarning(_(\"disabling cone pattern matching\"));\n+\thashmap_free(&pl->parent_hashmap, 1);\n+\thashmap_free(&pl->recursive_hashmap, 1);\n+\tpl->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct pattern_entry p;\n+\n+\t/* Check straight mapping */\n+\tp.pattern = pattern->buf;\n+\tp.patternlen = pattern->len;\n+\thashmap_entry_init(&p, memhash(p.pattern, p.patternlen));\n+\treturn !!hashmap_get(map, &p, NULL);\n+}\n+\n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer)\n+{\n+\tchar *slash_pos;\n+\n+\tstrbuf_setlen(buffer, 0);\n+\n+\tif (path[0] != '/')\n+\t\tstrbuf_addch(buffer, '/');\n+\n+\tstrbuf_addstr(buffer, path);\n+\n+\tslash_pos = strrchr(buffer->buf, '/');\n+\n+\twhile (slash_pos > buffer->buf) {\n+\t\tstrbuf_setlen(buffer, slash_pos - buffer->buf);\n+\n+\t\tif (hashmap_contains_path(map, buffer))\n+\t\t\treturn 1;\n+\n+\t\tslash_pos = strrchr(buffer->buf, '/');\n+\t}\n+\n+\treturn 0;\n+}\n+\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n {\n@@ -623,6 +762,8 @@ void add_pattern(const char *string, const char *base,\n \tALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);\n \tpl->patterns[pl->nr++] = pattern;\n \tpattern->pl = pl;\n+\n+\tadd_pattern_to_hashsets(pl, pattern);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -848,6 +989,9 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n \tpl->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1084,16 +1228,58 @@ enum pattern_match_result path_matches_pattern_list(\n \t\t\t\tstruct index_state *istate)\n {\n \tstruct path_pattern *pattern;\n-\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, pl, istate);\n-\tif (pattern) {\n-\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n-\t\t\treturn NOT_MATCHED;\n-\t\telse\n-\t\t\treturn MATCHED;\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = NOT_MATCHED;\n+\tconst char *slash_pos;\n+\n+\tif (!pl->use_cone_patterns) {\n+\t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\tdtype, pl, istate);\n+\t\tif (pattern) {\n+\t\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\t\treturn NOT_MATCHED;\n+\t\t\telse\n+\t\t\t\treturn MATCHED;\n+\t\t}\n+\n+\t\treturn UNDECIDED;\n+\t}\n+\n+\tif (pl->full_cone)\n+\t\treturn MATCHED;\n+\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\n+\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n \t}\n \n-\treturn UNDECIDED;\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tif (hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t pathname,\n+\t\t\t\t &parent_pathname))\n+\t\tresult = MATCHED;\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct path_pattern *last_matching_pattern_from_lists(\ndiff --git a/dir.h b/dir.h\nindex 608696c958..c6c188669a 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct path_pattern {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct pattern_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,26 @@ struct pattern_list {\n \tconst char *src;\n \n \tstruct path_pattern **patterns;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\tunsigned full_cone;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\n@@ -269,6 +297,9 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer);\n struct pattern_list *add_pattern_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src);\n int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 36fda5907b..b0d5aeb33a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -150,7 +150,8 @@ test_expect_success 'set sparse-checkout using --stdin' '\n test_expect_success 'cone mode: match patterns' '\n \tgit -C repo config --worktree core.sparseCheckoutCone true &&\n \trm -rf repo/a repo/folder1 repo/folder2 &&\n-\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n \tls repo >dir &&\n \tcat >expect <<-EOF &&\n@@ -161,6 +162,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info/ &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 26be8f3569..566df11309 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1479,6 +1479,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n \t\t\to->skip_sparse_checkout = 1;\n \t\telse\n-- \ngitgitgadget\n\n"},{"id":"383615","messageId":"69bd707e96bd5381b64c7f43e3e5672ac8655471.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 10/17] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:37Z","receivedAt":"2019-10-07T20:08:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout [init|set]'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckoutCone=true'.\n\nWhen running 'git sparse-checkout set' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nWhen testing 'git sparse-checkout set' in cone mode, check the\nerror stream to ensure we do not see any errors. Specifically,\nwe want to avoid the warning that the patterns do not match\nthe cone-mode patterns.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 161 +++++++++++++++++++++++++++--\n dir.c | 4 +-\n dir.h | 2 +\n t/t1091-sparse-checkout-builtin.sh | 49 +++++++++\n 4 files changed, 204 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex ab02acc125..b220f15741 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,6 +6,7 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n@@ -77,11 +78,13 @@ static int update_working_directory(void)\n enum sparse_checkout_mode {\n \tMODE_NONE = 0,\n \tMODE_FULL = 1,\n+\tMODE_CONE = 2,\n };\n \n static int sc_set_config(enum sparse_checkout_mode mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n \n \tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n \t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n@@ -100,9 +103,31 @@ static int sc_set_config(enum sparse_checkout_mode mode)\n \t\treturn 1;\n \t}\n \n+\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n+\t\t\t \"core.sparseCheckoutCone\", NULL);\n+\n+\tif (mode == MODE_CONE)\n+\t\targv_array_push(&cone_argv, \"true\");\n+\telse\n+\t\targv_array_push(&cone_argv, \"false\");\n+\n+\tif (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckoutCone\"));\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\n \n+static char const * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_init_opts {\n+\tint cone_mode;\n+} init_opts;\n+\n static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -110,8 +135,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n+\tint mode;\n \n-\tif (sc_set_config(MODE_FULL))\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = init_opts.cone_mode ? MODE_CONE : MODE_FULL;\n+\n+\tif (sc_set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -140,6 +178,72 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n+{\n+\tstruct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, e);\n+\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (slash == e->pattern)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct pattern_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get(&pl->parent_hashmap, e, NULL))\n+\t\t\thashmap_add(&pl->parent_hashmap, e);\n+\t}\n+}\n+\n+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\tstruct pattern_entry *entry;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_iter_init(&pl->parent_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter)))\n+\t\tstring_list_insert(&sl, entry->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (strlen(pattern))\n+\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_iter_init(&pl->recursive_hashmap, &iter);\n+\twhile ((entry = hashmap_iter_next(&iter)))\n+\t\tstring_list_insert(&sl, entry->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t}\n+}\n+\n static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n@@ -152,13 +256,33 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n-\twrite_patterns_to_file(fp, pl);\n+\n+\tif (core_sparse_checkout_cone)\n+\t\twrite_cone_to_file(fp, pl);\n+\telse\n+\t\twrite_patterns_to_file(fp, pl);\n+\n \tfclose(fp);\n \tfree(sparse_filename);\n \n \treturn update_working_directory();\n }\n \n+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n+{\n+\tstrbuf_trim(line);\n+\n+\tstrbuf_trim_trailing_dir_sep(line);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tif (line->buf[0] != '/')\n+\t\tstrbuf_insert(line, 0, \"/\", 1);\n+\n+\tinsert_recursive_pattern(pl, line);\n+}\n+\n static char const * const builtin_sparse_checkout_set_usage[] = {\n \tN_(\"git sparse-checkout set [--stdin|]\"),\n \tNULL\n@@ -188,17 +312,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n-\tif (set_opts.use_stdin) {\n+\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (!strbuf_getline(&line, stdin)) {\n-\t\t\tsize_t len;\n-\t\t\tchar *buf = strbuf_detach(&line, &len);\n-\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n+\t\tif (set_opts.use_stdin) {\n+\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++) {\n+\t\t\t\tstrbuf_setlen(&line, 0);\n+\t\t\t\tstrbuf_addstr(&line, argv[i]);\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n \t\t}\n \t} else {\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\tif (set_opts.use_stdin) {\n+\t\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\t}\n \t}\n \n \tresult = write_patterns_and_update(&pl);\ndiff --git a/dir.c b/dir.c\nindex 7ba4bc044e..2b240111aa 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -599,8 +599,8 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n-static int pl_hashmap_cmp(const void *unused_cmp_data,\n-\t\t\t const void *a, const void *b, const void *key)\n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const void *a, const void *b, const void *key)\n {\n \tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n \tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\ndiff --git a/dir.h b/dir.h\nindex c6c188669a..f7a2f000c3 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -297,6 +297,8 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const void *a, const void *b, const void *key);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b0d5aeb33a..db6371b079 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -185,4 +185,53 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and set' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n+\ttest_line_count = 0 err &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\t\t!/deep/*/\n+\t\t/deep/deeper1/\n+\t\t!/deep/deeper1/*/\n+\t\t/deep/deeper1/deepest/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\tls repo >dir &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383616","messageId":"2c53ea13d0ebfd64956064804d28e6e1646c6aed.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 07/17] trace2: add region in clear_ce_flags","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:34Z","receivedAt":"2019-10-07T20:08:44Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nWhen Git updates the working directory with the sparse-checkout\nfeature enabled, the unpack_trees() method calls clear_ce_flags()\nto update the skip-wortree bits on the cache entries. This\ncheck can be expensive, depending on the patterns used.\n\nAdd trace2 regions around the method, including some flag\ninformation, so we can get granular performance data during\nexperiments. This data will be used to measure improvements\nto the pattern-matching algorithms for sparse-checkout.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex cd548f4fa2..26be8f3569 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1404,15 +1404,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tpl, 0);\n+\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"383617","messageId":"3a677f32b6b914fcc800b619fd082da25c275422.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 13/17] read-tree: show progress by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:39Z","receivedAt":"2019-10-07T20:08:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe read-tree builtin has a --verbose option that signals to show\nprogress and other data while updating the index. Update this to\nbe on by default when stderr is a terminal window.\n\nThis will help tools like 'git sparse-checkout' to automatically\nbenefit from progress indicators when a user runs these commands.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex ca5e655d2f..69963d83dc 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \topts.head_idx = -1;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\topts.verbose_update = isatty(2);\n \n \tgit_config(git_read_tree_config, NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"383618","messageId":"e06349fcec6612c74bed35cc7ca3a039017c2845.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 11/17] unpack-trees: hash less in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:38Z","receivedAt":"2019-10-07T20:08:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature in \"cone mode\" can use the fact that\nthe recursive patterns are \"connected\" to the root via parent\npatterns to decide if a directory is entirely contained in the\nsparse-checkout or entirely removed.\n\nIn these cases, we can skip hashing the paths within those\ndirectories and simply set the skipworktree bit to the correct\nvalue.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 4 ++--\n dir.h | 1 +\n unpack-trees.c | 38 +++++++++++++++++++++++---------------\n 3 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 2b240111aa..eb6b2913ef 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1253,7 +1253,7 @@ enum pattern_match_result path_matches_pattern_list(\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n@@ -1275,7 +1275,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tif (hashmap_contains_parent(&pl->recursive_hashmap,\n \t\t\t\t pathname,\n \t\t\t\t &parent_pathname))\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \n done:\n \tstrbuf_release(&parent_pathname);\ndiff --git a/dir.h b/dir.h\nindex f7a2f000c3..c868129a54 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -262,6 +262,7 @@ enum pattern_match_result {\n \tUNDECIDED = -1,\n \tNOT_MATCHED = 0,\n \tMATCHED = 1,\n+\tMATCHED_RECURSIVE = 2,\n };\n \n /*\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 566df11309..b5cf591c38 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1280,15 +1280,17 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n \tint rc;\n-\tenum pattern_match_result ret;\n-\tret = path_matches_pattern_list(prefix->buf, prefix->len,\n-\t\t\t\t\tbasename, &dtype, pl, istate);\n+\tenum pattern_match_result ret, orig_ret;\n+\torig_ret = path_matches_pattern_list(prefix->buf, prefix->len,\n+\t\t\t\t\t basename, &dtype, pl, istate);\n \n \tstrbuf_addch(prefix, '/');\n \n \t/* If undecided, use matching result of parent dir in defval */\n-\tif (ret == UNDECIDED)\n+\tif (orig_ret == UNDECIDED)\n \t\tret = default_match;\n+\telse\n+\t\tret = orig_ret;\n \n \tfor (cache_end = cache; cache_end != cache + nr; cache_end++) {\n \t\tstruct cache_entry *ce = *cache_end;\n@@ -1296,17 +1298,23 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\tbreak;\n \t}\n \n-\t/*\n-\t * TODO: check pl, if there are no patterns that may conflict\n-\t * with ret (iow, we know in advance the incl/excl\n-\t * decision for the entire directory), clear flag here without\n-\t * calling clear_ce_flags_1(). That function will call\n-\t * the expensive path_matches_pattern_list() on every entry.\n-\t */\n-\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t prefix,\n-\t\t\t select_mask, clear_mask,\n-\t\t\t pl, ret);\n+\tif (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {\n+\t\tstruct cache_entry **ce = cache;\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\n+\t\twhile (ce < cache_end) {\n+\t\t\t(*ce)->ce_flags &= ~clear_mask;\n+\t\t\tce++;\n+\t\t}\n+\t} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\t} else {\n+\t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t prefix,\n+\t\t\t\t select_mask, clear_mask,\n+\t\t\t\t pl, ret);\n+\t}\n+\n \tstrbuf_setlen(prefix, prefix->len - 1);\n \treturn rc;\n }\n-- \ngitgitgadget\n\n"},{"id":"383619","messageId":"a66ec1affc22829d693090af2287692e032263d3.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 08/17] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:35Z","receivedAt":"2019-10-07T20:08:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new Boolean config option, core.sparseCheckoutCone, to\nindicate that we expect the sparse-checkout file to contain a\nmore limited set of patterns. This is a separate config setting\nfrom core.sparseCheckout to avoid breaking older clients by\nintroducing a tri-state option.\n\nThe config option does nothing right now, but will be expanded\nupon in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 10 ++++--\n Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n cache.h | 4 ++-\n config.c | 5 +++\n environment.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n 6 files changed, 81 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 75538d27e7..e2d343f738 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -591,8 +591,14 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\n+\tfor more information.\n+\n+core.sparseCheckoutCone::\n+\tEnables the \"cone mode\" of the sparse checkout feature. When the\n+\tsparse-checkout file contains a limited set of patterns, then this\n+\tmode provides significant performance advantages. See\n+\tlinkgit:git-sparse-checkout[1] for more information.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b12bf385ae..e2eb82ec45 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -88,6 +88,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+## CONE PATTERN SET\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+```\n+/*\n+!/*/\n+```\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+```\n+/*\n+!/*/\n+/A/\n+!/A/*/\n+/A/B/\n+!/A/B/*/\n+/A/B/C/\n+```\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/cache.h b/cache.h\nindex cf5d70c196..8e8ea67efa 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -911,12 +911,14 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex 296a6d9cc4..f65c74f5b7 100644\n--- a/config.c\n+++ b/config.c\n@@ -1329,6 +1329,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n+\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 89af47cb85..670d92bcc0 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -69,6 +69,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 52d24c66ba..36fda5907b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -147,6 +147,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"383620","messageId":"3ef32084f5ff55ddf56d241b26883e04b98296c9.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 12/17] unpack-trees: add progress to clear_ce_flags()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:39Z","receivedAt":"2019-10-07T20:08:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a large repository has many sparse-checkout patterns, the\nprocess for updating the skip-worktree bits can take long enough\nthat a user gets confused why nothing is happening. Update the\nclear_ce_flags() method to write progress.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 2 ++\n unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++--------------\n 2 files changed, 43 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8e8ea67efa..9d9d88a0f0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -328,6 +328,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n struct split_index;\n struct untracked_cache;\n+struct progress;\n \n struct index_state {\n \tstruct cache_entry **cache;\n@@ -350,6 +351,7 @@ struct index_state {\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n \tstruct mem_pool *ce_mem_pool;\n+\tstruct progress *progress;\n };\n \n /* Name hashing */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex b5cf591c38..edf0fb4673 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1266,7 +1266,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match);\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr);\n \n /* Whole directory matching */\n static int clear_ce_flags_dir(struct index_state *istate,\n@@ -1275,7 +1276,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\t char *basename,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n@@ -1312,7 +1314,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n \t\t\t\t prefix,\n \t\t\t\t select_mask, clear_mask,\n-\t\t\t\t pl, ret);\n+\t\t\t\t pl, ret,\n+\t\t\t\t progress_nr);\n \t}\n \n \tstrbuf_setlen(prefix, prefix->len - 1);\n@@ -1339,7 +1342,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end = cache + nr;\n \n@@ -1353,8 +1357,11 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tint len, dtype;\n \t\tenum pattern_match_result ret;\n \n+\t\tdisplay_progress(istate->progress, progress_nr);\n+\n \t\tif (select_mask && !(ce->ce_flags & select_mask)) {\n \t\t\tcache++;\n+\t\t\tprogress_nr++;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1375,20 +1382,26 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t\t\t\t prefix,\n \t\t\t\t\t\t prefix->buf + prefix->len - len,\n \t\t\t\t\t\t select_mask, clear_mask,\n-\t\t\t\t\t\t pl, default_match);\n+\t\t\t\t\t\t pl, default_match,\n+\t\t\t\t\t\t progress_nr);\n \n \t\t\t/* clear_c_f_dir eats a whole dir already? */\n \t\t\tif (processed) {\n \t\t\t\tcache += processed;\n+\t\t\t\tprogress_nr += processed;\n \t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n \t\t\t\tcontinue;\n \t\t\t}\n \n \t\t\tstrbuf_addch(prefix, '/');\n-\t\t\tcache += clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t\t\t\t prefix,\n-\t\t\t\t\t\t select_mask, clear_mask, pl,\n-\t\t\t\t\t\t default_match);\n+\t\t\tprocessed = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t\t\t prefix,\n+\t\t\t\t\t\t select_mask, clear_mask, pl,\n+\t\t\t\t\t\t default_match, progress_nr);\n+\n+\t\t\tcache += processed;\n+\t\t\tprogress_nr += processed;\n+\n \t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n \t\t\tcontinue;\n \t\t}\n@@ -1403,19 +1416,27 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tif (ret == MATCHED)\n \t\t\tce->ce_flags &= ~clear_mask;\n \t\tcache++;\n+\t\tprogress_nr++;\n \t}\n+\n+\tdisplay_progress(istate->progress, progress_nr);\n \treturn nr - (cache_end - cache);\n }\n \n static int clear_ce_flags(struct index_state *istate,\n \t\t\t int select_mask, int clear_mask,\n-\t\t\t struct pattern_list *pl)\n+\t\t\t struct pattern_list *pl,\n+\t\t\t int show_progress)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n \tchar label[100];\n \tint rval;\n \n \tstrbuf_reset(&prefix);\n+\tif (show_progress)\n+\t\tistate->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Updating index flags\"),\n+\t\t\t\t\tistate->cache_nr);\n \n \txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n \t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n@@ -1425,9 +1446,10 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n-\t\t\t\tpl, 0);\n+\t\t\t\tpl, 0, 0);\n \ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n \n+\tstop_progress(&istate->progress);\n \treturn rval;\n }\n \n@@ -1436,7 +1458,8 @@ static int clear_ce_flags(struct index_state *istate,\n */\n static void mark_new_skip_worktree(struct pattern_list *pl,\n \t\t\t\t struct index_state *istate,\n-\t\t\t\t int select_flag, int skip_wt_flag)\n+\t\t\t\t int select_flag, int skip_wt_flag,\n+\t\t\t\t int show_progress)\n {\n \tint i;\n \n@@ -1460,7 +1483,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \t * 2. Widen worktree according to sparse-checkout file.\n \t * Matched entries will have skip_wt_flag cleared (i.e. \"in\")\n \t */\n-\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl);\n+\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n static int verify_absent(const struct cache_entry *,\n@@ -1522,7 +1545,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n \t */\n \tif (!o->skip_sparse_checkout)\n-\t\tmark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n \n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n@@ -1587,7 +1611,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n-\t\tmark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, &o->result,\n+\t\t\t\t CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE,\n+\t\t\t\t o->verbose_update);\n \n \t\tret = 0;\n \t\tfor (i = 0; i < o->result.cache_nr; i++) {\n-- \ngitgitgadget\n\n"},{"id":"383621","messageId":"a6f17e9a77d86f8ec856ea08617d1c1af2853d54.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 15/17] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:41Z","receivedAt":"2019-10-07T20:08:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\nskip-worktree bits in the index and to update the working directory.\nThis extra process is overly complex, and prone to failure. It also\nrequires that we write our changes to the sparse-checkout file before\ntrying to update the index.\n\nRemove this extra process call by creating a direct call to\nunpack_trees() in the same way 'git read-tree -mu HEAD' does. In\nadition, provide an in-memory list of patterns so we can avoid\nreading from the sparse-checkout file. This allows us to test a\nproposed change to the file before writing to it.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 2 +-\n builtin/sparse-checkout.c | 85 +++++++++++++++++++++++++-----\n t/t1091-sparse-checkout-builtin.sh | 17 ++++++\n unpack-trees.c | 5 +-\n unpack-trees.h | 3 +-\n 5 files changed, 95 insertions(+), 17 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 69963d83dc..d7eeaa26ec 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \n \tif (opts.reset || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n-\t\t\tdie(\"You need to resolve your current index first\");\n+\t\t\tdie(_(\"You need to resolve your current index first\"));\n \t\tstage = opts.merge = 1;\n \t}\n \tresolve_undo_clear();\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 25786f8bb0..542d57fac6 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -7,6 +7,11 @@\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"cache.h\"\n+#include \"cache-tree.h\"\n+#include \"lockfile.h\"\n+#include \"resolve-undo.h\"\n+#include \"unpack-trees.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n@@ -60,18 +65,53 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n-static int update_working_directory(void)\n+static int update_working_directory(struct pattern_list *pl)\n {\n-\tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n-\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\tstruct unpack_trees_options o;\n+\tstruct lock_file lock_file = LOCK_INIT;\n+\tstruct object_id oid;\n+\tstruct tree *tree;\n+\tstruct tree_desc t;\n+\tstruct repository *r = the_repository;\n \n-\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n-\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n-\t\tresult = 1;\n+\tif (repo_read_index_unmerged(r))\n+\t\tdie(_(\"You need to resolve your current index first\"));\n+\n+\tif (get_oid(\"HEAD\", &oid))\n+\t\treturn 0;\n+\n+\ttree = parse_tree_indirect(&oid);\n+\tparse_tree(tree);\n+\tinit_tree_desc(&t, tree->buffer, tree->size);\n+\n+\tmemset(&o, 0, sizeof(o));\n+\to.verbose_update = isatty(2);\n+\to.merge = 1;\n+\to.update = 1;\n+\to.fn = oneway_merge;\n+\to.head_idx = -1;\n+\to.src_index = r->index;\n+\to.dst_index = r->index;\n+\to.skip_sparse_checkout = 0;\n+\to.pl = pl;\n+\to.keep_pattern_list = !!pl;\n+\n+\tresolve_undo_clear_index(r->index);\n+\tsetup_work_tree();\n+\n+\tcache_tree_free(&r->index->cache_tree);\n+\n+\trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n+\n+\tcore_apply_sparse_checkout = 1;\n+\tresult = unpack_trees(1, &t, &o);\n+\n+\tif (!result) {\n+\t\tprime_cache_tree(r, r->index, tree);\n+\t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n \t}\n \n-\targv_array_clear(&argv);\n \treturn result;\n }\n \n@@ -147,7 +187,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n-\tmode = init_opts.cone_mode ? MODE_CONE : MODE_FULL;\n+\tif (init_opts.cone_mode) {\n+\t\tmode = MODE_CONE;\n+\t\tcore_sparse_checkout_cone = 1;\n+\t} else\n+\t\tmode = MODE_FULL;\n \n \tif (sc_set_config(mode))\n \t\treturn 1;\n@@ -175,12 +219,14 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t}\n \n reset_dir:\n-\treturn update_working_directory();\n+\tcore_apply_sparse_checkout = 1;\n+\treturn update_working_directory(NULL);\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n {\n-\tstruct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n+\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n+\n \te->patternlen = path->len;\n \te->pattern = strbuf_detach(path, NULL);\n \thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n@@ -262,12 +308,21 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n-\n+\tint result;\n+\t\n \tif (!core_apply_sparse_checkout) {\n \t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n \t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n \t}\n \n+\tresult = update_working_directory(pl);\n+\n+\tif (result) {\n+\t\tclear_pattern_list(pl);\n+\t\tupdate_working_directory(NULL);\n+\t\treturn result;\n+\t}\n+\n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n \n@@ -277,9 +332,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \t\twrite_patterns_to_file(fp, pl);\n \n \tfclose(fp);\n+\n \tfree(sparse_filename);\n+\tclear_pattern_list(pl);\n \n-\treturn update_working_directory();\n+\treturn 0;\n }\n \n static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n@@ -330,6 +387,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tstruct strbuf line = STRBUF_INIT;\n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n \t\t\twhile (!strbuf_getline(&line, stdin))\n@@ -375,7 +433,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n\");\n \tfclose(fp);\n \n-\tif (update_working_directory())\n+\tcore_apply_sparse_checkout = 1;\n+\tif (update_working_directory(NULL))\n \t\tdie(_(\"error while refreshing working directory\"));\n \n \tunlink(sparse_filename);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ee4d361787..82eb5fb2f8 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -199,11 +199,13 @@ test_expect_success 'cone mode: init and set' '\n \t\ta\n \t\tdeep\n \tEOF\n+\ttest_cmp dir expect &&\n \tls repo/deep >dir &&\n \tcat >expect <<-EOF &&\n \t\ta\n \t\tdeeper1\n \tEOF\n+\ttest_cmp dir expect &&\n \tls repo/deep/deeper1 >dir &&\n \tcat >expect <<-EOF &&\n \t\ta\n@@ -245,4 +247,19 @@ test_expect_success 'cone mode: set with nested folders' '\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n \n+test_expect_success 'revert to old sparse-checkout on bad update' '\n+\techo update >repo/deep/deeper2/a &&\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp repo/.git/info/sparse-checkout expect &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\t\tdeeper2\n+\tEOF\n+\ttest_cmp dir expect\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex edf0fb4673..f0fee5adf2 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1508,7 +1508,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n-\tif (!o->skip_sparse_checkout) {\n+\tif (!o->skip_sparse_checkout && !o->pl) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n \t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n@@ -1681,7 +1681,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n done:\n \ttrace_performance_leave(\"unpack_trees\");\n-\tclear_pattern_list(&pl);\n+\tif (!o->keep_pattern_list)\n+\t\tclear_pattern_list(&pl);\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex f2eee0c7c5..ca94a421a5 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -59,7 +59,8 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run;\n+\t\t dry_run,\n+\t\t keep_pattern_list;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"383623","messageId":"56444a5498c8194186d294b3128d0ffdb3678296.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 14/17] sparse-checkout: sanitize for nested folders","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:40Z","receivedAt":"2019-10-07T20:08:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user provides folders A/ and A/B/ for inclusion in a cone-mode\nsparse-checkout file, the parsing logic will notice that A/ appears\nboth as a \"parent\" type pattern and as a \"recursive\" type pattern.\nThis is unexpected and hence will complain via a warning and revert\nto the old logic for checking sparse-checkout patterns.\n\nPrevent this from happening accidentally by sanitizing the folders\nfor this type of inclusion in the 'git sparse-checkout' builtin.\nThis happens in two ways:\n\n1. Do not include any parent patterns that also appear as recursive\n patterns.\n\n2. Do not include any recursive patterns deeper than other recursive\n patterns.\n\nIn order to minimize duplicate code for scanning parents, create\nhashmap_contains_parent() method. It takes a strbuf buffer to\navoid reallocating a buffer when calling in a tight loop.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 22 ++++++++++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b220f15741..25786f8bb0 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -212,10 +212,18 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstruct pattern_entry *entry;\n \tstruct hashmap_iter iter;\n \tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\tstruct strbuf parent_pattern = STRBUF_INIT;\n \n \thashmap_iter_init(&pl->parent_hashmap, &iter);\n-\twhile ((entry = hashmap_iter_next(&iter)))\n-\t\tstring_list_insert(&sl, entry->pattern);\n+\twhile ((entry = hashmap_iter_next(&iter))) {\n+\t\tif (hashmap_get(&pl->recursive_hashmap, entry, NULL))\n+\t\t\tcontinue;\n+\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t entry->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, entry->pattern);\n+\t}\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\n@@ -232,8 +240,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstring_list_clear(&sl, 0);\n \n \thashmap_iter_init(&pl->recursive_hashmap, &iter);\n-\twhile ((entry = hashmap_iter_next(&iter)))\n-\t\tstring_list_insert(&sl, entry->pattern);\n+\twhile ((entry = hashmap_iter_next(&iter))) {\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t entry->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, entry->pattern);\n+\t}\n+\n+\tstrbuf_release(&parent_pattern);\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex db6371b079..ee4d361787 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -234,4 +234,15 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: set with nested folders' '\n+\tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\tEOF\n+\ttest_cmp repo/.git/info/sparse-checkout expect\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383622","messageId":"8927494b8c418e43f5bbd6e1eb108be5a0a263fc.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 16/17] sparse-checkout: write using lockfile","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:42Z","receivedAt":"2019-10-07T20:08:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf two 'git sparse-checkout set' subcommands are launched at the\nsame time, the behavior can be unexpected as they compete to write\nthe sparse-checkout file and update the working directory.\n\nTake a lockfile around the writes to the sparse-checkout file. In\naddition, acquire this lock around the working directory update\nto avoid two commands updating the working directory in different\nways.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 15 ++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 2 files changed, 19 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 542d57fac6..9b313093cd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -308,6 +308,8 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \t\n \tif (!core_apply_sparse_checkout) {\n@@ -317,21 +319,28 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tresult = update_working_directory(pl);\n \n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfd = hold_lock_file_for_update(&lk, sparse_filename,\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\n+\tresult = update_working_directory(pl);\n \tif (result) {\n+\t\trollback_lock_file(&lk);\n+\t\tfree(sparse_filename);\n \t\tclear_pattern_list(pl);\n \t\tupdate_working_directory(NULL);\n \t\treturn result;\n \t}\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = fopen(sparse_filename, \"w\");\n+\tfp = fdopen(fd, \"w\");\n \n \tif (core_sparse_checkout_cone)\n \t\twrite_cone_to_file(fp, pl);\n \telse\n \t\twrite_patterns_to_file(fp, pl);\n \n-\tfclose(fp);\n+\tfflush(fp);\n+\tcommit_lock_file(&lk);\n \n \tfree(sparse_filename);\n \tclear_pattern_list(pl);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 82eb5fb2f8..f22a4afbea 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -262,4 +262,11 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_cmp dir expect\n '\n \n+test_expect_success 'fail when lock is taken' '\n+\ttest_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&\n+\ttouch repo/.git/info/sparse-checkout.lock &&\n+\ttest_must_fail git -C repo sparse-checkout set deep 2>err &&\n+\ttest_i18ngrep \"File exists\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"383624","messageId":"7f377c1407274aec87c6261f05bb9a944b659ba0.1570478905.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 17/17] sparse-checkout: cone mode should not interact with .gitignore","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-07T20:08:43Z","receivedAt":"2019-10-07T20:08:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the development of the sparse-checkout \"cone mode\" feature,\nan incorrect placement of the initializer for \"use_cone_patterns = 1\"\ncaused warnings to show up when a .gitignore file was present with\nnon-cone-mode patterns. This was fixed in the original commit\nintroducing the cone mode, but now we should add a test to avoid\nhitting this problem again in the future.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex f22a4afbea..ed9355384a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -269,4 +269,11 @@ test_expect_success 'fail when lock is taken' '\n \ttest_i18ngrep \"File exists\" err\n '\n \n+test_expect_success '.gitignore should not warn about cone mode' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\techo \"**/bin/*\" >repo/.gitignore &&\n+\tgit -C repo reset --hard 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"383937","messageId":"CABPp-BEc3s6UhMN2d+K7xbMCGG=QSP3cQSCob47OWcuFzzw6Zw@mail.gmail.com","threadId":"51691","inReplyTo":"30a0db68cd141bab5a9fb63e26e767579a42249c.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:01:07Z","receivedAt":"2019-10-11T22:01:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n> +SPARSE CHECKOUT\n> +----------------\n> +\n> +\"Sparse checkout\" allows populating the working directory sparsely.\n> +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n> +Git whether a file in the working directory is worth looking at. If\n> +the skip-worktree bit is set, then the file is ignored in the working\n> +directory. Git will not populate the contents of those files, which\n> +makes a sparse checkout helpful when working in a repository with many\n> +files, but only a few are important to the current user.\n> +\n> +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n> +skip-worktree reference bitmap. When Git updates the working\n> +directory, it updates the skip-worktree bits in the index based\n> +ont this file. The files matching the patterns in the file will\n\ns/ont/on/\n\n> +appear in the working directory, and the rest will not.\n> +\n> +## FULL PATTERN SET\n> +\n> +By default, the sparse-checkout file uses the same syntax as `.gitignore`\n> +files.\n> +\n> +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n> +files are included, you can also specify what files are _not_ included,\n> +using negative patterns. For example, to remove the file `unwanted`:\n> +\n> +----------------\n> +/*\n> +!unwanted\n> +----------------\n> +\n> +Another tricky thing is fully repopulating the working directory when you\n> +no longer want sparse checkout. You cannot just disable \"sparse\n> +checkout\" because skip-worktree bits are still in the index and your working\n> +directory is still sparsely populated. You should re-populate the working\n> +directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> +follows:\n> +\n> +----------------\n> +/*\n> +----------------\n> +\n> +Then you can disable sparse checkout. Sparse checkout support in 'git\n> +checkout' and similar commands is disabled by default. You need to\n> +set `core.sparseCheckout` to `true` in order to have sparse checkout\n> +support.\n\nLooks like these disappear by the end of the series, so no need to\ncomment on them. Thanks for all the fixes, other than the trivial\ntypo above, this patch looks good.\n"},{"id":"383940","messageId":"CABPp-BHd9sJ=HcM-VQaGpd5TV-A0JL5NWG6zdDReUJrMnE+FwA@mail.gmail.com","threadId":"51691","inReplyTo":"08bb6fb7f31b92395df4db10bf47e9a9c01257f4.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 02/17] sparse-checkout: create 'init' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:14:39Z","receivedAt":"2019-10-11T22:14:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n> ++\n> +The init subcommand also enables the 'extensions.worktreeConfig' setting\n> +and sets the `core.sparseCheckout` setting in the worktree-specific config\n> +file. This prevents the sparse-checkout feature from interfering with other\n> +worktrees.\n\nI'm afraid that might be mis-parsed by future readers. Perhaps something like:\n\nThe init subcommand also enables the `core.sparseCheckout` setting.\nTo avoid interfering with other worktrees, it first enables the\n`extensions.worktreeConfig` setting and makes sure to set the\n`core.sparseCheckout` setting in the worktree-specific config file.\n\n> +enum sparse_checkout_mode {\n> + MODE_NONE = 0,\n> + MODE_FULL = 1,\n> +};\n\nSo MODE_FULL is \"true\" and MODE_NONE is \"false\". MODE_NONE seems\nconfusing to me, but let's keep reading...\n\n> +\n> +static int sc_set_config(enum sparse_checkout_mode mode)\n> +{\n> + struct argv_array argv = ARGV_ARRAY_INIT;\n> +\n> + if (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n> + error(_(\"failed to set extensions.worktreeConfig setting\"));\n> + return 1;\n> + }\n> +\n> + argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n> +\n> + if (mode)\n> + argv_array_pushl(&argv, \"true\", NULL);\n> + else\n> + argv_array_pushl(&argv, \"false\", NULL);\n\nWait, what? MODE_FULL is used to specify that you want a sparse\ncheckout, and MODE_NONE is used to denote that you want a full (i.e.\nnon-sparse) checkout? These are *very* confusing names.\n\n\n> +static int sparse_checkout_init(int argc, const char **argv)\n> +{\n> + struct pattern_list pl;\n> + char *sparse_filename;\n> + FILE *fp;\n> + int res;\n> +\n> + if (sc_set_config(MODE_FULL))\n> + return 1;\n\nSeems confusing here too.\n\n\nEverything else in the patch looks good, though.\n"},{"id":"383941","messageId":"CABPp-BEcS2fT8XyrU+cghbN5YwO97HR5+xgi5=dGZPJ8ut+eUQ@mail.gmail.com","threadId":"51691","inReplyTo":"c8587a1fb0e335f9cbafdbb44b20d02a1e57886b.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/17] clone: add --sparse mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:20:38Z","receivedAt":"2019-10-11T22:20:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n> During the 'git sparse-checkout init' call, we must first look\n> to see if HEAD is valid, since 'git clone' does not have a valid\n> HEAD.\n\n...does not have a valid HEAD by the time git_sparse_checkout_init() is called?\n\n> The first checkout will create the HEAD ref and update the\n> working directory correctly.\n\nIs this checkout you reference a manual-initiated user checkout after\nthe clone, or the checkout performed as part of the clone? (I'm\nalmost certain it's the latter, but your wording makes me question.)\n"},{"id":"383943","messageId":"CABPp-BHTa1Bk=_pkaBk8FOLcHsTP-z3V=ETLeSu79KPgpzdD4A@mail.gmail.com","threadId":"51691","inReplyTo":"0c3e34ab-1eee-fe29-f584-d21047fd1457@gmail.com","subject":"Re: [PATCH v2 04/11] sparse-checkout: 'set' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:24:39Z","receivedAt":"2019-10-11T22:24:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 11:26 AM Derrick Stolee wrote:\n>\n> On 10/5/2019 8:30 PM, Elijah Newren wrote:\n> > On Sat, Oct 5, 2019 at 3:44 PM Elijah Newren wrote:\n> >>\n> >> On Thu, Sep 19, 2019 at 3:07 PM Derrick Stolee via GitGitGadget\n> >> wrote:\n> >>> +static int write_patterns_and_update(struct pattern_list *pl)\n> >>> +{\n> >>> + char *sparse_filename;\n> >>> + FILE *fp;\n> >>> +\n> >>> + sparse_filename = get_sparse_checkout_filename();\n> >>> + fp = fopen(sparse_filename, \"w\");\n> >>> + write_patterns_to_file(fp, pl);\n> >>> + fclose(fp);\n> >>> + free(sparse_filename);\n> >>> +\n> >>> + clear_pattern_list(pl);\n> >>\n> >> It seems slightly odd that pl is passed in but cleared in this\n> >> function rather than in the caller that created pl. Should this be\n> >> moved to the caller, or, alternatively, a comment added to explain\n> >> this side-effect for future callers of the function?\n> >>\n> >> The rest of the patch looked good to me.\n> >\n> > Actually, thought of something else. What if the user calls 'git\n> > sparse-checkout set ...' without first calling 'git sparse-checkout\n> > init'? Should that report an error to the user, a suggestion to\n> > follow it up with 'sparse-checkout init', or should it just call\n> > sc_set_config() behind the scenes and allow bypassing the init\n> > subcommand?\n>\n> Maybe a warning would suffice. I still think the workflow of the\n> following is most correct, and not difficult to recommend:\n>\n> * \"git sparse-checkout init [--cone]\" -OR- \"git clone --sparse\"\n> * git sparse-checkout set [stuff]\n> * git sparse-checkout disable\n\nRecommending the right thing is easy, but users will call things out\nof order despite documentation. If they call disable before init, I\nsee no problems that will lead to confusion. If they call set without\ncalling init, I can see them being surprised...so I commented on it\nand asked if we want a warning or whatever.\n"},{"id":"383945","messageId":"CABPp-BFaSh6O+eMsZ_ghMNUXdwh5kh2P9JF9msQSgz_XG2fENA@mail.gmail.com","threadId":"51691","inReplyTo":"6ce1d60b38a249f8d9f85d1b1a1ed556aef2469e.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 04/17] sparse-checkout: 'set' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:26:04Z","receivedAt":"2019-10-11T22:26:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The 'git sparse-checkout set' subcommand takes a list of patterns\n> as arguments and writes them to the sparse-checkout file. Then, it\n> updates the working directory using 'git read-tree -mu HEAD'.\n>\n> The 'set' subcommand will replace the entire contents of the\n> sparse-checkout file. The write_patterns_and_update() method is\n> extracted from cmd_sparse_checkout() to make it easier to implement\n> 'add' and/or 'remove' subcommands in the future.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 5 ++++\n> builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 19 +++++++++++++++\n> 3 files changed, 58 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index e095c4a98b..f4bd951550 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -39,6 +39,11 @@ and sets the `core.sparseCheckout` setting in the worktree-specific config\n> file. This prevents the sparse-checkout feature from interfering with other\n> worktrees.\n>\n> +'set'::\n> + Write a set of patterns to the sparse-checkout file, as given as\n> + a list of arguments following the 'set' subcommand. Update the\n> + working directory to match the new patterns.\n> +\n> SPARSE CHECKOUT\n> ----------------\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 3ecb7ac2e7..52d4f832f3 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> - N_(\"git sparse-checkout [init|list]\"),\n> + N_(\"git sparse-checkout [init|list|set] \"),\n> NULL\n> };\n>\n> @@ -140,6 +140,37 @@ static int sparse_checkout_init(int argc, const char **argv)\n> return update_working_directory();\n> }\n>\n> +static int write_patterns_and_update(struct pattern_list *pl)\n> +{\n> + char *sparse_filename;\n> + FILE *fp;\n> +\n> + sparse_filename = get_sparse_checkout_filename();\n> + fp = fopen(sparse_filename, \"w\");\n> + write_patterns_to_file(fp, pl);\n> + fclose(fp);\n> + free(sparse_filename);\n> +\n> + return update_working_directory();\n> +}\n> +\n> +static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> +{\n> + static const char *empty_base = \"\";\n> + int i;\n> + struct pattern_list pl;\n> + int result;\n> + memset(&pl, 0, sizeof(pl));\n> +\n> + for (i = 1; i < argc; i++)\n> + add_pattern(argv[i], empty_base, 0, &pl, 0);\n> +\n> + result = write_patterns_and_update(&pl);\n> +\n> + clear_pattern_list(&pl);\n> + return result;\n> +}\n> +\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> {\n> static struct option builtin_sparse_checkout_options[] = {\n> @@ -162,6 +193,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> return sparse_checkout_list(argc, argv);\n> if (!strcmp(argv[0], \"init\"))\n> return sparse_checkout_init(argc, argv);\n> + if (!strcmp(argv[0], \"set\"))\n> + return sparse_checkout_set(argc, argv, prefix);\n> }\n>\n> usage_with_options(builtin_sparse_checkout_usage,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index d4c145a3af..19e8673c6b 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -101,4 +101,23 @@ test_expect_success 'clone --sparse' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'set sparse-checkout using builtin' '\n> + git -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n> + cat >expect <<-EOF &&\n> + /*\n> + !/*/\n> + *folder*\n> + EOF\n> + git -C repo sparse-checkout list >actual &&\n> + test_cmp expect actual &&\n> + test_cmp expect repo/.git/info/sparse-checkout &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_done\n> --\n\nLooks good, thanks for the fixes. I'm still slightly worried about\nfolks not looking at the docs and calling sparse-checkout set without\ncalling init, and then being negatively surprised. It's a minor\nissue, but a warning might be helpful.\n"},{"id":"383946","messageId":"CABPp-BHc+iUq9NtTL4Mn1U-DhMmV2nX-_OTJr-Ltc7c3xDV-Ow@mail.gmail.com","threadId":"51691","inReplyTo":"0b1ed06bc8c27650086b10b587df943d3053b773.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:27:46Z","receivedAt":"2019-10-11T22:28:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The 'git sparse-checkout set' subcommand takes a list of patterns\n> and places them in the sparse-checkout file. Then, it updates the\n> working directory to match those patterns. For a large list of\n> patterns, the command-line call can get very cumbersome.\n>\n> Add a '--stdin' option to instead read patterns over standard in.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/sparse-checkout.c | 40 ++++++++++++++++++++++++++++--\n> t/t1091-sparse-checkout-builtin.sh | 27 ++++++++++++++++++++\n> 2 files changed, 65 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 52d4f832f3..68f3d8433e 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -145,6 +145,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> char *sparse_filename;\n> FILE *fp;\n>\n> + if (!core_apply_sparse_checkout) {\n> + warning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n> + warning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n> + }\n> +\n> sparse_filename = get_sparse_checkout_filename();\n> fp = fopen(sparse_filename, \"w\");\n> write_patterns_to_file(fp, pl);\n> @@ -154,16 +159,47 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> return update_working_directory();\n> }\n>\n> +static char const * const builtin_sparse_checkout_set_usage[] = {\n> + N_(\"git sparse-checkout set [--stdin|]\"),\n> + NULL\n> +};\n> +\n> +static struct sparse_checkout_set_opts {\n> + int use_stdin;\n> +} set_opts;\n> +\n> static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> {\n> static const char *empty_base = \"\";\n> int i;\n> struct pattern_list pl;\n> int result;\n> +\n> + static struct option builtin_sparse_checkout_set_options[] = {\n> + OPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n> + N_(\"read patterns from standard in\")),\n> + OPT_END(),\n> + };\n> +\n> memset(&pl, 0, sizeof(pl));\n>\n> - for (i = 1; i < argc; i++)\n> - add_pattern(argv[i], empty_base, 0, &pl, 0);\n> + argc = parse_options(argc, argv, prefix,\n> + builtin_sparse_checkout_set_options,\n> + builtin_sparse_checkout_set_usage,\n> + PARSE_OPT_KEEP_UNKNOWN);\n\nDoes this mean users can also spell it 'git sparse-checkout --stdin\nset', instead of the expected 'git sparse-checkout set --stdin'?\n\n> +\n> + if (set_opts.use_stdin) {\n> + struct strbuf line = STRBUF_INIT;\n> +\n> + while (!strbuf_getline(&line, stdin)) {\n> + size_t len;\n> + char *buf = strbuf_detach(&line, &len);\n> + add_pattern(buf, empty_base, 0, &pl, 0);\n> + }\n> + } else {\n> + for (i = 0; i < argc; i++)\n> + add_pattern(argv[i], empty_base, 0, &pl, 0);\n> + }\n>\n> result = write_patterns_and_update(&pl);\n>\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 19e8673c6b..2a0137fde3 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -101,6 +101,13 @@ test_expect_success 'clone --sparse' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'warn if core.sparseCheckout is disabled' '\n> + test_when_finished git -C repo config --worktree core.sparseCheckout true &&\n> + git -C repo config --worktree core.sparseCheckout false &&\n> + git -C repo sparse-checkout set folder1 2>err &&\n> + test_i18ngrep \"core.sparseCheckout is disabled\" err\n> +'\n> +\n> test_expect_success 'set sparse-checkout using builtin' '\n> git -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n> cat >expect <<-EOF &&\n> @@ -120,4 +127,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n> test_cmp expect dir\n> '\n>\n> +test_expect_success 'set sparse-checkout using --stdin' '\n> + cat >expect <<-EOF &&\n> + /*\n> + !/*/\n> + /folder1/\n> + /folder2/\n> + EOF\n> + git -C repo sparse-checkout set --stdin + git -C repo sparse-checkout list >actual &&\n> + test_cmp expect actual &&\n> + test_cmp expect repo/.git/info/sparse-checkout &&\n> + ls repo >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + folder1\n> + folder2\n> + EOF\n> + test_cmp expect dir\n> +'\n> +\n> test_done\n> --\n> gitgitgadget\n>\n"},{"id":"383947","messageId":"CABPp-BE=p1SnpLnOPpC=5ZsK5DtW7+DsgTwZRDWHor_p3Vx36w@mail.gmail.com","threadId":"51691","inReplyTo":"CABPp-BFaSh6O+eMsZ_ghMNUXdwh5kh2P9JF9msQSgz_XG2fENA@mail.gmail.com","subject":"Re: [PATCH v3 04/17] sparse-checkout: 'set' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-11T22:30:05Z","receivedAt":"2019-10-11T22:30:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Oct 11, 2019 at 3:26 PM Elijah Newren wrote:\n>\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n\n> Looks good, thanks for the fixes. I'm still slightly worried about\n> folks not looking at the docs and calling sparse-checkout set without\n> calling init, and then being negatively surprised. It's a minor\n> issue, but a warning might be helpful.\n\nLooks like you added that to patch 5, so nevermind.\n"},{"id":"383978","messageId":"CABPp-BH0pN5noDpFH286YFHC22+SNr9ZuyJxuy=WzNPQ=cyxfg@mail.gmail.com","threadId":"51691","inReplyTo":"3a677f32b6b914fcc800b619fd082da25c275422.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 13/17] read-tree: show progress by default","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-12T22:16:05Z","receivedAt":"2019-10-12T22:16:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The read-tree builtin has a --verbose option that signals to show\n> progress and other data while updating the index. Update this to\n> be on by default when stderr is a terminal window.\n>\n> This will help tools like 'git sparse-checkout' to automatically\n> benefit from progress indicators when a user runs these commands.\n\nThis change seems fine, but in patch 2 you said:\n\n> The use of running another process for 'git read-tree' is sub-\n> optimal. This will be removed in a later change.\n\nleaving me slightly confused about the goal/plan.\n"},{"id":"383980","messageId":"CABPp-BF+JWYZfDqp2Tn4AEKVp4b0YMA=Mbz4Nz62D-gGgiduYQ@mail.gmail.com","threadId":"51691","inReplyTo":"a6f17e9a77d86f8ec856ea08617d1c1af2853d54.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 15/17] sparse-checkout: update working directory in-process","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-12T22:57:14Z","receivedAt":"2019-10-12T22:57:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n> skip-worktree bits in the index and to update the working directory.\n> This extra process is overly complex, and prone to failure. It also\n> requires that we write our changes to the sparse-checkout file before\n> trying to update the index.\n>\n> Remove this extra process call by creating a direct call to\n> unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n> adition, provide an in-memory list of patterns so we can avoid\n\ns/adition/addition/\n\n> reading from the sparse-checkout file. This allows us to test a\n> proposed change to the file before writing to it.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/read-tree.c | 2 +-\n> builtin/sparse-checkout.c | 85 +++++++++++++++++++++++++-----\n> t/t1091-sparse-checkout-builtin.sh | 17 ++++++\n> unpack-trees.c | 5 +-\n> unpack-trees.h | 3 +-\n> 5 files changed, 95 insertions(+), 17 deletions(-)\n>\n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index 69963d83dc..d7eeaa26ec 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>\n> if (opts.reset || opts.merge || opts.prefix) {\n> if (read_cache_unmerged() && (opts.prefix || opts.merge))\n> - die(\"You need to resolve your current index first\");\n> + die(_(\"You need to resolve your current index first\"));\n\nA good change, but isn't this unrelated to the current commit?\n\n> stage = opts.merge = 1;\n> }\n> resolve_undo_clear();\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 25786f8bb0..542d57fac6 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -7,6 +7,11 @@\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n> #include \"string-list.h\"\n> +#include \"cache.h\"\n> +#include \"cache-tree.h\"\n> +#include \"lockfile.h\"\n> +#include \"resolve-undo.h\"\n> +#include \"unpack-trees.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> N_(\"git sparse-checkout [init|list|set|disable] \"),\n> @@ -60,18 +65,53 @@ static int sparse_checkout_list(int argc, const char **argv)\n> return 0;\n> }\n>\n> -static int update_working_directory(void)\n> +static int update_working_directory(struct pattern_list *pl)\n> {\n> - struct argv_array argv = ARGV_ARRAY_INIT;\n> int result = 0;\n> - argv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n> + struct unpack_trees_options o;\n> + struct lock_file lock_file = LOCK_INIT;\n> + struct object_id oid;\n> + struct tree *tree;\n> + struct tree_desc t;\n> + struct repository *r = the_repository;\n>\n> - if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> - error(_(\"failed to update index with new sparse-checkout paths\"));\n> - result = 1;\n> + if (repo_read_index_unmerged(r))\n> + die(_(\"You need to resolve your current index first\"));\n\nWell, at least that ensures that the user gets a good error message.\nI'm not sure I like the error, because e.g. if a user hits a conflict\nwhile merging in a sparse checkout and wants to return to a non-sparse\ncheckout because they think other files might help them resolve the\nconflicts, then they ought to be able to do it. Basically, unless\nthey are trying use sparsification to remove entries from the working\ndirectory that differ from the index (and conflicted entries always\ndiffer), then it seems like we should be able to support\nsparsification despite the presence of conflicts.\n\nYour series is long enough, doesn't make this problem any worse (and\nappears to make it slightly better), and so you really don't need to\ntackle that problem in this series. I'm just stating a gripe with\nsparse checkouts again. :-)\n\n[...]\n\n> static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n> {\n> - struct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n> + struct pattern_entry *e = xmalloc(sizeof(*e));\n\nThis is a good fix, but shouldn't it be squashed into the\n\"sparse-checkout: init and set in cone mode\" commit from earlier in\nyour series?\n\n> @@ -262,12 +308,21 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> {\n> char *sparse_filename;\n> FILE *fp;\n> -\n> + int result;\n> +\n\nTrailing whitespace that should be cleaned up.\n\n> if (!core_apply_sparse_checkout) {\n> warning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n> warning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n> }\n>\n> + result = update_working_directory(pl);\n> +\n> + if (result) {\n> + clear_pattern_list(pl);\n> + update_working_directory(NULL);\n> + return result;\n> + }\n> +\n> sparse_filename = get_sparse_checkout_filename();\n> fp = fopen(sparse_filename, \"w\");\n>\n> @@ -277,9 +332,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> write_patterns_to_file(fp, pl);\n>\n> fclose(fp);\n> +\n> free(sparse_filename);\n> + clear_pattern_list(pl);\n>\n> - return update_working_directory();\n> + return 0;\n> }\n>\n> static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n> @@ -330,6 +387,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> struct strbuf line = STRBUF_INIT;\n> hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n> hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n> + pl.use_cone_patterns = 1;\n>\n> if (set_opts.use_stdin) {\n> while (!strbuf_getline(&line, stdin))\n> @@ -375,7 +433,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> fprintf(fp, \"/*\\n\");\n> fclose(fp);\n>\n> - if (update_working_directory())\n> + core_apply_sparse_checkout = 1;\n> + if (update_working_directory(NULL))\n> die(_(\"error while refreshing working directory\"));\n>\n> unlink(sparse_filename);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index ee4d361787..82eb5fb2f8 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -199,11 +199,13 @@ test_expect_success 'cone mode: init and set' '\n> a\n> deep\n> EOF\n> + test_cmp dir expect &&\n> ls repo/deep >dir &&\n> cat >expect <<-EOF &&\n> a\n> deeper1\n> EOF\n> + test_cmp dir expect &&\n> ls repo/deep/deeper1 >dir &&\n> cat >expect <<-EOF &&\n> a\n> @@ -245,4 +247,19 @@ test_expect_success 'cone mode: set with nested folders' '\n> test_cmp repo/.git/info/sparse-checkout expect\n> '\n>\n> +test_expect_success 'revert to old sparse-checkout on bad update' '\n> + echo update >repo/deep/deeper2/a &&\n> + cp repo/.git/info/sparse-checkout expect &&\n> + test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n> + test_i18ngrep \"Cannot update sparse checkout\" err &&\n> + test_cmp repo/.git/info/sparse-checkout expect &&\n> + ls repo/deep >dir &&\n> + cat >expect <<-EOF &&\n> + a\n> + deeper1\n> + deeper2\n> + EOF\n> + test_cmp dir expect\n> +'\n> +\n> test_done\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index edf0fb4673..f0fee5adf2 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -1508,7 +1508,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n> memset(&pl, 0, sizeof(pl));\n> if (!core_apply_sparse_checkout || !o->update)\n> o->skip_sparse_checkout = 1;\n> - if (!o->skip_sparse_checkout) {\n> + if (!o->skip_sparse_checkout && !o->pl) {\n> char *sparse = git_pathdup(\"info/sparse-checkout\");\n> pl.use_cone_patterns = core_sparse_checkout_cone;\n> if (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n> @@ -1681,7 +1681,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>\n> done:\n> trace_performance_leave(\"unpack_trees\");\n> - clear_pattern_list(&pl);\n> + if (!o->keep_pattern_list)\n> + clear_pattern_list(&pl);\n> return ret;\n>\n> return_failed:\n> diff --git a/unpack-trees.h b/unpack-trees.h\n> index f2eee0c7c5..ca94a421a5 100644\n> --- a/unpack-trees.h\n> +++ b/unpack-trees.h\n> @@ -59,7 +59,8 @@ struct unpack_trees_options {\n> quiet,\n> exiting_early,\n> show_all_errors,\n> - dry_run;\n> + dry_run,\n> + keep_pattern_list;\n> const char *prefix;\n> int cache_bottom;\n> struct dir_struct *dir;\n> --\n\nThe rest looks reasonable.\n"},{"id":"383981","messageId":"CABPp-BGCNpZWoWk4coNEpA0zS2g0HXi5Kzb+QVusyLRt0fy=zw@mail.gmail.com","threadId":"51691","inReplyTo":"8927494b8c418e43f5bbd6e1eb108be5a0a263fc.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 16/17] sparse-checkout: write using lockfile","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-12T22:59:15Z","receivedAt":"2019-10-12T22:59:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> If two 'git sparse-checkout set' subcommands are launched at the\n> same time, the behavior can be unexpected as they compete to write\n> the sparse-checkout file and update the working directory.\n>\n> Take a lockfile around the writes to the sparse-checkout file. In\n> addition, acquire this lock around the working directory update\n> to avoid two commands updating the working directory in different\n> ways.\n\nWow, there's something I never would have thought to check. Did you\nhave folks run into this, or is this just some defensive programming?\nEither way, I'm impressed.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/sparse-checkout.c | 15 ++++++++++++---\n> t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n> 2 files changed, 19 insertions(+), 3 deletions(-)\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 542d57fac6..9b313093cd 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -308,6 +308,8 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> {\n> char *sparse_filename;\n> FILE *fp;\n> + int fd;\n> + struct lock_file lk = LOCK_INIT;\n> int result;\n>\n> if (!core_apply_sparse_checkout) {\n> @@ -317,21 +319,28 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>\n> result = update_working_directory(pl);\n>\n> + sparse_filename = get_sparse_checkout_filename();\n> + fd = hold_lock_file_for_update(&lk, sparse_filename,\n> + LOCK_DIE_ON_ERROR);\n> +\n> + result = update_working_directory(pl);\n> if (result) {\n> + rollback_lock_file(&lk);\n> + free(sparse_filename);\n> clear_pattern_list(pl);\n> update_working_directory(NULL);\n> return result;\n> }\n>\n> - sparse_filename = get_sparse_checkout_filename();\n> - fp = fopen(sparse_filename, \"w\");\n> + fp = fdopen(fd, \"w\");\n>\n> if (core_sparse_checkout_cone)\n> write_cone_to_file(fp, pl);\n> else\n> write_patterns_to_file(fp, pl);\n>\n> - fclose(fp);\n> + fflush(fp);\n> + commit_lock_file(&lk);\n>\n> free(sparse_filename);\n> clear_pattern_list(pl);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 82eb5fb2f8..f22a4afbea 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -262,4 +262,11 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n> test_cmp dir expect\n> '\n>\n> +test_expect_success 'fail when lock is taken' '\n> + test_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&\n> + touch repo/.git/info/sparse-checkout.lock &&\n> + test_must_fail git -C repo sparse-checkout set deep 2>err &&\n> + test_i18ngrep \"File exists\" err\n> +'\n> +\n> test_done\n> --\n> gitgitgadget\n>\n"},{"id":"383982","messageId":"CABPp-BHs+c83imzHUHarsJhFsNUQNkPCkDHLX8KH-2CJztGdGA@mail.gmail.com","threadId":"51691","inReplyTo":"7f377c1407274aec87c6261f05bb9a944b659ba0.1570478905.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 17/17] sparse-checkout: cone mode should not interact with .gitignore","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-12T23:00:40Z","receivedAt":"2019-10-12T23:00:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> During the development of the sparse-checkout \"cone mode\" feature,\n> an incorrect placement of the initializer for \"use_cone_patterns = 1\"\n> caused warnings to show up when a .gitignore file was present with\n> non-cone-mode patterns. This was fixed in the original commit\n> introducing the cone mode, but now we should add a test to avoid\n> hitting this problem again in the future.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n> 1 file changed, 7 insertions(+)\n>\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index f22a4afbea..ed9355384a 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -269,4 +269,11 @@ test_expect_success 'fail when lock is taken' '\n> test_i18ngrep \"File exists\" err\n> '\n>\n> +test_expect_success '.gitignore should not warn about cone mode' '\n> + git -C repo config --worktree core.sparseCheckoutCone true &&\n> + echo \"**/bin/*\" >repo/.gitignore &&\n> + git -C repo reset --hard 2>err &&\n> + test_i18ngrep ! \"disabling cone patterns\" err\n> +'\n> +\n> test_done\n> --\n\nMakes sense; thanks for adding good preventative tests.\n"},{"id":"383983","messageId":"CABPp-BHJeuEHBDkf93m9sfSZ4rZB7+eFejiAXOsjLEUu5eT5FA@mail.gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-12T23:22:11Z","receivedAt":"2019-10-12T23:22:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> This series makes the sparse-checkout feature more user-friendly. While\n> there, I also present a way to use a limited set of patterns to gain a\n> significant performance boost in very large repositories.\n>\n> Sparse-checkout is only documented as a subsection of the read-tree docs\n> [1], which makes the feature hard to discover. Users have trouble navigating\n> the feature, especially at clone time [2], and have even resorted to\n> creating their own helper tools [3].\n>\n> This series attempts to solve these problems using a new builtin. Here is a\n> sample workflow to give a feeling for how it can work:\n>\n> In an existing repo:\n>\n> $ git sparse-checkout init\n> $ ls\n> myFile1.txt myFile2.txt\n> $ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n> $ ls\n> myFile1.txt myFile2.txt myFolder\n> $ ls myFolder\n> a.c a.h\n> $ git sparse-checkout disable\n> $ ls\n> hiddenFolder myFile1.txt myFile2.txt myFolder\n>\n> At clone time:\n>\n> $ git clone --sparse origin repo\n> $ cd repo\n> $ ls\n> myFile1.txt myFile2.txt\n> $ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n> $ ls\n> myFile1.txt myFile2.txt myFolder\n>\n> Here are some more specific details:\n>\n> * git sparse-checkout init enables core.sparseCheckout and populates the\n> sparse-checkout file with patterns that match only the files at root.\n>\n>\n> * git clone learns the --sparse argument to run git sparse-checkout init\n> before the first checkout.\n>\n>\n> * git sparse-checkout set reads patterns from the arguments, or with\n> --stdin reads patterns from stdin one per line, then writes them to the\n> sparse-checkout file and refreshes the working directory.\n>\n>\n> * git sparse-checkout disable removes the patterns from the sparse-checkout\n> file, disables core.sparseCheckout, and refills the working directory.\n>\n>\n> * git sparse-checkout list lists the contents of the sparse-checkout file.\n>\n>\n>\n> The documentation for the sparse-checkout feature can now live primarily\n> with the git-sparse-checkout documentation.\n>\n> Cone Mode\n> =========\n>\n> What really got me interested in this area is a performance problem. If we\n> have N patterns in the sparse-checkout file and M entries in the index, then\n> we can perform up to O(N * M) pattern checks in clear_ce_flags(). This\n> quadratic growth is not sustainable in a repo with 1,000+ patterns and\n> 1,000,000+ index entries.\n>\n> To solve this problem, I propose a new, more restrictive mode to\n> sparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\n> matches at a directory level. This can then use hashsets for fast\n> performance -- O(M) instead of O(N*M). My hashset implementation is based on\n> the virtual filesystem hook in the VFS for Git custom code [4].\n>\n> In cone mode, a user specifies a list of folders which the user wants every\n> file inside. In addition, the cone adds all blobs that are siblings of the\n> folders in the directory path to that folder. This makes the directories\n> look \"hydrated\" as a user drills down to those recursively-closed folders.\n> These directories are called \"parent\" folders, as a file matches them only\n> if the file's immediate parent is that directory.\n>\n> When building a prototype of this feature, I used a separate file to contain\n> the list of recursively-closed folders and built the hashsets dynamically\n> based on that file. In this implementation, I tried to maximize the amount\n> of backwards-compatibility by storing all data in the sparse-checkout file\n> using patterns recognized by earlier Git versions.\n>\n> For example, if we add A/B/C as a recursive folder, then we add the\n> following patterns to the sparse-checkout file:\n>\n> /*\n> !/*/\n> /A/\n> !/A/*/\n> /A/B/\n> !/A/B/*/\n> /A/B/C/\n>\n> The alternating positive/negative patterns say \"include everything in this\n> folder, but exclude everything another level deeper\". The final pattern has\n> no matching negation, so is a recursively closed pattern.\n>\n> Note that I have some basic warnings to try and check that the\n> sparse-checkout file doesn't match what would be written by a cone-mode add.\n> In such a case, Git writes a warning to stderr and continues with the old\n> pattern matching algorithm. These checks are currently very barebones, and\n> would need to be updated with more robust checks for things like regex\n> characters in the middle of the pattern. As review moves forward (and if we\n> don't change the data storage) then we could spend more time on this.\n>\n> Thanks, -Stolee\n>\n> Updates in v2, relative to the RFC:\n>\n> * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n> adding 'add' and/or 'remove' subcommands later.\n>\n>\n> * 'set' reads from the arguments by default. '--stdin' option is available.\n>\n>\n> * A new performance-oriented commit is added at the end.\n>\n>\n> * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n> pattern.\n>\n>\n> * References to a \"bug\" (that was really a strange GVFS interaction in\n> microsoft/git) around deleting outside the cone are removed.\n>\n>\n>\n> Updates in v3:\n>\n> * The bad interaction with \"cone mode\" and .gitignore files is fixed. A\n> test is added in the last patch.\n>\n>\n> * Several patches are added that make the feature more robust. One\n> sanitizes user input, another few add progress indicators, and another\n> more prevent users from getting in bad states due to working directory\n> changes or concurrent processes.\n>\n>\n> * Updated several docs and commit messages according to feedback. Thanks,\n> Elijah!\n>\n\nI've read through v3, and at this point I'm mostly just finding small\nstuff to comment on; the patches are looking pretty good. One bigger\nitem I'd like to comment on, is that I'd really like this feature to\nbe explicitly labelled as experimental as per [1]. Not just because\nof edge and corner case bugs (at least one of which we know about and\nI've drummed on a few times), but much more importantly because I\nbelieve grep, log, diff, and other commands should by default pay\nattention to the sparsity patterns to limit their output for the user\nboth because that's the stuff the user is interested in and the files\noutside those paths are at best noise, and because it provides them an\nability to get the responsiveness of a small repository even while\nworking in a bigger one. See [2] for more details. If we don't mark\nthis command as experimental, I'm worried people may start coming to\nexpect whatever behavior they get with sparse checkouts and then we'll\nbe locked in due to backward compatibility, and users who want a good\nexperience will have to know to set a whole bunch of flags instead of\ngetting the right behavior by default.\n\n[1] https://public-inbox.org/git/b1444dab-24e5-6e4d-bea8-37abc433b546@gmail.com/\n[2] https://public-inbox.org/git/CABPp-BGuFhDwWZBRaD3nA8ui46wor-4=Ha1G1oApsfF8KNpfGQ@mail.gmail.com/\n\n> Things to leave for future patches:\n>\n> 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n> worktree-specific file.\n\nI'm a big fan of this, but it sounds like Junio isn't[2]. I tried to\nfollow up to get more details, but I'm worried his view might mean\nnon-sparse-worktrees-by-default, which in turn I'm worried will be\nunworkable with sufficiently large repos and/or not be very friendly\nto future users of both partial clones and sparse checkouts.\n\n[2] https://public-inbox.org/git/xmqqy2y3ejwe.fsf@gitster-ct.c.googlers.com/\n\n> 2. More robustness around detecting non-cone patterns with wildcards in the\n> middle of the line.\n\nMake sense.\n\n> 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n> (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n> super-valuable, as it only starts changing behavior when someone calls\n> 'git sparse-checkout set', but may be interesting.\n\nIt'd probably be nice for users, who would otherwise need to run:\n git clone --sparse ...\n git sparse-checkout init --cone\n git sparse-checkout set ...\n\nThe need to run init after doing a clone --sparse could possibly be\nseen as annoying. OR.... maybe those users will just set\ncore.sparseCheckoutCone=true in their global gitconfig, if the\ndocumentation points them to it or they think about it.\n\n\n\nThanks for all the work on this series! (And sorry my reviews always\ntake a while...)\n"},{"id":"384030","messageId":"91b9126d-8b50-8648-d5c1-25e4c6c68f8b@gmail.com","threadId":"51691","inReplyTo":"CABPp-BHd9sJ=HcM-VQaGpd5TV-A0JL5NWG6zdDReUJrMnE+FwA@mail.gmail.com","subject":"Re: [PATCH v3 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-14T20:22:30Z","receivedAt":"2019-10-14T20:22:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/11/2019 6:14 PM, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n> wrote:\n>> ++\n>> +The init subcommand also enables the 'extensions.worktreeConfig' setting\n>> +and sets the `core.sparseCheckout` setting in the worktree-specific config\n>> +file. This prevents the sparse-checkout feature from interfering with other\n>> +worktrees.\n> \n> I'm afraid that might be mis-parsed by future readers. Perhaps something like:\n> \n> The init subcommand also enables the `core.sparseCheckout` setting.\n\nI like the paragraph below, but the sentence above is repeated from\nthe earlier paragraph.\n\n> To avoid interfering with other worktrees, it first enables the\n> `extensions.worktreeConfig` setting and makes sure to set the\n> `core.sparseCheckout` setting in the worktree-specific config file.\n> \n>> +enum sparse_checkout_mode {\n>> + MODE_NONE = 0,\n>> + MODE_FULL = 1,\n>> +};\n> \n> So MODE_FULL is \"true\" and MODE_NONE is \"false\". MODE_NONE seems\n> confusing to me, but let's keep reading...\n> \n>> +\n>> +static int sc_set_config(enum sparse_checkout_mode mode)\n>> +{\n>> + struct argv_array argv = ARGV_ARRAY_INIT;\n>> +\n>> + if (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n>> + error(_(\"failed to set extensions.worktreeConfig setting\"));\n>> + return 1;\n>> + }\n>> +\n>> + argv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n>> +\n>> + if (mode)\n>> + argv_array_pushl(&argv, \"true\", NULL);\n>> + else\n>> + argv_array_pushl(&argv, \"false\", NULL);\n> \n> Wait, what? MODE_FULL is used to specify that you want a sparse\n> checkout, and MODE_NONE is used to denote that you want a full (i.e.\n> non-sparse) checkout? These are *very* confusing names.\n\nI understand they are confusing, hopefully it makes more sense with\nthe cone mode later.\n\n* NONE == \"No patterns at all\"\n* FULL == \"all patterns allowed\"\n* CONE == \"only cone patterns\" (appears later)\n\nSince this is just an internal detail, what if I switched it to\n\n* MODE_NO_PATTERNS\n* MODE_ALL_PATTERNS\n* MODE_CONE_PATTERNS\n\nWould that make more sense?\n\n>> +static int sparse_checkout_init(int argc, const char **argv)\n>> +{\n>> + struct pattern_list pl;\n>> + char *sparse_filename;\n>> + FILE *fp;\n>> + int res;\n>> +\n>> + if (sc_set_config(MODE_FULL))\n>> + return 1;\n> \n> Seems confusing here too.\n> \n> \n> Everything else in the patch looks good, though.\n\nThanks,\n-Stolee\n"},{"id":"384031","messageId":"4d961532-0d9a-2e7f-a2b9-a05d09ef7c83@gmail.com","threadId":"51691","inReplyTo":"CABPp-BHc+iUq9NtTL4Mn1U-DhMmV2nX-_OTJr-Ltc7c3xDV-Ow@mail.gmail.com","subject":"Re: [PATCH v3 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-14T20:28:33Z","receivedAt":"2019-10-14T20:28:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/11/2019 6:27 PM, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The 'git sparse-checkout set' subcommand takes a list of patterns\n>> and places them in the sparse-checkout file. Then, it updates the\n>> working directory to match those patterns. For a large list of\n>> patterns, the command-line call can get very cumbersome.\n>>\n>> Add a '--stdin' option to instead read patterns over standard in.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/sparse-checkout.c | 40 ++++++++++++++++++++++++++++--\n>> t/t1091-sparse-checkout-builtin.sh | 27 ++++++++++++++++++++\n>> 2 files changed, 65 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index 52d4f832f3..68f3d8433e 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -145,6 +145,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>> char *sparse_filename;\n>> FILE *fp;\n>>\n>> + if (!core_apply_sparse_checkout) {\n>> + warning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n>> + warning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n>> + }\n>> +\n>> sparse_filename = get_sparse_checkout_filename();\n>> fp = fopen(sparse_filename, \"w\");\n>> write_patterns_to_file(fp, pl);\n>> @@ -154,16 +159,47 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>> return update_working_directory();\n>> }\n>>\n>> +static char const * const builtin_sparse_checkout_set_usage[] = {\n>> + N_(\"git sparse-checkout set [--stdin|]\"),\n>> + NULL\n>> +};\n>> +\n>> +static struct sparse_checkout_set_opts {\n>> + int use_stdin;\n>> +} set_opts;\n>> +\n>> static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n>> {\n>> static const char *empty_base = \"\";\n>> int i;\n>> struct pattern_list pl;\n>> int result;\n>> +\n>> + static struct option builtin_sparse_checkout_set_options[] = {\n>> + OPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n>> + N_(\"read patterns from standard in\")),\n>> + OPT_END(),\n>> + };\n>> +\n>> memset(&pl, 0, sizeof(pl));\n>>\n>> - for (i = 1; i < argc; i++)\n>> - add_pattern(argv[i], empty_base, 0, &pl, 0);\n>> + argc = parse_options(argc, argv, prefix,\n>> + builtin_sparse_checkout_set_options,\n>> + builtin_sparse_checkout_set_usage,\n>> + PARSE_OPT_KEEP_UNKNOWN);\n> \n> Does this mean users can also spell it 'git sparse-checkout --stdin\n> set', instead of the expected 'git sparse-checkout set --stdin'?\n\nNo, because the parse_options() inside cmd_sparse_checkout() parses until\nit doesn't recognize an option. ('stdin' in your example). After we \"consume\"\nthe subcommand \"set\", we call this method and the parse_options() can then\nread the '--stdin'.\n\nHere is the output from my local command of 'git sparse-checkout --stdin set'\nat this commit:\n\n\t$ ./git sparse-checkout --stdin set\n\terror: unknown option `stdin'\n\tusage: git sparse-checkout [init|list|set] \n\nThanks,\n-Stolee\n"},{"id":"384032","messageId":"7887efb4-60de-f6d1-6cba-5ccb60645c52@gmail.com","threadId":"51691","inReplyTo":"CABPp-BH0pN5noDpFH286YFHC22+SNr9ZuyJxuy=WzNPQ=cyxfg@mail.gmail.com","subject":"Re: [PATCH v3 13/17] read-tree: show progress by default","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-14T20:31:30Z","receivedAt":"2019-10-14T20:31:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2019 6:16 PM, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The read-tree builtin has a --verbose option that signals to show\n>> progress and other data while updating the index. Update this to\n>> be on by default when stderr is a terminal window.\n>>\n>> This will help tools like 'git sparse-checkout' to automatically\n>> benefit from progress indicators when a user runs these commands.\n> \n> This change seems fine, but in patch 2 you said:\n> \n>> The use of running another process for 'git read-tree' is sub-\n>> optimal. This will be removed in a later change.\n> \n> leaving me slightly confused about the goal/plan.\n\nTrue, this is not necessary for the whole series. I created this\npatch as a way to show progress in our microsoft/git fork [1], then\nremoved the read-tree call in a later change [2]. When preparing v3,\nI took all of the changes together.\n\nI thought this was valuable on its own, for those users who are\nusing the old mechanisms for sparse-checkout updates.\n\nThanks,\n-Stolee\n\n[1] https://github.com/microsoft/git/pull/200\n\n[2] https://github.com/microsoft/git/pull/204\n"},{"id":"384033","messageId":"c12390e3-4126-98f4-396d-6dc57b15184b@gmail.com","threadId":"51691","inReplyTo":"CABPp-BF+JWYZfDqp2Tn4AEKVp4b0YMA=Mbz4Nz62D-gGgiduYQ@mail.gmail.com","subject":"Re: [PATCH v3 15/17] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-14T20:39:44Z","receivedAt":"2019-10-14T20:39:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2019 6:57 PM, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n>> skip-worktree bits in the index and to update the working directory.\n>> This extra process is overly complex, and prone to failure. It also\n>> requires that we write our changes to the sparse-checkout file before\n>> trying to update the index.\n>>\n>> Remove this extra process call by creating a direct call to\n>> unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n>> adition, provide an in-memory list of patterns so we can avoid\n> \n> s/adition/addition/\n> \n>> reading from the sparse-checkout file. This allows us to test a\n>> proposed change to the file before writing to it.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/read-tree.c | 2 +-\n>> builtin/sparse-checkout.c | 85 +++++++++++++++++++++++++-----\n>> t/t1091-sparse-checkout-builtin.sh | 17 ++++++\n>> unpack-trees.c | 5 +-\n>> unpack-trees.h | 3 +-\n>> 5 files changed, 95 insertions(+), 17 deletions(-)\n>>\n>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>> index 69963d83dc..d7eeaa26ec 100644\n>> --- a/builtin/read-tree.c\n>> +++ b/builtin/read-tree.c\n>> @@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>>\n>> if (opts.reset || opts.merge || opts.prefix) {\n>> if (read_cache_unmerged() && (opts.prefix || opts.merge))\n>> - die(\"You need to resolve your current index first\");\n>> + die(_(\"You need to resolve your current index first\"));\n> \n> A good change, but isn't this unrelated to the current commit?\n\nIt's related because I'm repeating the error in the sparse-checkout builtin, but\nit should be localized in both places.\n\n>> stage = opts.merge = 1;\n>> }\n>> resolve_undo_clear();\n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index 25786f8bb0..542d57fac6 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -7,6 +7,11 @@\n>> #include \"run-command.h\"\n>> #include \"strbuf.h\"\n>> #include \"string-list.h\"\n>> +#include \"cache.h\"\n>> +#include \"cache-tree.h\"\n>> +#include \"lockfile.h\"\n>> +#include \"resolve-undo.h\"\n>> +#include \"unpack-trees.h\"\n>>\n>> static char const * const builtin_sparse_checkout_usage[] = {\n>> N_(\"git sparse-checkout [init|list|set|disable] \"),\n>> @@ -60,18 +65,53 @@ static int sparse_checkout_list(int argc, const char **argv)\n>> return 0;\n>> }\n>>\n>> -static int update_working_directory(void)\n>> +static int update_working_directory(struct pattern_list *pl)\n>> {\n>> - struct argv_array argv = ARGV_ARRAY_INIT;\n>> int result = 0;\n>> - argv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n>> + struct unpack_trees_options o;\n>> + struct lock_file lock_file = LOCK_INIT;\n>> + struct object_id oid;\n>> + struct tree *tree;\n>> + struct tree_desc t;\n>> + struct repository *r = the_repository;\n>>\n>> - if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n>> - error(_(\"failed to update index with new sparse-checkout paths\"));\n>> - result = 1;\n>> + if (repo_read_index_unmerged(r))\n>> + die(_(\"You need to resolve your current index first\"));\n> \n> Well, at least that ensures that the user gets a good error message.\n> I'm not sure I like the error, because e.g. if a user hits a conflict\n> while merging in a sparse checkout and wants to return to a non-sparse\n> checkout because they think other files might help them resolve the\n> conflicts, then they ought to be able to do it. Basically, unless\n> they are trying use sparsification to remove entries from the working\n> directory that differ from the index (and conflicted entries always\n> differ), then it seems like we should be able to support\n> sparsification despite the presence of conflicts.\n> \n> Your series is long enough, doesn't make this problem any worse (and\n> appears to make it slightly better), and so you really don't need to\n> tackle that problem in this series. I'm just stating a gripe with\n> sparse checkouts again. :-)\n\nAbsolutely, we should revisit the entire feature and how it handles these\nconflicts in the best possible ways. As far as I can see, the only way these\nconflicts arise is if the user creates conflicting files _outside_ their\nsparse cone and then expand their cone. Finding all the strange cases\nwill require experimentation.\n \n> [...]\n> \n>> static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n>> {\n>> - struct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n>> + struct pattern_entry *e = xmalloc(sizeof(*e));\n> \n> This is a good fix, but shouldn't it be squashed into the\n> \"sparse-checkout: init and set in cone mode\" commit from earlier in\n> your series?\n\nYeah, I think I mis-applied a few fixups to this commit instead of an earlier one.\n\n>> @@ -262,12 +308,21 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>> {\n>> char *sparse_filename;\n>> FILE *fp;\n>> -\n>> + int result;\n>> +\n> \n> Trailing whitespace that should be cleaned up.\n\nThanks. Will do.\n\n> \n>> if (!core_apply_sparse_checkout) {\n>> warning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n>> warning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n>> }\n>>\n>> + result = update_working_directory(pl);\n>> +\n>> + if (result) {\n>> + clear_pattern_list(pl);\n>> + update_working_directory(NULL);\n>> + return result;\n>> + }\n>> +\n>> sparse_filename = get_sparse_checkout_filename();\n>> fp = fopen(sparse_filename, \"w\");\n>>\n>> @@ -277,9 +332,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>> write_patterns_to_file(fp, pl);\n>>\n>> fclose(fp);\n>> +\n>> free(sparse_filename);\n>> + clear_pattern_list(pl);\n>>\n>> - return update_working_directory();\n>> + return 0;\n>> }\n>>\n>> static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n>> @@ -330,6 +387,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n>> struct strbuf line = STRBUF_INIT;\n>> hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n>> hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n>> + pl.use_cone_patterns = 1;\n>>\n>> if (set_opts.use_stdin) {\n>> while (!strbuf_getline(&line, stdin))\n>> @@ -375,7 +433,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n>> fprintf(fp, \"/*\\n\");\n>> fclose(fp);\n>>\n>> - if (update_working_directory())\n>> + core_apply_sparse_checkout = 1;\n>> + if (update_working_directory(NULL))\n>> die(_(\"error while refreshing working directory\"));\n>>\n>> unlink(sparse_filename);\n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index ee4d361787..82eb5fb2f8 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -199,11 +199,13 @@ test_expect_success 'cone mode: init and set' '\n>> a\n>> deep\n>> EOF\n>> + test_cmp dir expect &&\n>> ls repo/deep >dir &&\n>> cat >expect <<-EOF &&\n>> a\n>> deeper1\n>> EOF\n>> + test_cmp dir expect &&\n>> ls repo/deep/deeper1 >dir &&\n>> cat >expect <<-EOF &&\n>> a\n>> @@ -245,4 +247,19 @@ test_expect_success 'cone mode: set with nested folders' '\n>> test_cmp repo/.git/info/sparse-checkout expect\n>> '\n>>\n>> +test_expect_success 'revert to old sparse-checkout on bad update' '\n>> + echo update >repo/deep/deeper2/a &&\n>> + cp repo/.git/info/sparse-checkout expect &&\n>> + test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n>> + test_i18ngrep \"Cannot update sparse checkout\" err &&\n>> + test_cmp repo/.git/info/sparse-checkout expect &&\n>> + ls repo/deep >dir &&\n>> + cat >expect <<-EOF &&\n>> + a\n>> + deeper1\n>> + deeper2\n>> + EOF\n>> + test_cmp dir expect\n>> +'\n>> +\n>> test_done\n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index edf0fb4673..f0fee5adf2 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -1508,7 +1508,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>> memset(&pl, 0, sizeof(pl));\n>> if (!core_apply_sparse_checkout || !o->update)\n>> o->skip_sparse_checkout = 1;\n>> - if (!o->skip_sparse_checkout) {\n>> + if (!o->skip_sparse_checkout && !o->pl) {\n>> char *sparse = git_pathdup(\"info/sparse-checkout\");\n>> pl.use_cone_patterns = core_sparse_checkout_cone;\n>> if (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n>> @@ -1681,7 +1681,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n>>\n>> done:\n>> trace_performance_leave(\"unpack_trees\");\n>> - clear_pattern_list(&pl);\n>> + if (!o->keep_pattern_list)\n>> + clear_pattern_list(&pl);\n>> return ret;\n>>\n>> return_failed:\n>> diff --git a/unpack-trees.h b/unpack-trees.h\n>> index f2eee0c7c5..ca94a421a5 100644\n>> --- a/unpack-trees.h\n>> +++ b/unpack-trees.h\n>> @@ -59,7 +59,8 @@ struct unpack_trees_options {\n>> quiet,\n>> exiting_early,\n>> show_all_errors,\n>> - dry_run;\n>> + dry_run,\n>> + keep_pattern_list;\n>> const char *prefix;\n>> int cache_bottom;\n>> struct dir_struct *dir;\n>> --\n> \n> The rest looks reasonable.\n> \n\n"},{"id":"384034","messageId":"0c8bcf25-3c32-8151-fa89-84170cb1aff4@gmail.com","threadId":"51691","inReplyTo":"CABPp-BGCNpZWoWk4coNEpA0zS2g0HXi5Kzb+QVusyLRt0fy=zw@mail.gmail.com","subject":"Re: [PATCH v3 16/17] sparse-checkout: write using lockfile","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-14T20:41:49Z","receivedAt":"2019-10-14T20:41:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/12/2019 6:59 PM, Elijah Newren wrote:\n> On Mon, Oct 7, 2019 at 1:08 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> If two 'git sparse-checkout set' subcommands are launched at the\n>> same time, the behavior can be unexpected as they compete to write\n>> the sparse-checkout file and update the working directory.\n>>\n>> Take a lockfile around the writes to the sparse-checkout file. In\n>> addition, acquire this lock around the working directory update\n>> to avoid two commands updating the working directory in different\n>> ways.\n> \n> Wow, there's something I never would have thought to check. Did you\n> have folks run into this, or is this just some defensive programming?\n> Either way, I'm impressed.\n\nThis is defensive programming thanks to Kevin Willford's careful\nreview [1].\n\n-Stolee\n\n[1] https://github.com/microsoft/git/pull/204#discussion_r330252848\n\n"},{"id":"384084","messageId":"65d26de1c2724750a51592d02466a4b9ecdea743.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:49Z","receivedAt":"2019-10-15T13:56:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nMake sure to use the `extensions.worktreeConfig` setting and write\nthe sparse checkout config to the worktree-specific config file.\nThis avoids confusing interactions with other worktrees.\n\nThe use of running another process for 'git read-tree' is sub-\noptimal. This will be removed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 11 ++++\n builtin/sparse-checkout.c | 79 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++\n 3 files changed, 130 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 46d3dc3cb1..d5fbbf17a0 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -28,6 +28,17 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n++\n+To avoid interfering with other worktrees, it first enables the\n+`extensions.worktreeConfig` setting and makes sure to set the\n+`core.sparseCheckout` setting in the worktree-specific config file.\n \n SPARSE CHECKOUT\n ----------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex eed9625a05..1d2327111a 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [list]\"),\n+\tN_(\"git sparse-checkout [init|list]\"),\n \tNULL\n };\n \n@@ -59,6 +59,81 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int update_working_directory(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+enum sparse_checkout_mode {\n+\tMODE_NO_PATTERNS = 0,\n+\tMODE_ALL_PATTERNS = 1,\n+};\n+\n+static int sc_set_config(enum sparse_checkout_mode mode)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n+\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n+\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n+\t\treturn 1;\n+\t}\n+\n+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n+\n+\tif (mode)\n+\t\targv_array_pushl(&argv, \"true\", NULL);\n+\telse\n+\t\targv_array_pushl(&argv, \"false\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\t\treturn 1;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\tfclose(fp);\n+\n+reset_dir:\n+\treturn update_working_directory();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -79,6 +154,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a9b04b1a88..c70085a759 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -47,4 +47,45 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384085","messageId":"63626e109767c268ee81b50abd21b95b1f1b5bfb.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:48Z","receivedAt":"2019-10-15T13:56:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na more restricted pattern set.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 87 +++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 50 +++++++++++++++\n 8 files changed, 228 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 89b3b79c1a..aebe7c0908 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -158,6 +158,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..46d3dc3cb1\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,87 @@\n+git-sparse-checkout(1)\n+=======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of directories\n+given by a list of prefixes.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of directories given by a list of prefixes.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+----------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it updates the skip-worktree bits in the index based\n+on this file. The files matching the patterns in the file will\n+appear in the working directory, and the rest will not.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are included, you can also specify what files are _not_ included,\n+using negative patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+checkout' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex de60c8e7aa..adefc229fe 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex 5cf5df69f7..2b25a80cde 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..eed9625a05\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,86 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout [list]\"),\n+\tNULL\n+};\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pl->nr; i++) {\n+\t\tstruct path_pattern *p = pl->patterns[i];\n+\n+\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", p->pattern);\n+\n+\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_patterns_to_file(stdout, &pl);\n+\tclear_pattern_list(&pl);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/git.c b/git.c\nindex ce6ab0ece2..7be7ad34bd 100644\n--- a/git.c\n+++ b/git.c\n@@ -572,6 +572,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..a9b04b1a88\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,50 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_line_count = 0 list &&\n+\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tgit -C repo sparse-checkout list >list &&\n+\tcat >expect <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\ttest_cmp expect list\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"384086","messageId":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:47Z","receivedAt":"2019-10-15T13:56:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"V4 UPDATE: Rebased on latest master to include ew/hashmap and\nds/include-exclude in the base.\n\nThis series makes the sparse-checkout feature more user-friendly. While\nthere, I also present a way to use a limited set of patterns to gain a\nsignificant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis series attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout set reads patterns from the arguments, or with\n --stdin reads patterns from stdin one per line, then writes them to the\n sparse-checkout file and refreshes the working directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\nUpdates in v2, relative to the RFC:\n\n * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n adding 'add' and/or 'remove' subcommands later.\n \n \n * 'set' reads from the arguments by default. '--stdin' option is available.\n \n \n * A new performance-oriented commit is added at the end.\n \n \n * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n pattern.\n \n \n * References to a \"bug\" (that was really a strange GVFS interaction in\n microsoft/git) around deleting outside the cone are removed.\n \n \n\nUpdates in v3:\n\n * The bad interaction with \"cone mode\" and .gitignore files is fixed. A\n test is added in the last patch.\n \n \n * Several patches are added that make the feature more robust. One\n sanitizes user input, another few add progress indicators, and another\n more prevent users from getting in bad states due to working directory\n changes or concurrent processes.\n \n \n * Updated several docs and commit messages according to feedback. Thanks,\n Elijah!\n \n \n\nUpdates in V4:\n\n * Updated hashmap API usage to respond to ew/hashmap\n \n \n * Responded to detailed review by Elijah. Thanks!\n \n \n * Marked the feature as experimental in git-sparse-checkout.txt the same\n way that git-switch.txt does.\n \n \n\nThings to leave for future patches:\n\n 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n worktree-specific file.\n \n \n 2. More robustness around detecting non-cone patterns with wildcards in the\n middle of the line.\n \n \n 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n super-valuable, as it only starts changing behavior when someone calls\n 'git sparse-checkout set', but may be interesting.\n \n \n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (16):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'set' subcommand\n sparse-checkout: add '--stdin' option to set subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and set in cone mode\n unpack-trees: hash less in cone mode\n unpack-trees: add progress to clear_ce_flags()\n read-tree: show progress by default\n sparse-checkout: sanitize for nested folders\n sparse-checkout: update working directory in-process\n sparse-checkout: write using lockfile\n sparse-checkout: cone mode should not interact with .gitignore\n\nJeff Hostetler (1):\n trace2: add region in clear_ce_flags\n\n .gitignore | 1 +\n Documentation/config/core.txt | 10 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 150 ++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/read-tree.c | 3 +-\n builtin/sparse-checkout.c | 482 ++++++++++++++++++++++++++\n cache.h | 6 +-\n config.c | 5 +\n dir.c | 207 ++++++++++-\n dir.h | 36 ++\n environment.c | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 279 +++++++++++++++\n unpack-trees.c | 110 ++++--\n unpack-trees.h | 3 +-\n 19 files changed, 1285 insertions(+), 48 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 108b97dc372828f0e72e56bbb40cae8e1e83ece6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n\nRange-diff vs v3:\n\n 1: 30a0db68cd ! 1: 63626e1097 sparse-checkout: create builtin with 'list' subcommand\n @@ -75,6 +75,8 @@\n +Initialize and modify the sparse-checkout configuration, which reduces\n +the checkout to a set of directories given by a list of prefixes.\n +\n ++THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n ++\n +\n +COMMANDS\n +--------\n @@ -96,7 +98,7 @@\n +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n +skip-worktree reference bitmap. When Git updates the working\n +directory, it updates the skip-worktree bits in the index based\n -+ont this file. The files matching the patterns in the file will\n ++on this file. The files matching the patterns in the file will\n +appear in the working directory, and the rest will not.\n +\n +## FULL PATTERN SET\n 2: 08bb6fb7f3 ! 2: 65d26de1c2 sparse-checkout: create 'init' subcommand\n @@ -32,10 +32,9 @@\n +\tby Git. Add patterns to the sparse-checkout file to\n +\trepopulate the working directory.\n ++\n -+The init subcommand also enables the 'extensions.worktreeConfig' setting\n -+and sets the `core.sparseCheckout` setting in the worktree-specific config\n -+file. This prevents the sparse-checkout feature from interfering with other\n -+worktrees.\n ++To avoid interfering with other worktrees, it first enables the\n ++`extensions.worktreeConfig` setting and makes sure to set the\n ++`core.sparseCheckout` setting in the worktree-specific config file.\n \n SPARSE CHECKOUT\n ----------------\n @@ -72,8 +71,8 @@\n +}\n +\n +enum sparse_checkout_mode {\n -+\tMODE_NONE = 0,\n -+\tMODE_FULL = 1,\n ++\tMODE_NO_PATTERNS = 0,\n ++\tMODE_ALL_PATTERNS = 1,\n +};\n +\n +static int sc_set_config(enum sparse_checkout_mode mode)\n @@ -107,7 +106,7 @@\n +\tFILE *fp;\n +\tint res;\n +\n -+\tif (sc_set_config(MODE_FULL))\n ++\tif (sc_set_config(MODE_ALL_PATTERNS))\n +\t\treturn 1;\n +\n +\tmemset(&pl, 0, sizeof(pl));\n 3: c8587a1fb0 ! 3: e59ed7128f clone: add --sparse mode\n @@ -22,8 +22,9 @@\n \n During the 'git sparse-checkout init' call, we must first look\n to see if HEAD is valid, since 'git clone' does not have a valid\n - HEAD. The first checkout will create the HEAD ref and update the\n - working directory correctly.\n + HEAD at the point where it initializes the sparse-checkout. The\n + following checkout within the clone command will create the HEAD\n + ref and update the working directory correctly.\n \n Signed-off-by: Derrick Stolee \n \n @@ -121,7 +122,7 @@\n \tint res;\n +\tstruct object_id oid;\n \n - \tif (sc_set_config(MODE_FULL))\n + \tif (sc_set_config(MODE_ALL_PATTERNS))\n \t\treturn 1;\n @@\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n 4: 6ce1d60b38 ! 4: 502b4b08f0 sparse-checkout: 'set' subcommand\n @@ -17,8 +17,8 @@\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n - file. This prevents the sparse-checkout feature from interfering with other\n - worktrees.\n + `extensions.worktreeConfig` setting and makes sure to set the\n + `core.sparseCheckout` setting in the worktree-specific config file.\n \n +'set'::\n +\tWrite a set of patterns to the sparse-checkout file, as given as\n 5: 0b1ed06bc8 = 5: 2852cf8e11 sparse-checkout: add '--stdin' option to set subcommand\n 6: 22b9bd21f4 ! 6: 55f95f290e sparse-checkout: create 'disable' subcommand\n @@ -24,7 +24,7 @@\n ----------------\n \n @@\n - ont this file. The files matching the patterns in the file will\n + on this file. The files matching the patterns in the file will\n appear in the working directory, and the rest will not.\n \n +To enable the sparse-checkout feature, run `git sparse-checkout init` to\n @@ -82,7 +82,7 @@\n +\tchar *sparse_filename;\n +\tFILE *fp;\n +\n -+\tif (sc_set_config(MODE_FULL))\n ++\tif (sc_set_config(MODE_ALL_PATTERNS))\n +\t\tdie(_(\"failed to change config\"));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n @@ -96,7 +96,7 @@\n +\tunlink(sparse_filename);\n +\tfree(sparse_filename);\n +\n -+\treturn sc_set_config(MODE_NONE);\n ++\treturn sc_set_config(MODE_NO_PATTERNS);\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n 7: 2c53ea13d0 = 7: 3d8f2f2007 trace2: add region in clear_ce_flags\n 8: a66ec1affc = 8: 03dc0ed716 sparse-checkout: add 'cone' mode\n 9: 431933bec6 ! 9: 28606a152c sparse-checkout: use hashmaps for cone patterns\n @@ -33,6 +33,8 @@\n While this example is contrived, it demonstrates how these\n patterns can slow the sparse-checkout feature.\n \n + Helped-by: Eric Wong \n + Helped-by: Johannes Schindelin \n Signed-off-by: Derrick Stolee \n \n diff --git a/dir.c b/dir.c\n @@ -43,10 +45,14 @@\n }\n \n +static int pl_hashmap_cmp(const void *unused_cmp_data,\n -+\t\t\t const void *a, const void *b, const void *key)\n ++\t\t\t const struct hashmap_entry *a,\n ++\t\t\t const struct hashmap_entry *b,\n ++\t\t\t const void *key)\n +{\n -+\tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n -+\tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\n ++\tconst struct pattern_entry *ee1 =\n ++\t\t\tcontainer_of(a, struct pattern_entry, ent);\n ++\tconst struct pattern_entry *ee2 =\n ++\t\t\tcontainer_of(b, struct pattern_entry, ent);\n +\n +\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n +\t\t\t ? ee1->patternlen\n @@ -91,10 +97,11 @@\n +\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n +\t\ttranslated->pattern = truncated;\n +\t\ttranslated->patternlen = given->patternlen - 2;\n -+\t\thashmap_entry_init(translated,\n ++\t\thashmap_entry_init(&translated->ent,\n +\t\t\t\t memhash(translated->pattern, translated->patternlen));\n +\n -+\t\tif (!hashmap_get(&pl->recursive_hashmap, translated, NULL)) {\n ++\t\tif (!hashmap_get_entry(&pl->recursive_hashmap,\n ++\t\t\t\t translated, ent, NULL)) {\n +\t\t\t/* We did not see the \"parent\" included */\n +\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n +\t\t\t\tgiven->pattern);\n @@ -103,8 +110,8 @@\n +\t\t\tgoto clear_hashmaps;\n +\t\t}\n +\n -+\t\thashmap_add(&pl->parent_hashmap, translated);\n -+\t\thashmap_remove(&pl->recursive_hashmap, translated, &data);\n ++\t\thashmap_add(&pl->parent_hashmap, &translated->ent);\n ++\t\thashmap_remove(&pl->recursive_hashmap, &translated->ent, &data);\n +\t\tfree(data);\n +\t\treturn;\n +\t}\n @@ -119,16 +126,16 @@\n +\n +\ttranslated->pattern = xstrdup(given->pattern);\n +\ttranslated->patternlen = given->patternlen;\n -+\thashmap_entry_init(translated,\n ++\thashmap_entry_init(&translated->ent,\n +\t\t\t memhash(translated->pattern, translated->patternlen));\n +\n -+\thashmap_add(&pl->recursive_hashmap, translated);\n ++\thashmap_add(&pl->recursive_hashmap, &translated->ent);\n +\n -+\tif (hashmap_get(&pl->parent_hashmap, translated, NULL)) {\n ++\tif (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) {\n +\t\t/* we already included this at the parent level */\n +\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n +\t\t\tgiven->pattern);\n -+\t\thashmap_remove(&pl->parent_hashmap, translated, &data);\n ++\t\thashmap_remove(&pl->parent_hashmap, &translated->ent, &data);\n +\t\tfree(data);\n +\t\tfree(translated);\n +\t}\n @@ -137,8 +144,8 @@\n +\n +clear_hashmaps:\n +\twarning(_(\"disabling cone pattern matching\"));\n -+\thashmap_free(&pl->parent_hashmap, 1);\n -+\thashmap_free(&pl->recursive_hashmap, 1);\n ++\thashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent);\n ++\thashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent);\n +\tpl->use_cone_patterns = 0;\n +}\n +\n @@ -150,8 +157,8 @@\n +\t/* Check straight mapping */\n +\tp.pattern = pattern->buf;\n +\tp.patternlen = pattern->len;\n -+\thashmap_entry_init(&p, memhash(p.pattern, p.patternlen));\n -+\treturn !!hashmap_get(map, &p, NULL);\n ++\thashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen));\n ++\treturn !!hashmap_get_entry(map, &p, ent, NULL);\n +}\n +\n +int hashmap_contains_parent(struct hashmap *map,\n 10: 69bd707e96 ! 10: 2ea3d8819b sparse-checkout: init and set in cone mode\n @@ -18,6 +18,8 @@\n we want to avoid the warning that the patterns do not match\n the cone-mode patterns.\n \n + Helped-by: Eric Wong \n + Helped-by: Johannes Schindelin \n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n @@ -33,9 +35,9 @@\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n @@\n enum sparse_checkout_mode {\n - \tMODE_NONE = 0,\n - \tMODE_FULL = 1,\n -+\tMODE_CONE = 2,\n + \tMODE_NO_PATTERNS = 0,\n + \tMODE_ALL_PATTERNS = 1,\n ++\tMODE_CONE_PATTERNS = 2,\n };\n \n static int sc_set_config(enum sparse_checkout_mode mode)\n @@ -52,7 +54,7 @@\n +\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n +\t\t\t \"core.sparseCheckoutCone\", NULL);\n +\n -+\tif (mode == MODE_CONE)\n ++\tif (mode == MODE_CONE_PATTERNS)\n +\t\targv_array_push(&cone_argv, \"true\");\n +\telse\n +\t\targv_array_push(&cone_argv, \"false\");\n @@ -83,7 +85,7 @@\n \tstruct object_id oid;\n +\tint mode;\n \n --\tif (sc_set_config(MODE_FULL))\n +-\tif (sc_set_config(MODE_ALL_PATTERNS))\n +\tstatic struct option builtin_sparse_checkout_init_options[] = {\n +\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n +\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n @@ -94,7 +96,7 @@\n +\t\t\t builtin_sparse_checkout_init_options,\n +\t\t\t builtin_sparse_checkout_init_usage, 0);\n +\n -+\tmode = init_opts.cone_mode ? MODE_CONE : MODE_FULL;\n ++\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n +\n +\tif (sc_set_config(mode))\n \t\treturn 1;\n @@ -106,12 +108,12 @@\n \n +static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n +{\n -+\tstruct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n ++\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n +\te->patternlen = path->len;\n +\te->pattern = strbuf_detach(path, NULL);\n -+\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n ++\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n +\n -+\thashmap_add(&pl->recursive_hashmap, e);\n ++\thashmap_add(&pl->recursive_hashmap, &e->ent);\n +\n +\twhile (e->patternlen) {\n +\t\tchar *slash = strrchr(e->pattern, '/');\n @@ -125,23 +127,22 @@\n +\t\te = xmalloc(sizeof(struct pattern_entry));\n +\t\te->patternlen = newlen;\n +\t\te->pattern = xstrndup(oldpattern, newlen);\n -+\t\thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n ++\t\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n +\n -+\t\tif (!hashmap_get(&pl->parent_hashmap, e, NULL))\n -+\t\t\thashmap_add(&pl->parent_hashmap, e);\n ++\t\tif (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))\n ++\t\t\thashmap_add(&pl->parent_hashmap, &e->ent);\n +\t}\n +}\n +\n +static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n +{\n +\tint i;\n -+\tstruct pattern_entry *entry;\n ++\tstruct pattern_entry *pe;\n +\tstruct hashmap_iter iter;\n +\tstruct string_list sl = STRING_LIST_INIT_DUP;\n +\n -+\thashmap_iter_init(&pl->parent_hashmap, &iter);\n -+\twhile ((entry = hashmap_iter_next(&iter)))\n -+\t\tstring_list_insert(&sl, entry->pattern);\n ++\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n ++\t\tstring_list_insert(&sl, pe->pattern);\n +\n +\tstring_list_sort(&sl);\n +\tstring_list_remove_duplicates(&sl, 0);\n @@ -157,9 +158,8 @@\n +\n +\tstring_list_clear(&sl, 0);\n +\n -+\thashmap_iter_init(&pl->recursive_hashmap, &iter);\n -+\twhile ((entry = hashmap_iter_next(&iter)))\n -+\t\tstring_list_insert(&sl, entry->pattern);\n ++\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n ++\t\tstring_list_insert(&sl, pe->pattern);\n +\n +\tstring_list_sort(&sl);\n +\tstring_list_remove_duplicates(&sl, 0);\n @@ -260,12 +260,16 @@\n }\n \n -static int pl_hashmap_cmp(const void *unused_cmp_data,\n --\t\t\t const void *a, const void *b, const void *key)\n +-\t\t\t const struct hashmap_entry *a,\n +-\t\t\t const struct hashmap_entry *b,\n +-\t\t\t const void *key)\n +int pl_hashmap_cmp(const void *unused_cmp_data,\n -+\t\t const void *a, const void *b, const void *key)\n ++\t\t const struct hashmap_entry *a,\n ++\t\t const struct hashmap_entry *b,\n ++\t\t const void *key)\n {\n - \tconst struct pattern_entry *ee1 = (const struct pattern_entry *)a;\n - \tconst struct pattern_entry *ee2 = (const struct pattern_entry *)b;\n + \tconst struct pattern_entry *ee1 =\n + \t\t\tcontainer_of(a, struct pattern_entry, ent);\n \n diff --git a/dir.h b/dir.h\n --- a/dir.h\n @@ -275,7 +279,9 @@\n \t\tconst char *name, int *dtype);\n \n +int pl_hashmap_cmp(const void *unused_cmp_data,\n -+\t\t const void *a, const void *b, const void *key);\n ++\t\t const struct hashmap_entry *a,\n ++\t\t const struct hashmap_entry *b,\n ++\t\t const void *key);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\n @@ -301,11 +307,13 @@\n +\t\ta\n +\t\tdeep\n +\tEOF\n ++\ttest_cmp expect dir &&\n +\tls repo/deep >dir &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n +\t\tdeeper1\n +\tEOF\n ++\ttest_cmp expect dir &&\n +\tls repo/deep/deeper1 >dir &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n 11: e06349fcec = 11: fefd1e1744 unpack-trees: hash less in cone mode\n 12: 3ef32084f5 = 12: 034fbd31bd unpack-trees: add progress to clear_ce_flags()\n 13: 3a677f32b6 = 13: 27aa9d22f0 read-tree: show progress by default\n 14: 56444a5498 ! 14: 44c3078029 sparse-checkout: sanitize for nested folders\n @@ -22,43 +22,44 @@\n hashmap_contains_parent() method. It takes a strbuf buffer to\n avoid reallocating a buffer when calling in a tight loop.\n \n + Helped-by: Eric Wong \n + Helped-by: Johannes Schindelin \n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n @@\n - \tstruct pattern_entry *entry;\n + \tstruct pattern_entry *pe;\n \tstruct hashmap_iter iter;\n \tstruct string_list sl = STRING_LIST_INIT_DUP;\n +\tstruct strbuf parent_pattern = STRBUF_INIT;\n \n - \thashmap_iter_init(&pl->parent_hashmap, &iter);\n --\twhile ((entry = hashmap_iter_next(&iter)))\n --\t\tstring_list_insert(&sl, entry->pattern);\n -+\twhile ((entry = hashmap_iter_next(&iter))) {\n -+\t\tif (hashmap_get(&pl->recursive_hashmap, entry, NULL))\n +-\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n +-\t\tstring_list_insert(&sl, pe->pattern);\n ++\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {\n ++\t\tif (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))\n +\t\t\tcontinue;\n +\n +\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n -+\t\t\t\t\t entry->pattern,\n ++\t\t\t\t\t pe->pattern,\n +\t\t\t\t\t &parent_pattern))\n -+\t\t\tstring_list_insert(&sl, entry->pattern);\n ++\t\t\tstring_list_insert(&sl, pe->pattern);\n +\t}\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\n @@\n + \n \tstring_list_clear(&sl, 0);\n \n - \thashmap_iter_init(&pl->recursive_hashmap, &iter);\n --\twhile ((entry = hashmap_iter_next(&iter)))\n --\t\tstring_list_insert(&sl, entry->pattern);\n -+\twhile ((entry = hashmap_iter_next(&iter))) {\n +-\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n +-\t\tstring_list_insert(&sl, pe->pattern);\n ++\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {\n +\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n -+\t\t\t\t\t entry->pattern,\n ++\t\t\t\t\t pe->pattern,\n +\t\t\t\t\t &parent_pattern))\n -+\t\t\tstring_list_insert(&sl, entry->pattern);\n ++\t\t\tstring_list_insert(&sl, pe->pattern);\n +\t}\n +\n +\tstrbuf_release(&parent_pattern);\n 15: a6f17e9a77 ! 15: 9ccec3ca9a sparse-checkout: update working directory in-process\n @@ -10,7 +10,7 @@\n \n Remove this extra process call by creating a direct call to\n unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n - adition, provide an in-memory list of patterns so we can avoid\n + addition, provide an in-memory list of patterns so we can avoid\n reading from the sparse-checkout file. This allows us to test a\n proposed change to the file before writing to it.\n \n @@ -109,12 +109,12 @@\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n --\tmode = init_opts.cone_mode ? MODE_CONE : MODE_FULL;\n +-\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n +\tif (init_opts.cone_mode) {\n -+\t\tmode = MODE_CONE;\n ++\t\tmode = MODE_CONE_PATTERNS;\n +\t\tcore_sparse_checkout_cone = 1;\n +\t} else\n -+\t\tmode = MODE_FULL;\n ++\t\tmode = MODE_ALL_PATTERNS;\n \n \tif (sc_set_config(mode))\n \t\treturn 1;\n @@ -128,20 +128,12 @@\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n - {\n --\tstruct pattern_entry *e = xmalloc(sizeof(struct pattern_entry));\n -+\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n -+\n - \te->patternlen = path->len;\n - \te->pattern = strbuf_detach(path, NULL);\n - \thashmap_entry_init(e, memhash(e->pattern, e->patternlen));\n @@\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n --\n +\tint result;\n -+\t\n + \n \tif (!core_apply_sparse_checkout) {\n \t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n \t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n @@ -193,20 +185,6 @@\n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n -@@\n - \t\ta\n - \t\tdeep\n - \tEOF\n -+\ttest_cmp dir expect &&\n - \tls repo/deep >dir &&\n - \tcat >expect <<-EOF &&\n - \t\ta\n - \t\tdeeper1\n - \tEOF\n -+\ttest_cmp dir expect &&\n - \tls repo/deep/deeper1 >dir &&\n - \tcat >expect <<-EOF &&\n - \t\ta\n @@\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n 16: 8927494b8c ! 16: d0421ef7b2 sparse-checkout: write using lockfile\n @@ -23,7 +23,7 @@\n +\tint fd;\n +\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n - \t\n + \n \tif (!core_apply_sparse_checkout) {\n @@\n \n 17: 7f377c1407 = 17: ed1f148763 sparse-checkout: cone mode should not interact with .gitignore\n\n-- \ngitgitgadget\n"},{"id":"384087","messageId":"e59ed7128fe22c7662c57e36261cad68c0515308.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 03/17] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:50Z","receivedAt":"2019-10-15T13:56:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nDuring the 'git sparse-checkout init' call, we must first look\nto see if HEAD is valid, since 'git clone' does not have a valid\nHEAD at the point where it initializes the sparse-checkout. The\nfollowing checkout within the clone command will create the HEAD\nref and update the working directory correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n builtin/sparse-checkout.c | 6 ++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 4 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 34011c2940..0fe91d2f04 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex c46ee29f0a..4348d962c9 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -59,6 +59,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 1d2327111a..4198995d46 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tstruct object_id oid;\n \n \tif (sc_set_config(MODE_ALL_PATTERNS))\n \t\treturn 1;\n@@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \tfclose(fp);\n \n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n reset_dir:\n \treturn update_working_directory();\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex c70085a759..d4c145a3af 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -88,4 +88,17 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384089","messageId":"502b4b08f0a801dd923c5a4e2a84808d9ae6e3c6.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 04/17] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:51Z","receivedAt":"2019-10-15T13:56:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nas arguments and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nThe 'set' subcommand will replace the entire contents of the\nsparse-checkout file. The write_patterns_and_update() method is\nextracted from cmd_sparse_checkout() to make it easier to implement\n'add' and/or 'remove' subcommands in the future.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 5 ++++\n builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 19 +++++++++++++++\n 3 files changed, 58 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex d5fbbf17a0..163f6c8db0 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -40,6 +40,11 @@ To avoid interfering with other worktrees, it first enables the\n `extensions.worktreeConfig` setting and makes sure to set the\n `core.sparseCheckout` setting in the worktree-specific config file.\n \n+'set'::\n+\tWrite a set of patterns to the sparse-checkout file, as given as\n+\ta list of arguments following the 'set' subcommand. Update the\n+\tworking directory to match the new patterns.\n+\n SPARSE CHECKOUT\n ----------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 4198995d46..2103cbe00c 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list]\"),\n+\tN_(\"git sparse-checkout [init|list|set] \"),\n \tNULL\n };\n \n@@ -140,6 +140,37 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_patterns_to_file(fp, pl);\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\treturn update_working_directory();\n+}\n+\n+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *empty_base = \"\";\n+\tint i;\n+\tstruct pattern_list pl;\n+\tint result;\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tfor (i = 1; i < argc; i++)\n+\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\n+\tresult = write_patterns_and_update(&pl);\n+\n+\tclear_pattern_list(&pl);\n+\treturn result;\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -162,6 +193,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"set\"))\n+\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex d4c145a3af..19e8673c6b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -101,4 +101,23 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using builtin' '\n+\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384088","messageId":"28606a152c79c4e399c5a9c77038888eda34341d.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 09/17] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:56Z","receivedAt":"2019-10-15T13:56:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/\n\t!/$folder/*/\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n\tcore.sparseCheckout=false: 0.21 s (0.00 s)\n\t core.sparseCheckout=true: 3.75 s (3.50 s)\n\t core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n dir.c | 207 +++++++++++++++++++++++++++--\n dir.h | 31 +++++\n t/t1091-sparse-checkout-builtin.sh | 11 +-\n unpack-trees.c | 1 +\n 4 files changed, 241 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..dfabf9982f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const struct hashmap_entry *a,\n+\t\t\t const struct hashmap_entry *b,\n+\t\t\t const void *key)\n+{\n+\tconst struct pattern_entry *ee1 =\n+\t\t\tcontainer_of(a, struct pattern_entry, ent);\n+\tconst struct pattern_entry *ee2 =\n+\t\t\tcontainer_of(b, struct pattern_entry, ent);\n+\n+\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n+\t\t\t ? ee1->patternlen\n+\t\t\t : ee2->patternlen;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n+}\n+\n+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n+{\n+\tstruct pattern_entry *translated;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!pl->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE &&\n+\t given->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 0;\n+\t\treturn;\n+\t}\n+\n+\tif (!given->flags && !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 1;\n+\t\treturn;\n+\t}\n+\n+\tif (given->patternlen > 2 &&\n+\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n+\t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tpl->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(given->pattern);\n+\t\ttruncated[given->patternlen - 2] = 0;\n+\n+\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\t\ttranslated->pattern = truncated;\n+\t\ttranslated->patternlen = given->patternlen - 2;\n+\t\thashmap_entry_init(&translated->ent,\n+\t\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->recursive_hashmap,\n+\t\t\t\t translated, ent, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\t\tgiven->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tfree(translated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&pl->parent_hashmap, &translated->ent);\n+\t\thashmap_remove(&pl->recursive_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE) {\n+\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\tgiven->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n+\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\n+\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->patternlen = given->patternlen;\n+\thashmap_entry_init(&translated->ent,\n+\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, &translated->ent);\n+\n+\tif (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) {\n+\t\t/* we already included this at the parent level */\n+\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\tgiven->pattern);\n+\t\thashmap_remove(&pl->parent_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\tfree(translated);\n+\t}\n+\n+\treturn;\n+\n+clear_hashmaps:\n+\twarning(_(\"disabling cone pattern matching\"));\n+\thashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent);\n+\thashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent);\n+\tpl->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct pattern_entry p;\n+\n+\t/* Check straight mapping */\n+\tp.pattern = pattern->buf;\n+\tp.patternlen = pattern->len;\n+\thashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen));\n+\treturn !!hashmap_get_entry(map, &p, ent, NULL);\n+}\n+\n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer)\n+{\n+\tchar *slash_pos;\n+\n+\tstrbuf_setlen(buffer, 0);\n+\n+\tif (path[0] != '/')\n+\t\tstrbuf_addch(buffer, '/');\n+\n+\tstrbuf_addstr(buffer, path);\n+\n+\tslash_pos = strrchr(buffer->buf, '/');\n+\n+\twhile (slash_pos > buffer->buf) {\n+\t\tstrbuf_setlen(buffer, slash_pos - buffer->buf);\n+\n+\t\tif (hashmap_contains_path(map, buffer))\n+\t\t\treturn 1;\n+\n+\t\tslash_pos = strrchr(buffer->buf, '/');\n+\t}\n+\n+\treturn 0;\n+}\n+\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n {\n@@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base,\n \tALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);\n \tpl->patterns[pl->nr++] = pattern;\n \tpattern->pl = pl;\n+\n+\tadd_pattern_to_hashsets(pl, pattern);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n \tpl->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list(\n \t\t\t\tstruct index_state *istate)\n {\n \tstruct path_pattern *pattern;\n-\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, pl, istate);\n-\tif (pattern) {\n-\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n-\t\t\treturn NOT_MATCHED;\n-\t\telse\n-\t\t\treturn MATCHED;\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = NOT_MATCHED;\n+\tconst char *slash_pos;\n+\n+\tif (!pl->use_cone_patterns) {\n+\t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\tdtype, pl, istate);\n+\t\tif (pattern) {\n+\t\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\t\treturn NOT_MATCHED;\n+\t\t\telse\n+\t\t\t\treturn MATCHED;\n+\t\t}\n+\n+\t\treturn UNDECIDED;\n+\t}\n+\n+\tif (pl->full_cone)\n+\t\treturn MATCHED;\n+\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\n+\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n \t}\n \n-\treturn UNDECIDED;\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tif (hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t pathname,\n+\t\t\t\t &parent_pathname))\n+\t\tresult = MATCHED;\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct path_pattern *last_matching_pattern_from_lists(\ndiff --git a/dir.h b/dir.h\nindex 2fbdef014f..f8edbca72b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct path_pattern {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct pattern_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,26 @@ struct pattern_list {\n \tconst char *src;\n \n \tstruct path_pattern **patterns;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\tunsigned full_cone;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\n@@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer);\n struct pattern_list *add_pattern_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src);\n int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 36fda5907b..b0d5aeb33a 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -150,7 +150,8 @@ test_expect_success 'set sparse-checkout using --stdin' '\n test_expect_success 'cone mode: match patterns' '\n \tgit -C repo config --worktree core.sparseCheckoutCone true &&\n \trm -rf repo/a repo/folder1 repo/folder2 &&\n-\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n \tls repo >dir &&\n \tcat >expect <<-EOF &&\n@@ -161,6 +162,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info/ &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 01a05ff66d..a90d71845d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n \t\t\to->skip_sparse_checkout = 1;\n \t\telse\n-- \ngitgitgadget\n\n"},{"id":"384090","messageId":"03dc0ed716d67f1f0bbedce90dc94dcfac8420c5.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 08/17] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:55Z","receivedAt":"2019-10-15T13:56:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new Boolean config option, core.sparseCheckoutCone, to\nindicate that we expect the sparse-checkout file to contain a\nmore limited set of patterns. This is a separate config setting\nfrom core.sparseCheckout to avoid breaking older clients by\nintroducing a tri-state option.\n\nThe config option does nothing right now, but will be expanded\nupon in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 10 ++++--\n Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n cache.h | 4 ++-\n config.c | 5 +++\n environment.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n 6 files changed, 81 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 852d2ba37a..bdbbee58b9 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -593,8 +593,14 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\n+\tfor more information.\n+\n+core.sparseCheckoutCone::\n+\tEnables the \"cone mode\" of the sparse checkout feature. When the\n+\tsparse-checkout file contains a limited set of patterns, then this\n+\tmode provides significant performance advantages. See\n+\tlinkgit:git-sparse-checkout[1] for more information.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b0c141b582..bb881aaa8e 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -89,6 +89,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+## CONE PATTERN SET\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+```\n+/*\n+!/*/\n+```\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+```\n+/*\n+!/*/\n+/A/\n+!/A/*/\n+/A/B/\n+!/A/B/*/\n+/A/B/C/\n+```\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/cache.h b/cache.h\nindex 04cabaac11..4980ee198e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -918,12 +918,14 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex e7052b3977..d75f88ca0c 100644\n--- a/config.c\n+++ b/config.c\n@@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n+\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex efa072680a..2a1a866659 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 52d24c66ba..36fda5907b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -147,6 +147,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"384091","messageId":"55f95f290ef77fa402f3e73f9c3045177bd70e84.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 06/17] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:53Z","receivedAt":"2019-10-15T13:56:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n 3 files changed, 52 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 163f6c8db0..b0c141b582 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -45,6 +45,10 @@ To avoid interfering with other worktrees, it first enables the\n \ta list of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ----------------\n \n@@ -62,6 +66,14 @@ directory, it updates the skip-worktree bits in the index based\n on this file. The files matching the patterns in the file will\n appear in the working directory, and the rest will not.\n \n+To enable the sparse-checkout feature, run `git sparse-checkout init` to\n+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`\n+config setting. Then, run `git sparse-checkout set` to modify the patterns in\n+the sparse-checkout file.\n+\n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -76,21 +88,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-checkout' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b747b78d34..78a80ce119 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout [init|list|set] \"),\n+\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n \tNULL\n };\n \n@@ -207,6 +207,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \treturn result;\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (update_working_directory())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn sc_set_config(MODE_NO_PATTERNS);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -231,6 +253,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"set\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2a0137fde3..52d24c66ba 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -147,4 +147,19 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384092","messageId":"2852cf8e11c73b1537cbfec969d1026a389fedc6.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:52Z","receivedAt":"2019-10-15T13:56:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nand places them in the sparse-checkout file. Then, it updates the\nworking directory to match those patterns. For a large list of\npatterns, the command-line call can get very cumbersome.\n\nAdd a '--stdin' option to instead read patterns over standard in.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 40 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 27 ++++++++++++++++++++\n 2 files changed, 65 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 2103cbe00c..b747b78d34 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -145,6 +145,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \tchar *sparse_filename;\n \tFILE *fp;\n \n+\tif (!core_apply_sparse_checkout) {\n+\t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n+\t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n+\t}\n+\n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n \twrite_patterns_to_file(fp, pl);\n@@ -154,16 +159,47 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static char const * const builtin_sparse_checkout_set_usage[] = {\n+\tN_(\"git sparse-checkout set [--stdin|]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_set_opts {\n+\tint use_stdin;\n+} set_opts;\n+\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tstatic const char *empty_base = \"\";\n \tint i;\n \tstruct pattern_list pl;\n \tint result;\n+\n+\tstatic struct option builtin_sparse_checkout_set_options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n+\t\t\t N_(\"read patterns from standard in\")),\n+\t\tOPT_END(),\n+\t};\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n-\tfor (i = 1; i < argc; i++)\n-\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_set_options,\n+\t\t\t builtin_sparse_checkout_set_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (set_opts.use_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tsize_t len;\n+\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t}\n \n \tresult = write_patterns_and_update(&pl);\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 19e8673c6b..2a0137fde3 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -101,6 +101,13 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'warn if core.sparseCheckout is disabled' '\n+\ttest_when_finished git -C repo config --worktree core.sparseCheckout true &&\n+\tgit -C repo config --worktree core.sparseCheckout false &&\n+\tgit -C repo sparse-checkout set folder1 2>err &&\n+\ttest_i18ngrep \"core.sparseCheckout is disabled\" err\n+'\n+\n test_expect_success 'set sparse-checkout using builtin' '\n \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n \tcat >expect <<-EOF &&\n@@ -120,4 +127,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using --stdin' '\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/folder1/\n+\t\t/folder2/\n+\tEOF\n+\tgit -C repo sparse-checkout set --stdin actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384093","messageId":"3d8f2f20077767da7f4198daaa44c6721fd01db2.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 07/17] trace2: add region in clear_ce_flags","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:54Z","receivedAt":"2019-10-15T13:56:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nWhen Git updates the working directory with the sparse-checkout\nfeature enabled, the unpack_trees() method calls clear_ce_flags()\nto update the skip-wortree bits on the cache entries. This\ncheck can be expensive, depending on the patterns used.\n\nAdd trace2 regions around the method, including some flag\ninformation, so we can get granular performance data during\nexperiments. This data will be used to measure improvements\nto the pattern-matching algorithms for sparse-checkout.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 33ea7810d8..01a05ff66d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tpl, 0);\n+\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"384094","messageId":"fefd1e17447dae54d98db6412ed3484be6a2292e.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 11/17] unpack-trees: hash less in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:58Z","receivedAt":"2019-10-15T13:56:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature in \"cone mode\" can use the fact that\nthe recursive patterns are \"connected\" to the root via parent\npatterns to decide if a directory is entirely contained in the\nsparse-checkout or entirely removed.\n\nIn these cases, we can skip hashing the paths within those\ndirectories and simply set the skipworktree bit to the correct\nvalue.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 4 ++--\n dir.h | 1 +\n unpack-trees.c | 38 +++++++++++++++++++++++---------------\n 3 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 35c1ca9e24..2ef92a50a0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list(\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n@@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tif (hashmap_contains_parent(&pl->recursive_hashmap,\n \t\t\t\t pathname,\n \t\t\t\t &parent_pathname))\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \n done:\n \tstrbuf_release(&parent_pathname);\ndiff --git a/dir.h b/dir.h\nindex 8e232085cd..77a43dbf89 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -264,6 +264,7 @@ enum pattern_match_result {\n \tUNDECIDED = -1,\n \tNOT_MATCHED = 0,\n \tMATCHED = 1,\n+\tMATCHED_RECURSIVE = 2,\n };\n \n /*\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a90d71845d..c0dca20865 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n \tint rc;\n-\tenum pattern_match_result ret;\n-\tret = path_matches_pattern_list(prefix->buf, prefix->len,\n-\t\t\t\t\tbasename, &dtype, pl, istate);\n+\tenum pattern_match_result ret, orig_ret;\n+\torig_ret = path_matches_pattern_list(prefix->buf, prefix->len,\n+\t\t\t\t\t basename, &dtype, pl, istate);\n \n \tstrbuf_addch(prefix, '/');\n \n \t/* If undecided, use matching result of parent dir in defval */\n-\tif (ret == UNDECIDED)\n+\tif (orig_ret == UNDECIDED)\n \t\tret = default_match;\n+\telse\n+\t\tret = orig_ret;\n \n \tfor (cache_end = cache; cache_end != cache + nr; cache_end++) {\n \t\tstruct cache_entry *ce = *cache_end;\n@@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\tbreak;\n \t}\n \n-\t/*\n-\t * TODO: check pl, if there are no patterns that may conflict\n-\t * with ret (iow, we know in advance the incl/excl\n-\t * decision for the entire directory), clear flag here without\n-\t * calling clear_ce_flags_1(). That function will call\n-\t * the expensive path_matches_pattern_list() on every entry.\n-\t */\n-\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t prefix,\n-\t\t\t select_mask, clear_mask,\n-\t\t\t pl, ret);\n+\tif (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {\n+\t\tstruct cache_entry **ce = cache;\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\n+\t\twhile (ce < cache_end) {\n+\t\t\t(*ce)->ce_flags &= ~clear_mask;\n+\t\t\tce++;\n+\t\t}\n+\t} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\t} else {\n+\t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t prefix,\n+\t\t\t\t select_mask, clear_mask,\n+\t\t\t\t pl, ret);\n+\t}\n+\n \tstrbuf_setlen(prefix, prefix->len - 1);\n \treturn rc;\n }\n-- \ngitgitgadget\n\n"},{"id":"384095","messageId":"ed1f148763437925248684b37a0c8b70f837d40b.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 17/17] sparse-checkout: cone mode should not interact with .gitignore","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:56:04Z","receivedAt":"2019-10-15T13:56:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the development of the sparse-checkout \"cone mode\" feature,\nan incorrect placement of the initializer for \"use_cone_patterns = 1\"\ncaused warnings to show up when a .gitignore file was present with\nnon-cone-mode patterns. This was fixed in the original commit\nintroducing the cone mode, but now we should add a test to avoid\nhitting this problem again in the future.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 48493e1012..14141b8f54 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -269,4 +269,11 @@ test_expect_success 'fail when lock is taken' '\n \ttest_i18ngrep \"File exists\" err\n '\n \n+test_expect_success '.gitignore should not warn about cone mode' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\techo \"**/bin/*\" >repo/.gitignore &&\n+\tgit -C repo reset --hard 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"384096","messageId":"2ea3d8819b35d8c5f3bce93c585614baa82ac57b.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 10/17] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:57Z","receivedAt":"2019-10-15T13:56:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout [init|set]'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckoutCone=true'.\n\nWhen running 'git sparse-checkout set' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nWhen testing 'git sparse-checkout set' in cone mode, check the\nerror stream to ensure we do not see any errors. Specifically,\nwe want to avoid the warning that the patterns do not match\nthe cone-mode patterns.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 159 +++++++++++++++++++++++++++--\n dir.c | 8 +-\n dir.h | 4 +\n t/t1091-sparse-checkout-builtin.sh | 51 +++++++++\n 4 files changed, 208 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 78a80ce119..95d9a90b84 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,6 +6,7 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n@@ -77,11 +78,13 @@ static int update_working_directory(void)\n enum sparse_checkout_mode {\n \tMODE_NO_PATTERNS = 0,\n \tMODE_ALL_PATTERNS = 1,\n+\tMODE_CONE_PATTERNS = 2,\n };\n \n static int sc_set_config(enum sparse_checkout_mode mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n \n \tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n \t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n@@ -100,9 +103,31 @@ static int sc_set_config(enum sparse_checkout_mode mode)\n \t\treturn 1;\n \t}\n \n+\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n+\t\t\t \"core.sparseCheckoutCone\", NULL);\n+\n+\tif (mode == MODE_CONE_PATTERNS)\n+\t\targv_array_push(&cone_argv, \"true\");\n+\telse\n+\t\targv_array_push(&cone_argv, \"false\");\n+\n+\tif (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckoutCone\"));\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\n \n+static char const * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_init_opts {\n+\tint cone_mode;\n+} init_opts;\n+\n static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -110,8 +135,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n+\tint mode;\n \n-\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\n+\tif (sc_set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -140,6 +178,70 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n+{\n+\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, &e->ent);\n+\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (slash == e->pattern)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct pattern_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))\n+\t\t\thashmap_add(&pl->parent_hashmap, &e->ent);\n+\t}\n+}\n+\n+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\tstruct pattern_entry *pe;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (strlen(pattern))\n+\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t}\n+}\n+\n static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n@@ -152,13 +254,33 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n-\twrite_patterns_to_file(fp, pl);\n+\n+\tif (core_sparse_checkout_cone)\n+\t\twrite_cone_to_file(fp, pl);\n+\telse\n+\t\twrite_patterns_to_file(fp, pl);\n+\n \tfclose(fp);\n \tfree(sparse_filename);\n \n \treturn update_working_directory();\n }\n \n+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n+{\n+\tstrbuf_trim(line);\n+\n+\tstrbuf_trim_trailing_dir_sep(line);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tif (line->buf[0] != '/')\n+\t\tstrbuf_insert(line, 0, \"/\", 1);\n+\n+\tinsert_recursive_pattern(pl, line);\n+}\n+\n static char const * const builtin_sparse_checkout_set_usage[] = {\n \tN_(\"git sparse-checkout set [--stdin|]\"),\n \tNULL\n@@ -188,17 +310,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n-\tif (set_opts.use_stdin) {\n+\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (!strbuf_getline(&line, stdin)) {\n-\t\t\tsize_t len;\n-\t\t\tchar *buf = strbuf_detach(&line, &len);\n-\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n+\t\tif (set_opts.use_stdin) {\n+\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++) {\n+\t\t\t\tstrbuf_setlen(&line, 0);\n+\t\t\t\tstrbuf_addstr(&line, argv[i]);\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n \t\t}\n \t} else {\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\tif (set_opts.use_stdin) {\n+\t\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\t}\n \t}\n \n \tresult = write_patterns_and_update(&pl);\ndiff --git a/dir.c b/dir.c\nindex dfabf9982f..35c1ca9e24 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n-static int pl_hashmap_cmp(const void *unused_cmp_data,\n-\t\t\t const struct hashmap_entry *a,\n-\t\t\t const struct hashmap_entry *b,\n-\t\t\t const void *key)\n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key)\n {\n \tconst struct pattern_entry *ee1 =\n \t\t\tcontainer_of(a, struct pattern_entry, ent);\ndiff --git a/dir.h b/dir.h\nindex f8edbca72b..8e232085cd 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b0d5aeb33a..3a4c0e6f5d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -185,4 +185,55 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and set' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n+\ttest_line_count = 0 err &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\t\t!/deep/*/\n+\t\t/deep/deeper1/\n+\t\t!/deep/deeper1/*/\n+\t\t/deep/deeper1/deepest/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\tls repo >dir &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384097","messageId":"d0421ef7b2b9c9c511dc72eacfc0b366fda5e38d.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 16/17] sparse-checkout: write using lockfile","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:56:03Z","receivedAt":"2019-10-15T13:56:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf two 'git sparse-checkout set' subcommands are launched at the\nsame time, the behavior can be unexpected as they compete to write\nthe sparse-checkout file and update the working directory.\n\nTake a lockfile around the writes to the sparse-checkout file. In\naddition, acquire this lock around the working directory update\nto avoid two commands updating the working directory in different\nways.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 15 ++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 2 files changed, 19 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 595955d4fc..b319966a79 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -305,6 +305,8 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \n \tif (!core_apply_sparse_checkout) {\n@@ -314,21 +316,28 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tresult = update_working_directory(pl);\n \n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfd = hold_lock_file_for_update(&lk, sparse_filename,\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\n+\tresult = update_working_directory(pl);\n \tif (result) {\n+\t\trollback_lock_file(&lk);\n+\t\tfree(sparse_filename);\n \t\tclear_pattern_list(pl);\n \t\tupdate_working_directory(NULL);\n \t\treturn result;\n \t}\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = fopen(sparse_filename, \"w\");\n+\tfp = fdopen(fd, \"w\");\n \n \tif (core_sparse_checkout_cone)\n \t\twrite_cone_to_file(fp, pl);\n \telse\n \t\twrite_patterns_to_file(fp, pl);\n \n-\tfclose(fp);\n+\tfflush(fp);\n+\tcommit_lock_file(&lk);\n \n \tfree(sparse_filename);\n \tclear_pattern_list(pl);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ecb5a63671..48493e1012 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -262,4 +262,11 @@ test_expect_success 'revert to old sparse-checkout on bad update' '\n \ttest_cmp dir expect\n '\n \n+test_expect_success 'fail when lock is taken' '\n+\ttest_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&\n+\ttouch repo/.git/info/sparse-checkout.lock &&\n+\ttest_must_fail git -C repo sparse-checkout set deep 2>err &&\n+\ttest_i18ngrep \"File exists\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384098","messageId":"9ccec3ca9a1efe038da6d69f076f32cc407bcf9f.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 15/17] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:56:02Z","receivedAt":"2019-10-15T13:56:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\nskip-worktree bits in the index and to update the working directory.\nThis extra process is overly complex, and prone to failure. It also\nrequires that we write our changes to the sparse-checkout file before\ntrying to update the index.\n\nRemove this extra process call by creating a direct call to\nunpack_trees() in the same way 'git read-tree -mu HEAD' does. In\naddition, provide an in-memory list of patterns so we can avoid\nreading from the sparse-checkout file. This allows us to test a\nproposed change to the file before writing to it.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 2 +-\n builtin/sparse-checkout.c | 80 ++++++++++++++++++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 15 ++++++\n unpack-trees.c | 5 +-\n unpack-trees.h | 3 +-\n 5 files changed, 90 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 69963d83dc..d7eeaa26ec 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \n \tif (opts.reset || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n-\t\t\tdie(\"You need to resolve your current index first\");\n+\t\t\tdie(_(\"You need to resolve your current index first\"));\n \t\tstage = opts.merge = 1;\n \t}\n \tresolve_undo_clear();\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 40b6c599dc..595955d4fc 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -7,6 +7,11 @@\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"cache.h\"\n+#include \"cache-tree.h\"\n+#include \"lockfile.h\"\n+#include \"resolve-undo.h\"\n+#include \"unpack-trees.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n@@ -60,18 +65,53 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n-static int update_working_directory(void)\n+static int update_working_directory(struct pattern_list *pl)\n {\n-\tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n-\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\tstruct unpack_trees_options o;\n+\tstruct lock_file lock_file = LOCK_INIT;\n+\tstruct object_id oid;\n+\tstruct tree *tree;\n+\tstruct tree_desc t;\n+\tstruct repository *r = the_repository;\n \n-\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n-\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n-\t\tresult = 1;\n+\tif (repo_read_index_unmerged(r))\n+\t\tdie(_(\"You need to resolve your current index first\"));\n+\n+\tif (get_oid(\"HEAD\", &oid))\n+\t\treturn 0;\n+\n+\ttree = parse_tree_indirect(&oid);\n+\tparse_tree(tree);\n+\tinit_tree_desc(&t, tree->buffer, tree->size);\n+\n+\tmemset(&o, 0, sizeof(o));\n+\to.verbose_update = isatty(2);\n+\to.merge = 1;\n+\to.update = 1;\n+\to.fn = oneway_merge;\n+\to.head_idx = -1;\n+\to.src_index = r->index;\n+\to.dst_index = r->index;\n+\to.skip_sparse_checkout = 0;\n+\to.pl = pl;\n+\to.keep_pattern_list = !!pl;\n+\n+\tresolve_undo_clear_index(r->index);\n+\tsetup_work_tree();\n+\n+\tcache_tree_free(&r->index->cache_tree);\n+\n+\trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n+\n+\tcore_apply_sparse_checkout = 1;\n+\tresult = unpack_trees(1, &t, &o);\n+\n+\tif (!result) {\n+\t\tprime_cache_tree(r, r->index, tree);\n+\t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n \t}\n \n-\targv_array_clear(&argv);\n \treturn result;\n }\n \n@@ -147,7 +187,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n-\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\tif (init_opts.cone_mode) {\n+\t\tmode = MODE_CONE_PATTERNS;\n+\t\tcore_sparse_checkout_cone = 1;\n+\t} else\n+\t\tmode = MODE_ALL_PATTERNS;\n \n \tif (sc_set_config(mode))\n \t\treturn 1;\n@@ -175,7 +219,8 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t}\n \n reset_dir:\n-\treturn update_working_directory();\n+\tcore_apply_sparse_checkout = 1;\n+\treturn update_working_directory(NULL);\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n@@ -260,12 +305,21 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint result;\n \n \tif (!core_apply_sparse_checkout) {\n \t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n \t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n \t}\n \n+\tresult = update_working_directory(pl);\n+\n+\tif (result) {\n+\t\tclear_pattern_list(pl);\n+\t\tupdate_working_directory(NULL);\n+\t\treturn result;\n+\t}\n+\n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n \n@@ -275,9 +329,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \t\twrite_patterns_to_file(fp, pl);\n \n \tfclose(fp);\n+\n \tfree(sparse_filename);\n+\tclear_pattern_list(pl);\n \n-\treturn update_working_directory();\n+\treturn 0;\n }\n \n static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n@@ -328,6 +384,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tstruct strbuf line = STRBUF_INIT;\n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n \t\t\twhile (!strbuf_getline(&line, stdin))\n@@ -373,7 +430,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n\");\n \tfclose(fp);\n \n-\tif (update_working_directory())\n+\tcore_apply_sparse_checkout = 1;\n+\tif (update_working_directory(NULL))\n \t\tdie(_(\"error while refreshing working directory\"));\n \n \tunlink(sparse_filename);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e752f3ba0f..ecb5a63671 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -247,4 +247,19 @@ test_expect_success 'cone mode: set with nested folders' '\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n \n+test_expect_success 'revert to old sparse-checkout on bad update' '\n+\techo update >repo/deep/deeper2/a &&\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp repo/.git/info/sparse-checkout expect &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\t\tdeeper2\n+\tEOF\n+\ttest_cmp dir expect\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8bb684ad62..3789a22cf0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n-\tif (!o->skip_sparse_checkout) {\n+\tif (!o->skip_sparse_checkout && !o->pl) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n \t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n@@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n done:\n \ttrace_performance_leave(\"unpack_trees\");\n-\tclear_pattern_list(&pl);\n+\tif (!o->keep_pattern_list)\n+\t\tclear_pattern_list(&pl);\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex f2eee0c7c5..ca94a421a5 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -59,7 +59,8 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run;\n+\t\t dry_run,\n+\t\t keep_pattern_list;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"384099","messageId":"44c30780290e31103d333b914b33882fc01b70ad.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 14/17] sparse-checkout: sanitize for nested folders","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:56:01Z","receivedAt":"2019-10-15T13:56:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user provides folders A/ and A/B/ for inclusion in a cone-mode\nsparse-checkout file, the parsing logic will notice that A/ appears\nboth as a \"parent\" type pattern and as a \"recursive\" type pattern.\nThis is unexpected and hence will complain via a warning and revert\nto the old logic for checking sparse-checkout patterns.\n\nPrevent this from happening accidentally by sanitizing the folders\nfor this type of inclusion in the 'git sparse-checkout' builtin.\nThis happens in two ways:\n\n1. Do not include any parent patterns that also appear as recursive\n patterns.\n\n2. Do not include any recursive patterns deeper than other recursive\n patterns.\n\nIn order to minimize duplicate code for scanning parents, create\nhashmap_contains_parent() method. It takes a strbuf buffer to\navoid reallocating a buffer when calling in a tight loop.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 22 ++++++++++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 95d9a90b84..40b6c599dc 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -212,9 +212,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstruct pattern_entry *pe;\n \tstruct hashmap_iter iter;\n \tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\tstruct strbuf parent_pattern = STRBUF_INIT;\n \n-\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {\n+\t\tif (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))\n+\t\t\tcontinue;\n+\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\n@@ -230,8 +238,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \n \tstring_list_clear(&sl, 0);\n \n-\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n+\n+\tstrbuf_release(&parent_pattern);\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 3a4c0e6f5d..e752f3ba0f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -236,4 +236,15 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: set with nested folders' '\n+\tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\tEOF\n+\ttest_cmp repo/.git/info/sparse-checkout expect\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384100","messageId":"034fbd31bdd614af4e1d98365079be5c44c4c8d7.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 12/17] unpack-trees: add progress to clear_ce_flags()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:55:59Z","receivedAt":"2019-10-15T13:56:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a large repository has many sparse-checkout patterns, the\nprocess for updating the skip-worktree bits can take long enough\nthat a user gets confused why nothing is happening. Update the\nclear_ce_flags() method to write progress.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 2 ++\n unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++--------------\n 2 files changed, 43 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4980ee198e..d3c89e7a53 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n struct split_index;\n struct untracked_cache;\n+struct progress;\n \n struct index_state {\n \tstruct cache_entry **cache;\n@@ -326,6 +327,7 @@ struct index_state {\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n \tstruct mem_pool *ce_mem_pool;\n+\tstruct progress *progress;\n };\n \n /* Name hashing */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex c0dca20865..8bb684ad62 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1269,7 +1269,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match);\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr);\n \n /* Whole directory matching */\n static int clear_ce_flags_dir(struct index_state *istate,\n@@ -1278,7 +1279,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\t char *basename,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n@@ -1315,7 +1317,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n \t\t\t\t prefix,\n \t\t\t\t select_mask, clear_mask,\n-\t\t\t\t pl, ret);\n+\t\t\t\t pl, ret,\n+\t\t\t\t progress_nr);\n \t}\n \n \tstrbuf_setlen(prefix, prefix->len - 1);\n@@ -1342,7 +1345,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end = cache + nr;\n \n@@ -1356,8 +1360,11 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tint len, dtype;\n \t\tenum pattern_match_result ret;\n \n+\t\tdisplay_progress(istate->progress, progress_nr);\n+\n \t\tif (select_mask && !(ce->ce_flags & select_mask)) {\n \t\t\tcache++;\n+\t\t\tprogress_nr++;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1378,20 +1385,26 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t\t\t\t prefix,\n \t\t\t\t\t\t prefix->buf + prefix->len - len,\n \t\t\t\t\t\t select_mask, clear_mask,\n-\t\t\t\t\t\t pl, default_match);\n+\t\t\t\t\t\t pl, default_match,\n+\t\t\t\t\t\t progress_nr);\n \n \t\t\t/* clear_c_f_dir eats a whole dir already? */\n \t\t\tif (processed) {\n \t\t\t\tcache += processed;\n+\t\t\t\tprogress_nr += processed;\n \t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n \t\t\t\tcontinue;\n \t\t\t}\n \n \t\t\tstrbuf_addch(prefix, '/');\n-\t\t\tcache += clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t\t\t\t prefix,\n-\t\t\t\t\t\t select_mask, clear_mask, pl,\n-\t\t\t\t\t\t default_match);\n+\t\t\tprocessed = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t\t\t prefix,\n+\t\t\t\t\t\t select_mask, clear_mask, pl,\n+\t\t\t\t\t\t default_match, progress_nr);\n+\n+\t\t\tcache += processed;\n+\t\t\tprogress_nr += processed;\n+\n \t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n \t\t\tcontinue;\n \t\t}\n@@ -1406,19 +1419,27 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tif (ret == MATCHED)\n \t\t\tce->ce_flags &= ~clear_mask;\n \t\tcache++;\n+\t\tprogress_nr++;\n \t}\n+\n+\tdisplay_progress(istate->progress, progress_nr);\n \treturn nr - (cache_end - cache);\n }\n \n static int clear_ce_flags(struct index_state *istate,\n \t\t\t int select_mask, int clear_mask,\n-\t\t\t struct pattern_list *pl)\n+\t\t\t struct pattern_list *pl,\n+\t\t\t int show_progress)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n \tchar label[100];\n \tint rval;\n \n \tstrbuf_reset(&prefix);\n+\tif (show_progress)\n+\t\tistate->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Updating index flags\"),\n+\t\t\t\t\tistate->cache_nr);\n \n \txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n \t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n@@ -1428,9 +1449,10 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n-\t\t\t\tpl, 0);\n+\t\t\t\tpl, 0, 0);\n \ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n \n+\tstop_progress(&istate->progress);\n \treturn rval;\n }\n \n@@ -1439,7 +1461,8 @@ static int clear_ce_flags(struct index_state *istate,\n */\n static void mark_new_skip_worktree(struct pattern_list *pl,\n \t\t\t\t struct index_state *istate,\n-\t\t\t\t int select_flag, int skip_wt_flag)\n+\t\t\t\t int select_flag, int skip_wt_flag,\n+\t\t\t\t int show_progress)\n {\n \tint i;\n \n@@ -1463,7 +1486,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \t * 2. Widen worktree according to sparse-checkout file.\n \t * Matched entries will have skip_wt_flag cleared (i.e. \"in\")\n \t */\n-\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl);\n+\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n static int verify_absent(const struct cache_entry *,\n@@ -1525,7 +1548,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n \t */\n \tif (!o->skip_sparse_checkout)\n-\t\tmark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n \n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n@@ -1590,7 +1614,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n-\t\tmark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, &o->result,\n+\t\t\t\t CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE,\n+\t\t\t\t o->verbose_update);\n \n \t\tret = 0;\n \t\tfor (i = 0; i < o->result.cache_nr; i++) {\n-- \ngitgitgadget\n\n"},{"id":"384101","messageId":"27aa9d22f0bc2a6d02dac5d35d19904845762c27.1571147765.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v4 13/17] read-tree: show progress by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-15T13:56:00Z","receivedAt":"2019-10-15T13:56:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe read-tree builtin has a --verbose option that signals to show\nprogress and other data while updating the index. Update this to\nbe on by default when stderr is a terminal window.\n\nThis will help tools like 'git sparse-checkout' to automatically\nbenefit from progress indicators when a user runs these commands.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex ca5e655d2f..69963d83dc 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \topts.head_idx = -1;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\topts.verbose_update = isatty(2);\n \n \tgit_config(git_read_tree_config, NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"384201","messageId":"CABPp-BEryfaeYhuUsiDTaYdRKpK6GRi7hgZ5XSTVkoHVkx2qQA@mail.gmail.com","threadId":"51691","inReplyTo":"63626e109767c268ee81b50abd21b95b1f1b5bfb.1571147765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-16T19:00:49Z","receivedAt":"2019-10-16T19:01:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 15, 2019 at 6:56 AM Derrick Stolee via GitGitGadget\n wrote:\n> +DESCRIPTION\n> +-----------\n> +\n> +Initialize and modify the sparse-checkout configuration, which reduces\n> +the checkout to a set of directories given by a list of prefixes.\n> +\n> +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n\nI think the wording needs to be a bit more detailed; you copied the\nwording from git-switch.txt, but usage of git-switch is not expected\nto modify the behavior of other commands. sparse-checkout, by\ncontrast, is designed to affect other commands: at the very least\ncheckout & switch, and likely will affect grep, diff, log, and a host\nof others. Perhaps something like:\n\nTHIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\nCOMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\nTHE FUTURE.\n"},{"id":"384211","messageId":"CABPp-BHkjx-qVNXAF4eiwa2Uer0_=epqhfOTMyigU_MpJ8xk4w@mail.gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-10-16T20:07:20Z","receivedAt":"2019-10-16T20:07:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Oct 15, 2019 at 6:56 AM Derrick Stolee via GitGitGadget\n wrote:\n> Updates in V4:\n>\n> * Updated hashmap API usage to respond to ew/hashmap\n>\n>\n> * Responded to detailed review by Elijah. Thanks!\n>\n>\n> * Marked the feature as experimental in git-sparse-checkout.txt the same\n> way that git-switch.txt does.\n\nI read through the range-diff, and it all looks good to me other than\none issue I flagged on patch 1.\n\nNice work!\n"},{"id":"384322","messageId":"1ea216d8-e706-ec4c-a3f3-954e57e0458c@jonsimons.org","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Jon Simons","fromEmail":"jon@jonsimons.org","sentAt":"2019-10-17T23:53:44Z","receivedAt":"2019-10-18T00:01:48Z","isPatch":true,"sender":{"key":"jon@jonsimons.org","avatar":"https://avatars.githubusercontent.com/u/118440?v=4"},"body":"On 10/15/19 6:55 AM, Derrick Stolee via GitGitGadget wrote:\n> V4 UPDATE: Rebased on latest master to include ew/hashmap and\n> ds/include-exclude in the base.\n\nI did a partial review of the v4 patches out of curiosity and I notice\nin sparse-checkout.c there are a couple of unchecked `fopen` call sites\nthat could be converted to `xfopen` to die gracefully upon error, and\none unchecked `fdopen` that can likewise be `xfdopen`.\n\n\n-Jon\n"},{"id":"384387","messageId":"20191018153112.GF29845@szeder.dev","threadId":"51691","inReplyTo":"28606a152c79c4e399c5a9c77038888eda34341d.1571147765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 09/17] sparse-checkout: use hashmaps for cone patterns","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-10-18T15:31:12Z","receivedAt":"2019-10-18T15:31:18Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 15, 2019 at 01:55:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Running 'git read-tree -mu HEAD' on this file had the following\n> performance:\n> \n> \tcore.sparseCheckout=false: 0.21 s (0.00 s)\n> \t core.sparseCheckout=true: 3.75 s (3.50 s)\n> \t core.sparseCheckout=cone: 0.23 s (0.01 s)\n\nThe previous patch added 'core.sparseCheckoutCone', so this last line\nshould be 'core.sparseCheckoutCone=true', shouldn't it?\n\n"},{"id":"384390","messageId":"20191018160737.GG29845@szeder.dev","threadId":"51691","inReplyTo":"63626e109767c268ee81b50abd21b95b1f1b5bfb.1571147765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-10-18T16:07:37Z","receivedAt":"2019-10-18T16:07:45Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 15, 2019 at 01:55:48PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The sparse-checkout feature is mostly hidden to users, as its\n> only documentation is supplementary information in the docs for\n> 'git read-tree'. In addition, users need to know how to edit the\n> .git/info/sparse-checkout file with the right patterns, then run\n> the appropriate 'git read-tree -mu HEAD' command. Keeping the\n> working directory in sync with the sparse-checkout file requires\n> care.\n> \n> Begin an effort to make the sparse-checkout feature a porcelain\n> feature by creating a new 'git sparse-checkout' builtin. This\n> builtin will be the preferred mechanism for manipulating the\n> sparse-checkout file and syncing the working directory.\n> \n> The documentation provided is adapted from the \"git read-tree\"\n> documentation with a few edits for clarity in the new context.\n> Extra sections are added to hint toward a future change to\n> a more restricted pattern set.\n> \n> Helped-by: Elijah Newren \n> Signed-off-by: Derrick Stolee \n> ---\n> .gitignore | 1 +\n> Documentation/git-read-tree.txt | 2 +-\n> Documentation/git-sparse-checkout.txt | 87 +++++++++++++++++++++++++++\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++\n> git.c | 1 +\n> t/t1091-sparse-checkout-builtin.sh | 50 +++++++++++++++\n\nYou need to add an entry for the new command to 'command-list.txt' as\nwell, so it will be included in 'git help -a' and completion (since\nit's intended to be a porcelain), etc.\n\n> 8 files changed, 228 insertions(+), 1 deletion(-)\n> create mode 100644 Documentation/git-sparse-checkout.txt\n> create mode 100644 builtin/sparse-checkout.c\n> create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> new file mode 100644\n> index 0000000000..46d3dc3cb1\n> --- /dev/null\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -0,0 +1,87 @@\n> +git-sparse-checkout(1)\n> +=======================\n\nThe ==== line is one character longer than the title; I think their\nlength should match. The other day the Documentation CI job failed\nbecause the length of these lines didn't match in one of the guides.\nStrangely, it doesn't complain about this one.\n\n> +\n> +NAME\n> +----\n> +git-sparse-checkout - Initialize and modify the sparse-checkout\n> +configuration, which reduces the checkout to a set of directories\n> +given by a list of prefixes.\n> +\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git sparse-checkout [options]'\n> +\n> +\n> +DESCRIPTION\n> +-----------\n> +\n> +Initialize and modify the sparse-checkout configuration, which reduces\n> +the checkout to a set of directories given by a list of prefixes.\n> +\n> +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n> +\n> +\n> +COMMANDS\n> +--------\n> +'list'::\n> +\tProvide a list of the contents in the sparse-checkout file.\n> +\n> +\n> +SPARSE CHECKOUT\n> +----------------\n\nThis is longer as well.\n\n> +\n> +\"Sparse checkout\" allows populating the working directory sparsely.\n> +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n> +Git whether a file in the working directory is worth looking at. If\n> +the skip-worktree bit is set, then the file is ignored in the working\n> +directory. Git will not populate the contents of those files, which\n> +makes a sparse checkout helpful when working in a repository with many\n> +files, but only a few are important to the current user.\n> +\n> +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n> +skip-worktree reference bitmap. When Git updates the working\n> +directory, it updates the skip-worktree bits in the index based\n> +on this file. The files matching the patterns in the file will\n> +appear in the working directory, and the rest will not.\n> +\n> +## FULL PATTERN SET\n> +\n> +By default, the sparse-checkout file uses the same syntax as `.gitignore`\n> +files.\n> +\n> +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n> +files are included, you can also specify what files are _not_ included,\n> +using negative patterns. For example, to remove the file `unwanted`:\n> +\n> +----------------\n> +/*\n> +!unwanted\n> +----------------\n> +\n> +Another tricky thing is fully repopulating the working directory when you\n> +no longer want sparse checkout. You cannot just disable \"sparse\n> +checkout\" because skip-worktree bits are still in the index and your working\n> +directory is still sparsely populated. You should re-populate the working\n> +directory with the `$GIT_DIR/info/sparse-checkout` file content as\n> +follows:\n> +\n> +----------------\n> +/*\n> +----------------\n> +\n> +Then you can disable sparse checkout. Sparse checkout support in 'git\n> +checkout' and similar commands is disabled by default. You need to\n> +set `core.sparseCheckout` to `true` in order to have sparse checkout\n> +support.\n> +\n> +SEE ALSO\n> +--------\n> +\n> +linkgit:git-read-tree[1]\n> +linkgit:gitignore[5]\n> +\n> +GIT\n> +---\n> +Part of the linkgit:git[1] suite\n\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> new file mode 100755\n> index 0000000000..a9b04b1a88\n> --- /dev/null\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -0,0 +1,50 @@\n> +#!/bin/sh\n> +\n> +test_description='sparse checkout builtin tests'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\t\techo \"initial\" >a &&\n> +\t\tmkdir folder1 folder2 deep &&\n> +\t\tmkdir deep/deeper1 deep/deeper2 &&\n> +\t\tmkdir deep/deeper1/deepest &&\n> +\t\tcp a folder1 &&\n> +\t\tcp a folder2 &&\n> +\t\tcp a deep &&\n> +\t\tcp a deep/deeper1 &&\n> +\t\tcp a deep/deeper2 &&\n> +\t\tcp a deep/deeper1/deepest &&\n\nThere are six 'cp' invocations here. Curious: does this mean that\nit's important that all those files are the same?\n\n> +\t\tgit add . &&\n> +\t\tgit commit -m \"initial commit\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'git sparse-checkout list (empty)' '\n> +\tgit -C repo sparse-checkout list >list 2>err &&\n> +\ttest_line_count = 0 list &&\n\nNit: test_must_be_empty might be a bit more idiomatic, dunno.\n\n> +\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n> +'\n> +\n> +test_expect_success 'git sparse-checkout list (populated)' '\n> +\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n> +\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n> +\t\t/folder1/*\n> +\t\t/deep/\n> +\t\t**/a\n> +\t\t!*bin*\n> +\tEOF\n> +\tgit -C repo sparse-checkout list >list &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/folder1/*\n> +\t\t/deep/\n> +\t\t**/a\n> +\t\t!*bin*\n> +\tEOF\n\nOTOH, here the content of the 'sparse-checkout' file and 'expect' must\nbe the same, but one has to carefully look through four lines of\npatterns to realize that they are indeed the same. I think in this\ncase the explicitness of 'cp' would be better.\n\n> +\ttest_cmp expect list\n> +'\n> +\n> +test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"384394","messageId":"20191018163116.GH29845@szeder.dev","threadId":"51691","inReplyTo":"55f95f290ef77fa402f3e73f9c3045177bd70e84.1571147765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/17] sparse-checkout: create 'disable' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-10-18T16:31:16Z","receivedAt":"2019-10-18T16:32:17Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 15, 2019 at 01:55:53PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index b747b78d34..78a80ce119 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n> \n> static char const * const builtin_sparse_checkout_usage[] = {\n> -\tN_(\"git sparse-checkout [init|list|set] \"),\n> +\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n\nIn the synopsis [] means optional, but I think the subcommands are not\noptional; however, the options are:\n\n git sparse-checkout (init|list|set|disable) []\n\nPerhaps it would be even better if each subcommand had its own\nsynopsis line in the short help, see e.g. 'git notes -h' or 'git\nremote -h'.\n\n"},{"id":"384408","messageId":"20191018202421.GJ29845@szeder.dev","threadId":"51691","inReplyTo":"9ccec3ca9a1efe038da6d69f076f32cc407bcf9f.1571147765.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 15/17] sparse-checkout: update working directory in-process","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-10-18T20:24:21Z","receivedAt":"2019-10-18T20:24:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 15, 2019 at 01:56:02PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n> skip-worktree bits in the index and to update the working directory.\n> This extra process is overly complex, and prone to failure. It also\n> requires that we write our changes to the sparse-checkout file before\n> trying to update the index.\n> \n> Remove this extra process call by creating a direct call to\n> unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n> addition, provide an in-memory list of patterns so we can avoid\n> reading from the sparse-checkout file. This allows us to test a\n> proposed change to the file before writing to it.\n\nStarting with this patch there is an issue with locking the index:\n\n $ git init\n Initialized empty Git repository in /home/szeder/src/git/tmp/SC/.git/\n $ >file\n $ git commit -m initial\n [master (root-commit) 5d80b9c] initial\n 1 file changed, 0 insertions(+), 0 deletions(-)\n create mode 100644 file\n $ ls .git/index.lock\n ls: cannot access '.git/index.lock': No such file or directory\n $ git sparse-checkout set nope\n warning: core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\n warning: run 'git sparse-checkout init' to enable the sparse-checkout feature\n error: Sparse checkout leaves no entry on working directory\n fatal: Unable to create '/home/szeder/src/git/tmp/SC/.git/index.lock':\n File exists.\n\n Another git process seems to be running in this repository, e.g.\n an editor opened by 'git commit'. Please make sure all processes\n are terminated then try again. If it still fails, a git process\n may have crashed in this repository earlier:\n remove the file manually to continue.\n $ ls .git/index.lock\n ls: cannot access '.git/index.lock': No such file or directory\n\n"},{"id":"384410","messageId":"20191018204032.GK29845@szeder.dev","threadId":"51691","inReplyTo":"20191018202421.GJ29845@szeder.dev","subject":"Re: [PATCH v4 15/17] sparse-checkout: update working directory in-process","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-10-18T20:40:32Z","receivedAt":"2019-10-18T20:40:38Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Oct 18, 2019 at 10:24:21PM +0200, SZEDER Gábor wrote:\n> On Tue, Oct 15, 2019 at 01:56:02PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > From: Derrick Stolee \n> > \n> > The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n> > skip-worktree bits in the index and to update the working directory.\n> > This extra process is overly complex, and prone to failure. It also\n> > requires that we write our changes to the sparse-checkout file before\n> > trying to update the index.\n> > \n> > Remove this extra process call by creating a direct call to\n> > unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n> > addition, provide an in-memory list of patterns so we can avoid\n> > reading from the sparse-checkout file. This allows us to test a\n> > proposed change to the file before writing to it.\n> \n> Starting with this patch there is an issue with locking the index:\n> \n> $ git init\n> Initialized empty Git repository in /home/szeder/src/git/tmp/SC/.git/\n> $ >file\n\n $ git add file\n\nForgot to copy that command...\n\n> $ git commit -m initial\n> [master (root-commit) 5d80b9c] initial\n> 1 file changed, 0 insertions(+), 0 deletions(-)\n> create mode 100644 file\n> $ ls .git/index.lock\n> ls: cannot access '.git/index.lock': No such file or directory\n> $ git sparse-checkout set nope\n> warning: core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\n> warning: run 'git sparse-checkout init' to enable the sparse-checkout feature\n> error: Sparse checkout leaves no entry on working directory\n> fatal: Unable to create '/home/szeder/src/git/tmp/SC/.git/index.lock':\n> File exists.\n> \n> Another git process seems to be running in this repository, e.g.\n> an editor opened by 'git commit'. Please make sure all processes\n> are terminated then try again. If it still fails, a git process\n> may have crashed in this repository earlier:\n> remove the file manually to continue.\n> $ ls .git/index.lock\n> ls: cannot access '.git/index.lock': No such file or directory\n\nI would add that building the previous patch and running the same\nsequence of commands works, in the sense that 'git sparse-checkout\nset' writes the non-existing filename to the 'sparse-checkout' file\nand it prints the same two warnings, and doesn't (seem to) attempt to\nupdate the working tree and the index.\n\n"},{"id":"384500","messageId":"6d12eb61-1dce-7d58-46c9-63cd75ca17bb@gmail.com","threadId":"51691","inReplyTo":"20191018160737.GG29845@szeder.dev","subject":"Re: [PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T11:42:51Z","receivedAt":"2019-10-21T11:42:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/18/2019 12:07 PM, SZEDER Gábor wrote:\n> On Tue, Oct 15, 2019 at 01:55:48PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The sparse-checkout feature is mostly hidden to users, as its\n>> only documentation is supplementary information in the docs for\n>> 'git read-tree'. In addition, users need to know how to edit the\n>> .git/info/sparse-checkout file with the right patterns, then run\n>> the appropriate 'git read-tree -mu HEAD' command. Keeping the\n>> working directory in sync with the sparse-checkout file requires\n>> care.\n>>\n>> Begin an effort to make the sparse-checkout feature a porcelain\n>> feature by creating a new 'git sparse-checkout' builtin. This\n>> builtin will be the preferred mechanism for manipulating the\n>> sparse-checkout file and syncing the working directory.\n>>\n>> The documentation provided is adapted from the \"git read-tree\"\n>> documentation with a few edits for clarity in the new context.\n>> Extra sections are added to hint toward a future change to\n>> a more restricted pattern set.\n>>\n>> Helped-by: Elijah Newren \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> .gitignore | 1 +\n>> Documentation/git-read-tree.txt | 2 +-\n>> Documentation/git-sparse-checkout.txt | 87 +++++++++++++++++++++++++++\n>> Makefile | 1 +\n>> builtin.h | 1 +\n>> builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++\n>> git.c | 1 +\n>> t/t1091-sparse-checkout-builtin.sh | 50 +++++++++++++++\n> \n> You need to add an entry for the new command to 'command-list.txt' as\n> well, so it will be included in 'git help -a' and completion (since\n> it's intended to be a porcelain), etc.\n> \n>> 8 files changed, 228 insertions(+), 1 deletion(-)\n>> create mode 100644 Documentation/git-sparse-checkout.txt\n>> create mode 100644 builtin/sparse-checkout.c\n>> create mode 100755 t/t1091-sparse-checkout-builtin.sh\n> \n> \n>> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n>> new file mode 100644\n>> index 0000000000..46d3dc3cb1\n>> --- /dev/null\n>> +++ b/Documentation/git-sparse-checkout.txt\n>> @@ -0,0 +1,87 @@\n>> +git-sparse-checkout(1)\n>> +=======================\n> \n> The ==== line is one character longer than the title; I think their\n> length should match. The other day the Documentation CI job failed\n> because the length of these lines didn't match in one of the guides.\n> Strangely, it doesn't complain about this one.\n\nInteresting. GitGitGadget runs a documentation build, and it has never\nfailed for me. Will fix.\n\n>> +\n>> +NAME\n>> +----\n>> +git-sparse-checkout - Initialize and modify the sparse-checkout\n>> +configuration, which reduces the checkout to a set of directories\n>> +given by a list of prefixes.\n>> +\n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git sparse-checkout [options]'\n>> +\n>> +\n>> +DESCRIPTION\n>> +-----------\n>> +\n>> +Initialize and modify the sparse-checkout configuration, which reduces\n>> +the checkout to a set of directories given by a list of prefixes.\n>> +\n>> +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n>> +\n>> +\n>> +COMMANDS\n>> +--------\n>> +'list'::\n>> +\tProvide a list of the contents in the sparse-checkout file.\n>> +\n>> +\n>> +SPARSE CHECKOUT\n>> +----------------\n> \n> This is longer as well.\n> \n>> +\n>> +\"Sparse checkout\" allows populating the working directory sparsely.\n>> +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n>> +Git whether a file in the working directory is worth looking at. If\n>> +the skip-worktree bit is set, then the file is ignored in the working\n>> +directory. Git will not populate the contents of those files, which\n>> +makes a sparse checkout helpful when working in a repository with many\n>> +files, but only a few are important to the current user.\n>> +\n>> +The `$GIT_DIR/info/sparse-checkout` file is used to define the\n>> +skip-worktree reference bitmap. When Git updates the working\n>> +directory, it updates the skip-worktree bits in the index based\n>> +on this file. The files matching the patterns in the file will\n>> +appear in the working directory, and the rest will not.\n>> +\n>> +## FULL PATTERN SET\n>> +\n>> +By default, the sparse-checkout file uses the same syntax as `.gitignore`\n>> +files.\n>> +\n>> +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n>> +files are included, you can also specify what files are _not_ included,\n>> +using negative patterns. For example, to remove the file `unwanted`:\n>> +\n>> +----------------\n>> +/*\n>> +!unwanted\n>> +----------------\n>> +\n>> +Another tricky thing is fully repopulating the working directory when you\n>> +no longer want sparse checkout. You cannot just disable \"sparse\n>> +checkout\" because skip-worktree bits are still in the index and your working\n>> +directory is still sparsely populated. You should re-populate the working\n>> +directory with the `$GIT_DIR/info/sparse-checkout` file content as\n>> +follows:\n>> +\n>> +----------------\n>> +/*\n>> +----------------\n>> +\n>> +Then you can disable sparse checkout. Sparse checkout support in 'git\n>> +checkout' and similar commands is disabled by default. You need to\n>> +set `core.sparseCheckout` to `true` in order to have sparse checkout\n>> +support.\n>> +\n>> +SEE ALSO\n>> +--------\n>> +\n>> +linkgit:git-read-tree[1]\n>> +linkgit:gitignore[5]\n>> +\n>> +GIT\n>> +---\n>> +Part of the linkgit:git[1] suite\n> \n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> new file mode 100755\n>> index 0000000000..a9b04b1a88\n>> --- /dev/null\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -0,0 +1,50 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='sparse checkout builtin tests'\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +test_expect_success 'setup' '\n>> +\tgit init repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\t\techo \"initial\" >a &&\n>> +\t\tmkdir folder1 folder2 deep &&\n>> +\t\tmkdir deep/deeper1 deep/deeper2 &&\n>> +\t\tmkdir deep/deeper1/deepest &&\n>> +\t\tcp a folder1 &&\n>> +\t\tcp a folder2 &&\n>> +\t\tcp a deep &&\n>> +\t\tcp a deep/deeper1 &&\n>> +\t\tcp a deep/deeper2 &&\n>> +\t\tcp a deep/deeper1/deepest &&\n> \n> There are six 'cp' invocations here. Curious: does this mean that\n> it's important that all those files are the same?\n\nThe content of the \"a\" file is not important at all to the\ntests. It only matters that there is a blob in each of these\ntrees.\n\n> \n>> +\t\tgit add . &&\n>> +\t\tgit commit -m \"initial commit\"\n>> +\t)\n>> +'\n>> +\n>> +test_expect_success 'git sparse-checkout list (empty)' '\n>> +\tgit -C repo sparse-checkout list >list 2>err &&\n>> +\ttest_line_count = 0 list &&\n> \n> Nit: test_must_be_empty might be a bit more idiomatic, dunno.\n> \n>> +\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n>> +'\n>> +\n>> +test_expect_success 'git sparse-checkout list (populated)' '\n>> +\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n>> +\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n>> +\t\t/folder1/*\n>> +\t\t/deep/\n>> +\t\t**/a\n>> +\t\t!*bin*\n>> +\tEOF\n>> +\tgit -C repo sparse-checkout list >list &&\n>> +\tcat >expect <<-EOF &&\n>> +\t\t/folder1/*\n>> +\t\t/deep/\n>> +\t\t**/a\n>> +\t\t!*bin*\n>> +\tEOF\n> \n> OTOH, here the content of the 'sparse-checkout' file and 'expect' must\n> be the same, but one has to carefully look through four lines of\n> patterns to realize that they are indeed the same. I think in this\n> case the explicitness of 'cp' would be better.\n\nThat's fine. We explicitly set the contents in the line above.\n\n>> +\ttest_cmp expect list\n>> +'\n>> +\n>> +test_done\n>> -- \n>> gitgitgadget\n>>\n\n"},{"id":"384501","messageId":"d063d58b-2e4b-5fbe-0aba-8086215fd9b9@gmail.com","threadId":"51691","inReplyTo":"20191018153112.GF29845@szeder.dev","subject":"Re: [PATCH v4 09/17] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T11:44:30Z","receivedAt":"2019-10-21T11:44:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/18/2019 11:31 AM, SZEDER Gábor wrote:\n> On Tue, Oct 15, 2019 at 01:55:56PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> Running 'git read-tree -mu HEAD' on this file had the following\n>> performance:\n>>\n>> \tcore.sparseCheckout=false: 0.21 s (0.00 s)\n>> \t core.sparseCheckout=true: 3.75 s (3.50 s)\n>> \t core.sparseCheckout=cone: 0.23 s (0.01 s)\n> \n> The previous patch added 'core.sparseCheckoutCone', so this last line\n> should be 'core.sparseCheckoutCone=true', shouldn't it?\n\nGood catch! Thanks.\n\n-Stolee\n\n\n"},{"id":"384502","messageId":"2dd417e0-2fc6-67ca-68c5-e9afceb7e496@gmail.com","threadId":"51691","inReplyTo":"20191018204032.GK29845@szeder.dev","subject":"Re: [PATCH v4 15/17] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T11:59:32Z","receivedAt":"2019-10-21T11:59:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/18/2019 4:40 PM, SZEDER Gábor wrote:\n> On Fri, Oct 18, 2019 at 10:24:21PM +0200, SZEDER Gábor wrote:\n>> On Tue, Oct 15, 2019 at 01:56:02PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>>>\n>>> The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n>>> skip-worktree bits in the index and to update the working directory.\n>>> This extra process is overly complex, and prone to failure. It also\n>>> requires that we write our changes to the sparse-checkout file before\n>>> trying to update the index.\n>>>\n>>> Remove this extra process call by creating a direct call to\n>>> unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n>>> addition, provide an in-memory list of patterns so we can avoid\n>>> reading from the sparse-checkout file. This allows us to test a\n>>> proposed change to the file before writing to it.\n>>\n>> Starting with this patch there is an issue with locking the index:\n>>\n>> $ git init\n>> Initialized empty Git repository in /home/szeder/src/git/tmp/SC/.git/\n>> $ >file\n> \n> $ git add file\n> \n> Forgot to copy that command...\n> \n>> $ git commit -m initial\n>> [master (root-commit) 5d80b9c] initial\n>> 1 file changed, 0 insertions(+), 0 deletions(-)\n>> create mode 100644 file\n>> $ ls .git/index.lock\n>> ls: cannot access '.git/index.lock': No such file or directory\n>> $ git sparse-checkout set nope\n>> warning: core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\n>> warning: run 'git sparse-checkout init' to enable the sparse-checkout feature\n>> error: Sparse checkout leaves no entry on working directory\n>> fatal: Unable to create '/home/szeder/src/git/tmp/SC/.git/index.lock':\n>> File exists.\n>>\n>> Another git process seems to be running in this repository, e.g.\n>> an editor opened by 'git commit'. Please make sure all processes\n>> are terminated then try again. If it still fails, a git process\n>> may have crashed in this repository earlier:\n>> remove the file manually to continue.\n>> $ ls .git/index.lock\n>> ls: cannot access '.git/index.lock': No such file or directory\n> \n> I would add that building the previous patch and running the same\n> sequence of commands works, in the sense that 'git sparse-checkout\n> set' writes the non-existing filename to the 'sparse-checkout' file\n> and it prints the same two warnings, and doesn't (seem to) attempt to\n> update the working tree and the index.\n\nThank you for catching this! The issue is that I was not rolling the\nindex back on this kind of failed update, and then trying to replay\nthe \"old\" sparse-checkout hit the existing .git/index.lock file.\n\nHere is the test I added that breaks on the current patch, but\npasses when adding a rollback_lock_file() call:\n\ntest_expect_success 'revert to old sparse-checkout on empty update' '\n\tgit init empty-test &&\n\t(\n\t\techo >file &&\n\t\tgit add file &&\n\t\tgit commit -m \"test\" &&\n\t\ttest_must_fail git sparse-checkout set nothing 2>err &&\n\t\ttest_i18ngrep \"Sparse checkout leaves no entry on working directory\" err &&\n\t\ttest_i18ngrep ! \".git/index.lock\" err &&\n\t\tgit sparse-checkout set file\n\t)\n'\n\nIt also has the following problem: because we are setting the\npatterns in-memory, this update acts like core.sparseCheckout is\nenabled, even when it is not. So, I'm finally convinced that\nthe 'set' command should enable the config setting if it is\ncalled even without 'init'.\n\nThanks,\n-Stolee\n\n"},{"id":"384503","messageId":"529d7656-d9d9-1a52-cea5-132ac8fdd2f7@gmail.com","threadId":"51691","inReplyTo":"1ea216d8-e706-ec4c-a3f3-954e57e0458c@jonsimons.org","subject":"Re: [PATCH v4 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T12:08:58Z","receivedAt":"2019-10-21T12:09:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/17/2019 7:53 PM, Jon Simons wrote:\n> On 10/15/19 6:55 AM, Derrick Stolee via GitGitGadget wrote:\n>> V4 UPDATE: Rebased on latest master to include ew/hashmap and\n>> ds/include-exclude in the base.\n> \n> I did a partial review of the v4 patches out of curiosity and I notice\n> in sparse-checkout.c there are a couple of unchecked `fopen` call sites\n> that could be converted to `xfopen` to die gracefully upon error, and\n> one unchecked `fdopen` that can likewise be `xfdopen`.\n\nThanks! Fixed for v5.\n\n-Stolee\n"},{"id":"384504","messageId":"01f02ca8-00a6-6b48-1cc5-345547c2d5e4@gmail.com","threadId":"51691","inReplyTo":"CABPp-BEryfaeYhuUsiDTaYdRKpK6GRi7hgZ5XSTVkoHVkx2qQA@mail.gmail.com","subject":"Re: [PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T12:10:41Z","receivedAt":"2019-10-21T12:10:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/16/2019 3:00 PM, Elijah Newren wrote:\n> On Tue, Oct 15, 2019 at 6:56 AM Derrick Stolee via GitGitGadget\n> wrote:\n>> +DESCRIPTION\n>> +-----------\n>> +\n>> +Initialize and modify the sparse-checkout configuration, which reduces\n>> +the checkout to a set of directories given by a list of prefixes.\n>> +\n>> +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n> \n> I think the wording needs to be a bit more detailed; you copied the\n> wording from git-switch.txt, but usage of git-switch is not expected\n> to modify the behavior of other commands. sparse-checkout, by\n> contrast, is designed to affect other commands: at the very least\n> checkout & switch, and likely will affect grep, diff, log, and a host\n> of others. Perhaps something like:\n> \n> THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n> COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n> THE FUTURE.\n\nThanks. I was taking your comment to mean \"this builtin is experimental\"\nbut what you really mean is \"the sparse-checkout feature is (still)\nexperimental\".\n\n-Stolee\n"},{"id":"384508","messageId":"a161cee0dfec76e7a08253083f488e2e6d26299e.1571666186.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:11Z","receivedAt":"2019-10-21T13:56:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nMake sure to use the `extensions.worktreeConfig` setting and write\nthe sparse checkout config to the worktree-specific config file.\nThis avoids confusing interactions with other worktrees.\n\nThe use of running another process for 'git read-tree' is sub-\noptimal. This will be removed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 11 ++++\n builtin/sparse-checkout.c | 79 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++\n 3 files changed, 130 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 9d6ca22917..930a361567 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -30,6 +30,17 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n++\n+To avoid interfering with other worktrees, it first enables the\n+`extensions.worktreeConfig` setting and makes sure to set the\n+`core.sparseCheckout` setting in the worktree-specific config file.\n \n SPARSE CHECKOUT\n ---------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 5717c9b2cb..77aa52ca01 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout list\"),\n+\tN_(\"git sparse-checkout (init|list)\"),\n \tNULL\n };\n \n@@ -59,6 +59,81 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int update_working_directory(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+enum sparse_checkout_mode {\n+\tMODE_NO_PATTERNS = 0,\n+\tMODE_ALL_PATTERNS = 1,\n+};\n+\n+static int sc_set_config(enum sparse_checkout_mode mode)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\n+\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n+\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n+\t\treturn 1;\n+\t}\n+\n+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n+\n+\tif (mode)\n+\t\targv_array_pushl(&argv, \"true\", NULL);\n+\telse\n+\t\targv_array_pushl(&argv, \"false\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\t\treturn 1;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = xfopen(sparse_filename, \"w\");\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\tfclose(fp);\n+\n+reset_dir:\n+\treturn update_working_directory();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -79,6 +154,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9b73d44907..cd56cc384b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -42,4 +42,45 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384510","messageId":"7d9d66a89f473244af3601e13caa713d929a202d.1571666186.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 04/17] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:13Z","receivedAt":"2019-10-21T13:56:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nas arguments and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nThe 'set' subcommand will replace the entire contents of the\nsparse-checkout file. The write_patterns_and_update() method is\nextracted from cmd_sparse_checkout() to make it easier to implement\n'add' and/or 'remove' subcommands in the future.\n\nIf the core.sparseCheckout config setting is disabled, then enable\nthe config setting in the worktree config. If we set the config\nthis way and the sparse-checkout fails, then re-disable the config\nsetting.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 6 ++++\n builtin/sparse-checkout.c | 45 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 32 +++++++++++++++++++\n 3 files changed, 82 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 930a361567..b933043b3d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -42,6 +42,12 @@ To avoid interfering with other worktrees, it first enables the\n `extensions.worktreeConfig` setting and makes sure to set the\n `core.sparseCheckout` setting in the worktree-specific config file.\n \n+'set'::\n+\tWrite a set of patterns to the sparse-checkout file, as given as\n+\ta list of arguments following the 'set' subcommand. Update the\n+\tworking directory to match the new patterns. Enable the\n+\tcore.sparseCheckout config setting if it is not already enabled.\n+\n SPARSE CHECKOUT\n ---------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 6c336b7ab3..834ee421f0 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list)\"),\n+\tN_(\"git sparse-checkout (init|list|set) \"),\n \tNULL\n };\n \n@@ -140,6 +140,47 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_patterns_to_file(fp, pl);\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\treturn update_working_directory();\n+}\n+\n+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *empty_base = \"\";\n+\tint i;\n+\tstruct pattern_list pl;\n+\tint result;\n+\tint set_config = 0;\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tfor (i = 1; i < argc; i++)\n+\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\n+\tif (!core_apply_sparse_checkout) {\n+\t\tsc_set_config(MODE_ALL_PATTERNS);\n+\t\tcore_apply_sparse_checkout = 1;\n+\t\tset_config = 1;\n+\t}\n+\n+\tresult = write_patterns_and_update(&pl);\n+\n+\tif (result && set_config)\n+\t\tsc_set_config(MODE_NO_PATTERNS);\n+\n+\tclear_pattern_list(&pl);\n+\treturn result;\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -162,6 +203,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"set\"))\n+\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex cb74715ca6..bf2dc55bb1 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set enables config' '\n+\tgit init empty-config &&\n+\t(\n+\t\tcd empty-config &&\n+\t\ttest_commit test file &&\n+\t\ttest_path_is_missing .git/config.worktree &&\n+\t\ttest_must_fail git sparse-checkout set nothing &&\n+\t\ttest_i18ngrep \"sparseCheckout = false\" .git/config.worktree &&\n+\t\tgit sparse-checkout set \"/*\" &&\n+\t\ttest_i18ngrep \"sparseCheckout = true\" .git/config.worktree\n+\t)\n+'\n+\n+test_expect_success 'set sparse-checkout using builtin' '\n+\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384509","messageId":"cd150853b7e4323c0a471d016800e614b3a62679.1571666186.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 03/17] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:12Z","receivedAt":"2019-10-21T13:56:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nDuring the 'git sparse-checkout init' call, we must first look\nto see if HEAD is valid, since 'git clone' does not have a valid\nHEAD at the point where it initializes the sparse-checkout. The\nfollowing checkout within the clone command will create the HEAD\nref and update the working directory correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n builtin/sparse-checkout.c | 6 ++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 4 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 34011c2940..0fe91d2f04 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex c46ee29f0a..4348d962c9 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -59,6 +59,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 77aa52ca01..6c336b7ab3 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tstruct object_id oid;\n \n \tif (sc_set_config(MODE_ALL_PATTERNS))\n \t\treturn 1;\n@@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \tfclose(fp);\n \n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n reset_dir:\n \treturn update_working_directory();\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex cd56cc384b..cb74715ca6 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -83,4 +83,17 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384511","messageId":"55c306a73a5e100b56b184c1ce5428c1259cfc2a.1571666186.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:10Z","receivedAt":"2019-10-21T13:56:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na more restricted pattern set.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 89 +++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 45 ++++++++++++++\n 9 files changed, 226 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 89b3b79c1a..aebe7c0908 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -158,6 +158,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..9d6ca22917\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,89 @@\n+git-sparse-checkout(1)\n+======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of directories\n+given by a list of prefixes.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of directories given by a list of prefixes.\n+\n+THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n+COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n+THE FUTURE.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+---------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it updates the skip-worktree bits in the index based\n+on this file. The files matching the patterns in the file will\n+appear in the working directory, and the rest will not.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are included, you can also specify what files are _not_ included,\n+using negative patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+checkout' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex de60c8e7aa..adefc229fe 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex 5cf5df69f7..2b25a80cde 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..5717c9b2cb\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,86 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout list\"),\n+\tNULL\n+};\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pl->nr; i++) {\n+\t\tstruct path_pattern *p = pl->patterns[i];\n+\n+\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", p->pattern);\n+\n+\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_patterns_to_file(stdout, &pl);\n+\tclear_pattern_list(&pl);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex a9ac72bef4..d3d28252b3 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -166,6 +166,7 @@ git-show-index plumbinginterrogators\n git-show-ref plumbinginterrogators\n git-sh-i18n purehelpers\n git-sh-setup purehelpers\n+git-sparse-checkout mainporcelain worktree\n git-stash mainporcelain\n git-stage complete\n git-status mainporcelain info\ndiff --git a/git.c b/git.c\nindex ce6ab0ece2..7be7ad34bd 100644\n--- a/git.c\n+++ b/git.c\n@@ -572,6 +572,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..9b73d44907\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_must_be_empty list &&\n+\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\tgit -C repo sparse-checkout list >list &&\n+\ttest_cmp expect list\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"384513","messageId":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v4.git.1571147764.gitgitgadget@gmail.com","subject":"[PATCH v5 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:09Z","receivedAt":"2019-10-21T13:56:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"V4 UPDATE: Rebased on latest master to include ew/hashmap and\nds/include-exclude in the base.\n\nThis series makes the sparse-checkout feature more user-friendly. While\nthere, I also present a way to use a limited set of patterns to gain a\nsignificant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis series attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout set reads patterns from the arguments, or with\n --stdin reads patterns from stdin one per line, then writes them to the\n sparse-checkout file and refreshes the working directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\nUpdates in v2, relative to the RFC:\n\n * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n adding 'add' and/or 'remove' subcommands later.\n \n \n * 'set' reads from the arguments by default. '--stdin' option is available.\n \n \n * A new performance-oriented commit is added at the end.\n \n \n * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n pattern.\n \n \n * References to a \"bug\" (that was really a strange GVFS interaction in\n microsoft/git) around deleting outside the cone are removed.\n \n \n\nUpdates in v3:\n\n * The bad interaction with \"cone mode\" and .gitignore files is fixed. A\n test is added in the last patch.\n \n \n * Several patches are added that make the feature more robust. One\n sanitizes user input, another few add progress indicators, and another\n more prevent users from getting in bad states due to working directory\n changes or concurrent processes.\n \n \n * Updated several docs and commit messages according to feedback. Thanks,\n Elijah!\n \n \n\nUpdates in V4:\n\n * Updated hashmap API usage to respond to ew/hashmap\n \n \n * Responded to detailed review by Elijah. Thanks!\n \n \n * Marked the feature as experimental in git-sparse-checkout.txt the same\n way that git-switch.txt does.\n \n \n\nUpdates in V5:\n\n * The 'set' subcommand now enables the core.sparseCheckout config setting\n (unless the checkout fails).\n \n \n * If the in-process unpack_trees() fails with the new patterns, the\n index.lock file is rolled back before the replay of the old\n sparse-checkout patterns.\n \n \n * Some documentation fixes, f(d)open->xf(d)open calls, and other nits.\n Thanks everyone!\n \n \n\nThings to leave for future patches:\n\n 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n worktree-specific file.\n \n \n 2. More robustness around detecting non-cone patterns with wildcards in the\n middle of the line.\n \n \n 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n super-valuable, as it only starts changing behavior when someone calls\n 'git sparse-checkout set', but may be interesting.\n \n \n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (16):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'set' subcommand\n sparse-checkout: add '--stdin' option to set subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and set in cone mode\n unpack-trees: hash less in cone mode\n unpack-trees: add progress to clear_ce_flags()\n read-tree: show progress by default\n sparse-checkout: sanitize for nested folders\n sparse-checkout: update working directory in-process\n sparse-checkout: write using lockfile\n sparse-checkout: cone mode should not interact with .gitignore\n\nJeff Hostetler (1):\n trace2: add region in clear_ce_flags\n\n .gitignore | 1 +\n Documentation/config/core.txt | 10 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 153 ++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/read-tree.c | 3 +-\n builtin/sparse-checkout.c | 488 ++++++++++++++++++++++++++\n cache.h | 6 +-\n command-list.txt | 1 +\n config.c | 5 +\n dir.c | 207 ++++++++++-\n dir.h | 36 ++\n environment.c | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 293 ++++++++++++++++\n unpack-trees.c | 110 ++++--\n unpack-trees.h | 3 +-\n 20 files changed, 1309 insertions(+), 48 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 108b97dc372828f0e72e56bbb40cae8e1e83ece6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n\nRange-diff vs v4:\n\n 1: 63626e1097 ! 1: 55c306a73a sparse-checkout: create builtin with 'list' subcommand\n @@ -54,7 +54,7 @@\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n +git-sparse-checkout(1)\n -+=======================\n ++======================\n +\n +NAME\n +----\n @@ -75,7 +75,9 @@\n +Initialize and modify the sparse-checkout configuration, which reduces\n +the checkout to a set of directories given by a list of prefixes.\n +\n -+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n ++THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n ++COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n ++THE FUTURE.\n +\n +\n +COMMANDS\n @@ -85,7 +87,7 @@\n +\n +\n +SPARSE CHECKOUT\n -+----------------\n ++---------------\n +\n +\"Sparse checkout\" allows populating the working directory sparsely.\n +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n @@ -180,7 +182,7 @@\n +#include \"strbuf.h\"\n +\n +static char const * const builtin_sparse_checkout_usage[] = {\n -+\tN_(\"git sparse-checkout [list]\"),\n ++\tN_(\"git sparse-checkout list\"),\n +\tNULL\n +};\n +\n @@ -257,6 +259,18 @@\n +\t\t\t builtin_sparse_checkout_options);\n +}\n \n + diff --git a/command-list.txt b/command-list.txt\n + --- a/command-list.txt\n + +++ b/command-list.txt\n +@@\n + git-show-ref plumbinginterrogators\n + git-sh-i18n purehelpers\n + git-sh-setup purehelpers\n ++git-sparse-checkout mainporcelain worktree\n + git-stash mainporcelain\n + git-stage complete\n + git-status mainporcelain info\n +\n diff --git a/git.c b/git.c\n --- a/git.c\n +++ b/git.c\n @@ -301,7 +315,7 @@\n +\n +test_expect_success 'git sparse-checkout list (empty)' '\n +\tgit -C repo sparse-checkout list >list 2>err &&\n -+\ttest_line_count = 0 list &&\n ++\ttest_must_be_empty list &&\n +\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n +'\n +\n @@ -313,13 +327,8 @@\n +\t\t**/a\n +\t\t!*bin*\n +\tEOF\n ++\tcp repo/.git/info/sparse-checkout expect &&\n +\tgit -C repo sparse-checkout list >list &&\n -+\tcat >expect <<-EOF &&\n -+\t\t/folder1/*\n -+\t\t/deep/\n -+\t\t**/a\n -+\t\t!*bin*\n -+\tEOF\n +\ttest_cmp expect list\n +'\n +\n 2: 65d26de1c2 ! 2: a161cee0df sparse-checkout: create 'init' subcommand\n @@ -37,7 +37,7 @@\n +`core.sparseCheckout` setting in the worktree-specific config file.\n \n SPARSE CHECKOUT\n - ----------------\n + ---------------\n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n @@ -46,8 +46,8 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n --\tN_(\"git sparse-checkout [list]\"),\n -+\tN_(\"git sparse-checkout [init|list]\"),\n +-\tN_(\"git sparse-checkout list\"),\n ++\tN_(\"git sparse-checkout (init|list)\"),\n \tNULL\n };\n \n @@ -121,7 +121,7 @@\n +\t}\n +\n +\t/* initial mode: all blobs at root */\n -+\tfp = fopen(sparse_filename, \"w\");\n ++\tfp = xfopen(sparse_filename, \"w\");\n +\tfree(sparse_filename);\n +\tfprintf(fp, \"/*\\n!/*/\\n\");\n +\tfclose(fp);\n 3: e59ed7128f = 3: cd150853b7 clone: add --sparse mode\n 4: 502b4b08f0 ! 4: 7d9d66a89f sparse-checkout: 'set' subcommand\n @@ -11,6 +11,11 @@\n extracted from cmd_sparse_checkout() to make it easier to implement\n 'add' and/or 'remove' subcommands in the future.\n \n + If the core.sparseCheckout config setting is disabled, then enable\n + the config setting in the worktree config. If we set the config\n + this way and the sparse-checkout fails, then re-disable the config\n + setting.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n @@ -23,10 +28,11 @@\n +'set'::\n +\tWrite a set of patterns to the sparse-checkout file, as given as\n +\ta list of arguments following the 'set' subcommand. Update the\n -+\tworking directory to match the new patterns.\n ++\tworking directory to match the new patterns. Enable the\n ++\tcore.sparseCheckout config setting if it is not already enabled.\n +\n SPARSE CHECKOUT\n - ----------------\n + ---------------\n \n \n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n @@ -36,8 +42,8 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n --\tN_(\"git sparse-checkout [init|list]\"),\n -+\tN_(\"git sparse-checkout [init|list|set] \"),\n +-\tN_(\"git sparse-checkout (init|list)\"),\n ++\tN_(\"git sparse-checkout (init|list|set) \"),\n \tNULL\n };\n \n @@ -65,13 +71,23 @@\n +\tint i;\n +\tstruct pattern_list pl;\n +\tint result;\n ++\tint set_config = 0;\n +\tmemset(&pl, 0, sizeof(pl));\n +\n +\tfor (i = 1; i < argc; i++)\n +\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\n ++\tif (!core_apply_sparse_checkout) {\n ++\t\tsc_set_config(MODE_ALL_PATTERNS);\n ++\t\tcore_apply_sparse_checkout = 1;\n ++\t\tset_config = 1;\n ++\t}\n ++\n +\tresult = write_patterns_and_update(&pl);\n +\n ++\tif (result && set_config)\n ++\t\tsc_set_config(MODE_NO_PATTERNS);\n ++\n +\tclear_pattern_list(&pl);\n +\treturn result;\n +}\n @@ -96,6 +112,19 @@\n \ttest_cmp expect dir\n '\n \n ++test_expect_success 'set enables config' '\n ++\tgit init empty-config &&\n ++\t(\n ++\t\tcd empty-config &&\n ++\t\ttest_commit test file &&\n ++\t\ttest_path_is_missing .git/config.worktree &&\n ++\t\ttest_must_fail git sparse-checkout set nothing &&\n ++\t\ttest_i18ngrep \"sparseCheckout = false\" .git/config.worktree &&\n ++\t\tgit sparse-checkout set \"/*\" &&\n ++\t\ttest_i18ngrep \"sparseCheckout = true\" .git/config.worktree\n ++\t)\n ++'\n ++\n +test_expect_success 'set sparse-checkout using builtin' '\n +\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n +\tcat >expect <<-EOF &&\n 5: 2852cf8e11 ! 5: 0e08898dcb sparse-checkout: add '--stdin' option to set subcommand\n @@ -14,18 +14,6 @@\n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n -@@\n - \tchar *sparse_filename;\n - \tFILE *fp;\n - \n -+\tif (!core_apply_sparse_checkout) {\n -+\t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n -+\t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n -+\t}\n -+\n - \tsparse_filename = get_sparse_checkout_filename();\n - \tfp = fopen(sparse_filename, \"w\");\n - \twrite_patterns_to_file(fp, pl);\n @@\n \treturn update_working_directory();\n }\n @@ -42,9 +30,10 @@\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tstatic const char *empty_base = \"\";\n - \tint i;\n +@@\n \tstruct pattern_list pl;\n \tint result;\n + \tint set_config = 0;\n +\n +\tstatic struct option builtin_sparse_checkout_set_options[] = {\n +\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n @@ -74,8 +63,8 @@\n +\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\t}\n \n - \tresult = write_patterns_and_update(&pl);\n - \n + \tif (!core_apply_sparse_checkout) {\n + \t\tsc_set_config(MODE_ALL_PATTERNS);\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n @@ -84,20 +73,6 @@\n \ttest_cmp expect dir\n '\n \n -+test_expect_success 'warn if core.sparseCheckout is disabled' '\n -+\ttest_when_finished git -C repo config --worktree core.sparseCheckout true &&\n -+\tgit -C repo config --worktree core.sparseCheckout false &&\n -+\tgit -C repo sparse-checkout set folder1 2>err &&\n -+\ttest_i18ngrep \"core.sparseCheckout is disabled\" err\n -+'\n -+\n - test_expect_success 'set sparse-checkout using builtin' '\n - \tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n - \tcat >expect <<-EOF &&\n -@@\n - \ttest_cmp expect dir\n - '\n - \n +test_expect_success 'set sparse-checkout using --stdin' '\n +\tcat >expect <<-EOF &&\n +\t\t/*\n 6: 55f95f290e ! 6: c48535cd5c sparse-checkout: create 'disable' subcommand\n @@ -13,15 +13,15 @@\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n - \ta list of arguments following the 'set' subcommand. Update the\n - \tworking directory to match the new patterns.\n + \tworking directory to match the new patterns. Enable the\n + \tcore.sparseCheckout config setting if it is not already enabled.\n \n +'disable'::\n +\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n +\t`false`, and restore the working directory to include all files.\n +\n SPARSE CHECKOUT\n - ----------------\n + ---------------\n \n @@\n on this file. The files matching the patterns in the file will\n @@ -68,8 +68,8 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n --\tN_(\"git sparse-checkout [init|list|set] \"),\n -+\tN_(\"git sparse-checkout [init|list|set|disable] \"),\n +-\tN_(\"git sparse-checkout (init|list|set) \"),\n ++\tN_(\"git sparse-checkout (init|list|set|disable) \"),\n \tNULL\n };\n \n @@ -86,7 +86,7 @@\n +\t\tdie(_(\"failed to change config\"));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n -+\tfp = fopen(sparse_filename, \"w\");\n ++\tfp = xfopen(sparse_filename, \"w\");\n +\tfprintf(fp, \"/*\\n\");\n +\tfclose(fp);\n +\n 7: 3d8f2f2007 = 7: 44b0245e65 trace2: add region in clear_ce_flags\n 8: 03dc0ed716 = 8: 121d878882 sparse-checkout: add 'cone' mode\n 9: 28606a152c ! 9: ff2a59bfff sparse-checkout: use hashmaps for cone patterns\n @@ -22,9 +22,9 @@\n Running 'git read-tree -mu HEAD' on this file had the following\n performance:\n \n - core.sparseCheckout=false: 0.21 s (0.00 s)\n - core.sparseCheckout=true: 3.75 s (3.50 s)\n - core.sparseCheckout=cone: 0.23 s (0.01 s)\n + core.sparseCheckout=false: 0.21 s (0.00 s)\n + core.sparseCheckout=true: 3.75 s (3.50 s)\n + core.sparseCheckoutCone=true: 0.23 s (0.01 s)\n \n The times in parentheses above correspond to the time spent\n in the first clear_ce_flags() call, according to the trace2\n 10: 2ea3d8819b ! 10: 0258ee8026 sparse-checkout: init and set in cone mode\n @@ -32,7 +32,7 @@\n +#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n - \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n + \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n @@\n enum sparse_checkout_mode {\n \tMODE_NO_PATTERNS = 0,\n @@ -250,7 +250,7 @@\n +\t\t}\n \t}\n \n - \tresult = write_patterns_and_update(&pl);\n + \tif (!core_apply_sparse_checkout) {\n \n diff --git a/dir.c b/dir.c\n --- a/dir.c\n 11: fefd1e1744 = 11: 7b5c5bad2d unpack-trees: hash less in cone mode\n 12: 034fbd31bd = 12: 386f4f8eb5 unpack-trees: add progress to clear_ce_flags()\n 13: 27aa9d22f0 = 13: a229e1ee0c read-tree: show progress by default\n 14: 44c3078029 = 14: f03bb23305 sparse-checkout: sanitize for nested folders\n 15: 9ccec3ca9a ! 15: d7af75672b sparse-checkout: update working directory in-process\n @@ -14,6 +14,12 @@\n reading from the sparse-checkout file. This allows us to test a\n proposed change to the file before writing to it.\n \n + An earlier version of this patch included a bug when the 'set' command\n + failed due to the \"Sparse checkout leaves no entry on working directory\"\n + error. It would not rollback the index.lock file, so the replay of the\n + old sparse-checkout specification would fail. A test in t1091 now\n + covers that scenario.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n @@ -43,7 +49,7 @@\n +#include \"unpack-trees.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n - \tN_(\"git sparse-checkout [init|list|set|disable] \"),\n + \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n @@\n \treturn 0;\n }\n @@ -64,6 +70,7 @@\n -\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n -\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n -\t\tresult = 1;\n +-\t}\n +\tif (repo_read_index_unmerged(r))\n +\t\tdie(_(\"You need to resolve your current index first\"));\n +\n @@ -99,7 +106,8 @@\n +\tif (!result) {\n +\t\tprime_cache_tree(r, r->index, tree);\n +\t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n - \t}\n ++\t} else\n ++\t\trollback_lock_file(&lock_file);\n \n -\targv_array_clear(&argv);\n \treturn result;\n @@ -133,12 +141,7 @@\n \tchar *sparse_filename;\n \tFILE *fp;\n +\tint result;\n - \n - \tif (!core_apply_sparse_checkout) {\n - \t\twarning(_(\"core.sparseCheckout is disabled, so changes to the sparse-checkout file will have no effect\"));\n - \t\twarning(_(\"run 'git sparse-checkout init' to enable the sparse-checkout feature\"));\n - \t}\n - \n ++\n +\tresult = update_working_directory(pl);\n +\n +\tif (result) {\n @@ -146,10 +149,9 @@\n +\t\tupdate_working_directory(NULL);\n +\t\treturn result;\n +\t}\n -+\n + \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n - \n @@\n \t\twrite_patterns_to_file(fp, pl);\n \n @@ -203,6 +205,19 @@\n +\tEOF\n +\ttest_cmp dir expect\n +'\n ++\n ++test_expect_success 'revert to old sparse-checkout on empty update' '\n ++\tgit init empty-test &&\n ++\t(\n ++\t\techo >file &&\n ++\t\tgit add file &&\n ++\t\tgit commit -m \"test\" &&\n ++\t\ttest_must_fail git sparse-checkout set nothing 2>err &&\n ++\t\ttest_i18ngrep \"Sparse checkout leaves no entry on working directory\" err &&\n ++\t\ttest_i18ngrep ! \".git/index.lock\" err &&\n ++\t\tgit sparse-checkout set file\n ++\t)\n ++'\n +\n test_done\n \n 16: d0421ef7b2 ! 16: 88eff318e0 sparse-checkout: write using lockfile\n @@ -24,9 +24,6 @@\n +\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \n - \tif (!core_apply_sparse_checkout) {\n -@@\n - \n \tresult = update_working_directory(pl);\n \n +\tsparse_filename = get_sparse_checkout_filename();\n @@ -44,7 +41,7 @@\n \n -\tsparse_filename = get_sparse_checkout_filename();\n -\tfp = fopen(sparse_filename, \"w\");\n -+\tfp = fdopen(fd, \"w\");\n ++\tfp = xfdopen(fd, \"w\");\n \n \tif (core_sparse_checkout_cone)\n \t\twrite_cone_to_file(fp, pl);\n @@ -62,7 +59,7 @@\n --- a/t/t1091-sparse-checkout-builtin.sh\n +++ b/t/t1091-sparse-checkout-builtin.sh\n @@\n - \ttest_cmp dir expect\n + \t)\n '\n \n +test_expect_success 'fail when lock is taken' '\n 17: ed1f148763 = 17: ab20c11d71 sparse-checkout: cone mode should not interact with .gitignore\n\n-- \ngitgitgadget\n"},{"id":"384512","messageId":"0e08898dcb42bd38ca3692b49a7e9f5763150c80.1571666186.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:14Z","receivedAt":"2019-10-21T13:56:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nand places them in the sparse-checkout file. Then, it updates the\nworking directory to match those patterns. For a large list of\npatterns, the command-line call can get very cumbersome.\n\nAdd a '--stdin' option to instead read patterns over standard in.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n 2 files changed, 53 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 834ee421f0..f2e2bd772d 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -154,6 +154,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static char const * const builtin_sparse_checkout_set_usage[] = {\n+\tN_(\"git sparse-checkout set [--stdin|]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_set_opts {\n+\tint use_stdin;\n+} set_opts;\n+\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tstatic const char *empty_base = \"\";\n@@ -161,10 +170,32 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \tstruct pattern_list pl;\n \tint result;\n \tint set_config = 0;\n+\n+\tstatic struct option builtin_sparse_checkout_set_options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n+\t\t\t N_(\"read patterns from standard in\")),\n+\t\tOPT_END(),\n+\t};\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n-\tfor (i = 1; i < argc; i++)\n-\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_set_options,\n+\t\t\t builtin_sparse_checkout_set_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (set_opts.use_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tsize_t len;\n+\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t}\n \n \tif (!core_apply_sparse_checkout) {\n \t\tsc_set_config(MODE_ALL_PATTERNS);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex bf2dc55bb1..a9ff5eb9ec 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using --stdin' '\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/folder1/\n+\t\t/folder2/\n+\tEOF\n+\tgit -C repo sparse-checkout set --stdin actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384515","messageId":"ff2a59bfffcac47071d40fb1fca36b4263698574.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 09/17] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:18Z","receivedAt":"2019-10-21T13:56:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/\n\t!/$folder/*/\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n core.sparseCheckout=false: 0.21 s (0.00 s)\n core.sparseCheckout=true: 3.75 s (3.50 s)\n core.sparseCheckoutCone=true: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n dir.c | 207 +++++++++++++++++++++++++++--\n dir.h | 31 +++++\n t/t1091-sparse-checkout-builtin.sh | 11 +-\n unpack-trees.c | 1 +\n 4 files changed, 241 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..dfabf9982f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const struct hashmap_entry *a,\n+\t\t\t const struct hashmap_entry *b,\n+\t\t\t const void *key)\n+{\n+\tconst struct pattern_entry *ee1 =\n+\t\t\tcontainer_of(a, struct pattern_entry, ent);\n+\tconst struct pattern_entry *ee2 =\n+\t\t\tcontainer_of(b, struct pattern_entry, ent);\n+\n+\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n+\t\t\t ? ee1->patternlen\n+\t\t\t : ee2->patternlen;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n+}\n+\n+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n+{\n+\tstruct pattern_entry *translated;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!pl->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE &&\n+\t given->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 0;\n+\t\treturn;\n+\t}\n+\n+\tif (!given->flags && !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 1;\n+\t\treturn;\n+\t}\n+\n+\tif (given->patternlen > 2 &&\n+\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n+\t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tpl->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(given->pattern);\n+\t\ttruncated[given->patternlen - 2] = 0;\n+\n+\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\t\ttranslated->pattern = truncated;\n+\t\ttranslated->patternlen = given->patternlen - 2;\n+\t\thashmap_entry_init(&translated->ent,\n+\t\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->recursive_hashmap,\n+\t\t\t\t translated, ent, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\t\tgiven->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tfree(translated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&pl->parent_hashmap, &translated->ent);\n+\t\thashmap_remove(&pl->recursive_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE) {\n+\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\tgiven->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n+\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\n+\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->patternlen = given->patternlen;\n+\thashmap_entry_init(&translated->ent,\n+\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, &translated->ent);\n+\n+\tif (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) {\n+\t\t/* we already included this at the parent level */\n+\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\tgiven->pattern);\n+\t\thashmap_remove(&pl->parent_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\tfree(translated);\n+\t}\n+\n+\treturn;\n+\n+clear_hashmaps:\n+\twarning(_(\"disabling cone pattern matching\"));\n+\thashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent);\n+\thashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent);\n+\tpl->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct pattern_entry p;\n+\n+\t/* Check straight mapping */\n+\tp.pattern = pattern->buf;\n+\tp.patternlen = pattern->len;\n+\thashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen));\n+\treturn !!hashmap_get_entry(map, &p, ent, NULL);\n+}\n+\n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer)\n+{\n+\tchar *slash_pos;\n+\n+\tstrbuf_setlen(buffer, 0);\n+\n+\tif (path[0] != '/')\n+\t\tstrbuf_addch(buffer, '/');\n+\n+\tstrbuf_addstr(buffer, path);\n+\n+\tslash_pos = strrchr(buffer->buf, '/');\n+\n+\twhile (slash_pos > buffer->buf) {\n+\t\tstrbuf_setlen(buffer, slash_pos - buffer->buf);\n+\n+\t\tif (hashmap_contains_path(map, buffer))\n+\t\t\treturn 1;\n+\n+\t\tslash_pos = strrchr(buffer->buf, '/');\n+\t}\n+\n+\treturn 0;\n+}\n+\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n {\n@@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base,\n \tALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);\n \tpl->patterns[pl->nr++] = pattern;\n \tpattern->pl = pl;\n+\n+\tadd_pattern_to_hashsets(pl, pattern);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n \tpl->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list(\n \t\t\t\tstruct index_state *istate)\n {\n \tstruct path_pattern *pattern;\n-\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, pl, istate);\n-\tif (pattern) {\n-\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n-\t\t\treturn NOT_MATCHED;\n-\t\telse\n-\t\t\treturn MATCHED;\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = NOT_MATCHED;\n+\tconst char *slash_pos;\n+\n+\tif (!pl->use_cone_patterns) {\n+\t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\tdtype, pl, istate);\n+\t\tif (pattern) {\n+\t\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\t\treturn NOT_MATCHED;\n+\t\t\telse\n+\t\t\t\treturn MATCHED;\n+\t\t}\n+\n+\t\treturn UNDECIDED;\n+\t}\n+\n+\tif (pl->full_cone)\n+\t\treturn MATCHED;\n+\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\n+\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n \t}\n \n-\treturn UNDECIDED;\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tif (hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t pathname,\n+\t\t\t\t &parent_pathname))\n+\t\tresult = MATCHED;\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct path_pattern *last_matching_pattern_from_lists(\ndiff --git a/dir.h b/dir.h\nindex 2fbdef014f..f8edbca72b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct path_pattern {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct pattern_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,26 @@ struct pattern_list {\n \tconst char *src;\n \n \tstruct path_pattern **patterns;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\tunsigned full_cone;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\n@@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer);\n struct pattern_list *add_pattern_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src);\n int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b3058d0e9b..9907278fc1 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -151,7 +151,8 @@ test_expect_success 'set sparse-checkout using --stdin' '\n test_expect_success 'cone mode: match patterns' '\n \tgit -C repo config --worktree core.sparseCheckoutCone true &&\n \trm -rf repo/a repo/folder1 repo/folder2 &&\n-\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n \tls repo >dir &&\n \tcat >expect <<-EOF &&\n@@ -162,6 +163,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info/ &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 01a05ff66d..a90d71845d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n \t\t\to->skip_sparse_checkout = 1;\n \t\telse\n-- \ngitgitgadget\n\n"},{"id":"384516","messageId":"7b5c5bad2d58041f634690e2e5581feaa4d0fd02.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 11/17] unpack-trees: hash less in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:20Z","receivedAt":"2019-10-21T13:56:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature in \"cone mode\" can use the fact that\nthe recursive patterns are \"connected\" to the root via parent\npatterns to decide if a directory is entirely contained in the\nsparse-checkout or entirely removed.\n\nIn these cases, we can skip hashing the paths within those\ndirectories and simply set the skipworktree bit to the correct\nvalue.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 4 ++--\n dir.h | 1 +\n unpack-trees.c | 38 +++++++++++++++++++++++---------------\n 3 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 35c1ca9e24..2ef92a50a0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list(\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n@@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tif (hashmap_contains_parent(&pl->recursive_hashmap,\n \t\t\t\t pathname,\n \t\t\t\t &parent_pathname))\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \n done:\n \tstrbuf_release(&parent_pathname);\ndiff --git a/dir.h b/dir.h\nindex 8e232085cd..77a43dbf89 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -264,6 +264,7 @@ enum pattern_match_result {\n \tUNDECIDED = -1,\n \tNOT_MATCHED = 0,\n \tMATCHED = 1,\n+\tMATCHED_RECURSIVE = 2,\n };\n \n /*\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a90d71845d..c0dca20865 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n \tint rc;\n-\tenum pattern_match_result ret;\n-\tret = path_matches_pattern_list(prefix->buf, prefix->len,\n-\t\t\t\t\tbasename, &dtype, pl, istate);\n+\tenum pattern_match_result ret, orig_ret;\n+\torig_ret = path_matches_pattern_list(prefix->buf, prefix->len,\n+\t\t\t\t\t basename, &dtype, pl, istate);\n \n \tstrbuf_addch(prefix, '/');\n \n \t/* If undecided, use matching result of parent dir in defval */\n-\tif (ret == UNDECIDED)\n+\tif (orig_ret == UNDECIDED)\n \t\tret = default_match;\n+\telse\n+\t\tret = orig_ret;\n \n \tfor (cache_end = cache; cache_end != cache + nr; cache_end++) {\n \t\tstruct cache_entry *ce = *cache_end;\n@@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\tbreak;\n \t}\n \n-\t/*\n-\t * TODO: check pl, if there are no patterns that may conflict\n-\t * with ret (iow, we know in advance the incl/excl\n-\t * decision for the entire directory), clear flag here without\n-\t * calling clear_ce_flags_1(). That function will call\n-\t * the expensive path_matches_pattern_list() on every entry.\n-\t */\n-\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t prefix,\n-\t\t\t select_mask, clear_mask,\n-\t\t\t pl, ret);\n+\tif (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {\n+\t\tstruct cache_entry **ce = cache;\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\n+\t\twhile (ce < cache_end) {\n+\t\t\t(*ce)->ce_flags &= ~clear_mask;\n+\t\t\tce++;\n+\t\t}\n+\t} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\t} else {\n+\t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t prefix,\n+\t\t\t\t select_mask, clear_mask,\n+\t\t\t\t pl, ret);\n+\t}\n+\n \tstrbuf_setlen(prefix, prefix->len - 1);\n \treturn rc;\n }\n-- \ngitgitgadget\n\n"},{"id":"384514","messageId":"121d878882609907cc8d13f9d76f2414b15618be.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 08/17] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:17Z","receivedAt":"2019-10-21T13:56:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new Boolean config option, core.sparseCheckoutCone, to\nindicate that we expect the sparse-checkout file to contain a\nmore limited set of patterns. This is a separate config setting\nfrom core.sparseCheckout to avoid breaking older clients by\nintroducing a tri-state option.\n\nThe config option does nothing right now, but will be expanded\nupon in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 10 ++++--\n Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++\n cache.h | 4 ++-\n config.c | 5 +++\n environment.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 14 ++++++++\n 6 files changed, 81 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 852d2ba37a..bdbbee58b9 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -593,8 +593,14 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\n+\tfor more information.\n+\n+core.sparseCheckoutCone::\n+\tEnables the \"cone mode\" of the sparse checkout feature. When the\n+\tsparse-checkout file contains a limited set of patterns, then this\n+\tmode provides significant performance advantages. See\n+\tlinkgit:git-sparse-checkout[1] for more information.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex f794d4797a..3931e4f2ad 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -92,6 +92,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+## CONE PATTERN SET\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+```\n+/*\n+!/*/\n+```\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+```\n+/*\n+!/*/\n+/A/\n+!/A/*/\n+/A/B/\n+!/A/B/*/\n+/A/B/C/\n+```\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/cache.h b/cache.h\nindex 04cabaac11..4980ee198e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -918,12 +918,14 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex e7052b3977..d75f88ca0c 100644\n--- a/config.c\n+++ b/config.c\n@@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n+\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex efa072680a..2a1a866659 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 583c232e9e..b3058d0e9b 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -148,6 +148,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"384517","messageId":"44b0245e65f54cbcec743184886d64a6b5ad87cf.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 07/17] trace2: add region in clear_ce_flags","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:16Z","receivedAt":"2019-10-21T13:56:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nWhen Git updates the working directory with the sparse-checkout\nfeature enabled, the unpack_trees() method calls clear_ce_flags()\nto update the skip-wortree bits on the cache entries. This\ncheck can be expensive, depending on the patterns used.\n\nAdd trace2 regions around the method, including some flag\ninformation, so we can get granular performance data during\nexperiments. This data will be used to measure improvements\nto the pattern-matching algorithms for sparse-checkout.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 33ea7810d8..01a05ff66d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tpl, 0);\n+\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"384518","messageId":"386f4f8eb5d12d9d089684841d57ea66447cf675.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 12/17] unpack-trees: add progress to clear_ce_flags()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:21Z","receivedAt":"2019-10-21T13:56:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a large repository has many sparse-checkout patterns, the\nprocess for updating the skip-worktree bits can take long enough\nthat a user gets confused why nothing is happening. Update the\nclear_ce_flags() method to write progress.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 2 ++\n unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++--------------\n 2 files changed, 43 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4980ee198e..d3c89e7a53 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n struct split_index;\n struct untracked_cache;\n+struct progress;\n \n struct index_state {\n \tstruct cache_entry **cache;\n@@ -326,6 +327,7 @@ struct index_state {\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n \tstruct mem_pool *ce_mem_pool;\n+\tstruct progress *progress;\n };\n \n /* Name hashing */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex c0dca20865..8bb684ad62 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1269,7 +1269,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match);\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr);\n \n /* Whole directory matching */\n static int clear_ce_flags_dir(struct index_state *istate,\n@@ -1278,7 +1279,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\t char *basename,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n@@ -1315,7 +1317,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n \t\t\t\t prefix,\n \t\t\t\t select_mask, clear_mask,\n-\t\t\t\t pl, ret);\n+\t\t\t\t pl, ret,\n+\t\t\t\t progress_nr);\n \t}\n \n \tstrbuf_setlen(prefix, prefix->len - 1);\n@@ -1342,7 +1345,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end = cache + nr;\n \n@@ -1356,8 +1360,11 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tint len, dtype;\n \t\tenum pattern_match_result ret;\n \n+\t\tdisplay_progress(istate->progress, progress_nr);\n+\n \t\tif (select_mask && !(ce->ce_flags & select_mask)) {\n \t\t\tcache++;\n+\t\t\tprogress_nr++;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1378,20 +1385,26 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t\t\t\t prefix,\n \t\t\t\t\t\t prefix->buf + prefix->len - len,\n \t\t\t\t\t\t select_mask, clear_mask,\n-\t\t\t\t\t\t pl, default_match);\n+\t\t\t\t\t\t pl, default_match,\n+\t\t\t\t\t\t progress_nr);\n \n \t\t\t/* clear_c_f_dir eats a whole dir already? */\n \t\t\tif (processed) {\n \t\t\t\tcache += processed;\n+\t\t\t\tprogress_nr += processed;\n \t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n \t\t\t\tcontinue;\n \t\t\t}\n \n \t\t\tstrbuf_addch(prefix, '/');\n-\t\t\tcache += clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t\t\t\t prefix,\n-\t\t\t\t\t\t select_mask, clear_mask, pl,\n-\t\t\t\t\t\t default_match);\n+\t\t\tprocessed = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t\t\t prefix,\n+\t\t\t\t\t\t select_mask, clear_mask, pl,\n+\t\t\t\t\t\t default_match, progress_nr);\n+\n+\t\t\tcache += processed;\n+\t\t\tprogress_nr += processed;\n+\n \t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n \t\t\tcontinue;\n \t\t}\n@@ -1406,19 +1419,27 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tif (ret == MATCHED)\n \t\t\tce->ce_flags &= ~clear_mask;\n \t\tcache++;\n+\t\tprogress_nr++;\n \t}\n+\n+\tdisplay_progress(istate->progress, progress_nr);\n \treturn nr - (cache_end - cache);\n }\n \n static int clear_ce_flags(struct index_state *istate,\n \t\t\t int select_mask, int clear_mask,\n-\t\t\t struct pattern_list *pl)\n+\t\t\t struct pattern_list *pl,\n+\t\t\t int show_progress)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n \tchar label[100];\n \tint rval;\n \n \tstrbuf_reset(&prefix);\n+\tif (show_progress)\n+\t\tistate->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Updating index flags\"),\n+\t\t\t\t\tistate->cache_nr);\n \n \txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n \t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n@@ -1428,9 +1449,10 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n-\t\t\t\tpl, 0);\n+\t\t\t\tpl, 0, 0);\n \ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n \n+\tstop_progress(&istate->progress);\n \treturn rval;\n }\n \n@@ -1439,7 +1461,8 @@ static int clear_ce_flags(struct index_state *istate,\n */\n static void mark_new_skip_worktree(struct pattern_list *pl,\n \t\t\t\t struct index_state *istate,\n-\t\t\t\t int select_flag, int skip_wt_flag)\n+\t\t\t\t int select_flag, int skip_wt_flag,\n+\t\t\t\t int show_progress)\n {\n \tint i;\n \n@@ -1463,7 +1486,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \t * 2. Widen worktree according to sparse-checkout file.\n \t * Matched entries will have skip_wt_flag cleared (i.e. \"in\")\n \t */\n-\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl);\n+\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n static int verify_absent(const struct cache_entry *,\n@@ -1525,7 +1548,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n \t */\n \tif (!o->skip_sparse_checkout)\n-\t\tmark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n \n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n@@ -1590,7 +1614,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n-\t\tmark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, &o->result,\n+\t\t\t\t CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE,\n+\t\t\t\t o->verbose_update);\n \n \t\tret = 0;\n \t\tfor (i = 0; i < o->result.cache_nr; i++) {\n-- \ngitgitgadget\n\n"},{"id":"384519","messageId":"0258ee80265f5f27a7de9b81eaf166648b4511d4.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 10/17] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:19Z","receivedAt":"2019-10-21T13:56:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout [init|set]'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckoutCone=true'.\n\nWhen running 'git sparse-checkout set' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nWhen testing 'git sparse-checkout set' in cone mode, check the\nerror stream to ensure we do not see any errors. Specifically,\nwe want to avoid the warning that the patterns do not match\nthe cone-mode patterns.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 159 +++++++++++++++++++++++++++--\n dir.c | 8 +-\n dir.h | 4 +\n t/t1091-sparse-checkout-builtin.sh | 51 +++++++++\n 4 files changed, 208 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 9fdcc6c4ef..6c3bf68c89 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,6 +6,7 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n@@ -77,11 +78,13 @@ static int update_working_directory(void)\n enum sparse_checkout_mode {\n \tMODE_NO_PATTERNS = 0,\n \tMODE_ALL_PATTERNS = 1,\n+\tMODE_CONE_PATTERNS = 2,\n };\n \n static int sc_set_config(enum sparse_checkout_mode mode)\n {\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n \n \tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n \t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n@@ -100,9 +103,31 @@ static int sc_set_config(enum sparse_checkout_mode mode)\n \t\treturn 1;\n \t}\n \n+\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n+\t\t\t \"core.sparseCheckoutCone\", NULL);\n+\n+\tif (mode == MODE_CONE_PATTERNS)\n+\t\targv_array_push(&cone_argv, \"true\");\n+\telse\n+\t\targv_array_push(&cone_argv, \"false\");\n+\n+\tif (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to enable core.sparseCheckoutCone\"));\n+\t\treturn 1;\n+\t}\n+\n \treturn 0;\n }\n \n+static char const * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_init_opts {\n+\tint cone_mode;\n+} init_opts;\n+\n static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -110,8 +135,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n+\tint mode;\n \n-\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\n+\tif (sc_set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -140,6 +178,70 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n+{\n+\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, &e->ent);\n+\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (slash == e->pattern)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct pattern_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))\n+\t\t\thashmap_add(&pl->parent_hashmap, &e->ent);\n+\t}\n+}\n+\n+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\tstruct pattern_entry *pe;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (strlen(pattern))\n+\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t}\n+}\n+\n static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n@@ -147,13 +249,33 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n-\twrite_patterns_to_file(fp, pl);\n+\n+\tif (core_sparse_checkout_cone)\n+\t\twrite_cone_to_file(fp, pl);\n+\telse\n+\t\twrite_patterns_to_file(fp, pl);\n+\n \tfclose(fp);\n \tfree(sparse_filename);\n \n \treturn update_working_directory();\n }\n \n+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n+{\n+\tstrbuf_trim(line);\n+\n+\tstrbuf_trim_trailing_dir_sep(line);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tif (line->buf[0] != '/')\n+\t\tstrbuf_insert(line, 0, \"/\", 1);\n+\n+\tinsert_recursive_pattern(pl, line);\n+}\n+\n static char const * const builtin_sparse_checkout_set_usage[] = {\n \tN_(\"git sparse-checkout set [--stdin|]\"),\n \tNULL\n@@ -184,17 +306,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n-\tif (set_opts.use_stdin) {\n+\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n-\n-\t\twhile (!strbuf_getline(&line, stdin)) {\n-\t\t\tsize_t len;\n-\t\t\tchar *buf = strbuf_detach(&line, &len);\n-\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n+\t\tif (set_opts.use_stdin) {\n+\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++) {\n+\t\t\t\tstrbuf_setlen(&line, 0);\n+\t\t\t\tstrbuf_addstr(&line, argv[i]);\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n \t\t}\n \t} else {\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\tif (set_opts.use_stdin) {\n+\t\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\t}\n \t}\n \n \tif (!core_apply_sparse_checkout) {\ndiff --git a/dir.c b/dir.c\nindex dfabf9982f..35c1ca9e24 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n-static int pl_hashmap_cmp(const void *unused_cmp_data,\n-\t\t\t const struct hashmap_entry *a,\n-\t\t\t const struct hashmap_entry *b,\n-\t\t\t const void *key)\n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key)\n {\n \tconst struct pattern_entry *ee1 =\n \t\t\tcontainer_of(a, struct pattern_entry, ent);\ndiff --git a/dir.h b/dir.h\nindex f8edbca72b..8e232085cd 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9907278fc1..ae99803d40 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -186,4 +186,55 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and set' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n+\ttest_line_count = 0 err &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\t\t!/deep/*/\n+\t\t/deep/deeper1/\n+\t\t!/deep/deeper1/*/\n+\t\t/deep/deeper1/deepest/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\tls repo >dir &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384521","messageId":"88eff318e04165e23406098bfdd61d52bdc604cb.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 16/17] sparse-checkout: write using lockfile","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:25Z","receivedAt":"2019-10-21T13:56:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf two 'git sparse-checkout set' subcommands are launched at the\nsame time, the behavior can be unexpected as they compete to write\nthe sparse-checkout file and update the working directory.\n\nTake a lockfile around the writes to the sparse-checkout file. In\naddition, acquire this lock around the working directory update\nto avoid two commands updating the working directory in different\nways.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 15 ++++++++++++---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 2 files changed, 19 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex ea2e7ee51b..76f65d8edd 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -306,25 +306,34 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \n \tresult = update_working_directory(pl);\n \n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfd = hold_lock_file_for_update(&lk, sparse_filename,\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\n+\tresult = update_working_directory(pl);\n \tif (result) {\n+\t\trollback_lock_file(&lk);\n+\t\tfree(sparse_filename);\n \t\tclear_pattern_list(pl);\n \t\tupdate_working_directory(NULL);\n \t\treturn result;\n \t}\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = fopen(sparse_filename, \"w\");\n+\tfp = xfdopen(fd, \"w\");\n \n \tif (core_sparse_checkout_cone)\n \t\twrite_cone_to_file(fp, pl);\n \telse\n \t\twrite_patterns_to_file(fp, pl);\n \n-\tfclose(fp);\n+\tfflush(fp);\n+\tcommit_lock_file(&lk);\n \n \tfree(sparse_filename);\n \tclear_pattern_list(pl);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a580b79595..2db706a9e4 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -276,4 +276,11 @@ test_expect_success 'revert to old sparse-checkout on empty update' '\n \t)\n '\n \n+test_expect_success 'fail when lock is taken' '\n+\ttest_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&\n+\ttouch repo/.git/info/sparse-checkout.lock &&\n+\ttest_must_fail git -C repo sparse-checkout set deep 2>err &&\n+\ttest_i18ngrep \"File exists\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384520","messageId":"d7af75672ba20ed208d51a35243b96ce10202a54.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 15/17] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:24Z","receivedAt":"2019-10-21T13:56:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\nskip-worktree bits in the index and to update the working directory.\nThis extra process is overly complex, and prone to failure. It also\nrequires that we write our changes to the sparse-checkout file before\ntrying to update the index.\n\nRemove this extra process call by creating a direct call to\nunpack_trees() in the same way 'git read-tree -mu HEAD' does. In\naddition, provide an in-memory list of patterns so we can avoid\nreading from the sparse-checkout file. This allows us to test a\nproposed change to the file before writing to it.\n\nAn earlier version of this patch included a bug when the 'set' command\nfailed due to the \"Sparse checkout leaves no entry on working directory\"\nerror. It would not rollback the index.lock file, so the replay of the\nold sparse-checkout specification would fail. A test in t1091 now\ncovers that scenario.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 2 +-\n builtin/sparse-checkout.c | 83 +++++++++++++++++++++++++-----\n t/t1091-sparse-checkout-builtin.sh | 28 ++++++++++\n unpack-trees.c | 5 +-\n unpack-trees.h | 3 +-\n 5 files changed, 105 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex 69963d83dc..d7eeaa26ec 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \n \tif (opts.reset || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n-\t\t\tdie(\"You need to resolve your current index first\");\n+\t\t\tdie(_(\"You need to resolve your current index first\"));\n \t\tstage = opts.merge = 1;\n \t}\n \tresolve_undo_clear();\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex b02e085495..ea2e7ee51b 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -7,6 +7,11 @@\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"cache.h\"\n+#include \"cache-tree.h\"\n+#include \"lockfile.h\"\n+#include \"resolve-undo.h\"\n+#include \"unpack-trees.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n@@ -60,18 +65,54 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n-static int update_working_directory(void)\n+static int update_working_directory(struct pattern_list *pl)\n {\n-\tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n-\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\tstruct unpack_trees_options o;\n+\tstruct lock_file lock_file = LOCK_INIT;\n+\tstruct object_id oid;\n+\tstruct tree *tree;\n+\tstruct tree_desc t;\n+\tstruct repository *r = the_repository;\n \n-\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n-\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n-\t\tresult = 1;\n-\t}\n+\tif (repo_read_index_unmerged(r))\n+\t\tdie(_(\"You need to resolve your current index first\"));\n+\n+\tif (get_oid(\"HEAD\", &oid))\n+\t\treturn 0;\n+\n+\ttree = parse_tree_indirect(&oid);\n+\tparse_tree(tree);\n+\tinit_tree_desc(&t, tree->buffer, tree->size);\n+\n+\tmemset(&o, 0, sizeof(o));\n+\to.verbose_update = isatty(2);\n+\to.merge = 1;\n+\to.update = 1;\n+\to.fn = oneway_merge;\n+\to.head_idx = -1;\n+\to.src_index = r->index;\n+\to.dst_index = r->index;\n+\to.skip_sparse_checkout = 0;\n+\to.pl = pl;\n+\to.keep_pattern_list = !!pl;\n+\n+\tresolve_undo_clear_index(r->index);\n+\tsetup_work_tree();\n+\n+\tcache_tree_free(&r->index->cache_tree);\n+\n+\trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n+\n+\tcore_apply_sparse_checkout = 1;\n+\tresult = unpack_trees(1, &t, &o);\n+\n+\tif (!result) {\n+\t\tprime_cache_tree(r, r->index, tree);\n+\t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n+\t} else\n+\t\trollback_lock_file(&lock_file);\n \n-\targv_array_clear(&argv);\n \treturn result;\n }\n \n@@ -147,7 +188,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n-\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\tif (init_opts.cone_mode) {\n+\t\tmode = MODE_CONE_PATTERNS;\n+\t\tcore_sparse_checkout_cone = 1;\n+\t} else\n+\t\tmode = MODE_ALL_PATTERNS;\n \n \tif (sc_set_config(mode))\n \t\treturn 1;\n@@ -175,7 +220,8 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t}\n \n reset_dir:\n-\treturn update_working_directory();\n+\tcore_apply_sparse_checkout = 1;\n+\treturn update_working_directory(NULL);\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n@@ -260,6 +306,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint result;\n+\n+\tresult = update_working_directory(pl);\n+\n+\tif (result) {\n+\t\tclear_pattern_list(pl);\n+\t\tupdate_working_directory(NULL);\n+\t\treturn result;\n+\t}\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n@@ -270,9 +325,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \t\twrite_patterns_to_file(fp, pl);\n \n \tfclose(fp);\n+\n \tfree(sparse_filename);\n+\tclear_pattern_list(pl);\n \n-\treturn update_working_directory();\n+\treturn 0;\n }\n \n static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n@@ -324,6 +381,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tstruct strbuf line = STRBUF_INIT;\n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n \t\t\twhile (!strbuf_getline(&line, stdin))\n@@ -378,7 +436,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n\");\n \tfclose(fp);\n \n-\tif (update_working_directory())\n+\tcore_apply_sparse_checkout = 1;\n+\tif (update_working_directory(NULL))\n \t\tdie(_(\"error while refreshing working directory\"));\n \n \tunlink(sparse_filename);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0792caeb7e..a580b79595 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -248,4 +248,32 @@ test_expect_success 'cone mode: set with nested folders' '\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n \n+test_expect_success 'revert to old sparse-checkout on bad update' '\n+\techo update >repo/deep/deeper2/a &&\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp repo/.git/info/sparse-checkout expect &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\t\tdeeper2\n+\tEOF\n+\ttest_cmp dir expect\n+'\n+\n+test_expect_success 'revert to old sparse-checkout on empty update' '\n+\tgit init empty-test &&\n+\t(\n+\t\techo >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"test\" &&\n+\t\ttest_must_fail git sparse-checkout set nothing 2>err &&\n+\t\ttest_i18ngrep \"Sparse checkout leaves no entry on working directory\" err &&\n+\t\ttest_i18ngrep ! \".git/index.lock\" err &&\n+\t\tgit sparse-checkout set file\n+\t)\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8bb684ad62..3789a22cf0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n-\tif (!o->skip_sparse_checkout) {\n+\tif (!o->skip_sparse_checkout && !o->pl) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n \t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n@@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n done:\n \ttrace_performance_leave(\"unpack_trees\");\n-\tclear_pattern_list(&pl);\n+\tif (!o->keep_pattern_list)\n+\t\tclear_pattern_list(&pl);\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex f2eee0c7c5..ca94a421a5 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -59,7 +59,8 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run;\n+\t\t dry_run,\n+\t\t keep_pattern_list;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"384522","messageId":"f03bb23305c01ddb7387b44add549bf22539002b.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 14/17] sparse-checkout: sanitize for nested folders","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:23Z","receivedAt":"2019-10-21T13:56:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user provides folders A/ and A/B/ for inclusion in a cone-mode\nsparse-checkout file, the parsing logic will notice that A/ appears\nboth as a \"parent\" type pattern and as a \"recursive\" type pattern.\nThis is unexpected and hence will complain via a warning and revert\nto the old logic for checking sparse-checkout patterns.\n\nPrevent this from happening accidentally by sanitizing the folders\nfor this type of inclusion in the 'git sparse-checkout' builtin.\nThis happens in two ways:\n\n1. Do not include any parent patterns that also appear as recursive\n patterns.\n\n2. Do not include any recursive patterns deeper than other recursive\n patterns.\n\nIn order to minimize duplicate code for scanning parents, create\nhashmap_contains_parent() method. It takes a strbuf buffer to\navoid reallocating a buffer when calling in a tight loop.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 22 ++++++++++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 6c3bf68c89..b02e085495 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -212,9 +212,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstruct pattern_entry *pe;\n \tstruct hashmap_iter iter;\n \tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\tstruct strbuf parent_pattern = STRBUF_INIT;\n \n-\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {\n+\t\tif (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))\n+\t\t\tcontinue;\n+\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\n@@ -230,8 +238,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \n \tstring_list_clear(&sl, 0);\n \n-\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n+\n+\tstrbuf_release(&parent_pattern);\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex ae99803d40..0792caeb7e 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -237,4 +237,15 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: set with nested folders' '\n+\tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\tEOF\n+\ttest_cmp repo/.git/info/sparse-checkout expect\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384524","messageId":"ab20c11d71a98def1c29e949cb267cdae822e5dc.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 17/17] sparse-checkout: cone mode should not interact with .gitignore","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:26Z","receivedAt":"2019-10-21T13:56:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the development of the sparse-checkout \"cone mode\" feature,\nan incorrect placement of the initializer for \"use_cone_patterns = 1\"\ncaused warnings to show up when a .gitignore file was present with\nnon-cone-mode patterns. This was fixed in the original commit\nintroducing the cone mode, but now we should add a test to avoid\nhitting this problem again in the future.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 2db706a9e4..4205e03449 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -283,4 +283,11 @@ test_expect_success 'fail when lock is taken' '\n \ttest_i18ngrep \"File exists\" err\n '\n \n+test_expect_success '.gitignore should not warn about cone mode' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\techo \"**/bin/*\" >repo/.gitignore &&\n+\tgit -C repo reset --hard 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"384523","messageId":"c48535cd5c00310403ffccac7507f399fca5a8a4.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 06/17] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:15Z","receivedAt":"2019-10-21T13:56:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n 3 files changed, 52 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex b933043b3d..f794d4797a 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -48,6 +48,10 @@ To avoid interfering with other worktrees, it first enables the\n \tworking directory to match the new patterns. Enable the\n \tcore.sparseCheckout config setting if it is not already enabled.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ---------------\n \n@@ -65,6 +69,14 @@ directory, it updates the skip-worktree bits in the index based\n on this file. The files matching the patterns in the file will\n appear in the working directory, and the rest will not.\n \n+To enable the sparse-checkout feature, run `git sparse-checkout init` to\n+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`\n+config setting. Then, run `git sparse-checkout set` to modify the patterns in\n+the sparse-checkout file.\n+\n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -79,21 +91,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-checkout' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex f2e2bd772d..9fdcc6c4ef 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list|set) \"),\n+\tN_(\"git sparse-checkout (init|list|set|disable) \"),\n \tNULL\n };\n \n@@ -212,6 +212,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \treturn result;\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (sc_set_config(MODE_ALL_PATTERNS))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = xfopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (update_working_directory())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn sc_set_config(MODE_NO_PATTERNS);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -236,6 +258,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"set\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex a9ff5eb9ec..583c232e9e 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -148,4 +148,19 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"384525","messageId":"a229e1ee0cb96c5f8c2c5d430641c386bc082a2d.1571666187.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v5 13/17] read-tree: show progress by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-10-21T13:56:22Z","receivedAt":"2019-10-21T13:56:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe read-tree builtin has a --verbose option that signals to show\nprogress and other data while updating the index. Update this to\nbe on by default when stderr is a terminal window.\n\nThis will help tools like 'git sparse-checkout' to automatically\nbenefit from progress indicators when a user runs these commands.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex ca5e655d2f..69963d83dc 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \topts.head_idx = -1;\n \topts.src_index = &the_index;\n \topts.dst_index = &the_index;\n+\topts.verbose_update = isatty(2);\n \n \tgit_config(git_read_tree_config, NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"384527","messageId":"3de0f590-c75c-2470-232f-73a07876df88@gmail.com","threadId":"51691","inReplyTo":"a229e1ee0cb96c5f8c2c5d430641c386bc082a2d.1571666187.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2019-10-21T15:04:25Z","receivedAt":"2019-10-21T15:04:34Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Stolee\n\nOn 21/10/2019 14:56, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The read-tree builtin has a --verbose option that signals to show\n> progress and other data while updating the index. Update this to\n> be on by default when stderr is a terminal window.\n> \n> This will help tools like 'git sparse-checkout' to automatically\n> benefit from progress indicators when a user runs these commands.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/read-tree.c | 1 +\n> 1 file changed, 1 insertion(+)\n> \n> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n> index ca5e655d2f..69963d83dc 100644\n> --- a/builtin/read-tree.c\n> +++ b/builtin/read-tree.c\n> @@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n> \topts.head_idx = -1;\n> \topts.src_index = &the_index;\n> \topts.dst_index = &the_index;\n> +\topts.verbose_update = isatty(2);\n\nI'm slightly wary of changing the output of plumbing commands like this. \nIf a script wants progress output it can already get it by passing \n--verbose. With this change a script that does not want that output now \nhas to pass --no-verbose. If 'git sparse-checkout' wants progress \nindicators why not put the isatty() check there and pass the appropriate \noption to read-tree?\n\nBest Wishes\n\nPhillip\n\n> \tgit_config(git_read_tree_config, NULL);\n> \n> \n"},{"id":"384528","messageId":"857e23df-c447-7297-df74-3546424386d9@gmail.com","threadId":"51691","inReplyTo":"3de0f590-c75c-2470-232f-73a07876df88@gmail.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-21T15:14:08Z","receivedAt":"2019-10-21T15:14:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/21/2019 11:04 AM, Phillip Wood wrote:\n> Hi Stolee\n> \n> On 21/10/2019 14:56, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The read-tree builtin has a --verbose option that signals to show\n>> progress and other data while updating the index. Update this to\n>> be on by default when stderr is a terminal window.\n>>\n>> This will help tools like 'git sparse-checkout' to automatically\n>> benefit from progress indicators when a user runs these commands.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>>   builtin/read-tree.c | 1 +\n>>   1 file changed, 1 insertion(+)\n>>\n>> diff --git a/builtin/read-tree.c b/builtin/read-tree.c\n>> index ca5e655d2f..69963d83dc 100644\n>> --- a/builtin/read-tree.c\n>> +++ b/builtin/read-tree.c\n>> @@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n>>       opts.head_idx = -1;\n>>       opts.src_index = &the_index;\n>>       opts.dst_index = &the_index;\n>> +    opts.verbose_update = isatty(2);\n> \n> I'm slightly wary of changing the output of plumbing commands like this. If a script wants progress output it can already get it by passing --verbose. With this change a script that does not want that output now has to pass --no-verbose.\n\nIf a script is calling this, then won't stderr not be a terminal window, and\nisatty(2) return 0? Or, if the script is run with stderr passing through to\na terminal, then the user would see progress while running the script, which\nseems like a side-effect but not one that will cause a broken script.\n\n> If 'git sparse-checkout' wants progress indicators why not put the isatty() check there and pass the appropriate option to read-tree?\n\nIt is not necessary for the sparse-checkout builtin by the end of the series,\nas the extra process is removed and unpack_trees() is called directly. However,\nit does seem like a helpful user-interaction for people still using the \"old\"\nway of interacting with the sparse-checkout feature.\n\nThanks,\n-Stolee\n"},{"id":"384669","messageId":"xmqqimogi0kq.fsf@gitster-ct.c.googlers.com","threadId":"51691","inReplyTo":"857e23df-c447-7297-df74-3546424386d9@gmail.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-10-23T03:48:37Z","receivedAt":"2019-10-23T03:48:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> I'm slightly wary of changing the output of plumbing commands\n>> like this. If a script wants progress output it can already get\n>> it by passing --verbose. With this change a script that does not\n>> want that output now has to pass --no-verbose.\n>\n> If a script is calling this, then won't stderr not be a terminal window, and\n> isatty(2) return 0?\n\nUnless the script tries to capture the error output and react\ndifferently depending on the error message from the plumbing (which\nis not localized), iow most of the time, standard error stream is\nleft unredirected and likely to be connected to the terminal if the\nscript is driven from a terminal command line.\n\n> Or, if the script is run with stderr passing through to\n> a terminal, then the user would see progress while running the script, which\n> seems like a side-effect but not one that will cause a broken script.\n\nIt will show unwanted output to the end users, no? That is the\ncomplaint about having to pass --no-verbose, if I understand\ncorrectly, if the script does not want to show the progress output.\n\n"},{"id":"384670","messageId":"xmqqeez4i0et.fsf@gitster-ct.c.googlers.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 00/17] New sparse-checkout builtin and \"cone\" mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-10-23T03:52:10Z","receivedAt":"2019-10-23T03:52:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> V4 UPDATE: Rebased on latest master to include ew/hashmap and\n> ds/include-exclude in the base.\n>\n> This series makes the sparse-checkout feature more user-friendly. While\n> there, I also present a way to use a limited set of patterns to gain a\n> significant performance boost in very large repositories.\n> ...\n> Updates in V5:\n>\n> * The 'set' subcommand now enables the core.sparseCheckout config setting\n> (unless the checkout fails).\n> \n> \n> * If the in-process unpack_trees() fails with the new patterns, the\n> index.lock file is rolled back before the replay of the old\n> sparse-checkout patterns.\n> \n> \n> * Some documentation fixes, f(d)open->xf(d)open calls, and other nits.\n> Thanks everyone!\n\nThanks. I quickly scanned the changes between the last round and\nthis one, and all I saw were pure improvements ;-) Not that I claim\nto have read the previous round very carefully, though.\n\nWill replace and queue.\n"},{"id":"384698","messageId":"07f6288a-21c3-bd61-ef74-fea129d12c76@gmail.com","threadId":"51691","inReplyTo":"xmqqimogi0kq.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-10-23T12:50:00Z","receivedAt":"2019-10-23T12:50:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/22/2019 11:48 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>>> I'm slightly wary of changing the output of plumbing commands\n>>> like this. If a script wants progress output it can already get\n>>> it by passing --verbose. With this change a script that does not\n>>> want that output now has to pass --no-verbose.\n>>\n>> If a script is calling this, then won't stderr not be a terminal window, and\n>> isatty(2) return 0?\n> \n> Unless the script tries to capture the error output and react\n> differently depending on the error message from the plumbing (which\n> is not localized), iow most of the time, standard error stream is\n> left unredirected and likely to be connected to the terminal if the\n> script is driven from a terminal command line.\n> \n>> Or, if the script is run with stderr passing through to\n>> a terminal, then the user would see progress while running the script, which\n>> seems like a side-effect but not one that will cause a broken script.\n> \n> It will show unwanted output to the end users, no? That is the\n> complaint about having to pass --no-verbose, if I understand\n> correctly, if the script does not want to show the progress output.\n\nI'm happy to have attempted the change and start this discussion. It\nsounds like this one patch could be ejected to no loss to the full\nseries.\n\nThanks,\n-Stolee\n"},{"id":"384755","messageId":"09df07f8-9a1f-9e00-31e8-63cbebd38397@gmail.com","threadId":"51691","inReplyTo":"xmqqimogi0kq.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2019-10-24T10:18:02Z","receivedAt":"2019-10-24T10:18:13Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 23/10/2019 04:48, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>>> I'm slightly wary of changing the output of plumbing commands\n>>> like this. If a script wants progress output it can already get\n>>> it by passing --verbose. With this change a script that does not\n>>> want that output now has to pass --no-verbose.\n>>\n>> If a script is calling this, then won't stderr not be a terminal window, and\n>> isatty(2) return 0?\n> \n> Unless the script tries to capture the error output and react\n> differently depending on the error message from the plumbing (which\n> is not localized), iow most of the time, standard error stream is\n> left unredirected and likely to be connected to the terminal if the\n> script is driven from a terminal command line.\n> \n>> Or, if the script is run with stderr passing through to\n>> a terminal, then the user would see progress while running the script, which\n>> seems like a side-effect but not one that will cause a broken script.\n> \n> It will show unwanted output to the end users, no? That is the\n> complaint about having to pass --no-verbose, if I understand\n> correctly, if the script does not want to show the progress output.\n\nYes that was my objection.\n\nBest Wishes\n\nPhillip\n\n\n"},{"id":"384756","messageId":"6c6b9838-af7b-7212-199f-a0a3f3f2ac77@gmail.com","threadId":"51691","inReplyTo":"07f6288a-21c3-bd61-ef74-fea129d12c76@gmail.com","subject":"Re: [PATCH v5 13/17] read-tree: show progress by default","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2019-10-24T10:18:46Z","receivedAt":"2019-10-24T10:18:51Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Stolee\n\nOn 23/10/2019 13:50, Derrick Stolee wrote:\n> On 10/22/2019 11:48 PM, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>>> I'm slightly wary of changing the output of plumbing commands\n>>>> like this. If a script wants progress output it can already get\n>>>> it by passing --verbose. With this change a script that does not\n>>>> want that output now has to pass --no-verbose.\n>>>\n>>> If a script is calling this, then won't stderr not be a terminal window, and\n>>> isatty(2) return 0?\n>>\n>> Unless the script tries to capture the error output and react\n>> differently depending on the error message from the plumbing (which\n>> is not localized), iow most of the time, standard error stream is\n>> left unredirected and likely to be connected to the terminal if the\n>> script is driven from a terminal command line.\n>>\n>>> Or, if the script is run with stderr passing through to\n>>> a terminal, then the user would see progress while running the script, which\n>>> seems like a side-effect but not one that will cause a broken script.\n>>\n>> It will show unwanted output to the end users, no? That is the\n>> complaint about having to pass --no-verbose, if I understand\n>> correctly, if the script does not want to show the progress output.\n> \n> I'm happy to have attempted the change and start this discussion. It\n> sounds like this one patch could be ejected to no loss to the full\n> series.\n\nThanks. Enabling --verbose by default on a tty might have made sense\noriginally but I'm worried doing it now will lead to confusing output\nwhere read-tree is a small part of a larger operation and the script was\nwritten on the assumption that read-tree will be quiet.\n\nBest Wishes\n\nPhillip\n\n> Thanks,\n> -Stolee\n> \n\n"},{"id":"386517","messageId":"20191119143308.GF23183@szeder.dev","threadId":"51691","inReplyTo":"a161cee0dfec76e7a08253083f488e2e6d26299e.1571666186.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T14:33:08Z","receivedAt":"2019-11-19T14:33:16Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Getting started with a sparse-checkout file can be daunting. Help\n> users start their sparse enlistment using 'git sparse-checkout init'.\n> This will set 'core.sparseCheckout=true' in their config, write\n> an initial set of patterns to the sparse-checkout file, and update\n> their working directory.\n\nReading this I was wandering what those \"initial set of patterns\"\nmight be ...\n\n> Make sure to use the `extensions.worktreeConfig` setting and write\n> the sparse checkout config to the worktree-specific config file.\n> This avoids confusing interactions with other worktrees.\n> \n> The use of running another process for 'git read-tree' is sub-\n> optimal. This will be removed in a later change.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 11 ++++\n> builtin/sparse-checkout.c | 79 ++++++++++++++++++++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++\n> 3 files changed, 130 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index 9d6ca22917..930a361567 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -30,6 +30,17 @@ COMMANDS\n> 'list'::\n> \tProvide a list of the contents in the sparse-checkout file.\n> \n> +'init'::\n> +\tEnable the `core.sparseCheckout` setting. If the\n> +\tsparse-checkout file does not exist, then populate it with\n> +\tpatterns that match every file in the root directory and\n> +\tno other directories, then will remove all directories tracked\n> +\tby Git. Add patterns to the sparse-checkout file to\n> +\trepopulate the working directory.\n\n... and then reading this I was wandering why these are those \"initial\nset of patterns\".\n\n> ++\n> +To avoid interfering with other worktrees, it first enables the\n> +`extensions.worktreeConfig` setting and makes sure to set the\n> +`core.sparseCheckout` setting in the worktree-specific config file.\n> \n> SPARSE CHECKOUT\n> ---------------\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 5717c9b2cb..77aa52ca01 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -8,7 +8,7 @@\n> #include \"strbuf.h\"\n> \n> static char const * const builtin_sparse_checkout_usage[] = {\n> -\tN_(\"git sparse-checkout list\"),\n> +\tN_(\"git sparse-checkout (init|list)\"),\n> \tNULL\n> };\n> \n> @@ -59,6 +59,81 @@ static int sparse_checkout_list(int argc, const char **argv)\n> \treturn 0;\n> }\n> \n> +static int update_working_directory(void)\n> +{\n> +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n> +\tint result = 0;\n> +\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n> +\n> +\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> +\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n> +\t\tresult = 1;\n> +\t}\n> +\n> +\targv_array_clear(&argv);\n> +\treturn result;\n> +}\n> +\n> +enum sparse_checkout_mode {\n> +\tMODE_NO_PATTERNS = 0,\n> +\tMODE_ALL_PATTERNS = 1,\n> +};\n> +\n> +static int sc_set_config(enum sparse_checkout_mode mode)\n\nNit: s/sc_//, perhaps? I suppose that \"sc\" prefix stands for \"sparse\ncheckout\", but this is a static function in\n'builtin/sparse-checkout.c', so it doesn't need a distinguising\nprefix. Even at the end of this patch series no other functions have\nthis \"sc\" prefix.\n\n> +{\n> +\tstruct argv_array argv = ARGV_ARRAY_INIT;\n\nThis 'argv_array' is not cleared at the end of the function, but...\n\n> +\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n> +\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n> +\n> +\tif (mode)\n> +\t\targv_array_pushl(&argv, \"true\", NULL);\n> +\telse\n> +\t\targv_array_pushl(&argv, \"false\", NULL);\n> +\n> +\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> +\t\terror(_(\"failed to enable core.sparseCheckout\"));\n> +\t\treturn 1;\n> +\t}\n\nWhy the external 'git config' invocation?\n\n git_config_set_in_file_gently(git_path(\"config.worktree\"),\n \"core.sparseCheckout\",\n mode ? \"true\" : \"false\")\n\n> +\n> +\treturn 0;\n> +}\n> +\n> +static int sparse_checkout_init(int argc, const char **argv)\n> +{\n> +\tstruct pattern_list pl;\n> +\tchar *sparse_filename;\n> +\tFILE *fp;\n> +\tint res;\n> +\n> +\tif (sc_set_config(MODE_ALL_PATTERNS))\n> +\t\treturn 1;\n> +\n> +\tmemset(&pl, 0, sizeof(pl));\n> +\n> +\tsparse_filename = get_sparse_checkout_filename();\n> +\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n> +\n> +\t/* If we already have a sparse-checkout file, use it. */\n> +\tif (res >= 0) {\n> +\t\tfree(sparse_filename);\n> +\t\tgoto reset_dir;\n> +\t}\n> +\n> +\t/* initial mode: all blobs at root */\n> +\tfp = xfopen(sparse_filename, \"w\");\n> +\tfree(sparse_filename);\n> +\tfprintf(fp, \"/*\\n!/*/\\n\");\n\nWhat if this fprintf() call were to fail?\n\n> +\tfclose(fp);\n> +\n> +reset_dir:\n> +\treturn update_working_directory();\n> +}\n> +\n> int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> {\n> \tstatic struct option builtin_sparse_checkout_options[] = {\n> @@ -79,6 +154,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n> \tif (argc > 0) {\n> \t\tif (!strcmp(argv[0], \"list\"))\n> \t\t\treturn sparse_checkout_list(argc, argv);\n> +\t\tif (!strcmp(argv[0], \"init\"))\n> +\t\t\treturn sparse_checkout_init(argc, argv);\n> \t}\n> \n> \tusage_with_options(builtin_sparse_checkout_usage,\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 9b73d44907..cd56cc384b 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -42,4 +42,45 @@ test_expect_success 'git sparse-checkout list (populated)' '\n> \ttest_cmp expect list\n> '\n> \n> +test_expect_success 'git sparse-checkout init' '\n> +\tgit -C repo sparse-checkout init &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\tEOF\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tgit -C repo config --list >config &&\n> +\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n\nWe have the 'test_cmp_config' helper function to check the expected\nvalue of configuration variables.\n\n> +\tls repo >dir &&\n> +\techo a >expect &&\n> +\ttest_cmp expect dir\n> +'\n> +\n> +test_expect_success 'git sparse-checkout list after init' '\n> +\tgit -C repo sparse-checkout list >actual &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\tEOF\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'init with existing sparse-checkout' '\n> +\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n> +\tgit -C repo sparse-checkout init &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\t\t*folder*\n> +\tEOF\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tls repo >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tfolder1\n> +\t\tfolder2\n> +\tEOF\n> +\ttest_cmp expect dir\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"386522","messageId":"20191119154640.GG23183@szeder.dev","threadId":"51691","inReplyTo":"7d9d66a89f473244af3601e13caa713d929a202d.1571666186.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 04/17] sparse-checkout: 'set' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T15:46:40Z","receivedAt":"2019-11-19T15:46:48Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The 'git sparse-checkout set' subcommand takes a list of patterns\n> as arguments and writes them to the sparse-checkout file. Then, it\n> updates the working directory using 'git read-tree -mu HEAD'.\n> \n> The 'set' subcommand will replace the entire contents of the\n> sparse-checkout file. The write_patterns_and_update() method is\n> extracted from cmd_sparse_checkout() to make it easier to implement\n> 'add' and/or 'remove' subcommands in the future.\n\nYeah, an 'add' subcommand would be great, because 'set' throwing away\nall the existing patterns can lead to some frustration:\n\n # Let's clone with sparse checkout\n $ git clone --sparse ...\n $ cd clone\n $ less README\n $ git sparse-checkout '/some/dir/*'\n # Erm, what was the next step?\n $ less README\n README: No such file or directory\n # Uhoh.\n\nHaving said that, being forced to use 'git sparse-checkout set' and\nspecify all patterns at once does have its own benefits: I needed like\n6 subdirectories to build that subset of a big project that I was\ninterested in, and now all the necessary patterns are saved in my Bash\nhistory, and I will be able to easily dig out the complete command in\nthe future. That wouldn't work with using the 'add' subcommand to add\none pattern at a time.\n\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index cb74715ca6..bf2dc55bb1 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' '\n> \ttest_cmp expect dir\n> '\n> \n> +test_expect_success 'set enables config' '\n> +\tgit init empty-config &&\n> +\t(\n> +\t\tcd empty-config &&\n> +\t\ttest_commit test file &&\n> +\t\ttest_path_is_missing .git/config.worktree &&\n> +\t\ttest_must_fail git sparse-checkout set nothing &&\n\nThis command prints the same error message twice:\n\n + test_must_fail git sparse-checkout set nothing\n error: Sparse checkout leaves no entry on working directory\n error: Sparse checkout leaves no entry on working directory\n\n\n> +\t\ttest_i18ngrep \"sparseCheckout = false\" .git/config.worktree &&\n\nThe contents of a configuration file surely can't ever be translated,\ncan it?!\n\nPlease use 'test_cmp_config'.\n\n> +\t\tgit sparse-checkout set \"/*\" &&\n> +\t\ttest_i18ngrep \"sparseCheckout = true\" .git/config.worktree\n> +\t)\n> +'\n> +\n> +test_expect_success 'set sparse-checkout using builtin' '\n> +\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\t\t*folder*\n> +\tEOF\n> +\tgit -C repo sparse-checkout list >actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tls repo >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tfolder1\n> +\t\tfolder2\n> +\tEOF\n> +\ttest_cmp expect dir\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"386523","messageId":"20191119161545.GH23183@szeder.dev","threadId":"51691","inReplyTo":"c48535cd5c00310403ffccac7507f399fca5a8a4.1571666187.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 06/17] sparse-checkout: create 'disable' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T16:15:45Z","receivedAt":"2019-11-19T16:15:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:15PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The instructions for disabling a sparse-checkout to a full\n> working directory are complicated and non-intuitive. Add a\n> subcommand, 'git sparse-checkout disable', to perform those\n> steps for the user.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n> builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n> t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n> 3 files changed, 52 insertions(+), 16 deletions(-)\n> \n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index b933043b3d..f794d4797a 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -48,6 +48,10 @@ To avoid interfering with other worktrees, it first enables the\n> \tworking directory to match the new patterns. Enable the\n> \tcore.sparseCheckout config setting if it is not already enabled.\n> \n> +'disable'::\n> +\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n> +\t`false`, and restore the working directory to include all files.\n\nI think it would make sense to leave the sparse-checkout file behind\nas-is, and only update the coonfiguration and the working tree. That\nway users could quickly go back to their last sparse checkout setup by\nsimply running 'git sparse-checkout init'.\n\n \n> +static int sparse_checkout_disable(int argc, const char **argv)\n> +{\n> +\tchar *sparse_filename;\n> +\tFILE *fp;\n> +\n> +\tif (sc_set_config(MODE_ALL_PATTERNS))\n> +\t\tdie(_(\"failed to change config\"));\n> +\n> +\tsparse_filename = get_sparse_checkout_filename();\n> +\tfp = xfopen(sparse_filename, \"w\");\n> +\tfprintf(fp, \"/*\\n\");\n> +\tfclose(fp);\n> +\n> +\tif (update_working_directory())\n> +\t\tdie(_(\"error while refreshing working directory\"));\n> +\n> +\tunlink(sparse_filename);\n> +\tfree(sparse_filename);\n> +\n> +\treturn sc_set_config(MODE_NO_PATTERNS);\n\nHrm. So disabling the sparse-checkout calls the same helper function\nto write the configuration as initializing or setting, but with a\ndifferent parameter. However, the error message in that function is:\n\n error(_(\"failed to enable core.sparseCheckout\"));\n\nSo if something goes wrong while disabling the sparse-checkout, the\nuser might get an error saying \"error: failed to enable\ncore.sparseCheckout\". That doesn't look quite right, does it?\n\n> +}\n"},{"id":"386537","messageId":"20191119170751.GI23183@szeder.dev","threadId":"51691","inReplyTo":"d7af75672ba20ed208d51a35243b96ce10202a54.1571666187.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 15/17] sparse-checkout: update working directory in-process","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T17:07:51Z","receivedAt":"2019-11-19T17:07:58Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:24PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\n> skip-worktree bits in the index and to update the working directory.\n> This extra process is overly complex, and prone to failure. It also\n> requires that we write our changes to the sparse-checkout file before\n> trying to update the index.\n> \n> Remove this extra process call by creating a direct call to\n> unpack_trees() in the same way 'git read-tree -mu HEAD' does. In\n> addition, provide an in-memory list of patterns so we can avoid\n> reading from the sparse-checkout file.\n\nOK, the way I understand the above two paragraphs is that after this\npatch it won't be necessary to write the updated patterns to the\n'sparse-checkout' file before calling unpack_trees(), and to me it\nimplies that it won't be necessary to write the \"include everything\"\npattern to that file during disabling sparse checkout.\n\n> @@ -378,7 +436,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> \tfprintf(fp, \"/*\\n\");\n> \tfclose(fp);\n\nHowever, as the patch context here shows we still write that \"include\neverything\" pattern to the 'sparse-checkout' file during disabling.\n\nFWIW, deleting those lines updating the 'sparse-checkout' file make\nthe 'sparse-checkout disable' test fail.\n\nDid I misunderstand what the commit message is trying to say?\n\n> -\tif (update_working_directory())\n> +\tcore_apply_sparse_checkout = 1;\n> +\tif (update_working_directory(NULL))\n> \t\tdie(_(\"error while refreshing working directory\"));\n> \n> \tunlink(sparse_filename);\n"},{"id":"386539","messageId":"20191119173643.GJ23183@szeder.dev","threadId":"51691","inReplyTo":"121d878882609907cc8d13f9d76f2414b15618be.1571666187.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 08/17] sparse-checkout: add 'cone' mode","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T17:36:43Z","receivedAt":"2019-11-19T17:36:50Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:17PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> index f794d4797a..3931e4f2ad 100644\n> --- a/Documentation/git-sparse-checkout.txt\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -92,6 +92,56 @@ using negative patterns. For example, to remove the file `unwanted`:\n> ----------------\n> \n> \n> +## CONE PATTERN SET\n> +\n> +The full pattern set allows for arbitrary pattern matches and complicated\n> +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n> +updating the index, where N is the number of patterns and M is the number\n> +of paths in the index. To combat this performance issue, a more restricted\n> +pattern set is allowed when `core.spareCheckoutCone` is enabled.\n> +\n> +The accepted patterns in the cone pattern set are:\n> +\n> +1. *Recursive:* All paths inside a directory are included.\n> +\n> +2. *Parent:* All files immediately inside a directory are included.\n> +\n> +In addition to the above two patterns, we also expect that all files in the\n> +root directory are included. If a recursive pattern is added, then all\n> +leading directories are added as parent patterns.\n> +\n> +By default, when running `git sparse-checkout init`, the root directory is\n> +added as a parent pattern. At this point, the sparse-checkout file contains\n> +the following patterns:\n> +\n> +```\n> +/*\n> +!/*/\n> +```\n> +\n> +This says \"include everything in root, but nothing two levels below root.\"\n> +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n> +`A/B` are added as parent patterns. The resulting sparse-checkout file is\n> +now\n> +\n> +```\n> +/*\n> +!/*/\n> +/A/\n> +!/A/*/\n> +/A/B/\n> +!/A/B/*/\n> +/A/B/C/\n> +```\n\nThis is a man page, not markdown. Those ## and ``` don't look as\nexpected in the generated documentation.\n\n"},{"id":"386540","messageId":"20191119175046.GK23183@szeder.dev","threadId":"51691","inReplyTo":"0258ee80265f5f27a7de9b81eaf166648b4511d4.1571666187.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/17] sparse-checkout: init and set in cone mode","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T17:50:46Z","receivedAt":"2019-11-19T17:50:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:19PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> To make the cone pattern set easy to use, update the behavior of\n> 'git sparse-checkout [init|set]'.\n> \n> Add '--cone' flag to 'git sparse-checkout init' to set the config\n> option 'core.sparseCheckoutCone=true'.\n\nIt's not necessary to run 'git sparse-checkout init' before running\n'git sparse-checkout set'. The description of the latter in the\ndocumentation is not explicit about it, but the commit message adding\nthe 'set' subcommand is, and there are several test cases that run\n'set' without a preceeding 'init.\n\nTherefore, I think the 'set' subcommand should get a '--cone' option\nas well.\n\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index 9907278fc1..ae99803d40 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -186,4 +186,55 @@ test_expect_success 'sparse-checkout disable' '\n> \ttest_cmp expect dir\n> '\n> \n> +test_expect_success 'cone mode: init and set' '\n> +\tgit -C repo sparse-checkout init --cone &&\n> +\tgit -C repo config --list >config &&\n> +\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n> +\tls repo >dir &&\n> +\techo a >expect &&\n> +\ttest_cmp expect dir &&\n> +\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n> +\ttest_line_count = 0 err &&\n\n'test_must_be_empty err' would be more idiomatic here as well.\n\n> +\tls repo >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tdeep\n> +\tEOF\n> +\ttest_cmp expect dir &&\n> +\tls repo/deep >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tdeeper1\n> +\tEOF\n> +\ttest_cmp expect dir &&\n> +\tls repo/deep/deeper1 >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tdeepest\n> +\tEOF\n> +\ttest_cmp expect dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\t\t/deep/\n> +\t\t!/deep/*/\n> +\t\t/deep/deeper1/\n> +\t\t!/deep/deeper1/*/\n> +\t\t/deep/deeper1/deepest/\n> +\tEOF\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n> +\t\tfolder1\n> +\t\tfolder2\n> +\tEOF\n> +\ttest_line_count = 0 err &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tfolder1\n> +\t\tfolder2\n> +\tEOF\n> +\tls repo >dir &&\n> +\ttest_cmp expect dir\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"386561","messageId":"20191119225133.GO23183@szeder.dev","threadId":"51691","inReplyTo":"55c306a73a5e100b56b184c1ce5428c1259cfc2a.1571666186.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 01/17] sparse-checkout: create builtin with 'list' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-19T22:51:33Z","receivedAt":"2019-11-19T22:51:40Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:10PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n> new file mode 100644\n> index 0000000000..9d6ca22917\n> --- /dev/null\n> +++ b/Documentation/git-sparse-checkout.txt\n> @@ -0,0 +1,89 @@\n> +git-sparse-checkout(1)\n> +======================\n> +\n> +NAME\n> +----\n> +git-sparse-checkout - Initialize and modify the sparse-checkout\n> +configuration, which reduces the checkout to a set of directories\n\ns/directories/paths/\n\n> +given by a list of prefixes.\n\ns/prefixes/patterns/\n\nE.g. 'README' is a file, not a directory, and it's not interpreted as\na prefix:\n\n $ git sparse-checkout set README\n $ find | grep README\n ./README\n ./Documentation/ABI/README\n ./Documentation/virt/kvm/devices/README\n ./arch/m68k/fpsp040/README\n [...]\n\n\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git sparse-checkout [options]'\n> +\n> +\n> +DESCRIPTION\n> +-----------\n> +\n> +Initialize and modify the sparse-checkout configuration, which reduces\n> +the checkout to a set of directories given by a list of prefixes.\n\nLikewise.\n\n"},{"id":"386639","messageId":"20191120141353.GA30837@szeder.dev","threadId":"51691","inReplyTo":"20191119143308.GF23183@szeder.dev","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-20T14:13:54Z","receivedAt":"2019-11-20T14:14:03Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Nov 19, 2019 at 03:33:08PM +0100, SZEDER Gábor wrote:\n> > +\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n> > +\n> > +\tif (mode)\n> > +\t\targv_array_pushl(&argv, \"true\", NULL);\n> > +\telse\n> > +\t\targv_array_pushl(&argv, \"false\", NULL);\n> > +\n> > +\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n> > +\t\terror(_(\"failed to enable core.sparseCheckout\"));\n> > +\t\treturn 1;\n> > +\t}\n> \n> Why the external 'git config' invocation?\n> \n> git_config_set_in_file_gently(git_path(\"config.worktree\"),\n> \"core.sparseCheckout\",\n> mode ? \"true\" : \"false\")\n\nHaving slept on it, I think it would be even better to pass NULL\ninstead of \"false\". \"false\", well, just sets the configuration\nvariable to false, but a NULL would properly unset it.\n\n"},{"id":"386713","messageId":"20191121112111.GQ23183@szeder.dev","threadId":"51691","inReplyTo":"0e08898dcb42bd38ca3692b49a7e9f5763150c80.1571666186.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T11:21:11Z","receivedAt":"2019-11-21T11:21:18Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:14PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The 'git sparse-checkout set' subcommand takes a list of patterns\n> and places them in the sparse-checkout file. Then, it updates the\n> working directory to match those patterns. For a large list of\n> patterns, the command-line call can get very cumbersome.\n> \n> Add a '--stdin' option to instead read patterns over standard in.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++++--\n> t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n\nNo documentation update.\n\n> 2 files changed, 53 insertions(+), 2 deletions(-)\n> \n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 834ee421f0..f2e2bd772d 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -154,6 +154,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n> \treturn update_working_directory();\n> }\n> \n> +static char const * const builtin_sparse_checkout_set_usage[] = {\n> +\tN_(\"git sparse-checkout set [--stdin|]\"),\n\nIn the usage string [...] denotes optional command line parameters.\nHowever, they are not optional, but either '--stdin' or at least one\npattern is required:\n\n $ git sparse-checkout set\n error: Sparse checkout leaves no entry on working directory\n error: Sparse checkout leaves no entry on working directory\n $ echo $?\n 255\n\nSo this should be (--stdin | ).\n\n> +\tNULL\n> +};\n> +\n> +static struct sparse_checkout_set_opts {\n> +\tint use_stdin;\n> +} set_opts;\n> +\n> static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> {\n> \tstatic const char *empty_base = \"\";\n> @@ -161,10 +170,32 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> \tstruct pattern_list pl;\n> \tint result;\n> \tint set_config = 0;\n> +\n> +\tstatic struct option builtin_sparse_checkout_set_options[] = {\n> +\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n> +\t\t\t N_(\"read patterns from standard in\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> \tmemset(&pl, 0, sizeof(pl));\n> \n> -\tfor (i = 1; i < argc; i++)\n> -\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n> +\targc = parse_options(argc, argv, prefix,\n> +\t\t\t builtin_sparse_checkout_set_options,\n> +\t\t\t builtin_sparse_checkout_set_usage,\n> +\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n> +\n> +\tif (set_opts.use_stdin) {\n> +\t\tstruct strbuf line = STRBUF_INIT;\n> +\n> +\t\twhile (!strbuf_getline(&line, stdin)) {\n\nThis reads patterns separated by a newline character.\n\nWhat if someone is doomed with pathnames containing newline\ncharacters, should we provide a '-z' option for \\0-separated patterns?\n\n $ touch foo bar $'foo\\nbar'\n $ git add .\n $ git commit -m 'filename with newline'\n [master (root-commit) 5cd7369] filename with newline\n 3 files changed, 0 insertions(+), 0 deletions(-)\n create mode 100644 bar\n create mode 100644 foo\n create mode 100644 \"foo\\nbar\"\n $ git sparse-checkout set foo\n $ ls\n foo\n $ git sparse-checkout set 'foo*'\n $ ls\n foo foo?bar\n $ git sparse-checkout set $'foo\\nbar'\n $ ls\n foo?bar\n # Looks good so far, but...\n $ cat .git/info/sparse-checkout \n foo\n bar\n $ git read-tree -um HEAD\n $ ls\n bar foo\n # Not so good after all.\n # Let's try to hand-edit the sparse-checkout file.\n $ echo $'\"foo\\\\nbar\"' >.git/info/sparse-checkout \n $ git read-tree -um HEAD\n error: Sparse checkout leaves no entry on working directory\n $ echo $'foo\\\\nbar'\n >.git/info/sparse-checkout \n $ git read-tree -um HEAD\n error: Sparse checkout leaves no entry on working directory\n $ echo $'foo\\\\\\nbar'\n >.git/info/sparse-checkout \n $ git read-tree -um HEAD\n $ ls\n bar\n # OK, I give up :)\n\nSo, it seems that the sparse-checkout file format doesn't support\npaths/patterns containing a newline character (or if it does, I\ncouldn't figure out how), and thus there is no use for a '-z' option.\n\nHowever, as shown above a newline in a pattern given as parameter\neventually leads to undesired behavior, so I think 'git\nsparse-checkout set $'foo\\nbar' should error out upfront.\n\n> +\t\t\tsize_t len;\n> +\t\t\tchar *buf = strbuf_detach(&line, &len);\n\nNit: this 'len' variable is unnecessary, as it's only used as an out\nvariable of this strbuf_detach() call, but strbuf_detach() accepts a\nNULL as its second parameter.\n\n> +\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n> +\t\t}\n> +\t} else {\n> +\t\tfor (i = 0; i < argc; i++)\n> +\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n> +\t}\n\nAccording to the usage string this subcommand needs either '--stdin'\nor a set of patterns, but not both, which is in line with my\ninterpretation of the commit message. I think this makes sense, and\nit's consistent with the '--stdin' option of other git commands.\nHowever, the above option parsing and if-else conditions allow both\n'--stdin' and patterns, silently ignoring the patterns, without\nerroring out:\n\n $ echo README | git sparse-checkout set --stdin Makefile\n $ echo $?\n 0\n $ find . -name README |wc -l\n 51\n $ find . -name Makefile |wc -l\n 0\n\n> \tif (!core_apply_sparse_checkout) {\n> \t\tsc_set_config(MODE_ALL_PATTERNS);\n> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> index bf2dc55bb1..a9ff5eb9ec 100755\n> --- a/t/t1091-sparse-checkout-builtin.sh\n> +++ b/t/t1091-sparse-checkout-builtin.sh\n> @@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n> \ttest_cmp expect dir\n> '\n> \n> +test_expect_success 'set sparse-checkout using --stdin' '\n> +\tcat >expect <<-EOF &&\n> +\t\t/*\n> +\t\t!/*/\n> +\t\t/folder1/\n> +\t\t/folder2/\n> +\tEOF\n> +\tgit -C repo sparse-checkout set --stdin +\tgit -C repo sparse-checkout list >actual &&\n> +\ttest_cmp expect actual &&\n> +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n> +\tls repo >dir &&\n> +\tcat >expect <<-EOF &&\n> +\t\ta\n> +\t\tfolder1\n> +\t\tfolder2\n> +\tEOF\n> +\ttest_cmp expect dir\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"386714","messageId":"20191121114936.GR23183@szeder.dev","threadId":"51691","inReplyTo":"a161cee0dfec76e7a08253083f488e2e6d26299e.1571666186.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T11:49:36Z","receivedAt":"2019-11-21T11:49:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> Getting started with a sparse-checkout file can be daunting. Help\n> users start their sparse enlistment using 'git sparse-checkout init'.\n> This will set 'core.sparseCheckout=true' in their config, write\n> an initial set of patterns to the sparse-checkout file, and update\n> their working directory.\n\nEnabling sparse-checkout can remove modified files:\n\n $ mkdir dir\n $ touch foo dir/bar\n $ git add .\n $ git commit -m Initial\n [master (root-commit) ecc81bd] Initial\n 2 files changed, 0 insertions(+), 0 deletions(-)\n create mode 100644 dir/bar\n create mode 100644 foo\n $ echo changes >dir/bar\n $ ~/src/git/git sparse-checkout init\n error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n error: failed to update index with new sparse-checkout paths\n $ cat dir/bar \n changes\n\nSo far so good, my changes are still there. Unfortunately, however:\n\n $ git add -u\n $ ~/src/git/git sparse-checkout init\n $ echo $?\n 0\n $ ls\n foo\n\nUh-oh, my changes are gone.\n\n"},{"id":"386715","messageId":"20191121123429.GS23183@szeder.dev","threadId":"51691","inReplyTo":"20191121114936.GR23183@szeder.dev","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T12:34:29Z","receivedAt":"2019-11-21T12:34:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Nov 21, 2019 at 12:49:36PM +0100, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> > Getting started with a sparse-checkout file can be daunting. Help\n> > users start their sparse enlistment using 'git sparse-checkout init'.\n> > This will set 'core.sparseCheckout=true' in their config, write\n> > an initial set of patterns to the sparse-checkout file, and update\n> > their working directory.\n> \n> Enabling sparse-checkout can remove modified files:\n> \n> $ mkdir dir\n> $ touch foo dir/bar\n> $ git add .\n> $ git commit -m Initial\n> [master (root-commit) ecc81bd] Initial\n> 2 files changed, 0 insertions(+), 0 deletions(-)\n> create mode 100644 dir/bar\n> create mode 100644 foo\n> $ echo changes >dir/bar\n> $ ~/src/git/git sparse-checkout init\n> error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n> error: failed to update index with new sparse-checkout paths\n\nAnd after this it leaves the 'sparse-checkout' file behind, which may\nor may not be desired:\n\n $ cat .git/info/sparse-checkout \n /*\n !/*/\n\n"},{"id":"386718","messageId":"64b230d4-d36f-5d92-7570-6d965db471a5@gmail.com","threadId":"51691","inReplyTo":"20191119154640.GG23183@szeder.dev","subject":"Re: [PATCH v5 04/17] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T13:04:35Z","receivedAt":"2019-11-21T13:04:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/19/2019 10:46 AM, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The 'git sparse-checkout set' subcommand takes a list of patterns\n>> as arguments and writes them to the sparse-checkout file. Then, it\n>> updates the working directory using 'git read-tree -mu HEAD'.\n>>\n>> The 'set' subcommand will replace the entire contents of the\n>> sparse-checkout file. The write_patterns_and_update() method is\n>> extracted from cmd_sparse_checkout() to make it easier to implement\n>> 'add' and/or 'remove' subcommands in the future.\n> \n> Yeah, an 'add' subcommand would be great, because 'set' throwing away\n> all the existing patterns can lead to some frustration:\n\nI plan to extend this feature when this series lands.\n\n> Having said that, being forced to use 'git sparse-checkout set' and\n> specify all patterns at once does have its own benefits: I needed like\n> 6 subdirectories to build that subset of a big project that I was\n> interested in, and now all the necessary patterns are saved in my Bash\n> history, and I will be able to easily dig out the complete command in\n> the future. That wouldn't work with using the 'add' subcommand to add\n> one pattern at a time.\n\nIn general, this \"set\" command is created first because tools can interact\nwith it more easily than \"add\" and \"remove\". Users would probably prefer the\n\"add\" and \"remove\" way to interact.\n \n>> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n>> index cb74715ca6..bf2dc55bb1 100755\n>> --- a/t/t1091-sparse-checkout-builtin.sh\n>> +++ b/t/t1091-sparse-checkout-builtin.sh\n>> @@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' '\n>> \ttest_cmp expect dir\n>> '\n>> \n>> +test_expect_success 'set enables config' '\n>> +\tgit init empty-config &&\n>> +\t(\n>> +\t\tcd empty-config &&\n>> +\t\ttest_commit test file &&\n>> +\t\ttest_path_is_missing .git/config.worktree &&\n>> +\t\ttest_must_fail git sparse-checkout set nothing &&\n> \n> This command prints the same error message twice:\n> \n> + test_must_fail git sparse-checkout set nothing\n> error: Sparse checkout leaves no entry on working directory\n> error: Sparse checkout leaves no entry on working directory\n\nAt this commit, I do not see two identical lines, but instead the second\nline says\n\n\terror: failed to update index with new sparse-checkout paths\n\n(that \"paths\" should probably be \"patterns\")\n\nThanks,\n-Stolee\n"},{"id":"386720","messageId":"7ad1d044-cb3f-9d9c-22de-a35c6e608c4e@gmail.com","threadId":"51691","inReplyTo":"20191121112111.GQ23183@szeder.dev","subject":"Re: [PATCH v5 05/17] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T13:15:48Z","receivedAt":"2019-11-21T13:15:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/2019 6:21 AM, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:14PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The 'git sparse-checkout set' subcommand takes a list of patterns\n>> and places them in the sparse-checkout file. Then, it updates the\n>> working directory to match those patterns. For a large list of\n>> patterns, the command-line call can get very cumbersome.\n>>\n>> Add a '--stdin' option to instead read patterns over standard in.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++++--\n>> t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++\n> \n> No documentation update.\n> \n>> 2 files changed, 53 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n>> index 834ee421f0..f2e2bd772d 100644\n>> --- a/builtin/sparse-checkout.c\n>> +++ b/builtin/sparse-checkout.c\n>> @@ -154,6 +154,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n>> \treturn update_working_directory();\n>> }\n>> \n>> +static char const * const builtin_sparse_checkout_set_usage[] = {\n>> +\tN_(\"git sparse-checkout set [--stdin|]\"),\n> \n> In the usage string [...] denotes optional command line parameters.\n> However, they are not optional, but either '--stdin' or at least one\n> pattern is required:\n> \n> $ git sparse-checkout set\n> error: Sparse checkout leaves no entry on working directory\n> error: Sparse checkout leaves no entry on working directory\n> $ echo $?\n> 255\n> \n> So this should be (--stdin | ).\n> \n>> +\tNULL\n>> +};\n>> +\n>> +static struct sparse_checkout_set_opts {\n>> +\tint use_stdin;\n>> +} set_opts;\n>> +\n>> static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n>> {\n>> \tstatic const char *empty_base = \"\";\n>> @@ -161,10 +170,32 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n>> \tstruct pattern_list pl;\n>> \tint result;\n>> \tint set_config = 0;\n>> +\n>> +\tstatic struct option builtin_sparse_checkout_set_options[] = {\n>> +\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n>> +\t\t\t N_(\"read patterns from standard in\")),\n>> +\t\tOPT_END(),\n>> +\t};\n>> +\n>> \tmemset(&pl, 0, sizeof(pl));\n>> \n>> -\tfor (i = 1; i < argc; i++)\n>> -\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n>> +\targc = parse_options(argc, argv, prefix,\n>> +\t\t\t builtin_sparse_checkout_set_options,\n>> +\t\t\t builtin_sparse_checkout_set_usage,\n>> +\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n>> +\n>> +\tif (set_opts.use_stdin) {\n>> +\t\tstruct strbuf line = STRBUF_INIT;\n>> +\n>> +\t\twhile (!strbuf_getline(&line, stdin)) {\n> \n> This reads patterns separated by a newline character.\n> \n> What if someone is doomed with pathnames containing newline\n> characters, should we provide a '-z' option for \\0-separated patterns?\n> \n> $ touch foo bar $'foo\\nbar'\n> $ git add .\n> $ git commit -m 'filename with newline'\n> [master (root-commit) 5cd7369] filename with newline\n> 3 files changed, 0 insertions(+), 0 deletions(-)\n> create mode 100644 bar\n> create mode 100644 foo\n> create mode 100644 \"foo\\nbar\"\n> $ git sparse-checkout set foo\n> $ ls\n> foo\n> $ git sparse-checkout set 'foo*'\n> $ ls\n> foo foo?bar\n> $ git sparse-checkout set $'foo\\nbar'\n> $ ls\n> foo?bar\n> # Looks good so far, but...\n> $ cat .git/info/sparse-checkout \n> foo\n> bar\n> $ git read-tree -um HEAD\n> $ ls\n> bar foo\n> # Not so good after all.\n> # Let's try to hand-edit the sparse-checkout file.\n> $ echo $'\"foo\\\\nbar\"' >.git/info/sparse-checkout \n> $ git read-tree -um HEAD\n> error: Sparse checkout leaves no entry on working directory\n> $ echo $'foo\\\\nbar'\n> >.git/info/sparse-checkout \n> $ git read-tree -um HEAD\n> error: Sparse checkout leaves no entry on working directory\n> $ echo $'foo\\\\\\nbar'\n> >.git/info/sparse-checkout \n> $ git read-tree -um HEAD\n> $ ls\n> bar\n> # OK, I give up :)\n> \n> So, it seems that the sparse-checkout file format doesn't support\n> paths/patterns containing a newline character (or if it does, I\n> couldn't figure out how), and thus there is no use for a '-z' option.\n> \n> However, as shown above a newline in a pattern given as parameter\n> eventually leads to undesired behavior, so I think 'git\n> sparse-checkout set $'foo\\nbar' should error out upfront.\n\nI will add this to the list of things to do as a follow-up. There are\nlots of ways users can abuse the command-line right now, especially in\ncone mode.\n\nThe goal right now is to get the typical usage figured out, then we can\nlook for atypical cases (such as newlines in filenames, which...who even\ndoes that?).\n\n>> +\t\t\tsize_t len;\n>> +\t\t\tchar *buf = strbuf_detach(&line, &len);\n> \n> Nit: this 'len' variable is unnecessary, as it's only used as an out\n> variable of this strbuf_detach() call, but strbuf_detach() accepts a\n> NULL as its second parameter.\n> \n>> +\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n>> +\t\t}\n>> +\t} else {\n>> +\t\tfor (i = 0; i < argc; i++)\n>> +\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n>> +\t}\n> \n> According to the usage string this subcommand needs either '--stdin'\n> or a set of patterns, but not both, which is in line with my\n> interpretation of the commit message. I think this makes sense, and\n> it's consistent with the '--stdin' option of other git commands.\n> However, the above option parsing and if-else conditions allow both\n> '--stdin' and patterns, silently ignoring the patterns, without\n> erroring out:\n> \n> $ echo README | git sparse-checkout set --stdin Makefile\n> $ echo $?\n> 0\n> $ find . -name README |wc -l\n> 51\n> $ find . -name Makefile |wc -l\n> 0\n\nI'll add this to the list, too.\n\nThanks,\n-Stolee\n\n"},{"id":"386721","messageId":"bdad792f-4050-691d-cfc1-f57e83d2b240@gmail.com","threadId":"51691","inReplyTo":"20191119161545.GH23183@szeder.dev","subject":"Re: [PATCH v5 06/17] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T13:17:51Z","receivedAt":"2019-11-21T13:17:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/19/2019 11:15 AM, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:15PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The instructions for disabling a sparse-checkout to a full\n>> working directory are complicated and non-intuitive. Add a\n>> subcommand, 'git sparse-checkout disable', to perform those\n>> steps for the user.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n>> builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n>> t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n>> 3 files changed, 52 insertions(+), 16 deletions(-)\n>>\n>> diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n>> index b933043b3d..f794d4797a 100644\n>> --- a/Documentation/git-sparse-checkout.txt\n>> +++ b/Documentation/git-sparse-checkout.txt\n>> @@ -48,6 +48,10 @@ To avoid interfering with other worktrees, it first enables the\n>> \tworking directory to match the new patterns. Enable the\n>> \tcore.sparseCheckout config setting if it is not already enabled.\n>> \n>> +'disable'::\n>> +\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n>> +\t`false`, and restore the working directory to include all files.\n> \n> I think it would make sense to leave the sparse-checkout file behind\n> as-is, and only update the coonfiguration and the working tree. That\n> way users could quickly go back to their last sparse checkout setup by\n> simply running 'git sparse-checkout init'.\n> \n> \n>> +static int sparse_checkout_disable(int argc, const char **argv)\n>> +{\n>> +\tchar *sparse_filename;\n>> +\tFILE *fp;\n>> +\n>> +\tif (sc_set_config(MODE_ALL_PATTERNS))\n>> +\t\tdie(_(\"failed to change config\"));\n>> +\n>> +\tsparse_filename = get_sparse_checkout_filename();\n>> +\tfp = xfopen(sparse_filename, \"w\");\n>> +\tfprintf(fp, \"/*\\n\");\n>> +\tfclose(fp);\n>> +\n>> +\tif (update_working_directory())\n>> +\t\tdie(_(\"error while refreshing working directory\"));\n>> +\n>> +\tunlink(sparse_filename);\n>> +\tfree(sparse_filename);\n>> +\n>> +\treturn sc_set_config(MODE_NO_PATTERNS);\n> \n> Hrm. So disabling the sparse-checkout calls the same helper function\n> to write the configuration as initializing or setting, but with a\n> different parameter. However, the error message in that function is:\n> \n> error(_(\"failed to enable core.sparseCheckout\"));\n> \n> So if something goes wrong while disabling the sparse-checkout, the\n> user might get an error saying \"error: failed to enable\n> core.sparseCheckout\". That doesn't look quite right, does it?\n\nBoth of your comments are valid, but will be easier to implement\nlater in the series, after \"sparse-checkout: update working directory\nin-process\" which previously did not interact with the \"disable\"\ncommand. I'll add a new commit that adds that integration point\nand should resolve your concerns raised here.\n\n-Stolee\n"},{"id":"386723","messageId":"20191121133628.GT23183@szeder.dev","threadId":"51691","inReplyTo":"64b230d4-d36f-5d92-7570-6d965db471a5@gmail.com","subject":"Re: [PATCH v5 04/17] sparse-checkout: 'set' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T13:36:28Z","receivedAt":"2019-11-21T13:36:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Nov 21, 2019 at 08:04:35AM -0500, Derrick Stolee wrote:\n> On 11/19/2019 10:46 AM, SZEDER Gábor wrote:\n> > On Mon, Oct 21, 2019 at 01:56:13PM +0000, Derrick Stolee via GitGitGadget wrote:\n> >> From: Derrick Stolee \n> >>\n> >> The 'git sparse-checkout set' subcommand takes a list of patterns\n> >> as arguments and writes them to the sparse-checkout file. Then, it\n> >> updates the working directory using 'git read-tree -mu HEAD'.\n> >>\n> >> The 'set' subcommand will replace the entire contents of the\n> >> sparse-checkout file. The write_patterns_and_update() method is\n> >> extracted from cmd_sparse_checkout() to make it easier to implement\n> >> 'add' and/or 'remove' subcommands in the future.\n> > \n> > Yeah, an 'add' subcommand would be great, because 'set' throwing away\n> > all the existing patterns can lead to some frustration:\n> \n> I plan to extend this feature when this series lands.\n> \n> > Having said that, being forced to use 'git sparse-checkout set' and\n> > specify all patterns at once does have its own benefits: I needed like\n> > 6 subdirectories to build that subset of a big project that I was\n> > interested in, and now all the necessary patterns are saved in my Bash\n> > history, and I will be able to easily dig out the complete command in\n> > the future. That wouldn't work with using the 'add' subcommand to add\n> > one pattern at a time.\n> \n> In general, this \"set\" command is created first because tools can interact\n> with it more easily than \"add\" and \"remove\". Users would probably prefer the\n> \"add\" and \"remove\" way to interact.\n\nMakes sense.\n\nHowever, I'd like to add that in the meantime I got to like the 'set'\nsubcommand more and more. I enabled-disabled sparse checkout a lot of\ntimes while testing and trying to poke holes, and to be able to set up\neverything with only one single command that I can easily recall from\nthe shell's history was a great help.\n\n> >> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n> >> index cb74715ca6..bf2dc55bb1 100755\n> >> --- a/t/t1091-sparse-checkout-builtin.sh\n> >> +++ b/t/t1091-sparse-checkout-builtin.sh\n> >> @@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' '\n> >> \ttest_cmp expect dir\n> >> '\n> >> \n> >> +test_expect_success 'set enables config' '\n> >> +\tgit init empty-config &&\n> >> +\t(\n> >> +\t\tcd empty-config &&\n> >> +\t\ttest_commit test file &&\n> >> +\t\ttest_path_is_missing .git/config.worktree &&\n> >> +\t\ttest_must_fail git sparse-checkout set nothing &&\n> > \n> > This command prints the same error message twice:\n> > \n> > + test_must_fail git sparse-checkout set nothing\n> > error: Sparse checkout leaves no entry on working directory\n> > error: Sparse checkout leaves no entry on working directory\n> \n> At this commit, I do not see two identical lines, but instead the second\n> line says\n> \n> \terror: failed to update index with new sparse-checkout paths\n\nYou're right, I must have looked at and copied the results from a test\nrun of the full patch series or even from 'next'. The error message\nchanges with commit efd9c53b6d (sparse-checkout: update working\ndirectory in-process, 2019-10-21) to what I shown above.\n\nInterestingly, at one point in my testing I managed to get a different\nerror three times:\n\n $ ~/src/git/git sparse-checkout set foo\n error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n\nHowever, when I later re-run the same sequence of commands that lead\nto that three errors I got that error only twice, and couldn't\nreproduce it since.\n\nStyle nit, seeing \"error: Sparse...\" and \"error: Entry...\": error\nmessages should start with a lower-case letter.\n\n> (that \"paths\" should probably be \"patterns\")\n\nThat would be better indeed, but that error message goes away by the\nend of the series anyway...\n\n"},{"id":"386724","messageId":"c6f641bd-8e26-b8c4-6456-65d480952ab2@gmail.com","threadId":"51691","inReplyTo":"20191119175046.GK23183@szeder.dev","subject":"Re: [PATCH v5 10/17] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T13:45:56Z","receivedAt":"2019-11-21T13:46:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/19/2019 12:50 PM, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:19PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> To make the cone pattern set easy to use, update the behavior of\n>> 'git sparse-checkout [init|set]'.\n>>\n>> Add '--cone' flag to 'git sparse-checkout init' to set the config\n>> option 'core.sparseCheckoutCone=true'.\n> \n> It's not necessary to run 'git sparse-checkout init' before running\n> 'git sparse-checkout set'. The description of the latter in the\n> documentation is not explicit about it, but the commit message adding\n> the 'set' subcommand is, and there are several test cases that run\n> 'set' without a preceeding 'init.\n> \n> Therefore, I think the 'set' subcommand should get a '--cone' option\n> as well.\n\nI will take this under advisement, and revisit it in the follow-up. The\nrecommended usage is to run 'init --cone' first, or to set\n'core.sparseCheckoutCone=true' before running the 'set' command. We can\nimprove this pattern to be more robust in the future.\n"},{"id":"386728","messageId":"147bf5a9-6ae5-23ea-a007-68b8046c94ea@gmail.com","threadId":"51691","inReplyTo":"20191121114936.GR23183@szeder.dev","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T14:28:59Z","receivedAt":"2019-11-21T14:29:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/2019 6:49 AM, SZEDER Gábor wrote:\n> On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> Getting started with a sparse-checkout file can be daunting. Help\n>> users start their sparse enlistment using 'git sparse-checkout init'.\n>> This will set 'core.sparseCheckout=true' in their config, write\n>> an initial set of patterns to the sparse-checkout file, and update\n>> their working directory.\n> \n> Enabling sparse-checkout can remove modified files:\n> \n> $ mkdir dir\n> $ touch foo dir/bar\n> $ git add .\n> $ git commit -m Initial\n> [master (root-commit) ecc81bd] Initial\n> 2 files changed, 0 insertions(+), 0 deletions(-)\n> create mode 100644 dir/bar\n> create mode 100644 foo\n> $ echo changes >dir/bar\n> $ ~/src/git/git sparse-checkout init\n> error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n> error: failed to update index with new sparse-checkout paths\n> $ cat dir/bar \n> changes\n> \n> So far so good, my changes are still there. Unfortunately, however:\n> \n> $ git add -u\n> $ ~/src/git/git sparse-checkout init\n> $ echo $?\n> 0\n> $ ls\n> foo\n> \n> Uh-oh, my changes are gone.\n\nHere, your changes are not \"lost\", they are staged, correct? A \"git status\"\nshould report that your changes are ready for committing. This seems to be\nthe expected behavior.\n\nAs to your other message about leaving the sparse-checkout file even when\nthe 'init' command fails, I'll fix that by using the in-process mechanism.\n\n-Stolee\n\n"},{"id":"386729","messageId":"20191121152744.GU23183@szeder.dev","threadId":"51691","inReplyTo":"147bf5a9-6ae5-23ea-a007-68b8046c94ea@gmail.com","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T15:27:44Z","receivedAt":"2019-11-21T15:27:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Nov 21, 2019 at 09:28:59AM -0500, Derrick Stolee wrote:\n> On 11/21/2019 6:49 AM, SZEDER Gábor wrote:\n> > On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n> >> Getting started with a sparse-checkout file can be daunting. Help\n> >> users start their sparse enlistment using 'git sparse-checkout init'.\n> >> This will set 'core.sparseCheckout=true' in their config, write\n> >> an initial set of patterns to the sparse-checkout file, and update\n> >> their working directory.\n> > \n> > Enabling sparse-checkout can remove modified files:\n> > \n> > $ mkdir dir\n> > $ touch foo dir/bar\n> > $ git add .\n> > $ git commit -m Initial\n> > [master (root-commit) ecc81bd] Initial\n> > 2 files changed, 0 insertions(+), 0 deletions(-)\n> > create mode 100644 dir/bar\n> > create mode 100644 foo\n> > $ echo changes >dir/bar\n> > $ ~/src/git/git sparse-checkout init\n> > error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n> > error: failed to update index with new sparse-checkout paths\n> > $ cat dir/bar \n> > changes\n> > \n> > So far so good, my changes are still there. Unfortunately, however:\n> > \n> > $ git add -u\n> > $ ~/src/git/git sparse-checkout init\n> > $ echo $?\n> > 0\n> > $ ls\n> > foo\n> > \n> > Uh-oh, my changes are gone.\n> \n> Here, your changes are not \"lost\", they are staged, correct? \n\nWell, yes, the changes were staged, so they must be in the object\ndatabase somewhere, the user just has to go through the dangling\nobjects reported by 'git fsck' to find and restore it... So from the\nperspective of an ordinary user they are lost.\n\n> A \"git status\"\n> should report that your changes are ready for committing. This seems to be\n> the expected behavior.\n\nNo, unfortunately enabling sparse-checkout did throw the staged\nchanges away:\n\n ~/test (master +)$ ~/src/git/git sparse-checkout init\n ~/test (master)$ git status \n On branch master\n nothing to commit, working tree clean\n\nNote also the shell prompt going from \"you have staged changes\" to\n\"working tree is clean\".\n\nBut wait, I thought that only changes to files that are excluded from\nthe sparse-checkout are thrown away, but as it turns out it throws\naway changes to files that are included in the sparse-checkout:\n\n ~/test (master #)$ echo original >foo\n ~/test (master #%)$ git add .\n ~/test (master +)$ git commit -m Initial\n [master (root-commit) 04cb2a2] Initial\n 1 file changed, 1 insertion(+)\n create mode 100644 foo\n ~/test (master)$ echo changes >foo \n ~/test (master *)$ ~/src/git/git sparse-checkout init\n ~/test (master *)$ cat foo \n changes\n\nSo far so good, but:\n\n ~/test (master *)$ ~/src/git/git sparse-checkout disable\n ~/test (master *)$ git add .\n ~/test (master +)$ ~/src/git/git sparse-checkout init\n ~/test (master)$ cat foo \n original\n\nThis is not really sparse-checkout-specific, because this is what 'git\nread-tree -um HEAD' always does on its own:\n\n ~/test (master)$ echo changes >foo \n ~/test (master *)$ git read-tree -um HEAD\n ~/test (master *)$ cat foo \n changes\n ~/test (master *)$ git add -u\n ~/test (master +)$ git read-tree -um HEAD\n ~/test (master)$ cat foo \n original\n\nThese issues are present at the end of the patch series as well, but\nthat is sort-of expected, because the later commit message states\nthat:\n\n Remove this extra process call by creating a direct call to\n unpack_trees() in the same way 'git read-tree -mu HEAD' does.\n\n"},{"id":"386730","messageId":"72a54009-5801-c92a-f108-faf38ebbdd4b@gmail.com","threadId":"51691","inReplyTo":"20191121152744.GU23183@szeder.dev","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T15:36:10Z","receivedAt":"2019-11-21T15:36:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/2019 10:27 AM, SZEDER Gábor wrote:\n> On Thu, Nov 21, 2019 at 09:28:59AM -0500, Derrick Stolee wrote:\n>> On 11/21/2019 6:49 AM, SZEDER Gábor wrote:\n>>> On Mon, Oct 21, 2019 at 01:56:11PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>>> Getting started with a sparse-checkout file can be daunting. Help\n>>>> users start their sparse enlistment using 'git sparse-checkout init'.\n>>>> This will set 'core.sparseCheckout=true' in their config, write\n>>>> an initial set of patterns to the sparse-checkout file, and update\n>>>> their working directory.\n>>>\n>>> Enabling sparse-checkout can remove modified files:\n>>>\n>>> $ mkdir dir\n>>> $ touch foo dir/bar\n>>> $ git add .\n>>> $ git commit -m Initial\n>>> [master (root-commit) ecc81bd] Initial\n>>> 2 files changed, 0 insertions(+), 0 deletions(-)\n>>> create mode 100644 dir/bar\n>>> create mode 100644 foo\n>>> $ echo changes >dir/bar\n>>> $ ~/src/git/git sparse-checkout init\n>>> error: Entry 'dir/bar' not uptodate. Cannot update sparse checkout.\n>>> error: failed to update index with new sparse-checkout paths\n>>> $ cat dir/bar \n>>> changes\n>>>\n>>> So far so good, my changes are still there. Unfortunately, however:\n>>>\n>>> $ git add -u\n>>> $ ~/src/git/git sparse-checkout init\n>>> $ echo $?\n>>> 0\n>>> $ ls\n>>> foo\n>>>\n>>> Uh-oh, my changes are gone.\n>>\n>> Here, your changes are not \"lost\", they are staged, correct? \n> \n> Well, yes, the changes were staged, so they must be in the object\n> database somewhere, the user just has to go through the dangling\n> objects reported by 'git fsck' to find and restore it... So from the\n> perspective of an ordinary user they are lost.\n> \n>> A \"git status\"\n>> should report that your changes are ready for committing. This seems to be\n>> the expected behavior.\n> \n> No, unfortunately enabling sparse-checkout did throw the staged\n> changes away:\n> \n> ~/test (master +)$ ~/src/git/git sparse-checkout init\n> ~/test (master)$ git status \n> On branch master\n> nothing to commit, working tree clean\n> \n> Note also the shell prompt going from \"you have staged changes\" to\n> \"working tree is clean\".\n> \n> But wait, I thought that only changes to files that are excluded from\n> the sparse-checkout are thrown away, but as it turns out it throws\n> away changes to files that are included in the sparse-checkout:\n> \n> ~/test (master #)$ echo original >foo\n> ~/test (master #%)$ git add .\n> ~/test (master +)$ git commit -m Initial\n> [master (root-commit) 04cb2a2] Initial\n> 1 file changed, 1 insertion(+)\n> create mode 100644 foo\n> ~/test (master)$ echo changes >foo \n> ~/test (master *)$ ~/src/git/git sparse-checkout init\n> ~/test (master *)$ cat foo \n> changes\n> \n> So far so good, but:\n> \n> ~/test (master *)$ ~/src/git/git sparse-checkout disable\n> ~/test (master *)$ git add .\n> ~/test (master +)$ ~/src/git/git sparse-checkout init\n> ~/test (master)$ cat foo \n> original\n> \n> This is not really sparse-checkout-specific, because this is what 'git\n> read-tree -um HEAD' always does on its own:\n> \n> ~/test (master)$ echo changes >foo \n> ~/test (master *)$ git read-tree -um HEAD\n> ~/test (master *)$ cat foo \n> changes\n> ~/test (master *)$ git add -u\n> ~/test (master +)$ git read-tree -um HEAD\n> ~/test (master)$ cat foo \n> original\n> \n> These issues are present at the end of the patch series as well, but\n> that is sort-of expected, because the later commit message states\n> that:\n> \n> Remove this extra process call by creating a direct call to\n> unpack_trees() in the same way 'git read-tree -mu HEAD' does.\n\nThanks for the additional details.\n\nThis series intended to make the existing sparse-checkout behavior\nmore useful to users by not requiring manual edits of the sparse-checkout\nfile followed by 'git read-tree' commands. However, there do appear\nto be some serious improvements that we can make in the future.\n\nKeeping staged changes seems important, and we can address that in\nthe near future.\n\n-Stolee\n\n"},{"id":"386735","messageId":"20191121163706.GV23183@szeder.dev","threadId":"51691","inReplyTo":"72a54009-5801-c92a-f108-faf38ebbdd4b@gmail.com","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-11-21T16:37:06Z","receivedAt":"2019-11-21T16:37:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Nov 21, 2019 at 10:36:10AM -0500, Derrick Stolee wrote:\n> > But wait, I thought that only changes to files that are excluded from\n> > the sparse-checkout are thrown away, but as it turns out it throws\n> > away changes to files that are included in the sparse-checkout:\n\nFor completeness, 'git sparse-checkout disable' throws away staged\nchanges as well, as it, too, runs 'git read-tree -um HEAD' (or its\nequivalent).\n\n> Thanks for the additional details.\n> \n> This series intended to make the existing sparse-checkout behavior\n> more useful to users by not requiring manual edits of the sparse-checkout\n> file followed by 'git read-tree' commands. However, there do appear\n> to be some serious improvements that we can make in the future.\n> \n> Keeping staged changes seems important, and we can address that in\n> the near future.\n\nI think that at least for now 'git sparse-checkout' should flat out\nrefuse to init/set/disable if the working tree is not clean (but still\nallow 'list', as that's a read-only operation), like the patch below.\nYeah, that way it wouldn't work in cases that appear to be safe\n(unstaged changes), but it would prevent the data loss until we\ncarefully consider the circumstances under which these operations can\nbe safely allowed.\n\n\n -- >8 --\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 76f65d8edd..4b05625c4c 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -12,6 +12,7 @@\n #include \"lockfile.h\"\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n+#include \"wt-status.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n@@ -188,6 +189,10 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n+\n \tif (init_opts.cone_mode) {\n \t\tmode = MODE_CONE_PATTERNS;\n \t\tcore_sparse_checkout_cone = 1;\n@@ -386,6 +391,10 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"set up sparse-checkout\"), NULL, 1, 0);\n+\n \tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n@@ -437,6 +446,10 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n+\n \tif (sc_set_config(MODE_ALL_PATTERNS))\n \t\tdie(_(\"failed to change config\"));\n \n\n"},{"id":"386736","messageId":"CABPp-BEVFyd9+22MhtREh57FYUJhBSj2vz3piUuk_hX8=Aq3UA@mail.gmail.com","threadId":"51691","inReplyTo":"20191121163706.GV23183@szeder.dev","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-11-21T17:01:19Z","receivedAt":"2019-11-21T17:01:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Nov 21, 2019 at 8:37 AM SZEDER Gábor wrote:\n>\n> On Thu, Nov 21, 2019 at 10:36:10AM -0500, Derrick Stolee wrote:\n> > > But wait, I thought that only changes to files that are excluded from\n> > > the sparse-checkout are thrown away, but as it turns out it throws\n> > > away changes to files that are included in the sparse-checkout:\n>\n> For completeness, 'git sparse-checkout disable' throws away staged\n> changes as well, as it, too, runs 'git read-tree -um HEAD' (or its\n> equivalent).\n>\n> > Thanks for the additional details.\n> >\n> > This series intended to make the existing sparse-checkout behavior\n> > more useful to users by not requiring manual edits of the sparse-checkout\n> > file followed by 'git read-tree' commands. However, there do appear\n> > to be some serious improvements that we can make in the future.\n> >\n> > Keeping staged changes seems important, and we can address that in\n> > the near future.\n>\n> I think that at least for now 'git sparse-checkout' should flat out\n> refuse to init/set/disable if the working tree is not clean (but still\n> allow 'list', as that's a read-only operation), like the patch below.\n> Yeah, that way it wouldn't work in cases that appear to be safe\n> (unstaged changes), but it would prevent the data loss until we\n> carefully consider the circumstances under which these operations can\n> be safely allowed.\n\nA big +1 for this from me.\n\nWe had an unfortunately large number of mis-merging and dataloss bugs\nin the merge machinery that were slowly fixed over the course of more\nthan a decade[1], due to the fact that builtin/merge required\nindex==HEAD and did so by placing a comment in the code notifying\nfolks that the individual merge strategies were responsible to enforce\nit -- and, in practice, they *all* forgot to do so unless and until we\ndiscovered bugs. So, count me as a strongly in favor of just\npreventatively enforcing safe conditions and then later relaxing them\nin special conditions if it can be proven safe.\n\n[1] https://public-inbox.org/git/20190725174611.14802-4-newren@gmail.com/\n\n> -- >8 --\n>\n> diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n> index 76f65d8edd..4b05625c4c 100644\n> --- a/builtin/sparse-checkout.c\n> +++ b/builtin/sparse-checkout.c\n> @@ -12,6 +12,7 @@\n> #include \"lockfile.h\"\n> #include \"resolve-undo.h\"\n> #include \"unpack-trees.h\"\n> +#include \"wt-status.h\"\n>\n> static char const * const builtin_sparse_checkout_usage[] = {\n> N_(\"git sparse-checkout (init|list|set|disable) \"),\n> @@ -188,6 +189,10 @@ static int sparse_checkout_init(int argc, const char **argv)\n> builtin_sparse_checkout_init_options,\n> builtin_sparse_checkout_init_usage, 0);\n>\n> + repo_read_index(the_repository);\n> + require_clean_work_tree(the_repository,\n> + N_(\"initialize sparse-checkout\"), NULL, 1, 0);\n> +\n> if (init_opts.cone_mode) {\n> mode = MODE_CONE_PATTERNS;\n> core_sparse_checkout_cone = 1;\n> @@ -386,6 +391,10 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n> builtin_sparse_checkout_set_usage,\n> PARSE_OPT_KEEP_UNKNOWN);\n>\n> + repo_read_index(the_repository);\n> + require_clean_work_tree(the_repository,\n> + N_(\"set up sparse-checkout\"), NULL, 1, 0);\n> +\n> if (core_sparse_checkout_cone) {\n> struct strbuf line = STRBUF_INIT;\n> hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n> @@ -437,6 +446,10 @@ static int sparse_checkout_disable(int argc, const char **argv)\n> char *sparse_filename;\n> FILE *fp;\n>\n> + repo_read_index(the_repository);\n> + require_clean_work_tree(the_repository,\n> + N_(\"disable sparse-checkout\"), NULL, 1, 0);\n> +\n> if (sc_set_config(MODE_ALL_PATTERNS))\n> die(_(\"failed to change config\"));\n"},{"id":"386737","messageId":"42d485d5-9b9d-6ec3-0b52-0da916faf3ad@gmail.com","threadId":"51691","inReplyTo":"CABPp-BEVFyd9+22MhtREh57FYUJhBSj2vz3piUuk_hX8=Aq3UA@mail.gmail.com","subject":"Re: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-11-21T17:15:23Z","receivedAt":"2019-11-21T17:15:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/21/2019 12:01 PM, Elijah Newren wrote:\n> On Thu, Nov 21, 2019 at 8:37 AM SZEDER Gábor wrote:\n>>\n>> On Thu, Nov 21, 2019 at 10:36:10AM -0500, Derrick Stolee wrote:\n>>>> But wait, I thought that only changes to files that are excluded from\n>>>> the sparse-checkout are thrown away, but as it turns out it throws\n>>>> away changes to files that are included in the sparse-checkout:\n>>\n>> For completeness, 'git sparse-checkout disable' throws away staged\n>> changes as well, as it, too, runs 'git read-tree -um HEAD' (or its\n>> equivalent).\n>>\n>>> Thanks for the additional details.\n>>>\n>>> This series intended to make the existing sparse-checkout behavior\n>>> more useful to users by not requiring manual edits of the sparse-checkout\n>>> file followed by 'git read-tree' commands. However, there do appear\n>>> to be some serious improvements that we can make in the future.\n>>>\n>>> Keeping staged changes seems important, and we can address that in\n>>> the near future.\n>>\n>> I think that at least for now 'git sparse-checkout' should flat out\n>> refuse to init/set/disable if the working tree is not clean (but still\n>> allow 'list', as that's a read-only operation), like the patch below.\n>> Yeah, that way it wouldn't work in cases that appear to be safe\n>> (unstaged changes), but it would prevent the data loss until we\n>> carefully consider the circumstances under which these operations can\n>> be safely allowed.\n> \n> A big +1 for this from me.\n> \n> We had an unfortunately large number of mis-merging and dataloss bugs\n> in the merge machinery that were slowly fixed over the course of more\n> than a decade[1], due to the fact that builtin/merge required\n> index==HEAD and did so by placing a comment in the code notifying\n> folks that the individual merge strategies were responsible to enforce\n> it -- and, in practice, they *all* forgot to do so unless and until we\n> discovered bugs. So, count me as a strongly in favor of just\n> preventatively enforcing safe conditions and then later relaxing them\n> in special conditions if it can be proven safe.\n> \n> [1] https://public-inbox.org/git/20190725174611.14802-4-newren@gmail.com/\n\nSounds good. Thanks, both.\n\n-Stolee\n"},{"id":"386745","messageId":"fcfe477813cfe91c1e84a224e6736c9c238ca29e.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 02/19] sparse-checkout: create 'init' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:34Z","receivedAt":"2019-11-21T22:05:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGetting started with a sparse-checkout file can be daunting. Help\nusers start their sparse enlistment using 'git sparse-checkout init'.\nThis will set 'core.sparseCheckout=true' in their config, write\nan initial set of patterns to the sparse-checkout file, and update\ntheir working directory.\n\nMake sure to use the `extensions.worktreeConfig` setting and write\nthe sparse checkout config to the worktree-specific config file.\nThis avoids confusing interactions with other worktrees.\n\nThe use of running another process for 'git read-tree' is sub-\noptimal. This will be removed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 11 ++++\n builtin/sparse-checkout.c | 75 ++++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 40 ++++++++++++++\n 3 files changed, 125 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 87ffcbbcb0..491be1345f 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -30,6 +30,17 @@ COMMANDS\n 'list'::\n \tProvide a list of the contents in the sparse-checkout file.\n \n+'init'::\n+\tEnable the `core.sparseCheckout` setting. If the\n+\tsparse-checkout file does not exist, then populate it with\n+\tpatterns that match every file in the root directory and\n+\tno other directories, then will remove all directories tracked\n+\tby Git. Add patterns to the sparse-checkout file to\n+\trepopulate the working directory.\n++\n+To avoid interfering with other worktrees, it first enables the\n+`extensions.worktreeConfig` setting and makes sure to set the\n+`core.sparseCheckout` setting in the worktree-specific config file.\n \n SPARSE CHECKOUT\n ---------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 5717c9b2cb..fcf97e9df8 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout list\"),\n+\tN_(\"git sparse-checkout (init|list)\"),\n \tNULL\n };\n \n@@ -59,6 +59,77 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n+static int update_working_directory(void)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n+enum sparse_checkout_mode {\n+\tMODE_NO_PATTERNS = 0,\n+\tMODE_ALL_PATTERNS = 1,\n+};\n+\n+static int set_config(enum sparse_checkout_mode mode)\n+{\n+\tconst char *config_path;\n+\n+\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n+\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n+\t\treturn 1;\n+\t}\n+\n+\tconfig_path = git_path(\"config.worktree\");\n+\tgit_config_set_in_file_gently(config_path,\n+\t\t\t\t \"core.sparseCheckout\",\n+\t\t\t\t mode ? \"true\" : NULL);\n+\n+\treturn 0;\n+}\n+\n+static int sparse_checkout_init(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\tint res;\n+\n+\tif (set_config(MODE_ALL_PATTERNS))\n+\t\treturn 1;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\n+\t/* If we already have a sparse-checkout file, use it. */\n+\tif (res >= 0) {\n+\t\tfree(sparse_filename);\n+\t\tgoto reset_dir;\n+\t}\n+\n+\t/* initial mode: all blobs at root */\n+\tfp = xfopen(sparse_filename, \"w\");\n+\tif (!fp)\n+\t\tdie(_(\"failed to open '%s'\"), sparse_filename);\n+\n+\tfree(sparse_filename);\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\tfclose(fp);\n+\n+reset_dir:\n+\treturn update_working_directory();\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -79,6 +150,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"list\"))\n \t\t\treturn sparse_checkout_list(argc, argv);\n+\t\tif (!strcmp(argv[0], \"init\"))\n+\t\t\treturn sparse_checkout_init(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 9b73d44907..21143c529c 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -42,4 +42,44 @@ test_expect_success 'git sparse-checkout list (populated)' '\n \ttest_cmp expect list\n '\n \n+test_expect_success 'git sparse-checkout init' '\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\ttest_cmp_config -C repo true core.sparsecheckout &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n+test_expect_success 'git sparse-checkout list after init' '\n+\tgit -C repo sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'init with existing sparse-checkout' '\n+\techo \"*folder*\" >> repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout init &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386746","messageId":"81a15260695f83cf46185ed5357dd4f0e5dae384.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 01/19] sparse-checkout: create builtin with 'list' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:33Z","receivedAt":"2019-11-21T22:05:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature is mostly hidden to users, as its\nonly documentation is supplementary information in the docs for\n'git read-tree'. In addition, users need to know how to edit the\n.git/info/sparse-checkout file with the right patterns, then run\nthe appropriate 'git read-tree -mu HEAD' command. Keeping the\nworking directory in sync with the sparse-checkout file requires\ncare.\n\nBegin an effort to make the sparse-checkout feature a porcelain\nfeature by creating a new 'git sparse-checkout' builtin. This\nbuiltin will be the preferred mechanism for manipulating the\nsparse-checkout file and syncing the working directory.\n\nThe documentation provided is adapted from the \"git read-tree\"\ndocumentation with a few edits for clarity in the new context.\nExtra sections are added to hint toward a future change to\na more restricted pattern set.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 89 +++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 45 ++++++++++++++\n 9 files changed, 226 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 89b3b79c1a..aebe7c0908 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -158,6 +158,7 @@\n /git-show-branch\n /git-show-index\n /git-show-ref\n+/git-sparse-checkout\n /git-stage\n /git-stash\n /git-status\ndiff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt\nindex d271842608..da33f84f33 100644\n--- a/Documentation/git-read-tree.txt\n+++ b/Documentation/git-read-tree.txt\n@@ -436,7 +436,7 @@ support.\n SEE ALSO\n --------\n linkgit:git-write-tree[1]; linkgit:git-ls-files[1];\n-linkgit:gitignore[5]\n+linkgit:gitignore[5]; linkgit:git-sparse-checkout[1];\n \n GIT\n ---\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nnew file mode 100644\nindex 0000000000..87ffcbbcb0\n--- /dev/null\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -0,0 +1,89 @@\n+git-sparse-checkout(1)\n+======================\n+\n+NAME\n+----\n+git-sparse-checkout - Initialize and modify the sparse-checkout\n+configuration, which reduces the checkout to a set of paths\n+given by a list of atterns.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git sparse-checkout [options]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Initialize and modify the sparse-checkout configuration, which reduces\n+the checkout to a set of paths given by a list of patterns.\n+\n+THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n+COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n+THE FUTURE.\n+\n+\n+COMMANDS\n+--------\n+'list'::\n+\tProvide a list of the contents in the sparse-checkout file.\n+\n+\n+SPARSE CHECKOUT\n+---------------\n+\n+\"Sparse checkout\" allows populating the working directory sparsely.\n+It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell\n+Git whether a file in the working directory is worth looking at. If\n+the skip-worktree bit is set, then the file is ignored in the working\n+directory. Git will not populate the contents of those files, which\n+makes a sparse checkout helpful when working in a repository with many\n+files, but only a few are important to the current user.\n+\n+The `$GIT_DIR/info/sparse-checkout` file is used to define the\n+skip-worktree reference bitmap. When Git updates the working\n+directory, it updates the skip-worktree bits in the index based\n+on this file. The files matching the patterns in the file will\n+appear in the working directory, and the rest will not.\n+\n+## FULL PATTERN SET\n+\n+By default, the sparse-checkout file uses the same syntax as `.gitignore`\n+files.\n+\n+While `$GIT_DIR/info/sparse-checkout` is usually used to specify what\n+files are included, you can also specify what files are _not_ included,\n+using negative patterns. For example, to remove the file `unwanted`:\n+\n+----------------\n+/*\n+!unwanted\n+----------------\n+\n+Another tricky thing is fully repopulating the working directory when you\n+no longer want sparse checkout. You cannot just disable \"sparse\n+checkout\" because skip-worktree bits are still in the index and your working\n+directory is still sparsely populated. You should re-populate the working\n+directory with the `$GIT_DIR/info/sparse-checkout` file content as\n+follows:\n+\n+----------------\n+/*\n+----------------\n+\n+Then you can disable sparse checkout. Sparse checkout support in 'git\n+checkout' and similar commands is disabled by default. You need to\n+set `core.sparseCheckout` to `true` in order to have sparse checkout\n+support.\n+\n+SEE ALSO\n+--------\n+\n+linkgit:git-read-tree[1]\n+linkgit:gitignore[5]\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex de60c8e7aa..adefc229fe 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\n BUILTIN_OBJS += builtin/show-index.o\n BUILTIN_OBJS += builtin/show-ref.o\n+BUILTIN_OBJS += builtin/sparse-checkout.o\n BUILTIN_OBJS += builtin/stash.o\n BUILTIN_OBJS += builtin/stripspace.o\n BUILTIN_OBJS += builtin/submodule--helper.o\ndiff --git a/builtin.h b/builtin.h\nindex 5cf5df69f7..2b25a80cde 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\n int cmd_show_branch(int argc, const char **argv, const char *prefix);\n int cmd_show_index(int argc, const char **argv, const char *prefix);\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix);\n int cmd_status(int argc, const char **argv, const char *prefix);\n int cmd_stash(int argc, const char **argv, const char *prefix);\n int cmd_stripspace(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nnew file mode 100644\nindex 0000000000..5717c9b2cb\n--- /dev/null\n+++ b/builtin/sparse-checkout.c\n@@ -0,0 +1,86 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"parse-options.h\"\n+#include \"pathspec.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+\n+static char const * const builtin_sparse_checkout_usage[] = {\n+\tN_(\"git sparse-checkout list\"),\n+\tNULL\n+};\n+\n+static char *get_sparse_checkout_filename(void)\n+{\n+\treturn git_pathdup(\"info/sparse-checkout\");\n+}\n+\n+static void write_patterns_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\n+\tfor (i = 0; i < pl->nr; i++) {\n+\t\tstruct path_pattern *p = pl->patterns[i];\n+\n+\t\tif (p->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\tfprintf(fp, \"!\");\n+\n+\t\tfprintf(fp, \"%s\", p->pattern);\n+\n+\t\tif (p->flags & PATTERN_FLAG_MUSTBEDIR)\n+\t\t\tfprintf(fp, \"/\");\n+\n+\t\tfprintf(fp, \"\\n\");\n+\t}\n+}\n+\n+static int sparse_checkout_list(int argc, const char **argv)\n+{\n+\tstruct pattern_list pl;\n+\tchar *sparse_filename;\n+\tint res;\n+\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tres = add_patterns_from_file_to_list(sparse_filename, \"\", 0, &pl, NULL);\n+\tfree(sparse_filename);\n+\n+\tif (res < 0) {\n+\t\twarning(_(\"this worktree is not sparse (sparse-checkout file may not exist)\"));\n+\t\treturn 0;\n+\t}\n+\n+\twrite_patterns_to_file(stdout, &pl);\n+\tclear_pattern_list(&pl);\n+\n+\treturn 0;\n+}\n+\n+int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_sparse_checkout_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t\t builtin_sparse_checkout_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_options,\n+\t\t\t builtin_sparse_checkout_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"list\"))\n+\t\t\treturn sparse_checkout_list(argc, argv);\n+\t}\n+\n+\tusage_with_options(builtin_sparse_checkout_usage,\n+\t\t\t builtin_sparse_checkout_options);\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex a9ac72bef4..d3d28252b3 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -166,6 +166,7 @@ git-show-index plumbinginterrogators\n git-show-ref plumbinginterrogators\n git-sh-i18n purehelpers\n git-sh-setup purehelpers\n+git-sparse-checkout mainporcelain worktree\n git-stash mainporcelain\n git-stage complete\n git-status mainporcelain info\ndiff --git a/git.c b/git.c\nindex ce6ab0ece2..7be7ad34bd 100644\n--- a/git.c\n+++ b/git.c\n@@ -572,6 +572,7 @@ static struct cmd_struct commands[] = {\n \t{ \"show-branch\", cmd_show_branch, RUN_SETUP },\n \t{ \"show-index\", cmd_show_index },\n \t{ \"show-ref\", cmd_show_ref, RUN_SETUP },\n+\t{ \"sparse-checkout\", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"stage\", cmd_add, RUN_SETUP | NEED_WORK_TREE },\n \t/*\n \t * NEEDSWORK: Until the builtin stash is thoroughly robust and no\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nnew file mode 100755\nindex 0000000000..9b73d44907\n--- /dev/null\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -0,0 +1,45 @@\n+#!/bin/sh\n+\n+test_description='sparse checkout builtin tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\techo \"initial\" >a &&\n+\t\tmkdir folder1 folder2 deep &&\n+\t\tmkdir deep/deeper1 deep/deeper2 &&\n+\t\tmkdir deep/deeper1/deepest &&\n+\t\tcp a folder1 &&\n+\t\tcp a folder2 &&\n+\t\tcp a deep &&\n+\t\tcp a deep/deeper1 &&\n+\t\tcp a deep/deeper2 &&\n+\t\tcp a deep/deeper1/deepest &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"initial commit\"\n+\t)\n+'\n+\n+test_expect_success 'git sparse-checkout list (empty)' '\n+\tgit -C repo sparse-checkout list >list 2>err &&\n+\ttest_must_be_empty list &&\n+\ttest_i18ngrep \"this worktree is not sparse (sparse-checkout file may not exist)\" err\n+'\n+\n+test_expect_success 'git sparse-checkout list (populated)' '\n+\ttest_when_finished rm -f repo/.git/info/sparse-checkout &&\n+\tcat >repo/.git/info/sparse-checkout <<-EOF &&\n+\t\t/folder1/*\n+\t\t/deep/\n+\t\t**/a\n+\t\t!*bin*\n+\tEOF\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\tgit -C repo sparse-checkout list >list &&\n+\ttest_cmp expect list\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"386747","messageId":"7ef5f2ef693153a0f786e1d913d4258059cd587a.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 03/19] clone: add --sparse mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:35Z","receivedAt":"2019-11-21T22:05:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen someone wants to clone a large repository, but plans to work\nusing a sparse-checkout file, they either need to do a full\ncheckout first and then reduce the patterns they included, or\nclone with --no-checkout, set up their patterns, and then run\na checkout manually. This requires knowing a lot about the repo\nshape and how sparse-checkout works.\n\nAdd a new '--sparse' option to 'git clone' that initializes the\nsparse-checkout file to include the following patterns:\n\n\t/*\n\t!/*/\n\nThese patterns include every file in the root directory, but\nno directories. This allows a repo to include files like a\nREADME or a bootstrapping script to grow enlistments from that\npoint.\n\nDuring the 'git sparse-checkout init' call, we must first look\nto see if HEAD is valid, since 'git clone' does not have a valid\nHEAD at the point where it initializes the sparse-checkout. The\nfollowing checkout within the clone command will create the HEAD\nref and update the working directory correctly.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-clone.txt | 8 +++++++-\n builtin/clone.c | 27 +++++++++++++++++++++++++++\n builtin/sparse-checkout.c | 6 ++++++\n t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++\n 4 files changed, 53 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 34011c2940..0fe91d2f04 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -15,7 +15,7 @@ SYNOPSIS\n \t [--dissociate] [--separate-git-dir ]\n \t [--depth ] [--[no-]single-branch] [--no-tags]\n \t [--recurse-submodules[=]] [--[no-]shallow-submodules]\n-\t [--[no-]remote-submodules] [--jobs ] [--] \n+\t [--[no-]remote-submodules] [--jobs ] [--sparse] [--] \n \t []\n \n DESCRIPTION\n@@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.\n \tused, neither remote-tracking branches nor the related\n \tconfiguration variables are created.\n \n+--sparse::\n+\tInitialize the sparse-checkout file so the working\n+\tdirectory starts with only the files in the root\n+\tof the repository. The sparse-checkout file can be\n+\tmodified to grow the working directory as needed.\n+\n --mirror::\n \tSet up a mirror of the source repository. This implies `--bare`.\n \tCompared to `--bare`, `--mirror` not only maps local branches of the\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex c46ee29f0a..4348d962c9 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -59,6 +59,7 @@ static const char *real_git_dir;\n static char *option_upload_pack = \"git-upload-pack\";\n static int option_verbosity;\n static int option_progress = -1;\n+static int option_sparse_checkout;\n static enum transport_family family;\n static struct string_list option_config = STRING_LIST_INIT_NODUP;\n static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;\n@@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = {\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_BOOL(0, \"remote-submodules\", &option_remote_submodules,\n \t\t N_(\"any cloned submodules will use their remote-tracking branch\")),\n+\tOPT_BOOL(0, \"sparse\", &option_sparse_checkout,\n+\t\t N_(\"initialize sparse-checkout file to include only files at root\")),\n \tOPT_END()\n };\n \n@@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote,\n \t}\n }\n \n+static int git_sparse_checkout_init(const char *repo)\n+{\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint result = 0;\n+\targv_array_pushl(&argv, \"-C\", repo, \"sparse-checkout\", \"init\", NULL);\n+\n+\t/*\n+\t * We must apply the setting in the current process\n+\t * for the later checkout to use the sparse-checkout file.\n+\t */\n+\tcore_apply_sparse_checkout = 1;\n+\n+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n+\t\terror(_(\"failed to initialize sparse-checkout\"));\n+\t\tresult = 1;\n+\t}\n+\n+\targv_array_clear(&argv);\n+\treturn result;\n+}\n+\n static int checkout(int submodule_progress)\n {\n \tstruct object_id oid;\n@@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_required_reference.nr || option_optional_reference.nr)\n \t\tsetup_reference();\n \n+\tif (option_sparse_checkout && git_sparse_checkout_init(repo))\n+\t\treturn 1;\n+\n \tremote = remote_get(option_origin);\n \n \tstrbuf_addf(&default_refspec, \"+%s*:%s*\", src_ref_prefix,\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex fcf97e9df8..e3418fbe2a 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -102,6 +102,7 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tchar *sparse_filename;\n \tFILE *fp;\n \tint res;\n+\tstruct object_id oid;\n \n \tif (set_config(MODE_ALL_PATTERNS))\n \t\treturn 1;\n@@ -126,6 +127,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n \tfclose(fp);\n \n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\t/* assume we are in a fresh repo */\n+\t\treturn 0;\n+\t}\n+\n reset_dir:\n \treturn update_working_directory();\n }\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 21143c529c..78c20cb7e3 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -82,4 +82,17 @@ test_expect_success 'init with existing sparse-checkout' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'clone --sparse' '\n+\tgit clone --sparse repo clone &&\n+\tgit -C clone sparse-checkout list >actual &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\tEOF\n+\ttest_cmp expect actual &&\n+\tls clone >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386748","messageId":"248fc172acc1cb3d0fdb5c3cf4ac91a51d0c85d9.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 04/19] sparse-checkout: 'set' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:36Z","receivedAt":"2019-11-21T22:05:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nas arguments and writes them to the sparse-checkout file. Then, it\nupdates the working directory using 'git read-tree -mu HEAD'.\n\nThe 'set' subcommand will replace the entire contents of the\nsparse-checkout file. The write_patterns_and_update() method is\nextracted from cmd_sparse_checkout() to make it easier to implement\n'add' and/or 'remove' subcommands in the future.\n\nIf the core.sparseCheckout config setting is disabled, then enable\nthe config setting in the worktree config. If we set the config\nthis way and the sparse-checkout fails, then re-disable the config\nsetting.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 6 ++++\n builtin/sparse-checkout.c | 47 +++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 33 +++++++++++++++++++\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 491be1345f..ca62669b8c 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -42,6 +42,12 @@ To avoid interfering with other worktrees, it first enables the\n `extensions.worktreeConfig` setting and makes sure to set the\n `core.sparseCheckout` setting in the worktree-specific config file.\n \n+'set'::\n+\tWrite a set of patterns to the sparse-checkout file, as given as\n+\ta list of arguments following the 'set' subcommand. Update the\n+\tworking directory to match the new patterns. Enable the\n+\tcore.sparseCheckout config setting if it is not already enabled.\n+\n SPARSE CHECKOUT\n ---------------\n \ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex e3418fbe2a..95cbd0a42c 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list)\"),\n+\tN_(\"git sparse-checkout (init|list|set) \"),\n \tNULL\n };\n \n@@ -66,7 +66,7 @@ static int update_working_directory(void)\n \targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n \n \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n-\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n+\t\terror(_(\"failed to update index with new sparse-checkout patterns\"));\n \t\tresult = 1;\n \t}\n \n@@ -136,6 +136,47 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\twrite_patterns_to_file(fp, pl);\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\treturn update_working_directory();\n+}\n+\n+static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *empty_base = \"\";\n+\tint i;\n+\tstruct pattern_list pl;\n+\tint result;\n+\tint changed_config = 0;\n+\tmemset(&pl, 0, sizeof(pl));\n+\n+\tfor (i = 1; i < argc; i++)\n+\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\n+\tif (!core_apply_sparse_checkout) {\n+\t\tset_config(MODE_ALL_PATTERNS);\n+\t\tcore_apply_sparse_checkout = 1;\n+\t\tchanged_config = 1;\n+\t}\n+\n+\tresult = write_patterns_and_update(&pl);\n+\n+\tif (result && changed_config)\n+\t\tset_config(MODE_NO_PATTERNS);\n+\n+\tclear_pattern_list(&pl);\n+\treturn result;\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -158,6 +199,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_list(argc, argv);\n \t\tif (!strcmp(argv[0], \"init\"))\n \t\t\treturn sparse_checkout_init(argc, argv);\n+\t\tif (!strcmp(argv[0], \"set\"))\n+\t\t\treturn sparse_checkout_set(argc, argv, prefix);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 78c20cb7e3..72d8bc5c25 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -95,4 +95,37 @@ test_expect_success 'clone --sparse' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set enables config' '\n+\tgit init empty-config &&\n+\t(\n+\t\tcd empty-config &&\n+\t\ttest_commit test file &&\n+\t\ttest_path_is_missing .git/config.worktree &&\n+\t\ttest_must_fail git sparse-checkout set nothing &&\n+\t\ttest_path_is_file .git/config.worktree &&\n+\t\ttest_must_fail git config core.sparseCheckout &&\n+\t\tgit sparse-checkout set \"/*\" &&\n+\t\ttest_cmp_config true core.sparseCheckout\n+\t)\n+'\n+\n+test_expect_success 'set sparse-checkout using builtin' '\n+\tgit -C repo sparse-checkout set \"/*\" \"!/*/\" \"*folder*\" &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t*folder*\n+\tEOF\n+\tgit -C repo sparse-checkout list >actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386749","messageId":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v5.git.1571666186.gitgitgadget@gmail.com","subject":"[PATCH v6 00/19] New sparse-checkout builtin and \"cone\" mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:32Z","receivedAt":"2019-11-21T22:05:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series makes the sparse-checkout feature more user-friendly. While\nthere, I also present a way to use a limited set of patterns to gain a\nsignificant performance boost in very large repositories.\n\nSparse-checkout is only documented as a subsection of the read-tree docs\n[1], which makes the feature hard to discover. Users have trouble navigating\nthe feature, especially at clone time [2], and have even resorted to\ncreating their own helper tools [3].\n\nThis series attempts to solve these problems using a new builtin. Here is a\nsample workflow to give a feeling for how it can work:\n\nIn an existing repo:\n\n$ git sparse-checkout init\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n$ ls myFolder\na.c a.h\n$ git sparse-checkout disable\n$ ls\nhiddenFolder myFile1.txt myFile2.txt myFolder\n\nAt clone time:\n\n$ git clone --sparse origin repo\n$ cd repo\n$ ls\nmyFile1.txt myFile2.txt\n$ git sparse-checkout set \"/*\" \"!/*/\" /myFolder/\n$ ls\nmyFile1.txt myFile2.txt myFolder\n\nHere are some more specific details:\n\n * git sparse-checkout init enables core.sparseCheckout and populates the\n sparse-checkout file with patterns that match only the files at root.\n \n \n * git clone learns the --sparse argument to run git sparse-checkout init \n before the first checkout.\n \n \n * git sparse-checkout set reads patterns from the arguments, or with\n --stdin reads patterns from stdin one per line, then writes them to the\n sparse-checkout file and refreshes the working directory.\n \n \n * git sparse-checkout disable removes the patterns from the sparse-checkout\n file, disables core.sparseCheckout, and refills the working directory.\n \n \n * git sparse-checkout list lists the contents of the sparse-checkout file.\n \n \n\nThe documentation for the sparse-checkout feature can now live primarily\nwith the git-sparse-checkout documentation.\n\nCone Mode\n=========\n\nWhat really got me interested in this area is a performance problem. If we\nhave N patterns in the sparse-checkout file and M entries in the index, then\nwe can perform up to O(N * M) pattern checks in clear_ce_flags(). This\nquadratic growth is not sustainable in a repo with 1,000+ patterns and\n1,000,000+ index entries.\n\nTo solve this problem, I propose a new, more restrictive mode to\nsparse-checkout: \"cone mode\". In this mode, all patterns are based on prefix\nmatches at a directory level. This can then use hashsets for fast\nperformance -- O(M) instead of O(N*M). My hashset implementation is based on\nthe virtual filesystem hook in the VFS for Git custom code [4].\n\nIn cone mode, a user specifies a list of folders which the user wants every\nfile inside. In addition, the cone adds all blobs that are siblings of the\nfolders in the directory path to that folder. This makes the directories\nlook \"hydrated\" as a user drills down to those recursively-closed folders.\nThese directories are called \"parent\" folders, as a file matches them only\nif the file's immediate parent is that directory.\n\nWhen building a prototype of this feature, I used a separate file to contain\nthe list of recursively-closed folders and built the hashsets dynamically\nbased on that file. In this implementation, I tried to maximize the amount\nof backwards-compatibility by storing all data in the sparse-checkout file\nusing patterns recognized by earlier Git versions.\n\nFor example, if we add A/B/C as a recursive folder, then we add the\nfollowing patterns to the sparse-checkout file:\n\n/*\n!/*/\n/A/\n!/A/*/\n/A/B/\n!/A/B/*/\n/A/B/C/\n\nThe alternating positive/negative patterns say \"include everything in this\nfolder, but exclude everything another level deeper\". The final pattern has\nno matching negation, so is a recursively closed pattern.\n\nNote that I have some basic warnings to try and check that the\nsparse-checkout file doesn't match what would be written by a cone-mode add.\nIn such a case, Git writes a warning to stderr and continues with the old\npattern matching algorithm. These checks are currently very barebones, and\nwould need to be updated with more robust checks for things like regex\ncharacters in the middle of the pattern. As review moves forward (and if we\ndon't change the data storage) then we could spend more time on this.\n\nThanks, -Stolee\n\nUpdates in v2, relative to the RFC:\n\n * Instead of an 'add' subcommand, use a 'set' subcommand. We can consider\n adding 'add' and/or 'remove' subcommands later.\n \n \n * 'set' reads from the arguments by default. '--stdin' option is available.\n \n \n * A new performance-oriented commit is added at the end.\n \n \n * Patterns no longer end with a trailing asterisk except for the first \"/*\"\n pattern.\n \n \n * References to a \"bug\" (that was really a strange GVFS interaction in\n microsoft/git) around deleting outside the cone are removed.\n \n \n\nUpdates in v3:\n\n * The bad interaction with \"cone mode\" and .gitignore files is fixed. A\n test is added in the last patch.\n \n \n * Several patches are added that make the feature more robust. One\n sanitizes user input, another few add progress indicators, and another\n more prevent users from getting in bad states due to working directory\n changes or concurrent processes.\n \n \n * Updated several docs and commit messages according to feedback. Thanks,\n Elijah!\n \n \n\nUpdates in V4:\n\n * Updated hashmap API usage to respond to ew/hashmap\n \n \n * Responded to detailed review by Elijah. Thanks!\n \n \n * Marked the feature as experimental in git-sparse-checkout.txt the same\n way that git-switch.txt does.\n \n \n\nUpdates in V5:\n\n * The 'set' subcommand now enables the core.sparseCheckout config setting\n (unless the checkout fails).\n \n \n * If the in-process unpack_trees() fails with the new patterns, the\n index.lock file is rolled back before the replay of the old\n sparse-checkout patterns.\n \n \n * Some documentation fixes, f(d)open->xf(d)open calls, and other nits.\n Thanks everyone!\n \n \n\nUpdates in V6:\n\n * The init, set, and disable commands now require a clean status.\n \n \n * Git config is now set in-process instead of via a run_command call.\n \n \n * The working directory was being updated twice, leading to multiple errors\n being shown if the working directory would become empty.\n \n \n * Before, only the 'set' command used the in-process workdir update. Now\n 'init' and 'disable' also use this in-process code, which removes some\n error cases.\n \n \n\nThings to leave for future patches:\n\n 1. Integrate in 'git worktree add' to copy the sparse-checkout file to a\n worktree-specific file.\n \n \n 2. More robustness around detecting non-cone patterns with wildcards in the\n middle of the line.\n \n \n 3. 'git clone --sparse-cone' to clone into \"cone mode\" sparse-checkouts\n (i.e. set 'core.sparseCheckoutCone=true'). This may not be\n super-valuable, as it only starts changing behavior when someone calls\n 'git sparse-checkout set', but may be interesting.\n \n \n 4. Make the working-directory update not modify the staging environment.\n Block only if it would lose work-in-progress.\n \n \n 5. Some robustness things can be saved for later, such as including pattern\n arguments next to \"--stdin\", \"set --cone\", etc.\n \n \n\n[1] https://git-scm.com/docs/git-read-tree#_sparse_checkoutSparse-checkout\ndocumentation in git-read-tree.\n\n[2] https://stackoverflow.com/a/4909267/127088Is it possible to do a sparse\ncheckout without checking out the whole repository first?\n\n[3] http://www.marcoyuen.com/articles/2016/06/07/git-sparse.htmlA blog post\nof a user's extra \"git-sparse\" helper.\n\n[4] \nhttps://github.com/git/git/compare/fc5fd706ff733392053e6180086a4d7f96acc2af...01204f24c5349aa2fb0c474546d768946d315dab\nThe virtual filesystem hook in microsoft/git.\n\nDerrick Stolee (18):\n sparse-checkout: create builtin with 'list' subcommand\n sparse-checkout: create 'init' subcommand\n clone: add --sparse mode\n sparse-checkout: 'set' subcommand\n sparse-checkout: add '--stdin' option to set subcommand\n sparse-checkout: create 'disable' subcommand\n sparse-checkout: add 'cone' mode\n sparse-checkout: use hashmaps for cone patterns\n sparse-checkout: init and set in cone mode\n unpack-trees: hash less in cone mode\n unpack-trees: add progress to clear_ce_flags()\n sparse-checkout: sanitize for nested folders\n sparse-checkout: update working directory in-process\n sparse-checkout: use in-process update for disable subcommand\n sparse-checkout: write using lockfile\n sparse-checkout: cone mode should not interact with .gitignore\n sparse-checkout: update working directory in-process for 'init'\n sparse-checkout: check for dirty status\n\nJeff Hostetler (1):\n trace2: add region in clear_ce_flags\n\n .gitignore | 1 +\n Documentation/config/core.txt | 10 +-\n Documentation/git-clone.txt | 8 +-\n Documentation/git-read-tree.txt | 2 +-\n Documentation/git-sparse-checkout.txt | 161 +++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/clone.c | 27 ++\n builtin/read-tree.c | 2 +-\n builtin/sparse-checkout.c | 489 ++++++++++++++++++++++++++\n cache.h | 6 +-\n command-list.txt | 1 +\n config.c | 5 +\n dir.c | 207 ++++++++++-\n dir.h | 36 ++\n environment.c | 1 +\n git.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 307 ++++++++++++++++\n unpack-trees.c | 110 ++++--\n unpack-trees.h | 3 +-\n 20 files changed, 1331 insertions(+), 48 deletions(-)\n create mode 100644 Documentation/git-sparse-checkout.txt\n create mode 100644 builtin/sparse-checkout.c\n create mode 100755 t/t1091-sparse-checkout-builtin.sh\n\n\nbase-commit: 108b97dc372828f0e72e56bbb40cae8e1e83ece6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-316%2Fderrickstolee%2Fsparse-checkout%2Fupstream-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-316/derrickstolee/sparse-checkout/upstream-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/316\n\nRange-diff vs v5:\n\n 1: 55c306a73a ! 1: 81a1526069 sparse-checkout: create builtin with 'list' subcommand\n @@ -59,8 +59,8 @@\n +NAME\n +----\n +git-sparse-checkout - Initialize and modify the sparse-checkout\n -+configuration, which reduces the checkout to a set of directories\n -+given by a list of prefixes.\n ++configuration, which reduces the checkout to a set of paths\n ++given by a list of atterns.\n +\n +\n +SYNOPSIS\n @@ -73,7 +73,7 @@\n +-----------\n +\n +Initialize and modify the sparse-checkout configuration, which reduces\n -+the checkout to a set of directories given by a list of prefixes.\n ++the checkout to a set of paths given by a list of patterns.\n +\n +THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER\n +COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN\n 2: a161cee0df ! 2: fcfe477813 sparse-checkout: create 'init' subcommand\n @@ -75,26 +75,19 @@\n +\tMODE_ALL_PATTERNS = 1,\n +};\n +\n -+static int sc_set_config(enum sparse_checkout_mode mode)\n ++static int set_config(enum sparse_checkout_mode mode)\n +{\n -+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n ++\tconst char *config_path;\n +\n +\tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n +\t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n +\t\treturn 1;\n +\t}\n +\n -+\targv_array_pushl(&argv, \"config\", \"--worktree\", \"core.sparseCheckout\", NULL);\n -+\n -+\tif (mode)\n -+\t\targv_array_pushl(&argv, \"true\", NULL);\n -+\telse\n -+\t\targv_array_pushl(&argv, \"false\", NULL);\n -+\n -+\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n -+\t\terror(_(\"failed to enable core.sparseCheckout\"));\n -+\t\treturn 1;\n -+\t}\n ++\tconfig_path = git_path(\"config.worktree\");\n ++\tgit_config_set_in_file_gently(config_path,\n ++\t\t\t\t \"core.sparseCheckout\",\n ++\t\t\t\t mode ? \"true\" : NULL);\n +\n +\treturn 0;\n +}\n @@ -106,7 +99,7 @@\n +\tFILE *fp;\n +\tint res;\n +\n -+\tif (sc_set_config(MODE_ALL_PATTERNS))\n ++\tif (set_config(MODE_ALL_PATTERNS))\n +\t\treturn 1;\n +\n +\tmemset(&pl, 0, sizeof(pl));\n @@ -122,6 +115,9 @@\n +\n +\t/* initial mode: all blobs at root */\n +\tfp = xfopen(sparse_filename, \"w\");\n ++\tif (!fp)\n ++\t\tdie(_(\"failed to open '%s'\"), sparse_filename);\n ++\n +\tfree(sparse_filename);\n +\tfprintf(fp, \"/*\\n!/*/\\n\");\n +\tfclose(fp);\n @@ -157,8 +153,7 @@\n +\t\t!/*/\n +\tEOF\n +\ttest_cmp expect repo/.git/info/sparse-checkout &&\n -+\tgit -C repo config --list >config &&\n -+\ttest_i18ngrep \"core.sparsecheckout=true\" config &&\n ++\ttest_cmp_config -C repo true core.sparsecheckout &&\n +\tls repo >dir &&\n +\techo a >expect &&\n +\ttest_cmp expect dir\n 3: cd150853b7 ! 3: 7ef5f2ef69 clone: add --sparse mode\n @@ -122,7 +122,7 @@\n \tint res;\n +\tstruct object_id oid;\n \n - \tif (sc_set_config(MODE_ALL_PATTERNS))\n + \tif (set_config(MODE_ALL_PATTERNS))\n \t\treturn 1;\n @@\n \tfprintf(fp, \"/*\\n!/*/\\n\");\n 4: 7d9d66a89f ! 4: 248fc172ac sparse-checkout: 'set' subcommand\n @@ -47,6 +47,15 @@\n \tNULL\n };\n \n +@@\n + \targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n + \n + \tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n +-\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n ++\t\terror(_(\"failed to update index with new sparse-checkout patterns\"));\n + \t\tresult = 1;\n + \t}\n + \n @@\n \treturn update_working_directory();\n }\n @@ -71,22 +80,22 @@\n +\tint i;\n +\tstruct pattern_list pl;\n +\tint result;\n -+\tint set_config = 0;\n ++\tint changed_config = 0;\n +\tmemset(&pl, 0, sizeof(pl));\n +\n +\tfor (i = 1; i < argc; i++)\n +\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n +\n +\tif (!core_apply_sparse_checkout) {\n -+\t\tsc_set_config(MODE_ALL_PATTERNS);\n ++\t\tset_config(MODE_ALL_PATTERNS);\n +\t\tcore_apply_sparse_checkout = 1;\n -+\t\tset_config = 1;\n ++\t\tchanged_config = 1;\n +\t}\n +\n +\tresult = write_patterns_and_update(&pl);\n +\n -+\tif (result && set_config)\n -+\t\tsc_set_config(MODE_NO_PATTERNS);\n ++\tif (result && changed_config)\n ++\t\tset_config(MODE_NO_PATTERNS);\n +\n +\tclear_pattern_list(&pl);\n +\treturn result;\n @@ -119,9 +128,10 @@\n +\t\ttest_commit test file &&\n +\t\ttest_path_is_missing .git/config.worktree &&\n +\t\ttest_must_fail git sparse-checkout set nothing &&\n -+\t\ttest_i18ngrep \"sparseCheckout = false\" .git/config.worktree &&\n ++\t\ttest_path_is_file .git/config.worktree &&\n ++\t\ttest_must_fail git config core.sparseCheckout &&\n +\t\tgit sparse-checkout set \"/*\" &&\n -+\t\ttest_i18ngrep \"sparseCheckout = true\" .git/config.worktree\n ++\t\ttest_cmp_config true core.sparseCheckout\n +\t)\n +'\n +\n 5: 0e08898dcb ! 5: 751798216f sparse-checkout: add '--stdin' option to set subcommand\n @@ -11,6 +11,20 @@\n \n Signed-off-by: Derrick Stolee \n \n + diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n + --- a/Documentation/git-sparse-checkout.txt\n + +++ b/Documentation/git-sparse-checkout.txt\n +@@\n + \ta list of arguments following the 'set' subcommand. Update the\n + \tworking directory to match the new patterns. Enable the\n + \tcore.sparseCheckout config setting if it is not already enabled.\n +++\n ++When the `--stdin` option is provided, the patterns are read from\n ++standard in as a newline-delimited list instead of from the arguments.\n + \n + SPARSE CHECKOUT\n + ---------------\n +\n diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\n --- a/builtin/sparse-checkout.c\n +++ b/builtin/sparse-checkout.c\n @@ -19,7 +33,7 @@\n }\n \n +static char const * const builtin_sparse_checkout_set_usage[] = {\n -+\tN_(\"git sparse-checkout set [--stdin|]\"),\n ++\tN_(\"git sparse-checkout set (--stdin | )\"),\n +\tNULL\n +};\n +\n @@ -33,7 +47,7 @@\n @@\n \tstruct pattern_list pl;\n \tint result;\n - \tint set_config = 0;\n + \tint changed_config = 0;\n +\n +\tstatic struct option builtin_sparse_checkout_set_options[] = {\n +\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n @@ -54,8 +68,7 @@\n +\t\tstruct strbuf line = STRBUF_INIT;\n +\n +\t\twhile (!strbuf_getline(&line, stdin)) {\n -+\t\t\tsize_t len;\n -+\t\t\tchar *buf = strbuf_detach(&line, &len);\n ++\t\t\tchar *buf = strbuf_detach(&line, NULL);\n +\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n +\t\t}\n +\t} else {\n @@ -64,7 +77,7 @@\n +\t}\n \n \tif (!core_apply_sparse_checkout) {\n - \t\tsc_set_config(MODE_ALL_PATTERNS);\n + \t\tset_config(MODE_ALL_PATTERNS);\n \n diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\n --- a/t/t1091-sparse-checkout-builtin.sh\n 6: c48535cd5c ! 6: 6431141e03 sparse-checkout: create 'disable' subcommand\n @@ -13,8 +13,8 @@\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n @@\n - \tworking directory to match the new patterns. Enable the\n - \tcore.sparseCheckout config setting if it is not already enabled.\n + When the `--stdin` option is provided, the patterns are read from\n + standard in as a newline-delimited list instead of from the arguments.\n \n +'disable'::\n +\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n @@ -82,7 +82,7 @@\n +\tchar *sparse_filename;\n +\tFILE *fp;\n +\n -+\tif (sc_set_config(MODE_ALL_PATTERNS))\n ++\tif (set_config(MODE_ALL_PATTERNS))\n +\t\tdie(_(\"failed to change config\"));\n +\n +\tsparse_filename = get_sparse_checkout_filename();\n @@ -96,7 +96,7 @@\n +\tunlink(sparse_filename);\n +\tfree(sparse_filename);\n +\n -+\treturn sc_set_config(MODE_NO_PATTERNS);\n ++\treturn set_config(MODE_NO_PATTERNS);\n +}\n +\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n @@ -123,7 +123,7 @@\n +\tgit -C repo sparse-checkout disable &&\n +\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n +\tgit -C repo config --list >config &&\n -+\ttest_i18ngrep \"core.sparsecheckout=false\" config &&\n ++\ttest_must_fail git config core.sparseCheckout &&\n +\tls repo >dir &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n 7: 44b0245e65 = 7: 0bc87c1a88 trace2: add region in clear_ce_flags\n 8: 121d878882 ! 8: 3b9e11d7f2 sparse-checkout: add 'cone' mode\n @@ -42,11 +42,23 @@\n diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\n --- a/Documentation/git-sparse-checkout.txt\n +++ b/Documentation/git-sparse-checkout.txt\n +@@\n + To repopulate the working directory with all files, use the\n + `git sparse-checkout disable` command.\n + \n +-## FULL PATTERN SET\n ++\n ++FULL PATTERN SET\n ++----------------\n + \n + By default, the sparse-checkout file uses the same syntax as `.gitignore`\n + files.\n @@\n ----------------\n \n \n -+## CONE PATTERN SET\n ++CONE PATTERN SET\n ++----------------\n +\n +The full pattern set allows for arbitrary pattern matches and complicated\n +inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n @@ -68,17 +80,17 @@\n +added as a parent pattern. At this point, the sparse-checkout file contains\n +the following patterns:\n +\n -+```\n ++----------------\n +/*\n +!/*/\n -+```\n ++----------------\n +\n +This says \"include everything in root, but nothing two levels below root.\"\n +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n +`A/B` are added as parent patterns. The resulting sparse-checkout file is\n +now\n +\n -+```\n ++----------------\n +/*\n +!/*/\n +/A/\n @@ -86,7 +98,7 @@\n +/A/B/\n +!/A/B/*/\n +/A/B/C/\n -+```\n ++----------------\n +\n +Here, order matters, so the negative patterns are overridden by the positive\n +patterns that appear lower in the file.\n 9: ff2a59bfff = 9: eca0bbf930 sparse-checkout: use hashmaps for cone patterns\n 10: 0258ee8026 ! 10: 951e622aac sparse-checkout: init and set in cone mode\n @@ -3,7 +3,7 @@\n sparse-checkout: init and set in cone mode\n \n To make the cone pattern set easy to use, update the behavior of\n - 'git sparse-checkout [init|set]'.\n + 'git sparse-checkout (init|set)'.\n \n Add '--cone' flag to 'git sparse-checkout init' to set the config\n option 'core.sparseCheckoutCone=true'.\n @@ -34,35 +34,78 @@\n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n @@\n + \treturn result;\n + }\n + \n ++static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n ++{\n ++\tint i;\n ++\tstruct pattern_entry *pe;\n ++\tstruct hashmap_iter iter;\n ++\tstruct string_list sl = STRING_LIST_INIT_DUP;\n ++\n ++\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n ++\t\tstring_list_insert(&sl, pe->pattern);\n ++\n ++\tstring_list_sort(&sl);\n ++\tstring_list_remove_duplicates(&sl, 0);\n ++\n ++\tfprintf(fp, \"/*\\n!/*/\\n\");\n ++\n ++\tfor (i = 0; i < sl.nr; i++) {\n ++\t\tchar *pattern = sl.items[i].string;\n ++\n ++\t\tif (strlen(pattern))\n ++\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n ++\t}\n ++\n ++\tstring_list_clear(&sl, 0);\n ++\n ++\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n ++\t\tstring_list_insert(&sl, pe->pattern);\n ++\n ++\tstring_list_sort(&sl);\n ++\tstring_list_remove_duplicates(&sl, 0);\n ++\n ++\tfor (i = 0; i < sl.nr; i++) {\n ++\t\tchar *pattern = sl.items[i].string;\n ++\t\tfprintf(fp, \"%s/\\n\", pattern);\n ++\t}\n ++}\n ++\n ++static int write_patterns_and_update(struct pattern_list *pl)\n ++{\n ++\tchar *sparse_filename;\n ++\tFILE *fp;\n ++\n ++\tsparse_filename = get_sparse_checkout_filename();\n ++\tfp = fopen(sparse_filename, \"w\");\n ++\n ++\tif (core_sparse_checkout_cone)\n ++\t\twrite_cone_to_file(fp, pl);\n ++\telse\n ++\t\twrite_patterns_to_file(fp, pl);\n ++\n ++\tfclose(fp);\n ++\tfree(sparse_filename);\n ++\n ++\treturn update_working_directory();\n ++}\n ++\n enum sparse_checkout_mode {\n \tMODE_NO_PATTERNS = 0,\n \tMODE_ALL_PATTERNS = 1,\n +\tMODE_CONE_PATTERNS = 2,\n };\n \n - static int sc_set_config(enum sparse_checkout_mode mode)\n - {\n - \tstruct argv_array argv = ARGV_ARRAY_INIT;\n -+\tstruct argv_array cone_argv = ARGV_ARRAY_INIT;\n - \n - \tif (git_config_set_gently(\"extensions.worktreeConfig\", \"true\")) {\n - \t\terror(_(\"failed to set extensions.worktreeConfig setting\"));\n + static int set_config(enum sparse_checkout_mode mode)\n @@\n - \t\treturn 1;\n - \t}\n + \t\t\t\t \"core.sparseCheckout\",\n + \t\t\t\t mode ? \"true\" : NULL);\n \n -+\targv_array_pushl(&cone_argv, \"config\", \"--worktree\",\n -+\t\t\t \"core.sparseCheckoutCone\", NULL);\n -+\n -+\tif (mode == MODE_CONE_PATTERNS)\n -+\t\targv_array_push(&cone_argv, \"true\");\n -+\telse\n -+\t\targv_array_push(&cone_argv, \"false\");\n -+\n -+\tif (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) {\n -+\t\terror(_(\"failed to enable core.sparseCheckoutCone\"));\n -+\t\treturn 1;\n -+\t}\n ++\tgit_config_set_in_file_gently(config_path,\n ++\t\t\t\t \"core.sparseCheckoutCone\",\n ++\t\t\t\t mode == MODE_CONE_PATTERNS ? \"true\" : NULL);\n +\n \treturn 0;\n }\n @@ -85,7 +128,7 @@\n \tstruct object_id oid;\n +\tint mode;\n \n --\tif (sc_set_config(MODE_ALL_PATTERNS))\n +-\tif (set_config(MODE_ALL_PATTERNS))\n +\tstatic struct option builtin_sparse_checkout_init_options[] = {\n +\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n +\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n @@ -98,7 +141,7 @@\n +\n +\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n +\n -+\tif (sc_set_config(mode))\n ++\tif (set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n @@ -106,15 +149,24 @@\n \treturn update_working_directory();\n }\n \n +-static int write_patterns_and_update(struct pattern_list *pl)\n +static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n -+{\n + {\n +-\tchar *sparse_filename;\n +-\tFILE *fp;\n +\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n +\te->patternlen = path->len;\n +\te->pattern = strbuf_detach(path, NULL);\n +\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n -+\n + \n +-\tsparse_filename = get_sparse_checkout_filename();\n +-\tfp = fopen(sparse_filename, \"w\");\n +-\twrite_patterns_to_file(fp, pl);\n +-\tfclose(fp);\n +-\tfree(sparse_filename);\n +\thashmap_add(&pl->recursive_hashmap, &e->ent);\n -+\n + \n +-\treturn update_working_directory();\n +\twhile (e->patternlen) {\n +\t\tchar *slash = strrchr(e->pattern, '/');\n +\t\tchar *oldpattern = e->pattern;\n @@ -134,62 +186,6 @@\n +\t}\n +}\n +\n -+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n -+{\n -+\tint i;\n -+\tstruct pattern_entry *pe;\n -+\tstruct hashmap_iter iter;\n -+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n -+\n -+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n -+\t\tstring_list_insert(&sl, pe->pattern);\n -+\n -+\tstring_list_sort(&sl);\n -+\tstring_list_remove_duplicates(&sl, 0);\n -+\n -+\tfprintf(fp, \"/*\\n!/*/\\n\");\n -+\n -+\tfor (i = 0; i < sl.nr; i++) {\n -+\t\tchar *pattern = sl.items[i].string;\n -+\n -+\t\tif (strlen(pattern))\n -+\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n -+\t}\n -+\n -+\tstring_list_clear(&sl, 0);\n -+\n -+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n -+\t\tstring_list_insert(&sl, pe->pattern);\n -+\n -+\tstring_list_sort(&sl);\n -+\tstring_list_remove_duplicates(&sl, 0);\n -+\n -+\tfor (i = 0; i < sl.nr; i++) {\n -+\t\tchar *pattern = sl.items[i].string;\n -+\t\tfprintf(fp, \"%s/\\n\", pattern);\n -+\t}\n -+}\n -+\n - static int write_patterns_and_update(struct pattern_list *pl)\n - {\n - \tchar *sparse_filename;\n -@@\n - \n - \tsparse_filename = get_sparse_checkout_filename();\n - \tfp = fopen(sparse_filename, \"w\");\n --\twrite_patterns_to_file(fp, pl);\n -+\n -+\tif (core_sparse_checkout_cone)\n -+\t\twrite_cone_to_file(fp, pl);\n -+\telse\n -+\t\twrite_patterns_to_file(fp, pl);\n -+\n - \tfclose(fp);\n - \tfree(sparse_filename);\n - \n - \treturn update_working_directory();\n - }\n - \n +static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n +{\n +\tstrbuf_trim(line);\n @@ -203,11 +199,9 @@\n +\t\tstrbuf_insert(line, 0, \"/\", 1);\n +\n +\tinsert_recursive_pattern(pl, line);\n -+}\n -+\n + }\n + \n static char const * const builtin_sparse_checkout_set_usage[] = {\n - \tN_(\"git sparse-checkout set [--stdin|]\"),\n - \tNULL\n @@\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n @@ -215,10 +209,9 @@\n -\tif (set_opts.use_stdin) {\n +\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n --\n + \n -\t\twhile (!strbuf_getline(&line, stdin)) {\n --\t\t\tsize_t len;\n --\t\t\tchar *buf = strbuf_detach(&line, &len);\n +-\t\t\tchar *buf = strbuf_detach(&line, NULL);\n -\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n +\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n +\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n @@ -301,7 +294,7 @@\n +\techo a >expect &&\n +\ttest_cmp expect dir &&\n +\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n -+\ttest_line_count = 0 err &&\n ++\ttest_must_be_empty err &&\n +\tls repo >dir &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n @@ -334,7 +327,7 @@\n +\t\tfolder1\n +\t\tfolder2\n +\tEOF\n -+\ttest_line_count = 0 err &&\n ++\ttest_must_be_empty err &&\n +\tcat >expect <<-EOF &&\n +\t\ta\n +\t\tfolder1\n 11: 7b5c5bad2d = 11: 745910bb8e unpack-trees: hash less in cone mode\n 12: 386f4f8eb5 = 12: 3d0f951d33 unpack-trees: add progress to clear_ce_flags()\n 13: a229e1ee0c < -: ---------- read-tree: show progress by default\n 14: f03bb23305 = 13: 2703b344f6 sparse-checkout: sanitize for nested folders\n 15: d7af75672b ! 14: 63e01c2021 sparse-checkout: update working directory in-process\n @@ -68,11 +68,11 @@\n +\tstruct repository *r = the_repository;\n \n -\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n --\t\terror(_(\"failed to update index with new sparse-checkout paths\"));\n +-\t\terror(_(\"failed to update index with new sparse-checkout patterns\"));\n -\t\tresult = 1;\n -\t}\n +\tif (repo_read_index_unmerged(r))\n -+\t\tdie(_(\"You need to resolve your current index first\"));\n ++\t\tdie(_(\"you need to resolve your current index first\"));\n +\n +\tif (get_oid(\"HEAD\", &oid))\n +\t\treturn 0;\n @@ -113,29 +113,6 @@\n \treturn result;\n }\n \n -@@\n - \t\t\t builtin_sparse_checkout_init_options,\n - \t\t\t builtin_sparse_checkout_init_usage, 0);\n - \n --\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n -+\tif (init_opts.cone_mode) {\n -+\t\tmode = MODE_CONE_PATTERNS;\n -+\t\tcore_sparse_checkout_cone = 1;\n -+\t} else\n -+\t\tmode = MODE_ALL_PATTERNS;\n - \n - \tif (sc_set_config(mode))\n - \t\treturn 1;\n -@@\n - \t}\n - \n - reset_dir:\n --\treturn update_working_directory();\n -+\tcore_apply_sparse_checkout = 1;\n -+\treturn update_working_directory(NULL);\n - }\n - \n - static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n @@\n {\n \tchar *sparse_filename;\n @@ -164,9 +141,32 @@\n +\treturn 0;\n }\n \n - static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n + enum sparse_checkout_mode {\n @@\n - \t\tstruct strbuf line = STRBUF_INIT;\n + \t\t\t builtin_sparse_checkout_init_options,\n + \t\t\t builtin_sparse_checkout_init_usage, 0);\n + \n +-\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n ++\tif (init_opts.cone_mode) {\n ++\t\tmode = MODE_CONE_PATTERNS;\n ++\t\tcore_sparse_checkout_cone = 1;\n ++\t} else\n ++\t\tmode = MODE_ALL_PATTERNS;\n + \n + \tif (set_config(mode))\n + \t\treturn 1;\n +@@\n + \t}\n + \n + reset_dir:\n +-\treturn update_working_directory();\n ++\tcore_apply_sparse_checkout = 1;\n ++\treturn update_working_directory(NULL);\n + }\n + \n + static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n +@@\n + \n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n +\t\tpl.use_cone_patterns = 1;\n -: ---------- > 15: cc6773e49b sparse-checkout: use in-process update for disable subcommand\n 16: 88eff318e0 ! 16: a0dca0056a sparse-checkout: write using lockfile\n @@ -24,12 +24,11 @@\n +\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \n - \tresult = update_working_directory(pl);\n - \n +-\tresult = update_working_directory(pl);\n +\tsparse_filename = get_sparse_checkout_filename();\n +\tfd = hold_lock_file_for_update(&lk, sparse_filename,\n +\t\t\t\t LOCK_DIE_ON_ERROR);\n -+\n + \n +\tresult = update_working_directory(pl);\n \tif (result) {\n +\t\trollback_lock_file(&lk);\n 17: ab20c11d71 = 17: 4097d8f6de sparse-checkout: cone mode should not interact with .gitignore\n -: ---------- > 18: c62d9750ef sparse-checkout: update working directory in-process for 'init'\n -: ---------- > 19: 7577ffc034 sparse-checkout: check for dirty status\n\n-- \ngitgitgadget\n"},{"id":"386752","messageId":"eca0bbf930918bc0417c1ad73a9f75aee8c5cfb3.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 09/19] sparse-checkout: use hashmaps for cone patterns","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:41Z","receivedAt":"2019-11-21T22:05:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parent and recursive patterns allowed by the \"cone mode\"\noption in sparse-checkout are restrictive enough that we\ncan avoid using the regex parsing. Everything is based on\nprefix matches, so we can use hashsets to store the prefixes\nfrom the sparse-checkout file. When checking a path, we can\nstrip path entries from the path and check the hashset for\nan exact match.\n\nAs a test, I created a cone-mode sparse-checkout file for the\nLinux repository that actually includes every file. This was\nconstructed by taking every folder in the Linux repo and creating\nthe pattern pairs here:\n\n\t/$folder/\n\t!/$folder/*/\n\nThis resulted in a sparse-checkout file sith 8,296 patterns.\nRunning 'git read-tree -mu HEAD' on this file had the following\nperformance:\n\n core.sparseCheckout=false: 0.21 s (0.00 s)\n core.sparseCheckout=true: 3.75 s (3.50 s)\n core.sparseCheckoutCone=true: 0.23 s (0.01 s)\n\nThe times in parentheses above correspond to the time spent\nin the first clear_ce_flags() call, according to the trace2\nperformance traces.\n\nWhile this example is contrived, it demonstrates how these\npatterns can slow the sparse-checkout feature.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n dir.c | 207 +++++++++++++++++++++++++++--\n dir.h | 31 +++++\n t/t1091-sparse-checkout-builtin.sh | 11 +-\n unpack-trees.c | 1 +\n 4 files changed, 241 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..dfabf9982f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n+static int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t\t const struct hashmap_entry *a,\n+\t\t\t const struct hashmap_entry *b,\n+\t\t\t const void *key)\n+{\n+\tconst struct pattern_entry *ee1 =\n+\t\t\tcontainer_of(a, struct pattern_entry, ent);\n+\tconst struct pattern_entry *ee2 =\n+\t\t\tcontainer_of(b, struct pattern_entry, ent);\n+\n+\tsize_t min_len = ee1->patternlen <= ee2->patternlen\n+\t\t\t ? ee1->patternlen\n+\t\t\t : ee2->patternlen;\n+\n+\treturn strncmp(ee1->pattern, ee2->pattern, min_len);\n+}\n+\n+static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given)\n+{\n+\tstruct pattern_entry *translated;\n+\tchar *truncated;\n+\tchar *data = NULL;\n+\n+\tif (!pl->use_cone_patterns)\n+\t\treturn;\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE &&\n+\t given->flags & PATTERN_FLAG_MUSTBEDIR &&\n+\t !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 0;\n+\t\treturn;\n+\t}\n+\n+\tif (!given->flags && !strcmp(given->pattern, \"/*\")) {\n+\t\tpl->full_cone = 1;\n+\t\treturn;\n+\t}\n+\n+\tif (given->patternlen > 2 &&\n+\t !strcmp(given->pattern + given->patternlen - 2, \"/*\")) {\n+\t\tif (!(given->flags & PATTERN_FLAG_NEGATIVE)) {\n+\t\t\t/* Not a cone pattern. */\n+\t\t\tpl->use_cone_patterns = 0;\n+\t\t\twarning(_(\"unrecognized pattern: '%s'\"), given->pattern);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\ttruncated = xstrdup(given->pattern);\n+\t\ttruncated[given->patternlen - 2] = 0;\n+\n+\t\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\t\ttranslated->pattern = truncated;\n+\t\ttranslated->patternlen = given->patternlen - 2;\n+\t\thashmap_entry_init(&translated->ent,\n+\t\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->recursive_hashmap,\n+\t\t\t\t translated, ent, NULL)) {\n+\t\t\t/* We did not see the \"parent\" included */\n+\t\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\t\tgiven->pattern);\n+\t\t\tfree(truncated);\n+\t\t\tfree(translated);\n+\t\t\tgoto clear_hashmaps;\n+\t\t}\n+\n+\t\thashmap_add(&pl->parent_hashmap, &translated->ent);\n+\t\thashmap_remove(&pl->recursive_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\treturn;\n+\t}\n+\n+\tif (given->flags & PATTERN_FLAG_NEGATIVE) {\n+\t\twarning(_(\"unrecognized negative pattern: '%s'\"),\n+\t\t\tgiven->pattern);\n+\t\tgoto clear_hashmaps;\n+\t}\n+\n+\ttranslated = xmalloc(sizeof(struct pattern_entry));\n+\n+\ttranslated->pattern = xstrdup(given->pattern);\n+\ttranslated->patternlen = given->patternlen;\n+\thashmap_entry_init(&translated->ent,\n+\t\t\t memhash(translated->pattern, translated->patternlen));\n+\n+\thashmap_add(&pl->recursive_hashmap, &translated->ent);\n+\n+\tif (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) {\n+\t\t/* we already included this at the parent level */\n+\t\twarning(_(\"your sparse-checkout file may have issues: pattern '%s' is repeated\"),\n+\t\t\tgiven->pattern);\n+\t\thashmap_remove(&pl->parent_hashmap, &translated->ent, &data);\n+\t\tfree(data);\n+\t\tfree(translated);\n+\t}\n+\n+\treturn;\n+\n+clear_hashmaps:\n+\twarning(_(\"disabling cone pattern matching\"));\n+\thashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent);\n+\thashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent);\n+\tpl->use_cone_patterns = 0;\n+}\n+\n+static int hashmap_contains_path(struct hashmap *map,\n+\t\t\t\t struct strbuf *pattern)\n+{\n+\tstruct pattern_entry p;\n+\n+\t/* Check straight mapping */\n+\tp.pattern = pattern->buf;\n+\tp.patternlen = pattern->len;\n+\thashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen));\n+\treturn !!hashmap_get_entry(map, &p, ent, NULL);\n+}\n+\n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer)\n+{\n+\tchar *slash_pos;\n+\n+\tstrbuf_setlen(buffer, 0);\n+\n+\tif (path[0] != '/')\n+\t\tstrbuf_addch(buffer, '/');\n+\n+\tstrbuf_addstr(buffer, path);\n+\n+\tslash_pos = strrchr(buffer->buf, '/');\n+\n+\twhile (slash_pos > buffer->buf) {\n+\t\tstrbuf_setlen(buffer, slash_pos - buffer->buf);\n+\n+\t\tif (hashmap_contains_path(map, buffer))\n+\t\t\treturn 1;\n+\n+\t\tslash_pos = strrchr(buffer->buf, '/');\n+\t}\n+\n+\treturn 0;\n+}\n+\n void add_pattern(const char *string, const char *base,\n \t\t int baselen, struct pattern_list *pl, int srcpos)\n {\n@@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base,\n \tALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc);\n \tpl->patterns[pl->nr++] = pattern;\n \tpattern->pl = pl;\n+\n+\tadd_pattern_to_hashsets(pl, pattern);\n }\n \n static int read_skip_worktree_file_from_index(const struct index_state *istate,\n@@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size,\n \tint i, lineno = 1;\n \tchar *entry;\n \n+\thashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n \tpl->filebuf = buf;\n \n \tif (skip_utf8_bom(&buf, size))\n@@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list(\n \t\t\t\tstruct index_state *istate)\n {\n \tstruct path_pattern *pattern;\n-\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n-\t\t\t\t\t\t dtype, pl, istate);\n-\tif (pattern) {\n-\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n-\t\t\treturn NOT_MATCHED;\n-\t\telse\n-\t\t\treturn MATCHED;\n+\tstruct strbuf parent_pathname = STRBUF_INIT;\n+\tint result = NOT_MATCHED;\n+\tconst char *slash_pos;\n+\n+\tif (!pl->use_cone_patterns) {\n+\t\tpattern = last_matching_pattern_from_list(pathname, pathlen, basename,\n+\t\t\t\t\t\t\tdtype, pl, istate);\n+\t\tif (pattern) {\n+\t\t\tif (pattern->flags & PATTERN_FLAG_NEGATIVE)\n+\t\t\t\treturn NOT_MATCHED;\n+\t\t\telse\n+\t\t\t\treturn MATCHED;\n+\t\t}\n+\n+\t\treturn UNDECIDED;\n+\t}\n+\n+\tif (pl->full_cone)\n+\t\treturn MATCHED;\n+\n+\tstrbuf_addch(&parent_pathname, '/');\n+\tstrbuf_add(&parent_pathname, pathname, pathlen);\n+\n+\tif (hashmap_contains_path(&pl->recursive_hashmap,\n+\t\t\t\t &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tslash_pos = strrchr(parent_pathname.buf, '/');\n+\n+\tif (slash_pos == parent_pathname.buf) {\n+\t\t/* include every file in root */\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n \t}\n \n-\treturn UNDECIDED;\n+\tstrbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf);\n+\n+\tif (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) {\n+\t\tresult = MATCHED;\n+\t\tgoto done;\n+\t}\n+\n+\tif (hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t pathname,\n+\t\t\t\t &parent_pathname))\n+\t\tresult = MATCHED;\n+\n+done:\n+\tstrbuf_release(&parent_pathname);\n+\treturn result;\n }\n \n static struct path_pattern *last_matching_pattern_from_lists(\ndiff --git a/dir.h b/dir.h\nindex 2fbdef014f..f8edbca72b 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -4,6 +4,7 @@\n /* See Documentation/technical/api-directory-listing.txt */\n \n #include \"cache.h\"\n+#include \"hashmap.h\"\n #include \"strbuf.h\"\n \n struct dir_entry {\n@@ -37,6 +38,13 @@ struct path_pattern {\n \tint srcpos;\n };\n \n+/* used for hashmaps for cone patterns */\n+struct pattern_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *pattern;\n+\tsize_t patternlen;\n+};\n+\n /*\n * Each excludes file will be parsed into a fresh exclude_list which\n * is appended to the relevant exclude_list_group (either EXC_DIRS or\n@@ -55,6 +63,26 @@ struct pattern_list {\n \tconst char *src;\n \n \tstruct path_pattern **patterns;\n+\n+\t/*\n+\t * While scanning the excludes, we attempt to match the patterns\n+\t * with a more restricted set that allows us to use hashsets for\n+\t * matching logic, which is faster than the linear lookup in the\n+\t * excludes array above. If non-zero, that check succeeded.\n+\t */\n+\tunsigned use_cone_patterns;\n+\tunsigned full_cone;\n+\n+\t/*\n+\t * Stores paths where everything starting with those paths\n+\t * is included.\n+\t */\n+\tstruct hashmap recursive_hashmap;\n+\n+\t/*\n+\t * Used to check single-level parents of blobs.\n+\t */\n+\tstruct hashmap parent_hashmap;\n };\n \n /*\n@@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int hashmap_contains_parent(struct hashmap *map,\n+\t\t\t const char *path,\n+\t\t\t struct strbuf *buffer);\n struct pattern_list *add_pattern_list(struct dir_struct *dir,\n \t\t\t\t int group_type, const char *src);\n int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 0b2715db52..1ed003ac1d 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -151,7 +151,8 @@ test_expect_success 'set sparse-checkout using --stdin' '\n test_expect_success 'cone mode: match patterns' '\n \tgit -C repo config --worktree core.sparseCheckoutCone true &&\n \trm -rf repo/a repo/folder1 repo/folder2 &&\n-\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err &&\n \tgit -C repo reset --hard &&\n \tls repo >dir &&\n \tcat >expect <<-EOF &&\n@@ -162,6 +163,14 @@ test_expect_success 'cone mode: match patterns' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: warn on bad pattern' '\n+\ttest_when_finished mv sparse-checkout repo/.git/info/ &&\n+\tcp repo/.git/info/sparse-checkout . &&\n+\techo \"!/deep/deeper/*\" >>repo/.git/info/sparse-checkout &&\n+\tgit -C repo read-tree -mu HEAD 2>err &&\n+\ttest_i18ngrep \"unrecognized negative pattern\" err\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 01a05ff66d..a90d71845d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n \t\t\to->skip_sparse_checkout = 1;\n \t\telse\n-- \ngitgitgadget\n\n"},{"id":"386750","messageId":"3b9e11d7f2a7ec7b9e0250e603e5de42f10efb92.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 08/19] sparse-checkout: add 'cone' mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:40Z","receivedAt":"2019-11-21T22:05:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature can have quadratic performance as\nthe number of patterns and number of entries in the index grow.\nIf there are 1,000 patterns and 1,000,000 entries, this time can\nbe very significant.\n\nCreate a new Boolean config option, core.sparseCheckoutCone, to\nindicate that we expect the sparse-checkout file to contain a\nmore limited set of patterns. This is a separate config setting\nfrom core.sparseCheckout to avoid breaking older clients by\nintroducing a tri-state option.\n\nThe config option does nothing right now, but will be expanded\nupon in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 10 ++++-\n Documentation/git-sparse-checkout.txt | 55 ++++++++++++++++++++++++++-\n cache.h | 4 +-\n config.c | 5 +++\n environment.c | 1 +\n t/t1091-sparse-checkout-builtin.sh | 14 +++++++\n 6 files changed, 85 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 852d2ba37a..bdbbee58b9 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -593,8 +593,14 @@ core.multiPackIndex::\n \tmulti-pack-index design document].\n \n core.sparseCheckout::\n-\tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n-\tlinkgit:git-read-tree[1] for more information.\n+\tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\n+\tfor more information.\n+\n+core.sparseCheckoutCone::\n+\tEnables the \"cone mode\" of the sparse checkout feature. When the\n+\tsparse-checkout file contains a limited set of patterns, then this\n+\tmode provides significant performance advantages. See\n+\tlinkgit:git-sparse-checkout[1] for more information.\n \n core.abbrev::\n \tSet the length object names are abbreviated to. If\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex c2cb19f80d..8535f0cf40 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -80,7 +80,9 @@ the sparse-checkout file.\n To repopulate the working directory with all files, use the\n `git sparse-checkout disable` command.\n \n-## FULL PATTERN SET\n+\n+FULL PATTERN SET\n+----------------\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n files.\n@@ -95,6 +97,57 @@ using negative patterns. For example, to remove the file `unwanted`:\n ----------------\n \n \n+CONE PATTERN SET\n+----------------\n+\n+The full pattern set allows for arbitrary pattern matches and complicated\n+inclusion/exclusion rules. These can result in O(N*M) pattern matches when\n+updating the index, where N is the number of patterns and M is the number\n+of paths in the index. To combat this performance issue, a more restricted\n+pattern set is allowed when `core.spareCheckoutCone` is enabled.\n+\n+The accepted patterns in the cone pattern set are:\n+\n+1. *Recursive:* All paths inside a directory are included.\n+\n+2. *Parent:* All files immediately inside a directory are included.\n+\n+In addition to the above two patterns, we also expect that all files in the\n+root directory are included. If a recursive pattern is added, then all\n+leading directories are added as parent patterns.\n+\n+By default, when running `git sparse-checkout init`, the root directory is\n+added as a parent pattern. At this point, the sparse-checkout file contains\n+the following patterns:\n+\n+----------------\n+/*\n+!/*/\n+----------------\n+\n+This says \"include everything in root, but nothing two levels below root.\"\n+If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and\n+`A/B` are added as parent patterns. The resulting sparse-checkout file is\n+now\n+\n+----------------\n+/*\n+!/*/\n+/A/\n+!/A/*/\n+/A/B/\n+!/A/B/*/\n+/A/B/C/\n+----------------\n+\n+Here, order matters, so the negative patterns are overridden by the positive\n+patterns that appear lower in the file.\n+\n+If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file\n+expecting patterns of these types. Git will warn if the patterns do not match.\n+If the patterns do match the expected format, then Git will use faster hash-\n+based algorithms to compute inclusion in the sparse-checkout.\n+\n SEE ALSO\n --------\n \ndiff --git a/cache.h b/cache.h\nindex 04cabaac11..4980ee198e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -918,12 +918,14 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n-extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\n extern const char *core_fsmonitor;\n \n+int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n+\n /*\n * Include broken refs in all ref iterations, which will\n * generally choke dangerous operations rather than letting\ndiff --git a/config.c b/config.c\nindex e7052b3977..d75f88ca0c 100644\n--- a/config.c\n+++ b/config.c\n@@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.sparsecheckoutcone\")) {\n+\t\tcore_sparse_checkout_cone = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.precomposeunicode\")) {\n \t\tprecomposed_unicode = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex efa072680a..2a1a866659 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_sparse_checkout_cone;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex c385c62c92..0b2715db52 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -148,6 +148,20 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: match patterns' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\trm -rf repo/a repo/folder1 repo/folder2 &&\n+\tgit -C repo read-tree -mu HEAD &&\n+\tgit -C repo reset --hard &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_expect_success 'sparse-checkout disable' '\n \tgit -C repo sparse-checkout disable &&\n \ttest_path_is_missing repo/.git/info/sparse-checkout &&\n-- \ngitgitgadget\n\n"},{"id":"386751","messageId":"6431141e0385c2d700fd213d9ad8238f75f1cab3.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 06/19] sparse-checkout: create 'disable' subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:38Z","receivedAt":"2019-11-21T22:05:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe instructions for disabling a sparse-checkout to a full\nworking directory are complicated and non-intuitive. Add a\nsubcommand, 'git sparse-checkout disable', to perform those\nsteps for the user.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 27 ++++++++++++---------------\n builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++-\n t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++\n 3 files changed, 52 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex a724eae09c..c2cb19f80d 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -51,6 +51,10 @@ To avoid interfering with other worktrees, it first enables the\n When the `--stdin` option is provided, the patterns are read from\n standard in as a newline-delimited list instead of from the arguments.\n \n+'disable'::\n+\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n+\t`false`, and restore the working directory to include all files.\n+\n SPARSE CHECKOUT\n ---------------\n \n@@ -68,6 +72,14 @@ directory, it updates the skip-worktree bits in the index based\n on this file. The files matching the patterns in the file will\n appear in the working directory, and the rest will not.\n \n+To enable the sparse-checkout feature, run `git sparse-checkout init` to\n+initialize a simple sparse-checkout file and enable the `core.sparseCheckout`\n+config setting. Then, run `git sparse-checkout set` to modify the patterns in\n+the sparse-checkout file.\n+\n+To repopulate the working directory with all files, use the\n+`git sparse-checkout disable` command.\n+\n ## FULL PATTERN SET\n \n By default, the sparse-checkout file uses the same syntax as `.gitignore`\n@@ -82,21 +94,6 @@ using negative patterns. For example, to remove the file `unwanted`:\n !unwanted\n ----------------\n \n-Another tricky thing is fully repopulating the working directory when you\n-no longer want sparse checkout. You cannot just disable \"sparse\n-checkout\" because skip-worktree bits are still in the index and your working\n-directory is still sparsely populated. You should re-populate the working\n-directory with the `$GIT_DIR/info/sparse-checkout` file content as\n-follows:\n-\n-----------------\n-/*\n-----------------\n-\n-Then you can disable sparse checkout. Sparse checkout support in 'git\n-checkout' and similar commands is disabled by default. You need to\n-set `core.sparseCheckout` to `true` in order to have sparse checkout\n-support.\n \n SEE ALSO\n --------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 82bff0020d..e3a8d3460a 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -8,7 +8,7 @@\n #include \"strbuf.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n-\tN_(\"git sparse-checkout (init|list|set) \"),\n+\tN_(\"git sparse-checkout (init|list|set|disable) \"),\n \tNULL\n };\n \n@@ -207,6 +207,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \treturn result;\n }\n \n+static int sparse_checkout_disable(int argc, const char **argv)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tif (set_config(MODE_ALL_PATTERNS))\n+\t\tdie(_(\"failed to change config\"));\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = xfopen(sparse_filename, \"w\");\n+\tfprintf(fp, \"/*\\n\");\n+\tfclose(fp);\n+\n+\tif (update_working_directory())\n+\t\tdie(_(\"error while refreshing working directory\"));\n+\n+\tunlink(sparse_filename);\n+\tfree(sparse_filename);\n+\n+\treturn set_config(MODE_NO_PATTERNS);\n+}\n+\n int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_sparse_checkout_options[] = {\n@@ -231,6 +253,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix)\n \t\t\treturn sparse_checkout_init(argc, argv);\n \t\tif (!strcmp(argv[0], \"set\"))\n \t\t\treturn sparse_checkout_set(argc, argv, prefix);\n+\t\tif (!strcmp(argv[0], \"disable\"))\n+\t\t\treturn sparse_checkout_disable(argc, argv);\n \t}\n \n \tusage_with_options(builtin_sparse_checkout_usage,\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 07e73b4674..c385c62c92 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -148,4 +148,19 @@ test_expect_success 'set sparse-checkout using --stdin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'sparse-checkout disable' '\n+\tgit -C repo sparse-checkout disable &&\n+\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\tgit -C repo config --list >config &&\n+\ttest_must_fail git config core.sparseCheckout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386753","messageId":"751798216f35a4d26563898b639d0757eca9c5ff.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 05/19] sparse-checkout: add '--stdin' option to set subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:37Z","receivedAt":"2019-11-21T22:05:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout set' subcommand takes a list of patterns\nand places them in the sparse-checkout file. Then, it updates the\nworking directory to match those patterns. For a large list of\npatterns, the command-line call can get very cumbersome.\n\nAdd a '--stdin' option to instead read patterns over standard in.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 3 +++\n builtin/sparse-checkout.c | 34 +++++++++++++++++++++++++--\n t/t1091-sparse-checkout-builtin.sh | 20 ++++++++++++++++\n 3 files changed, 55 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex ca62669b8c..a724eae09c 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -47,6 +47,9 @@ To avoid interfering with other worktrees, it first enables the\n \ta list of arguments following the 'set' subcommand. Update the\n \tworking directory to match the new patterns. Enable the\n \tcore.sparseCheckout config setting if it is not already enabled.\n++\n+When the `--stdin` option is provided, the patterns are read from\n+standard in as a newline-delimited list instead of from the arguments.\n \n SPARSE CHECKOUT\n ---------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 95cbd0a42c..82bff0020d 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -150,6 +150,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \treturn update_working_directory();\n }\n \n+static char const * const builtin_sparse_checkout_set_usage[] = {\n+\tN_(\"git sparse-checkout set (--stdin | )\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_set_opts {\n+\tint use_stdin;\n+} set_opts;\n+\n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n \tstatic const char *empty_base = \"\";\n@@ -157,10 +166,31 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \tstruct pattern_list pl;\n \tint result;\n \tint changed_config = 0;\n+\n+\tstatic struct option builtin_sparse_checkout_set_options[] = {\n+\t\tOPT_BOOL(0, \"stdin\", &set_opts.use_stdin,\n+\t\t\t N_(\"read patterns from standard in\")),\n+\t\tOPT_END(),\n+\t};\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n-\tfor (i = 1; i < argc; i++)\n-\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_sparse_checkout_set_options,\n+\t\t\t builtin_sparse_checkout_set_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (set_opts.use_stdin) {\n+\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\tchar *buf = strbuf_detach(&line, NULL);\n+\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < argc; i++)\n+\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t}\n \n \tif (!core_apply_sparse_checkout) {\n \t\tset_config(MODE_ALL_PATTERNS);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 72d8bc5c25..07e73b4674 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'set sparse-checkout using --stdin' '\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/folder1/\n+\t\t/folder2/\n+\tEOF\n+\tgit -C repo sparse-checkout set --stdin actual &&\n+\ttest_cmp expect actual &&\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386754","messageId":"0bc87c1a88f5419a9e11681e1d3c079b2426def2.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 07/19] trace2: add region in clear_ce_flags","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:39Z","receivedAt":"2019-11-21T22:05:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nWhen Git updates the working directory with the sparse-checkout\nfeature enabled, the unpack_trees() method calls clear_ce_flags()\nto update the skip-wortree bits on the cache entries. This\ncheck can be expensive, depending on the patterns used.\n\nAdd trace2 regions around the method, including some flag\ninformation, so we can get granular performance data during\nexperiments. This data will be used to measure improvements\nto the pattern-matching algorithms for sparse-checkout.\n\nSigned-off-by: Jeff Hostetler \nSigned-off-by: Derrick Stolee \n---\n unpack-trees.c | 10 +++++++++-\n 1 file changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 33ea7810d8..01a05ff66d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t struct pattern_list *pl)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n+\tchar label[100];\n+\tint rval;\n \n \tstrbuf_reset(&prefix);\n \n-\treturn clear_ce_flags_1(istate,\n+\txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n+\t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n+\ttrace2_region_enter(\"unpack_trees\", label, the_repository);\n+\trval = clear_ce_flags_1(istate,\n \t\t\t\tistate->cache,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n \t\t\t\tpl, 0);\n+\ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n+\n+\treturn rval;\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"386755","messageId":"951e622aacdb2069177038bff6eb1a0330d46f34.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 10/19] sparse-checkout: init and set in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:42Z","receivedAt":"2019-11-21T22:05:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo make the cone pattern set easy to use, update the behavior of\n'git sparse-checkout (init|set)'.\n\nAdd '--cone' flag to 'git sparse-checkout init' to set the config\noption 'core.sparseCheckoutCone=true'.\n\nWhen running 'git sparse-checkout set' in cone mode, a user only\nneeds to supply a list of recursive folder matches. Git will\nautomatically add the necessary parent matches for the leading\ndirectories.\n\nWhen testing 'git sparse-checkout set' in cone mode, check the\nerror stream to ensure we do not see any errors. Specifically,\nwe want to avoid the warning that the patterns do not match\nthe cone-mode patterns.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 163 ++++++++++++++++++++++++++---\n dir.c | 8 +-\n dir.h | 4 +\n t/t1091-sparse-checkout-builtin.sh | 51 +++++++++\n 4 files changed, 206 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex e3a8d3460a..85cc801f03 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -6,6 +6,7 @@\n #include \"repository.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n+#include \"string-list.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n@@ -74,9 +75,65 @@ static int update_working_directory(void)\n \treturn result;\n }\n \n+static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n+{\n+\tint i;\n+\tstruct pattern_entry *pe;\n+\tstruct hashmap_iter iter;\n+\tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\n+\t\tif (strlen(pattern))\n+\t\t\tfprintf(fp, \"%s/\\n!%s/*/\\n\", pattern, pattern);\n+\t}\n+\n+\tstring_list_clear(&sl, 0);\n+\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n+\t\tstring_list_insert(&sl, pe->pattern);\n+\n+\tstring_list_sort(&sl);\n+\tstring_list_remove_duplicates(&sl, 0);\n+\n+\tfor (i = 0; i < sl.nr; i++) {\n+\t\tchar *pattern = sl.items[i].string;\n+\t\tfprintf(fp, \"%s/\\n\", pattern);\n+\t}\n+}\n+\n+static int write_patterns_and_update(struct pattern_list *pl)\n+{\n+\tchar *sparse_filename;\n+\tFILE *fp;\n+\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfp = fopen(sparse_filename, \"w\");\n+\n+\tif (core_sparse_checkout_cone)\n+\t\twrite_cone_to_file(fp, pl);\n+\telse\n+\t\twrite_patterns_to_file(fp, pl);\n+\n+\tfclose(fp);\n+\tfree(sparse_filename);\n+\n+\treturn update_working_directory();\n+}\n+\n enum sparse_checkout_mode {\n \tMODE_NO_PATTERNS = 0,\n \tMODE_ALL_PATTERNS = 1,\n+\tMODE_CONE_PATTERNS = 2,\n };\n \n static int set_config(enum sparse_checkout_mode mode)\n@@ -93,9 +150,22 @@ static int set_config(enum sparse_checkout_mode mode)\n \t\t\t\t \"core.sparseCheckout\",\n \t\t\t\t mode ? \"true\" : NULL);\n \n+\tgit_config_set_in_file_gently(config_path,\n+\t\t\t\t \"core.sparseCheckoutCone\",\n+\t\t\t\t mode == MODE_CONE_PATTERNS ? \"true\" : NULL);\n+\n \treturn 0;\n }\n \n+static char const * const builtin_sparse_checkout_init_usage[] = {\n+\tN_(\"git sparse-checkout init [--cone]\"),\n+\tNULL\n+};\n+\n+static struct sparse_checkout_init_opts {\n+\tint cone_mode;\n+} init_opts;\n+\n static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n@@ -103,8 +173,21 @@ static int sparse_checkout_init(int argc, const char **argv)\n \tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n+\tint mode;\n \n-\tif (set_config(MODE_ALL_PATTERNS))\n+\tstatic struct option builtin_sparse_checkout_init_options[] = {\n+\t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n+\t\t\t N_(\"initialize the sparse-checkout in cone mode\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_sparse_checkout_init_options,\n+\t\t\t builtin_sparse_checkout_init_usage, 0);\n+\n+\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\n+\tif (set_config(mode))\n \t\treturn 1;\n \n \tmemset(&pl, 0, sizeof(pl));\n@@ -136,18 +219,47 @@ static int sparse_checkout_init(int argc, const char **argv)\n \treturn update_working_directory();\n }\n \n-static int write_patterns_and_update(struct pattern_list *pl)\n+static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n {\n-\tchar *sparse_filename;\n-\tFILE *fp;\n+\tstruct pattern_entry *e = xmalloc(sizeof(*e));\n+\te->patternlen = path->len;\n+\te->pattern = strbuf_detach(path, NULL);\n+\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = fopen(sparse_filename, \"w\");\n-\twrite_patterns_to_file(fp, pl);\n-\tfclose(fp);\n-\tfree(sparse_filename);\n+\thashmap_add(&pl->recursive_hashmap, &e->ent);\n \n-\treturn update_working_directory();\n+\twhile (e->patternlen) {\n+\t\tchar *slash = strrchr(e->pattern, '/');\n+\t\tchar *oldpattern = e->pattern;\n+\t\tsize_t newlen;\n+\n+\t\tif (slash == e->pattern)\n+\t\t\tbreak;\n+\n+\t\tnewlen = slash - e->pattern;\n+\t\te = xmalloc(sizeof(struct pattern_entry));\n+\t\te->patternlen = newlen;\n+\t\te->pattern = xstrndup(oldpattern, newlen);\n+\t\thashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen));\n+\n+\t\tif (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL))\n+\t\t\thashmap_add(&pl->parent_hashmap, &e->ent);\n+\t}\n+}\n+\n+static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl)\n+{\n+\tstrbuf_trim(line);\n+\n+\tstrbuf_trim_trailing_dir_sep(line);\n+\n+\tif (!line->len)\n+\t\treturn;\n+\n+\tif (line->buf[0] != '/')\n+\t\tstrbuf_insert(line, 0, \"/\", 1);\n+\n+\tinsert_recursive_pattern(pl, line);\n }\n \n static char const * const builtin_sparse_checkout_set_usage[] = {\n@@ -180,16 +292,35 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_sparse_checkout_set_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n-\tif (set_opts.use_stdin) {\n+\tif (core_sparse_checkout_cone) {\n \t\tstruct strbuf line = STRBUF_INIT;\n \n-\t\twhile (!strbuf_getline(&line, stdin)) {\n-\t\t\tchar *buf = strbuf_detach(&line, NULL);\n-\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\n+\t\tif (set_opts.use_stdin) {\n+\t\t\twhile (!strbuf_getline(&line, stdin))\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++) {\n+\t\t\t\tstrbuf_setlen(&line, 0);\n+\t\t\t\tstrbuf_addstr(&line, argv[i]);\n+\t\t\t\tstrbuf_to_cone_pattern(&line, &pl);\n+\t\t\t}\n \t\t}\n \t} else {\n-\t\tfor (i = 0; i < argc; i++)\n-\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\tif (set_opts.use_stdin) {\n+\t\t\tstruct strbuf line = STRBUF_INIT;\n+\n+\t\t\twhile (!strbuf_getline(&line, stdin)) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tchar *buf = strbuf_detach(&line, &len);\n+\t\t\t\tadd_pattern(buf, empty_base, 0, &pl, 0);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tfor (i = 0; i < argc; i++)\n+\t\t\t\tadd_pattern(argv[i], empty_base, 0, &pl, 0);\n+\t\t}\n \t}\n \n \tif (!core_apply_sparse_checkout) {\ndiff --git a/dir.c b/dir.c\nindex dfabf9982f..35c1ca9e24 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern,\n \t*patternlen = len;\n }\n \n-static int pl_hashmap_cmp(const void *unused_cmp_data,\n-\t\t\t const struct hashmap_entry *a,\n-\t\t\t const struct hashmap_entry *b,\n-\t\t\t const void *key)\n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key)\n {\n \tconst struct pattern_entry *ee1 =\n \t\t\tcontainer_of(a, struct pattern_entry, ent);\ndiff --git a/dir.h b/dir.h\nindex f8edbca72b..8e232085cd 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir,\n \t\tstruct index_state *istate,\n \t\tconst char *name, int *dtype);\n \n+int pl_hashmap_cmp(const void *unused_cmp_data,\n+\t\t const struct hashmap_entry *a,\n+\t\t const struct hashmap_entry *b,\n+\t\t const void *key);\n int hashmap_contains_parent(struct hashmap *map,\n \t\t\t const char *path,\n \t\t\t struct strbuf *buffer);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 1ed003ac1d..fbd46c3f61 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -186,4 +186,55 @@ test_expect_success 'sparse-checkout disable' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: init and set' '\n+\tgit -C repo sparse-checkout init --cone &&\n+\tgit -C repo config --list >config &&\n+\ttest_i18ngrep \"core.sparsecheckoutcone=true\" config &&\n+\tls repo >dir &&\n+\techo a >expect &&\n+\ttest_cmp expect dir &&\n+\tgit -C repo sparse-checkout set deep/deeper1/deepest/ 2>err &&\n+\ttest_must_be_empty err &&\n+\tls repo >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeep\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tls repo/deep/deeper1 >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeepest\n+\tEOF\n+\ttest_cmp expect dir &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\t\t!/deep/*/\n+\t\t/deep/deeper1/\n+\t\t!/deep/deeper1/*/\n+\t\t/deep/deeper1/deepest/\n+\tEOF\n+\ttest_cmp expect repo/.git/info/sparse-checkout &&\n+\tgit -C repo sparse-checkout set --stdin 2>err <<-EOF &&\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\ttest_must_be_empty err &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tfolder1\n+\t\tfolder2\n+\tEOF\n+\tls repo >dir &&\n+\ttest_cmp expect dir\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386756","messageId":"745910bb8e7afe9cf0c66e0d8f873a00b506fdfa.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 11/19] unpack-trees: hash less in cone mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:43Z","receivedAt":"2019-11-21T22:05:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout feature in \"cone mode\" can use the fact that\nthe recursive patterns are \"connected\" to the root via parent\npatterns to decide if a directory is entirely contained in the\nsparse-checkout or entirely removed.\n\nIn these cases, we can skip hashing the paths within those\ndirectories and simply set the skipworktree bit to the correct\nvalue.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 4 ++--\n dir.h | 1 +\n unpack-trees.c | 38 +++++++++++++++++++++++---------------\n 3 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 35c1ca9e24..2ef92a50a0 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list(\n \n \tif (hashmap_contains_path(&pl->recursive_hashmap,\n \t\t\t\t &parent_pathname)) {\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \t\tgoto done;\n \t}\n \n@@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list(\n \tif (hashmap_contains_parent(&pl->recursive_hashmap,\n \t\t\t\t pathname,\n \t\t\t\t &parent_pathname))\n-\t\tresult = MATCHED;\n+\t\tresult = MATCHED_RECURSIVE;\n \n done:\n \tstrbuf_release(&parent_pathname);\ndiff --git a/dir.h b/dir.h\nindex 8e232085cd..77a43dbf89 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -264,6 +264,7 @@ enum pattern_match_result {\n \tUNDECIDED = -1,\n \tNOT_MATCHED = 0,\n \tMATCHED = 1,\n+\tMATCHED_RECURSIVE = 2,\n };\n \n /*\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex a90d71845d..c0dca20865 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n \tint rc;\n-\tenum pattern_match_result ret;\n-\tret = path_matches_pattern_list(prefix->buf, prefix->len,\n-\t\t\t\t\tbasename, &dtype, pl, istate);\n+\tenum pattern_match_result ret, orig_ret;\n+\torig_ret = path_matches_pattern_list(prefix->buf, prefix->len,\n+\t\t\t\t\t basename, &dtype, pl, istate);\n \n \tstrbuf_addch(prefix, '/');\n \n \t/* If undecided, use matching result of parent dir in defval */\n-\tif (ret == UNDECIDED)\n+\tif (orig_ret == UNDECIDED)\n \t\tret = default_match;\n+\telse\n+\t\tret = orig_ret;\n \n \tfor (cache_end = cache; cache_end != cache + nr; cache_end++) {\n \t\tstruct cache_entry *ce = *cache_end;\n@@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\tbreak;\n \t}\n \n-\t/*\n-\t * TODO: check pl, if there are no patterns that may conflict\n-\t * with ret (iow, we know in advance the incl/excl\n-\t * decision for the entire directory), clear flag here without\n-\t * calling clear_ce_flags_1(). That function will call\n-\t * the expensive path_matches_pattern_list() on every entry.\n-\t */\n-\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t prefix,\n-\t\t\t select_mask, clear_mask,\n-\t\t\t pl, ret);\n+\tif (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) {\n+\t\tstruct cache_entry **ce = cache;\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\n+\t\twhile (ce < cache_end) {\n+\t\t\t(*ce)->ce_flags &= ~clear_mask;\n+\t\t\tce++;\n+\t\t}\n+\t} else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) {\n+\t\trc = (cache_end - cache) / sizeof(struct cache_entry *);\n+\t} else {\n+\t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t prefix,\n+\t\t\t\t select_mask, clear_mask,\n+\t\t\t\t pl, ret);\n+\t}\n+\n \tstrbuf_setlen(prefix, prefix->len - 1);\n \treturn rc;\n }\n-- \ngitgitgadget\n\n"},{"id":"386757","messageId":"2703b344f6f7195ec309454cda46f642df31a3d9.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 13/19] sparse-checkout: sanitize for nested folders","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:45Z","receivedAt":"2019-11-21T22:05:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf a user provides folders A/ and A/B/ for inclusion in a cone-mode\nsparse-checkout file, the parsing logic will notice that A/ appears\nboth as a \"parent\" type pattern and as a \"recursive\" type pattern.\nThis is unexpected and hence will complain via a warning and revert\nto the old logic for checking sparse-checkout patterns.\n\nPrevent this from happening accidentally by sanitizing the folders\nfor this type of inclusion in the 'git sparse-checkout' builtin.\nThis happens in two ways:\n\n1. Do not include any parent patterns that also appear as recursive\n patterns.\n\n2. Do not include any recursive patterns deeper than other recursive\n patterns.\n\nIn order to minimize duplicate code for scanning parents, create\nhashmap_contains_parent() method. It takes a strbuf buffer to\navoid reallocating a buffer when calling in a tight loop.\n\nHelped-by: Eric Wong \nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 22 ++++++++++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++\n 2 files changed, 29 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 85cc801f03..55b337ad8e 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -81,9 +81,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \tstruct pattern_entry *pe;\n \tstruct hashmap_iter iter;\n \tstruct string_list sl = STRING_LIST_INIT_DUP;\n+\tstruct strbuf parent_pattern = STRBUF_INIT;\n \n-\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) {\n+\t\tif (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL))\n+\t\t\tcontinue;\n+\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\n@@ -99,8 +107,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl)\n \n \tstring_list_clear(&sl, 0);\n \n-\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent)\n-\t\tstring_list_insert(&sl, pe->pattern);\n+\thashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) {\n+\t\tif (!hashmap_contains_parent(&pl->recursive_hashmap,\n+\t\t\t\t\t pe->pattern,\n+\t\t\t\t\t &parent_pattern))\n+\t\t\tstring_list_insert(&sl, pe->pattern);\n+\t}\n+\n+\tstrbuf_release(&parent_pattern);\n \n \tstring_list_sort(&sl);\n \tstring_list_remove_duplicates(&sl, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex fbd46c3f61..b88d08da98 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -237,4 +237,15 @@ test_expect_success 'cone mode: init and set' '\n \ttest_cmp expect dir\n '\n \n+test_expect_success 'cone mode: set with nested folders' '\n+\tgit -C repo sparse-checkout set deep deep/deeper1/deepest 2>err &&\n+\ttest_line_count = 0 err &&\n+\tcat >expect <<-EOF &&\n+\t\t/*\n+\t\t!/*/\n+\t\t/deep/\n+\tEOF\n+\ttest_cmp repo/.git/info/sparse-checkout expect\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386758","messageId":"3d0f951d33a64dffa2f09651eca6416ee883acab.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 12/19] unpack-trees: add progress to clear_ce_flags()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:44Z","receivedAt":"2019-11-21T22:05:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen a large repository has many sparse-checkout patterns, the\nprocess for updating the skip-worktree bits can take long enough\nthat a user gets confused why nothing is happening. Update the\nclear_ce_flags() method to write progress.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 2 ++\n unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++--------------\n 2 files changed, 43 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4980ee198e..d3c89e7a53 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n struct split_index;\n struct untracked_cache;\n+struct progress;\n \n struct index_state {\n \tstruct cache_entry **cache;\n@@ -326,6 +327,7 @@ struct index_state {\n \tuint64_t fsmonitor_last_update;\n \tstruct ewah_bitmap *fsmonitor_dirty;\n \tstruct mem_pool *ce_mem_pool;\n+\tstruct progress *progress;\n };\n \n /* Name hashing */\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex c0dca20865..8bb684ad62 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1269,7 +1269,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match);\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr);\n \n /* Whole directory matching */\n static int clear_ce_flags_dir(struct index_state *istate,\n@@ -1278,7 +1279,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\t\t char *basename,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end;\n \tint dtype = DT_DIR;\n@@ -1315,7 +1317,8 @@ static int clear_ce_flags_dir(struct index_state *istate,\n \t\trc = clear_ce_flags_1(istate, cache, cache_end - cache,\n \t\t\t\t prefix,\n \t\t\t\t select_mask, clear_mask,\n-\t\t\t\t pl, ret);\n+\t\t\t\t pl, ret,\n+\t\t\t\t progress_nr);\n \t}\n \n \tstrbuf_setlen(prefix, prefix->len - 1);\n@@ -1342,7 +1345,8 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t struct strbuf *prefix,\n \t\t\t int select_mask, int clear_mask,\n \t\t\t struct pattern_list *pl,\n-\t\t\t enum pattern_match_result default_match)\n+\t\t\t enum pattern_match_result default_match,\n+\t\t\t int progress_nr)\n {\n \tstruct cache_entry **cache_end = cache + nr;\n \n@@ -1356,8 +1360,11 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tint len, dtype;\n \t\tenum pattern_match_result ret;\n \n+\t\tdisplay_progress(istate->progress, progress_nr);\n+\n \t\tif (select_mask && !(ce->ce_flags & select_mask)) {\n \t\t\tcache++;\n+\t\t\tprogress_nr++;\n \t\t\tcontinue;\n \t\t}\n \n@@ -1378,20 +1385,26 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\t\t\t\t prefix,\n \t\t\t\t\t\t prefix->buf + prefix->len - len,\n \t\t\t\t\t\t select_mask, clear_mask,\n-\t\t\t\t\t\t pl, default_match);\n+\t\t\t\t\t\t pl, default_match,\n+\t\t\t\t\t\t progress_nr);\n \n \t\t\t/* clear_c_f_dir eats a whole dir already? */\n \t\t\tif (processed) {\n \t\t\t\tcache += processed;\n+\t\t\t\tprogress_nr += processed;\n \t\t\t\tstrbuf_setlen(prefix, prefix->len - len);\n \t\t\t\tcontinue;\n \t\t\t}\n \n \t\t\tstrbuf_addch(prefix, '/');\n-\t\t\tcache += clear_ce_flags_1(istate, cache, cache_end - cache,\n-\t\t\t\t\t\t prefix,\n-\t\t\t\t\t\t select_mask, clear_mask, pl,\n-\t\t\t\t\t\t default_match);\n+\t\t\tprocessed = clear_ce_flags_1(istate, cache, cache_end - cache,\n+\t\t\t\t\t\t prefix,\n+\t\t\t\t\t\t select_mask, clear_mask, pl,\n+\t\t\t\t\t\t default_match, progress_nr);\n+\n+\t\t\tcache += processed;\n+\t\t\tprogress_nr += processed;\n+\n \t\t\tstrbuf_setlen(prefix, prefix->len - len - 1);\n \t\t\tcontinue;\n \t\t}\n@@ -1406,19 +1419,27 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\tif (ret == MATCHED)\n \t\t\tce->ce_flags &= ~clear_mask;\n \t\tcache++;\n+\t\tprogress_nr++;\n \t}\n+\n+\tdisplay_progress(istate->progress, progress_nr);\n \treturn nr - (cache_end - cache);\n }\n \n static int clear_ce_flags(struct index_state *istate,\n \t\t\t int select_mask, int clear_mask,\n-\t\t\t struct pattern_list *pl)\n+\t\t\t struct pattern_list *pl,\n+\t\t\t int show_progress)\n {\n \tstatic struct strbuf prefix = STRBUF_INIT;\n \tchar label[100];\n \tint rval;\n \n \tstrbuf_reset(&prefix);\n+\tif (show_progress)\n+\t\tistate->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Updating index flags\"),\n+\t\t\t\t\tistate->cache_nr);\n \n \txsnprintf(label, sizeof(label), \"clear_ce_flags(0x%08lx,0x%08lx)\",\n \t\t (unsigned long)select_mask, (unsigned long)clear_mask);\n@@ -1428,9 +1449,10 @@ static int clear_ce_flags(struct index_state *istate,\n \t\t\t\tistate->cache_nr,\n \t\t\t\t&prefix,\n \t\t\t\tselect_mask, clear_mask,\n-\t\t\t\tpl, 0);\n+\t\t\t\tpl, 0, 0);\n \ttrace2_region_leave(\"unpack_trees\", label, the_repository);\n \n+\tstop_progress(&istate->progress);\n \treturn rval;\n }\n \n@@ -1439,7 +1461,8 @@ static int clear_ce_flags(struct index_state *istate,\n */\n static void mark_new_skip_worktree(struct pattern_list *pl,\n \t\t\t\t struct index_state *istate,\n-\t\t\t\t int select_flag, int skip_wt_flag)\n+\t\t\t\t int select_flag, int skip_wt_flag,\n+\t\t\t\t int show_progress)\n {\n \tint i;\n \n@@ -1463,7 +1486,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl,\n \t * 2. Widen worktree according to sparse-checkout file.\n \t * Matched entries will have skip_wt_flag cleared (i.e. \"in\")\n \t */\n-\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl);\n+\tclear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress);\n }\n \n static int verify_absent(const struct cache_entry *,\n@@ -1525,7 +1548,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries\n \t */\n \tif (!o->skip_sparse_checkout)\n-\t\tmark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, o->src_index, 0,\n+\t\t\t\t CE_NEW_SKIP_WORKTREE, o->verbose_update);\n \n \tif (!dfc)\n \t\tdfc = xcalloc(1, cache_entry_size(0));\n@@ -1590,7 +1614,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \t\t * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE\n \t\t * so apply_sparse_checkout() won't attempt to remove it from worktree\n \t\t */\n-\t\tmark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE);\n+\t\tmark_new_skip_worktree(o->pl, &o->result,\n+\t\t\t\t CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE,\n+\t\t\t\t o->verbose_update);\n \n \t\tret = 0;\n \t\tfor (i = 0; i < o->result.cache_nr; i++) {\n-- \ngitgitgadget\n\n"},{"id":"386759","messageId":"4097d8f6dec0cc25e356c17a8cda73d3c32a4894.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 17/19] sparse-checkout: cone mode should not interact with .gitignore","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:49Z","receivedAt":"2019-11-21T22:05:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nDuring the development of the sparse-checkout \"cone mode\" feature,\nan incorrect placement of the initializer for \"use_cone_patterns = 1\"\ncaused warnings to show up when a .gitignore file was present with\nnon-cone-mode patterns. This was fixed in the original commit\nintroducing the cone mode, but now we should add a test to avoid\nhitting this problem again in the future.\n\nSigned-off-by: Derrick Stolee \n---\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex f074b7f3be..e61ddb4ad5 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -284,4 +284,11 @@ test_expect_success 'fail when lock is taken' '\n \ttest_i18ngrep \"File exists\" err\n '\n \n+test_expect_success '.gitignore should not warn about cone mode' '\n+\tgit -C repo config --worktree core.sparseCheckoutCone true &&\n+\techo \"**/bin/*\" >repo/.gitignore &&\n+\tgit -C repo reset --hard 2>err &&\n+\ttest_i18ngrep ! \"disabling cone patterns\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386760","messageId":"63e01c202162689e5b8c9adff8ce140219a0bebe.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 14/19] sparse-checkout: update working directory in-process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:46Z","receivedAt":"2019-11-21T22:05:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe sparse-checkout builtin used 'git read-tree -mu HEAD' to update the\nskip-worktree bits in the index and to update the working directory.\nThis extra process is overly complex, and prone to failure. It also\nrequires that we write our changes to the sparse-checkout file before\ntrying to update the index.\n\nRemove this extra process call by creating a direct call to\nunpack_trees() in the same way 'git read-tree -mu HEAD' does. In\naddition, provide an in-memory list of patterns so we can avoid\nreading from the sparse-checkout file. This allows us to test a\nproposed change to the file before writing to it.\n\nAn earlier version of this patch included a bug when the 'set' command\nfailed due to the \"Sparse checkout leaves no entry on working directory\"\nerror. It would not rollback the index.lock file, so the replay of the\nold sparse-checkout specification would fail. A test in t1091 now\ncovers that scenario.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/read-tree.c | 2 +-\n builtin/sparse-checkout.c | 83 +++++++++++++++++++++++++-----\n t/t1091-sparse-checkout-builtin.sh | 28 ++++++++++\n unpack-trees.c | 5 +-\n unpack-trees.h | 3 +-\n 5 files changed, 105 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/read-tree.c b/builtin/read-tree.c\nindex ca5e655d2f..af7424b94c 100644\n--- a/builtin/read-tree.c\n+++ b/builtin/read-tree.c\n@@ -185,7 +185,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix)\n \n \tif (opts.reset || opts.merge || opts.prefix) {\n \t\tif (read_cache_unmerged() && (opts.prefix || opts.merge))\n-\t\t\tdie(\"You need to resolve your current index first\");\n+\t\t\tdie(_(\"You need to resolve your current index first\"));\n \t\tstage = opts.merge = 1;\n \t}\n \tresolve_undo_clear();\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 55b337ad8e..a5d32e4702 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -7,6 +7,11 @@\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"string-list.h\"\n+#include \"cache.h\"\n+#include \"cache-tree.h\"\n+#include \"lockfile.h\"\n+#include \"resolve-undo.h\"\n+#include \"unpack-trees.h\"\n \n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n@@ -60,18 +65,54 @@ static int sparse_checkout_list(int argc, const char **argv)\n \treturn 0;\n }\n \n-static int update_working_directory(void)\n+static int update_working_directory(struct pattern_list *pl)\n {\n-\tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tint result = 0;\n-\targv_array_pushl(&argv, \"read-tree\", \"-m\", \"-u\", \"HEAD\", NULL);\n+\tstruct unpack_trees_options o;\n+\tstruct lock_file lock_file = LOCK_INIT;\n+\tstruct object_id oid;\n+\tstruct tree *tree;\n+\tstruct tree_desc t;\n+\tstruct repository *r = the_repository;\n \n-\tif (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {\n-\t\terror(_(\"failed to update index with new sparse-checkout patterns\"));\n-\t\tresult = 1;\n-\t}\n+\tif (repo_read_index_unmerged(r))\n+\t\tdie(_(\"you need to resolve your current index first\"));\n+\n+\tif (get_oid(\"HEAD\", &oid))\n+\t\treturn 0;\n+\n+\ttree = parse_tree_indirect(&oid);\n+\tparse_tree(tree);\n+\tinit_tree_desc(&t, tree->buffer, tree->size);\n+\n+\tmemset(&o, 0, sizeof(o));\n+\to.verbose_update = isatty(2);\n+\to.merge = 1;\n+\to.update = 1;\n+\to.fn = oneway_merge;\n+\to.head_idx = -1;\n+\to.src_index = r->index;\n+\to.dst_index = r->index;\n+\to.skip_sparse_checkout = 0;\n+\to.pl = pl;\n+\to.keep_pattern_list = !!pl;\n+\n+\tresolve_undo_clear_index(r->index);\n+\tsetup_work_tree();\n+\n+\tcache_tree_free(&r->index->cache_tree);\n+\n+\trepo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR);\n+\n+\tcore_apply_sparse_checkout = 1;\n+\tresult = unpack_trees(1, &t, &o);\n+\n+\tif (!result) {\n+\t\tprime_cache_tree(r, r->index, tree);\n+\t\twrite_locked_index(r->index, &lock_file, COMMIT_LOCK);\n+\t} else\n+\t\trollback_lock_file(&lock_file);\n \n-\targv_array_clear(&argv);\n \treturn result;\n }\n \n@@ -129,6 +170,15 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint result;\n+\n+\tresult = update_working_directory(pl);\n+\n+\tif (result) {\n+\t\tclear_pattern_list(pl);\n+\t\tupdate_working_directory(NULL);\n+\t\treturn result;\n+\t}\n \n \tsparse_filename = get_sparse_checkout_filename();\n \tfp = fopen(sparse_filename, \"w\");\n@@ -139,9 +189,11 @@ static int write_patterns_and_update(struct pattern_list *pl)\n \t\twrite_patterns_to_file(fp, pl);\n \n \tfclose(fp);\n+\n \tfree(sparse_filename);\n+\tclear_pattern_list(pl);\n \n-\treturn update_working_directory();\n+\treturn 0;\n }\n \n enum sparse_checkout_mode {\n@@ -199,7 +251,11 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n \n-\tmode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS;\n+\tif (init_opts.cone_mode) {\n+\t\tmode = MODE_CONE_PATTERNS;\n+\t\tcore_sparse_checkout_cone = 1;\n+\t} else\n+\t\tmode = MODE_ALL_PATTERNS;\n \n \tif (set_config(mode))\n \t\treturn 1;\n@@ -230,7 +286,8 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t}\n \n reset_dir:\n-\treturn update_working_directory();\n+\tcore_apply_sparse_checkout = 1;\n+\treturn update_working_directory(NULL);\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n@@ -311,6 +368,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \n \t\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \t\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\t\tpl.use_cone_patterns = 1;\n \n \t\tif (set_opts.use_stdin) {\n \t\t\twhile (!strbuf_getline(&line, stdin))\n@@ -365,7 +423,8 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tfprintf(fp, \"/*\\n\");\n \tfclose(fp);\n \n-\tif (update_working_directory())\n+\tcore_apply_sparse_checkout = 1;\n+\tif (update_working_directory(NULL))\n \t\tdie(_(\"error while refreshing working directory\"));\n \n \tunlink(sparse_filename);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b88d08da98..53aeb5980f 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -248,4 +248,32 @@ test_expect_success 'cone mode: set with nested folders' '\n \ttest_cmp repo/.git/info/sparse-checkout expect\n '\n \n+test_expect_success 'revert to old sparse-checkout on bad update' '\n+\techo update >repo/deep/deeper2/a &&\n+\tcp repo/.git/info/sparse-checkout expect &&\n+\ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n+\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_cmp repo/.git/info/sparse-checkout expect &&\n+\tls repo/deep >dir &&\n+\tcat >expect <<-EOF &&\n+\t\ta\n+\t\tdeeper1\n+\t\tdeeper2\n+\tEOF\n+\ttest_cmp dir expect\n+'\n+\n+test_expect_success 'revert to old sparse-checkout on empty update' '\n+\tgit init empty-test &&\n+\t(\n+\t\techo >file &&\n+\t\tgit add file &&\n+\t\tgit commit -m \"test\" &&\n+\t\ttest_must_fail git sparse-checkout set nothing 2>err &&\n+\t\ttest_i18ngrep \"Sparse checkout leaves no entry on working directory\" err &&\n+\t\ttest_i18ngrep ! \".git/index.lock\" err &&\n+\t\tgit sparse-checkout set file\n+\t)\n+'\n+\n test_done\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 8bb684ad62..3789a22cf0 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tmemset(&pl, 0, sizeof(pl));\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n-\tif (!o->skip_sparse_checkout) {\n+\tif (!o->skip_sparse_checkout && !o->pl) {\n \t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n \t\tpl.use_cone_patterns = core_sparse_checkout_cone;\n \t\tif (add_patterns_from_file_to_list(sparse, \"\", 0, &pl, NULL) < 0)\n@@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \n done:\n \ttrace_performance_leave(\"unpack_trees\");\n-\tclear_pattern_list(&pl);\n+\tif (!o->keep_pattern_list)\n+\t\tclear_pattern_list(&pl);\n \treturn ret;\n \n return_failed:\ndiff --git a/unpack-trees.h b/unpack-trees.h\nindex f2eee0c7c5..ca94a421a5 100644\n--- a/unpack-trees.h\n+++ b/unpack-trees.h\n@@ -59,7 +59,8 @@ struct unpack_trees_options {\n \t\t quiet,\n \t\t exiting_early,\n \t\t show_all_errors,\n-\t\t dry_run;\n+\t\t dry_run,\n+\t\t keep_pattern_list;\n \tconst char *prefix;\n \tint cache_bottom;\n \tstruct dir_struct *dir;\n-- \ngitgitgadget\n\n"},{"id":"386761","messageId":"a0dca0056a0a20b5cf50e1af90c4c124ac7c9560.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 16/19] sparse-checkout: write using lockfile","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:48Z","receivedAt":"2019-11-21T22:05:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf two 'git sparse-checkout set' subcommands are launched at the\nsame time, the behavior can be unexpected as they compete to write\nthe sparse-checkout file and update the working directory.\n\nTake a lockfile around the writes to the sparse-checkout file. In\naddition, acquire this lock around the working directory update\nto avoid two commands updating the working directory in different\nways.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 15 +++++++++++----\n t/t1091-sparse-checkout-builtin.sh | 7 +++++++\n 2 files changed, 18 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex a11ea65599..9a620ff014 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -170,25 +170,32 @@ static int write_patterns_and_update(struct pattern_list *pl)\n {\n \tchar *sparse_filename;\n \tFILE *fp;\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n \tint result;\n \n-\tresult = update_working_directory(pl);\n+\tsparse_filename = get_sparse_checkout_filename();\n+\tfd = hold_lock_file_for_update(&lk, sparse_filename,\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n \n+\tresult = update_working_directory(pl);\n \tif (result) {\n+\t\trollback_lock_file(&lk);\n+\t\tfree(sparse_filename);\n \t\tclear_pattern_list(pl);\n \t\tupdate_working_directory(NULL);\n \t\treturn result;\n \t}\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = fopen(sparse_filename, \"w\");\n+\tfp = xfdopen(fd, \"w\");\n \n \tif (core_sparse_checkout_cone)\n \t\twrite_cone_to_file(fp, pl);\n \telse\n \t\twrite_patterns_to_file(fp, pl);\n \n-\tfclose(fp);\n+\tfflush(fp);\n+\tcommit_lock_file(&lk);\n \n \tfree(sparse_filename);\n \tclear_pattern_list(pl);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex b8f18e2a09..f074b7f3be 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -277,4 +277,11 @@ test_expect_success 'revert to old sparse-checkout on empty update' '\n \t)\n '\n \n+test_expect_success 'fail when lock is taken' '\n+\ttest_when_finished rm -rf repo/.git/info/sparse-checkout.lock &&\n+\ttouch repo/.git/info/sparse-checkout.lock &&\n+\ttest_must_fail git -C repo sparse-checkout set deep 2>err &&\n+\ttest_i18ngrep \"File exists\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"386762","messageId":"cc6773e49b70681a9efbaa42d5a78011a0cafb38.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 15/19] sparse-checkout: use in-process update for disable subcommand","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:47Z","receivedAt":"2019-11-21T22:05:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout disable' subcommand returns a user to a\nfull working directory. The old process for doing this required\nupdating the sparse-checkout file with the \"/*\" pattern and then\nupdating the working directory with core.sparseCheckout enabled.\nFinally, the sparse-checkout file could be removed and the config\nsetting disabled.\n\nHowever, it is valuable to keep a user's sparse-checkout file\nintact so they can re-enable the sparse-checkout they previously\nused with 'git sparse-checkout init'. This is now possible with\nthe in-process mechanism for updating the working directory.\n\nReported-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-sparse-checkout.txt | 6 ++++--\n builtin/sparse-checkout.c | 25 ++++++++++++-------------\n t/t1091-sparse-checkout-builtin.sh | 3 ++-\n 3 files changed, 18 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt\nindex 8535f0cf40..b975285673 100644\n--- a/Documentation/git-sparse-checkout.txt\n+++ b/Documentation/git-sparse-checkout.txt\n@@ -52,8 +52,10 @@ When the `--stdin` option is provided, the patterns are read from\n standard in as a newline-delimited list instead of from the arguments.\n \n 'disable'::\n-\tRemove the sparse-checkout file, set `core.sparseCheckout` to\n-\t`false`, and restore the working directory to include all files.\n+\tDisable the `core.sparseCheckout` config setting, and restore the\n+\tworking directory to include all files. Leaves the sparse-checkout\n+\tfile intact so a later 'git sparse-checkout init' command may\n+\treturn the working directory to the same state.\n \n SPARSE CHECKOUT\n ---------------\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex a5d32e4702..a11ea65599 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -412,24 +412,23 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \n static int sparse_checkout_disable(int argc, const char **argv)\n {\n-\tchar *sparse_filename;\n-\tFILE *fp;\n+\tstatic const char *empty_base = \"\";\n+\tstruct pattern_list pl;\n+\tstruct strbuf match_all = STRBUF_INIT;\n \n-\tif (set_config(MODE_ALL_PATTERNS))\n-\t\tdie(_(\"failed to change config\"));\n+\tmemset(&pl, 0, sizeof(pl));\n+\thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n+\thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\n+\tpl.use_cone_patterns = 0;\n+\tcore_apply_sparse_checkout = 1;\n \n-\tsparse_filename = get_sparse_checkout_filename();\n-\tfp = xfopen(sparse_filename, \"w\");\n-\tfprintf(fp, \"/*\\n\");\n-\tfclose(fp);\n+\tstrbuf_addstr(&match_all, \"/*\");\n+\tadd_pattern(strbuf_detach(&match_all, NULL), empty_base, 0, &pl, 0);\n \n-\tcore_apply_sparse_checkout = 1;\n-\tif (update_working_directory(NULL))\n+\tif (update_working_directory(&pl))\n \t\tdie(_(\"error while refreshing working directory\"));\n \n-\tunlink(sparse_filename);\n-\tfree(sparse_filename);\n-\n+\tclear_pattern_list(&pl);\n \treturn set_config(MODE_NO_PATTERNS);\n }\n \ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex 53aeb5980f..b8f18e2a09 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -172,8 +172,9 @@ test_expect_success 'cone mode: warn on bad pattern' '\n '\n \n test_expect_success 'sparse-checkout disable' '\n+\ttest_when_finished rm -rf repo/.git/info/sparse-checkout &&\n \tgit -C repo sparse-checkout disable &&\n-\ttest_path_is_missing repo/.git/info/sparse-checkout &&\n+\ttest_path_is_file repo/.git/info/sparse-checkout &&\n \tgit -C repo config --list >config &&\n \ttest_must_fail git config core.sparseCheckout &&\n \tls repo >dir &&\n-- \ngitgitgadget\n\n"},{"id":"386763","messageId":"c62d9750efe992f13cb2fa8c71120ffa0f850ae3.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 18/19] sparse-checkout: update working directory in-process for 'init'","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:50Z","receivedAt":"2019-11-21T22:05:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git sparse-checkout init' subcommand previously wrote directly\nto the sparse-checkout file and then updated the working directory.\nThis may fail if there are modified files not included in the initial\npattern set. However, that left a populated sparse-checkout file.\n\nUse the in-process working directory update to guarantee that the\ninit subcommand only changes the sparse-checkout file if the working\ndirectory update succeeds.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 36 ++++++++++++++++++++----------------\n 1 file changed, 20 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 9a620ff014..5dbfb2a7e0 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -13,6 +13,8 @@\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n \n+static const char *empty_base = \"\";\n+\n static char const * const builtin_sparse_checkout_usage[] = {\n \tN_(\"git sparse-checkout (init|list|set|disable) \"),\n \tNULL\n@@ -243,10 +245,10 @@ static int sparse_checkout_init(int argc, const char **argv)\n {\n \tstruct pattern_list pl;\n \tchar *sparse_filename;\n-\tFILE *fp;\n \tint res;\n \tstruct object_id oid;\n \tint mode;\n+\tstruct strbuf pattern = STRBUF_INIT;\n \n \tstatic struct option builtin_sparse_checkout_init_options[] = {\n \t\tOPT_BOOL(0, \"cone\", &init_opts.cone_mode,\n@@ -275,26 +277,30 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t/* If we already have a sparse-checkout file, use it. */\n \tif (res >= 0) {\n \t\tfree(sparse_filename);\n-\t\tgoto reset_dir;\n+\t\tcore_apply_sparse_checkout = 1;\n+\t\treturn update_working_directory(NULL);\n \t}\n \n-\t/* initial mode: all blobs at root */\n-\tfp = xfopen(sparse_filename, \"w\");\n-\tif (!fp)\n-\t\tdie(_(\"failed to open '%s'\"), sparse_filename);\n+\tif (get_oid(\"HEAD\", &oid)) {\n+\t\tFILE *fp;\n \n-\tfree(sparse_filename);\n-\tfprintf(fp, \"/*\\n!/*/\\n\");\n-\tfclose(fp);\n+\t\t/* assume we are in a fresh repo, but update the sparse-checkout file */\n+\t\tfp = xfopen(sparse_filename, \"w\");\n+\t\tif (!fp)\n+\t\t\tdie(_(\"failed to open '%s'\"), sparse_filename);\n \n-\tif (get_oid(\"HEAD\", &oid)) {\n-\t\t/* assume we are in a fresh repo */\n+\t\tfree(sparse_filename);\n+\t\tfprintf(fp, \"/*\\n!/*/\\n\");\n+\t\tfclose(fp);\n \t\treturn 0;\n \t}\n \n-reset_dir:\n-\tcore_apply_sparse_checkout = 1;\n-\treturn update_working_directory(NULL);\n+\tstrbuf_addstr(&pattern, \"/*\");\n+\tadd_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);\n+\tstrbuf_addstr(&pattern, \"!/*/\");\n+\tadd_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0);\n+\n+\treturn write_patterns_and_update(&pl);\n }\n \n static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path)\n@@ -351,7 +357,6 @@ static struct sparse_checkout_set_opts {\n \n static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n {\n-\tstatic const char *empty_base = \"\";\n \tint i;\n \tstruct pattern_list pl;\n \tint result;\n@@ -419,7 +424,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \n static int sparse_checkout_disable(int argc, const char **argv)\n {\n-\tstatic const char *empty_base = \"\";\n \tstruct pattern_list pl;\n \tstruct strbuf match_all = STRBUF_INIT;\n \n-- \ngitgitgadget\n\n"},{"id":"386764","messageId":"7577ffc034be90bd60da17e8f3189d999e7f8149.1574373892.git.gitgitgadget@gmail.com","threadId":"51691","inReplyTo":"pull.316.v6.git.1574373891.gitgitgadget@gmail.com","subject":"[PATCH v6 19/19] sparse-checkout: check for dirty status","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-21T22:04:51Z","receivedAt":"2019-11-21T22:05:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe index-merge performed by 'git sparse-checkout' will erase any staged\nchanges, which can lead to data loss. Prevent these attempts by requiring\na clean 'git status' output.\n\nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n builtin/sparse-checkout.c | 13 +++++++++++++\n t/t1091-sparse-checkout-builtin.sh | 15 ++++++++++++++-\n 2 files changed, 27 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c\nindex 5dbfb2a7e0..a542d617a5 100644\n--- a/builtin/sparse-checkout.c\n+++ b/builtin/sparse-checkout.c\n@@ -12,6 +12,7 @@\n #include \"lockfile.h\"\n #include \"resolve-undo.h\"\n #include \"unpack-trees.h\"\n+#include \"wt-status.h\"\n \n static const char *empty_base = \"\";\n \n@@ -256,6 +257,10 @@ static int sparse_checkout_init(int argc, const char **argv)\n \t\tOPT_END(),\n \t};\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"initialize sparse-checkout\"), NULL, 1, 0);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_sparse_checkout_init_options,\n \t\t\t builtin_sparse_checkout_init_usage, 0);\n@@ -368,6 +373,10 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix)\n \t\tOPT_END(),\n \t};\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"set sparse-checkout patterns\"), NULL, 1, 0);\n+\n \tmemset(&pl, 0, sizeof(pl));\n \n \targc = parse_options(argc, argv, prefix,\n@@ -427,6 +436,10 @@ static int sparse_checkout_disable(int argc, const char **argv)\n \tstruct pattern_list pl;\n \tstruct strbuf match_all = STRBUF_INIT;\n \n+\trepo_read_index(the_repository);\n+\trequire_clean_work_tree(the_repository,\n+\t\t\t\tN_(\"disable sparse-checkout\"), NULL, 1, 0);\n+\n \tmemset(&pl, 0, sizeof(pl));\n \thashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0);\n \thashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0);\ndiff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh\nindex e61ddb4ad5..d5e2892526 100755\n--- a/t/t1091-sparse-checkout-builtin.sh\n+++ b/t/t1091-sparse-checkout-builtin.sh\n@@ -250,10 +250,11 @@ test_expect_success 'cone mode: set with nested folders' '\n '\n \n test_expect_success 'revert to old sparse-checkout on bad update' '\n+\ttest_when_finished git -C repo reset --hard &&\n \techo update >repo/deep/deeper2/a &&\n \tcp repo/.git/info/sparse-checkout expect &&\n \ttest_must_fail git -C repo sparse-checkout set deep/deeper1 2>err &&\n-\ttest_i18ngrep \"Cannot update sparse checkout\" err &&\n+\ttest_i18ngrep \"cannot set sparse-checkout patterns\" err &&\n \ttest_cmp repo/.git/info/sparse-checkout expect &&\n \tls repo/deep >dir &&\n \tcat >expect <<-EOF &&\n@@ -291,4 +292,16 @@ test_expect_success '.gitignore should not warn about cone mode' '\n \ttest_i18ngrep ! \"disabling cone patterns\" err\n '\n \n+test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' '\n+\tgit clone repo dirty &&\n+\techo dirty >dirty/folder1/a &&\n+\ttest_must_fail git -C dirty sparse-checkout init &&\n+\ttest_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\ttest_must_fail git -C dirty sparse-checkout disable &&\n+\tgit -C dirty reset --hard &&\n+\tgit -C dirty sparse-checkout init &&\n+\tgit -C dirty sparse-checkout set /folder2/* /deep/deeper1/* &&\n+\tgit -C dirty sparse-checkout disable\n+'\n+\n test_done\n-- \ngitgitgadget\n"}]}