{"thread":{"id":"49719","subject":"[RFC v1] Add virtual file system settings and hook proc","startedAt":"2018-10-30T19:16:50Z","lastAt":"2019-01-28T19:00:16Z","messageCount":21,"participants":["Ben Peart","Junio C Hamano","Duy Nguyen","brian m. carlson","Ævar Arnfjörð Bjarmason","Johannes Schindelin","SZEDER Gábor"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362011","messageId":"20181030191608.18716-1-peartben@gmail.com","threadId":"49719","inReplyTo":null,"subject":"[RFC v1] Add virtual file system settings and hook proc","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-10-30T19:16:08Z","receivedAt":"2018-10-30T19:16:50Z","isPatch":false,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"From: Ben Peart \n\nOn index load, clear/set the skip worktree bits based on the virtual\nfile system data. Use virtual file system data to update skip-worktree\nbit in unpack-trees. Use virtual file system data to exclude files and\nfolders not explicitly requested.\n\nSigned-off-by: Ben Peart \n---\n\nWe have taken several steps to make git perform well on very large repos.\nSome of those steps include: improving underlying algorithms, utilizing\nmulti-threading where possible, and simplifying the behavior of some commands.\nThese changes typically benefit all git repos to varying degrees. While\nthese optimizations all help, they are insufficient to provide adequate\nperformance on the very large repos we often work with.\n\nTo make git perform well on the very largest repos, we had to make more\nsignificant changes. The biggest performance win by far is the work we have\ndone to make git operations O(modified) instead of O(size of repo). This\ntakes advantage of the fact that the number of files a developer has modified\nis a tiny fraction of the overall repo size.\n\nWe accomplished this by utilizing the existing internal logic for the skip\nworktree bit and excludes to tell git to ignore all files and folders other\nthan those that have been modified. This logic is driven by an external\nprocess that monitors writes to the repo and communicates the list of files\nand folders with changes to git via the virtual file system hook in this patch.\n\nThe external process maintains a list of files and folders that have been\nmodified. When git runs, it requests the list of files and folders that\nhave been modified via the virtual file system hook. Git then sets/clears\nthe skip-worktree bit on the cache entries and builds a hashmap of the\nmodified files/folders that is used by the excludes logic to avoid scanning\nthe entire repo looking for changes and untracked files.\n\nWith this system, we have been able to make local git command performance on\nextremely large repos (millions of files, 1/2 million folders) entirely\nmanageable (30 second checkout, 3.5 seconds status, 4 second add, 7 second\ncommit, etc).\n\nOur desire is to eliminate all custom patches in our fork of git. To that\nend, I'm submitting this as an RFC to see how much interest there is and how\nmuch willingness to take this type of change into git.\n\nNotes:\n Base Ref: master\n Web-Diff: https://github.com/benpeart/git/commit/c06b290d2f\n Checkout: git fetch https://github.com/benpeart/git virtual-filesystem-v1 && git checkout c06b290d2f\n\n Documentation/config.txt | 8 +\n Documentation/githooks.txt | 20 ++\n Makefile | 1 +\n cache.h | 1 +\n config.c | 37 +++-\n config.h | 1 +\n dir.c | 34 +++-\n environment.c | 1 +\n read-cache.c | 2 +\n t/README | 3 +\n t/t1092-virtualfilesystem.sh | 354 +++++++++++++++++++++++++++++++++++\n t/t1092/virtualfilesystem | 23 +++\n unpack-trees.c | 23 ++-\n virtualfilesystem.c | 308 ++++++++++++++++++++++++++++++\n virtualfilesystem.h | 25 +++\n 15 files changed, 833 insertions(+), 8 deletions(-)\n create mode 100755 t/t1092-virtualfilesystem.sh\n create mode 100755 t/t1092/virtualfilesystem\n create mode 100644 virtualfilesystem.c\n create mode 100644 virtualfilesystem.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 09e95e9e98..dd4b834375 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -441,6 +441,14 @@ core.fsmonitor::\n \tavoiding unnecessary processing of files that have not changed.\n \tSee the \"fsmonitor-watchman\" section of linkgit:githooks[5].\n \n+core.virtualFilesystem::\n+\tIf set, the value of this variable is used as a command which\n+\twill identify all files and directories that are present in\n+\tthe working directory. Git will only track and update files\n+\tlisted in the virtual file system. Using the virtual file system\n+\twill supersede the sparse-checkout settings which will be ignored.\n+\tSee the \"virtual file system\" section of linkgit:githooks[6].\n+\n core.trustctime::\n \tIf false, the ctime differences between the index and the\n \tworking tree are ignored; useful when the inode change time\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 959044347e..87f9ad2a77 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -492,6 +492,26 @@ This hook is invoked by `git-p4 submit`. It takes no parameters and nothing\n from standard input. Exiting with non-zero status from this script prevent\n `git-p4 submit` from launching. Run `git-p4 submit --help` for details.\n \n+virtualFilesystem\n+~~~~~~~~~~~~~~~~~~\n+\n+\"Virtual File System\" allows populating the working directory sparsely.\n+The projection data is typically automatically generated by an external\n+process. Git will limit what files it checks for changes as well as which\n+directories are checked for untracked files based on the path names given.\n+Git will also only update those files listed in the projection.\n+\n+The hook is invoked when the configuration option core.virtualFilesystem\n+is set. It takes one argument, a version (currently 1).\n+\n+The hook should output to stdout the list of all files in the working\n+directory that git should track. The paths are relative to the root\n+of the working directory and are separated by a single NUL. Full paths\n+('dir1/a.txt') as well as directories are supported (ie 'dir1/').\n+\n+The exit status determines whether git will use the data from the\n+hook. On error, git will abort the command with an error message.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex b08d5ea258..bcb404c28d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1009,6 +1009,7 @@ LIB_OBJS += utf8.o\n LIB_OBJS += varint.o\n LIB_OBJS += version.o\n LIB_OBJS += versioncmp.o\n+LIB_OBJS += virtualfilesystem.o\n LIB_OBJS += walker.o\n LIB_OBJS += wildmatch.o\n LIB_OBJS += worktree.o\ndiff --git a/cache.h b/cache.h\nindex f7fabdde8f..d5815397c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -881,6 +881,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern const char *core_virtualfilesystem;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\ndiff --git a/config.c b/config.c\nindex 4051e38823..96e05ee0f1 100644\n--- a/config.c\n+++ b/config.c\n@@ -1325,7 +1325,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t}\n \n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n-\t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n+\t\t/* virtual file system relies on the sparse checkout logic so force it on */\n+\t\tif (core_virtualfilesystem)\n+\t\t\tcore_apply_sparse_checkout = 1;\n+\t\telse\n+\t\t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\n \t}\n \n@@ -2307,6 +2311,37 @@ int git_config_get_index_threads(void)\n \treturn 0; /* auto */\n }\n \n+int git_config_get_virtualfilesystem(void)\n+{\n+\tif (git_config_get_pathname(\"core.virtualfilesystem\", &core_virtualfilesystem))\n+\t\tcore_virtualfilesystem = getenv(\"GIT_TEST_VIRTUALFILESYSTEM\");\n+\n+\tif (core_virtualfilesystem && !*core_virtualfilesystem)\n+\t\tcore_virtualfilesystem = NULL;\n+\n+\tif (core_virtualfilesystem) {\n+\t\t/*\n+\t\t * Some git commands spawn helpers and redirect the index to a different\n+\t\t * location. These include \"difftool -d\" and the sequencer\n+\t\t * (i.e. `git rebase -i`, `git cherry-pick` and `git revert`) and others.\n+\t\t * In those instances we don't want to update their temporary index with\n+\t\t * our virtualization data.\n+\t\t */\n+\t\tchar *default_index_file = xstrfmt(\"%s/%s\", the_repository->gitdir, \"index\");\n+\t\tint should_run_hook = !strcmp(default_index_file, the_repository->index_file);\n+\n+\t\tfree(default_index_file);\n+\t\tif (should_run_hook) {\n+\t\t\t/* virtual file system relies on the sparse checkout logic so force it on */\n+\t\t\tcore_apply_sparse_checkout = 1;\n+\t\t\treturn 1;\n+\t\t}\n+\t\tcore_virtualfilesystem = NULL;\n+\t}\n+\n+\treturn 0;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex a06027e69b..2ed36d5e26 100644\n--- a/config.h\n+++ b/config.h\n@@ -251,6 +251,7 @@ extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n extern int git_config_get_fsmonitor(void);\n extern int git_config_get_index_threads(void);\n+extern int git_config_get_virtualfilesystem(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/dir.c b/dir.c\nindex 47c2fca8dc..3097b0e446 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -21,6 +21,7 @@\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n #include \"submodule-config.h\"\n+#include \"virtualfilesystem.h\"\n \n /*\n * Tells read_directory_recursive how a file or directory should be treated.\n@@ -1109,6 +1110,18 @@ int is_excluded_from_list(const char *pathname,\n \t\t\t struct exclude_list *el, struct index_state *istate)\n {\n \tstruct exclude *exclude;\n+\n+\t/*\n+\t * The virtual file system data is used to prevent git from traversing\n+\t * any part of the tree that is not in the virtual file system. Return\n+\t * 1 to exclude the entry if it is not found in the virtual file system,\n+\t * else fall through to the regular excludes logic as it may further exclude.\n+\t */\n+\tif (*dtype == DT_UNKNOWN)\n+\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n+\tif (is_excluded_from_virtualfilesystem(pathname, pathlen, *dtype) > 0)\n+\t\treturn 1;\n+\n \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n \t\t\t\t\t\t dtype, el, istate);\n \tif (exclude)\n@@ -1324,8 +1337,20 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n int is_excluded(struct dir_struct *dir, struct index_state *istate,\n \t\tconst char *pathname, int *dtype_p)\n {\n-\tstruct exclude *exclude =\n-\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n+\tstruct exclude *exclude;\n+\n+\t/*\n+\t * The virtual file system data is used to prevent git from traversing\n+\t * any part of the tree that is not in the virtual file system. Return\n+\t * 1 to exclude the entry if it is not found in the virtual file system,\n+\t * else fall through to the regular excludes logic as it may further exclude.\n+\t */\n+\tif (*dtype_p == DT_UNKNOWN)\n+\t\t*dtype_p = get_dtype(NULL, istate, pathname, strlen(pathname));\n+\tif (is_excluded_from_virtualfilesystem(pathname, strlen(pathname), *dtype_p) > 0)\n+\t\treturn 1;\n+\n+\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n \tif (exclude)\n \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n \treturn 0;\n@@ -1678,6 +1703,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n+\tif (is_excluded_from_virtualfilesystem(path->buf, path->len, dtype) > 0)\n+\t\treturn path_excluded;\n+\n \t/*\n \t * When we are looking at a directory P in the working tree,\n \t * there are three cases:\n@@ -2018,6 +2046,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* add the path to the appropriate result list */\n \t\tswitch (state) {\n \t\tcase path_excluded:\n+\t\t\tif (is_excluded_from_virtualfilesystem(path.buf, path.len, DT_DIR) > 0)\n+\t\t\t\tbreak;\n \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\ndiff --git a/environment.c b/environment.c\nindex 3f3c8746c2..eea50b71a7 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -68,6 +68,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+const char *core_virtualfilesystem;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\ndiff --git a/read-cache.c b/read-cache.c\nindex d57958233e..afe09a0343 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -25,6 +25,7 @@\n #include \"fsmonitor.h\"\n #include \"thread-utils.h\"\n #include \"progress.h\"\n+#include \"virtualfilesystem.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1890,6 +1891,7 @@ static void post_read_index_from(struct index_state *istate)\n \ttweak_untracked_cache(istate);\n \ttweak_split_index(istate);\n \ttweak_fsmonitor(istate);\n+\tapply_virtualfilesystem(istate);\n }\n \n static size_t estimate_cache_size_from_compressed(unsigned int entries)\ndiff --git a/t/README b/t/README\nindex 8847489640..b79c77c8b5 100644\n--- a/t/README\n+++ b/t/README\n@@ -343,6 +343,9 @@ of the index for the whole test suite by bypassing the default number of\n cache entries and thread minimums. Setting this to 1 will make the\n index loading single threaded.\n \n+GIT_TEST_VIRTUALFILESYSTEM=$PWD/t1092/virtualfilesystem exercises the\n+virtual file system code path for limiting what git pays attention to.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1092-virtualfilesystem.sh b/t/t1092-virtualfilesystem.sh\nnew file mode 100755\nindex 0000000000..a0f4200360\n--- /dev/null\n+++ b/t/t1092-virtualfilesystem.sh\n@@ -0,0 +1,354 @@\n+#!/bin/sh\n+\n+test_description='virtual file system tests'\n+\n+. ./test-lib.sh\n+\n+# We need total control of the virtual file system hook\n+sane_unset GIT_TEST_VIRTUALFILESYSTEM\n+\n+clean_repo () {\n+\trm .git/index &&\n+\tgit -c core.virtualfilesystem= reset --hard HEAD &&\n+\tgit -c core.virtualfilesystem= clean -fd &&\n+\ttouch untracked.txt &&\n+\ttouch dir1/untracked.txt &&\n+\ttouch dir2/untracked.txt\n+}\n+\n+test_expect_success 'setup' '\n+\tmkdir -p .git/hooks/ &&\n+\tcat > .gitignore <<-\\EOF &&\n+\t\t.gitignore\n+\t\texpect*\n+\t\tactual*\n+\tEOF\n+\tmkdir -p dir1 &&\n+\ttouch dir1/file1.txt &&\n+\ttouch dir1/file2.txt &&\n+\tmkdir -p dir2 &&\n+\ttouch dir2/file1.txt &&\n+\ttouch dir2/file2.txt &&\n+\tgit add . &&\n+\tgit commit -m \"initial\" &&\n+\tgit config --local core.virtualfilesystem .git/hooks/virtualfilesystem\n+'\n+\n+test_expect_success 'test hook parameters and version' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tif test \"$#\" -ne 1\n+\t\tthen\n+\t\t\techo \"$0: Exactly 1 argument expected\" >&2\n+\t\t\texit 2\n+\t\tfi\n+\n+\t\tif test \"$1\" != 1\n+\t\tthen\n+\t\t\techo \"$0: Unsupported hook version.\" >&2\n+\t\t\texit 1\n+\t\tfi\n+\tEOF\n+\tgit status &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\texit 3\n+\tEOF\n+\ttest_must_fail git status\n+'\n+\n+test_expect_success 'verify status is clean' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir2/file1.txt\\0\"\n+\tEOF\n+\trm -f .git/index &&\n+\tgit checkout -f &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir2/file1.txt\\0\"\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\t\tprintf \"dir1/file2.txt\\0\"\n+\tEOF\n+\tgit status > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tOn branch master\n+\t\tnothing to commit, working tree clean\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify skip-worktree bit is set for absolute path' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\tEOF\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tH dir1/file1.txt\n+\t\tS dir1/file2.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify skip-worktree bit is cleared for absolute path' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file2.txt\\0\"\n+\tEOF\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tS dir1/file1.txt\n+\t\tH dir1/file2.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify folder wild cards' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/\\0\"\n+\tEOF\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tH dir1/file1.txt\n+\t\tH dir1/file2.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify folders not included are ignored' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\t\tprintf \"dir1/file2.txt\\0\"\n+\tEOF\n+\tmkdir -p dir1/dir2 &&\n+\ttouch dir1/a &&\n+\ttouch dir1/b &&\n+\ttouch dir1/dir2/a &&\n+\ttouch dir1/dir2/b &&\n+\tgit add . &&\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tH dir1/file1.txt\n+\t\tH dir1/file2.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify including one file doesnt include the rest' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\t\tprintf \"dir1/file2.txt\\0\"\n+\t\tprintf \"dir1/dir2/a\\0\"\n+\tEOF\n+\tmkdir -p dir1/dir2 &&\n+\ttouch dir1/a &&\n+\ttouch dir1/b &&\n+\ttouch dir1/dir2/a &&\n+\ttouch dir1/dir2/b &&\n+\tgit add . &&\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tH dir1/dir2/a\n+\t\tH dir1/file1.txt\n+\t\tH dir1/file2.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify files not listed are ignored by git clean -f -x' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"untracked.txt\\0\"\n+\t\tprintf \"dir1/\\0\"\n+\tEOF\n+\tmkdir -p dir3 &&\n+\ttouch dir3/untracked.txt &&\n+\tgit clean -f -x &&\n+\ttest ! -f untracked.txt &&\n+\ttest -d dir1 &&\n+\ttest -f dir1/file1.txt &&\n+\ttest -f dir1/file2.txt &&\n+\ttest ! -f dir1/untracked.txt &&\n+\ttest -f dir2/file1.txt &&\n+\ttest -f dir2/file2.txt &&\n+\ttest -f dir2/untracked.txt &&\n+\ttest -d dir3 &&\n+\ttest -f dir3/untracked.txt\n+'\n+\n+test_expect_success 'verify files not listed are ignored by git clean -f -d -x' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"untracked.txt\\0\"\n+\t\tprintf \"dir1/\\0\"\n+\t\tprintf \"dir3/\\0\"\n+\tEOF\n+\tmkdir -p dir3 &&\n+\ttouch dir3/untracked.txt &&\n+\tgit clean -f -d -x &&\n+\ttest ! -f untracked.txt &&\n+\ttest -d dir1 &&\n+\ttest -f dir1/file1.txt &&\n+\ttest -f dir1/file2.txt &&\n+\ttest ! -f dir1/untracked.txt &&\n+\ttest -f dir2/file1.txt &&\n+\ttest -f dir2/file2.txt &&\n+\ttest -f dir2/untracked.txt &&\n+\ttest ! -d dir3 &&\n+\ttest ! -f dir3/untracked.txt\n+'\n+\n+test_expect_success 'verify folder entries include all files' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/\\0\"\n+\tEOF\n+\tmkdir -p dir1/dir2 &&\n+\ttouch dir1/a &&\n+\ttouch dir1/b &&\n+\ttouch dir1/dir2/a &&\n+\ttouch dir1/dir2/b &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t?? dir1/a\n+\t\t?? dir1/b\n+\t\t?? dir1/dir2/a\n+\t\t?? dir1/dir2/b\n+\t\t?? dir1/untracked.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'verify case insensitivity of virtual file system entries' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/a\\0\"\n+\t\tprintf \"Dir1/Dir2/a\\0\"\n+\t\tprintf \"DIR2/\\0\"\n+\tEOF\n+\tmkdir -p dir1/dir2 &&\n+\ttouch dir1/a &&\n+\ttouch dir1/b &&\n+\ttouch dir1/dir2/a &&\n+\ttouch dir1/dir2/b &&\n+\tgit -c core.ignorecase=false status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t?? dir1/a\n+\tEOF\n+\ttest_cmp expected actual &&\n+\tgit -c core.ignorecase=true status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t?? dir1/a\n+\t\t?? dir1/dir2/a\n+\t\t?? dir2/untracked.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'on file created' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file3.txt\\0\"\n+\tEOF\n+\ttouch dir1/file3.txt &&\n+\tgit add . &&\n+\tgit ls-files -v > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\tS dir1/file1.txt\n+\t\tS dir1/file2.txt\n+\t\tH dir1/file3.txt\n+\t\tS dir2/file1.txt\n+\t\tS dir2/file2.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'on file renamed' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\t\tprintf \"dir1/file3.txt\\0\"\n+\tEOF\n+\tmv dir1/file1.txt dir1/file3.txt &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t D dir1/file1.txt\n+\t\t?? dir1/file3.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'on file deleted' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\tEOF\n+\trm dir1/file1.txt &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t D dir1/file1.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'on file overwritten' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\tEOF\n+\techo \"overwritten\" > dir1/file1.txt &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t M dir1/file1.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'on folder created' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir1/dir1/\\0\"\n+\tEOF\n+\tmkdir -p dir1/dir1 &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\tEOF\n+\ttest_cmp expected actual &&\n+\tgit clean -fd &&\n+\ttest ! -d \"/dir1/dir1\"\n+'\n+\n+test_expect_success 'on folder renamed' '\n+\tclean_repo &&\n+\twrite_script .git/hooks/virtualfilesystem <<-\\EOF &&\n+\t\tprintf \"dir3/\\0\"\n+\t\tprintf \"dir1/file1.txt\\0\"\n+\t\tprintf \"dir1/file2.txt\\0\"\n+\t\tprintf \"dir3/file1.txt\\0\"\n+\t\tprintf \"dir3/file2.txt\\0\"\n+\tEOF\n+\tmv dir1 dir3 &&\n+\tgit status -su > actual &&\n+\tcat > expected <<-\\EOF &&\n+\t\t D dir1/file1.txt\n+\t\t D dir1/file2.txt\n+\t\t?? dir3/file1.txt\n+\t\t?? dir3/file2.txt\n+\t\t?? dir3/untracked.txt\n+\tEOF\n+\ttest_cmp expected actual\n+'\n+\n+test_done\ndiff --git a/t/t1092/virtualfilesystem b/t/t1092/virtualfilesystem\nnew file mode 100755\nindex 0000000000..3f29c5f91f\n--- /dev/null\n+++ b/t/t1092/virtualfilesystem\n@@ -0,0 +1,23 @@\n+#!/bin/sh\n+#\n+# An test hook script to integrate with git to test the virtual file system.\n+#\n+# The hook is passed a version (currently 1) and outputs to stdout all files\n+# that git should paid attention to. Paths must be relative to the root of\n+# the working tree and separated by a single NUL.\n+#\n+\n+if test \"$#\" -ne 1\n+then\n+\techo \"$0: exactly 1 argument expected\" >&2\n+\texit 2\n+fi\n+\n+if test \"$1\" != 1\n+then\n+\techo \"Unsupported core.virtualfilesystem hook version.\" >&2\n+\texit 1\n+fi\n+\n+#find -type f -printf '%P\\0'\n+find -type d -printf '%P/\\0'\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 7570df481b..ee3cda2e94 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -18,6 +18,7 @@\n #include \"fsmonitor.h\"\n #include \"object-store.h\"\n #include \"fetch-object.h\"\n+#include \"virtualfilesystem.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -1363,6 +1364,14 @@ static int clear_ce_flags_1(struct index_state *istate,\n \t\t\tcontinue;\n \t\t}\n \n+\t\t/* if it's not in the virtual file system, exit early */\n+\t\tif (core_virtualfilesystem) {\n+\t\t\tif (is_included_in_virtualfilesystem(ce->name, ce->ce_namelen) > 0)\n+\t\t\t\tce->ce_flags &= ~clear_mask;\n+\t\t\tcache++;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (prefix->len && strncmp(ce->name, prefix->buf, prefix->len))\n \t\t\tbreak;\n \n@@ -1481,12 +1490,16 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options\n \tif (!core_apply_sparse_checkout || !o->update)\n \t\to->skip_sparse_checkout = 1;\n \tif (!o->skip_sparse_checkout) {\n-\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n-\t\tif (add_excludes_from_file_to_list(sparse, \"\", 0, &el, NULL) < 0)\n-\t\t\to->skip_sparse_checkout = 1;\n-\t\telse\n+\t\tif (core_virtualfilesystem) {\n \t\t\to->el = ⪙\n-\t\tfree(sparse);\n+\t\t} else {\n+\t\t\tchar *sparse = git_pathdup(\"info/sparse-checkout\");\n+\t\t\tif (add_excludes_from_file_to_list(sparse, \"\", 0, &el, NULL) < 0)\n+\t\t\t\to->skip_sparse_checkout = 1;\n+\t\t\telse\n+\t\t\t\to->el = ⪙\n+\t\t\tfree(sparse);\n+\t\t}\n \t}\n \n \tmemset(&o->result, 0, sizeof(o->result));\ndiff --git a/virtualfilesystem.c b/virtualfilesystem.c\nnew file mode 100644\nindex 0000000000..0d45803dbc\n--- /dev/null\n+++ b/virtualfilesystem.c\n@@ -0,0 +1,308 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"hashmap.h\"\n+#include \"run-command.h\"\n+#include \"virtualfilesystem.h\"\n+\n+#define HOOK_INTERFACE_VERSION\t(1)\n+\n+static struct strbuf virtual_filesystem_data = STRBUF_INIT;\n+static struct hashmap virtual_filesystem_hashmap;\n+static struct hashmap parent_directory_hashmap;\n+\n+struct virtualfilesystem {\n+\tstruct hashmap_entry ent; /* must be the first member! */\n+\tconst char *pattern;\n+\tint patternlen;\n+};\n+\n+static unsigned int(*vfshash)(const void *buf, size_t len);\n+static int(*vfscmp)(const char *a, const char *b, size_t len);\n+\n+static int vfs_hashmap_cmp(const void *unused_cmp_data,\n+\tconst void *a, const void *b, const void *key)\n+{\n+\tconst struct virtualfilesystem *vfs1 = a;\n+\tconst struct virtualfilesystem *vfs2 = b;\n+\n+\treturn vfscmp(vfs1->pattern, vfs2->pattern, vfs1->patternlen);\n+}\n+\n+static void get_virtual_filesystem_data(struct strbuf *vfs_data)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint err;\n+\n+\tstrbuf_init(vfs_data, 0);\n+\n+\targv_array_push(&cp.args, core_virtualfilesystem);\n+\targv_array_pushf(&cp.args, \"%d\", HOOK_INTERFACE_VERSION);\n+\tcp.use_shell = 1;\n+\tcp.dir = get_git_work_tree();\n+\n+\terr = capture_command(&cp, vfs_data, 1024);\n+\tif (err)\n+\t\tdie(\"unable to load virtual file system\");\n+}\n+\n+static int check_includes_hashmap(struct hashmap *map, const char *pattern, int patternlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct virtualfilesystem vfs;\n+\tchar *slash;\n+\n+\t/* Check straight mapping */\n+\tstrbuf_reset(&sb);\n+\tstrbuf_add(&sb, pattern, patternlen);\n+\tvfs.pattern = sb.buf;\n+\tvfs.patternlen = sb.len;\n+\thashmap_entry_init(&vfs, vfshash(vfs.pattern, vfs.patternlen));\n+\tif (hashmap_get(map, &vfs, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 1;\n+\t}\n+\n+\t/*\n+\t * Check to see if it matches a directory or any path\n+\t * underneath it. In other words, 'a/b/foo.txt' will match\n+\t * '/', 'a/', and 'a/b/'.\n+\t */\n+\tslash = strchr(sb.buf, '/');\n+\twhile (slash) {\n+\t\tvfs.pattern = sb.buf;\n+\t\tvfs.patternlen = slash - sb.buf + 1;\n+\t\thashmap_entry_init(&vfs, vfshash(vfs.pattern, vfs.patternlen));\n+\t\tif (hashmap_get(map, &vfs, NULL)) {\n+\t\t\tstrbuf_release(&sb);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tslash = strchr(slash + 1, '/');\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 0;\n+}\n+\n+static void includes_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tstruct virtualfilesystem *vfs;\n+\n+\tvfs = xmalloc(sizeof(struct virtualfilesystem));\n+\tvfs->pattern = pattern;\n+\tvfs->patternlen = patternlen;\n+\thashmap_entry_init(vfs, vfshash(vfs->pattern, vfs->patternlen));\n+\thashmap_add(map, vfs);\n+}\n+\n+static void initialize_includes_hashmap(struct hashmap *map, struct strbuf *vfs_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the virtual file system data we can use to look\n+\t * for cache entry matches quickly\n+\t */\n+\tvfshash = ignore_case ? memihash : memhash;\n+\tvfscmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, vfs_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = vfs_data->buf;\n+\tlen = vfs_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tincludes_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Return 1 if the requested item is found in the virtual file system,\n+ * 0 for not found and -1 for undecided.\n+ */\n+int is_included_in_virtualfilesystem(const char *pathname, int pathlen)\n+{\n+\tif (!core_virtualfilesystem)\n+\t\treturn -1;\n+\n+\tif (!virtual_filesystem_hashmap.tablesize && virtual_filesystem_data.len)\n+\t\tinitialize_includes_hashmap(&virtual_filesystem_hashmap, &virtual_filesystem_data);\n+\tif (!virtual_filesystem_hashmap.tablesize)\n+\t\treturn -1;\n+\n+\treturn check_includes_hashmap(&virtual_filesystem_hashmap, pathname, pathlen);\n+}\n+\n+static void parent_directory_hashmap_add(struct hashmap *map, const char *pattern, const int patternlen)\n+{\n+\tchar *slash;\n+\tstruct virtualfilesystem *vfs;\n+\n+\t/*\n+\t * Add any directories leading up to the file as the excludes logic\n+\t * needs to match directories leading up to the files as well. Detect\n+\t * and prevent unnecessary duplicate entries which will be common.\n+\t */\n+\tif (patternlen > 1) {\n+\t\tslash = strchr(pattern + 1, '/');\n+\t\twhile (slash) {\n+\t\t\tvfs = xmalloc(sizeof(struct virtualfilesystem));\n+\t\t\tvfs->pattern = pattern;\n+\t\t\tvfs->patternlen = slash - pattern + 1;\n+\t\t\thashmap_entry_init(vfs, vfshash(vfs->pattern, vfs->patternlen));\n+\t\t\tif (hashmap_get(map, vfs, NULL))\n+\t\t\t\tfree(vfs);\n+\t\t\telse\n+\t\t\t\thashmap_add(map, vfs);\n+\t\t\tslash = strchr(slash + 1, '/');\n+\t\t}\n+\t}\n+}\n+\n+static void initialize_parent_directory_hashmap(struct hashmap *map, struct strbuf *vfs_data)\n+{\n+\tchar *buf, *entry;\n+\tsize_t len;\n+\tint i;\n+\n+\t/*\n+\t * Build a hashmap of the parent directories contained in the virtual\n+\t * file system data we can use to look for matches quickly\n+\t */\n+\tvfshash = ignore_case ? memihash : memhash;\n+\tvfscmp = ignore_case ? strncasecmp : strncmp;\n+\thashmap_init(map, vfs_hashmap_cmp, NULL, 0);\n+\n+\tentry = buf = vfs_data->buf;\n+\tlen = vfs_data->len;\n+\tfor (i = 0; i < len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tparent_directory_hashmap_add(map, entry, buf + i - entry);\n+\t\t\tentry = buf + i + 1;\n+\t\t}\n+\t}\n+}\n+\n+static int check_directory_hashmap(struct hashmap *map, const char *pathname, int pathlen)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct virtualfilesystem vfs;\n+\n+\t/* Check for directory */\n+\tstrbuf_reset(&sb);\n+\tstrbuf_add(&sb, pathname, pathlen);\n+\tstrbuf_addch(&sb, '/');\n+\tvfs.pattern = sb.buf;\n+\tvfs.patternlen = sb.len;\n+\thashmap_entry_init(&vfs, vfshash(vfs.pattern, vfs.patternlen));\n+\tif (hashmap_get(map, &vfs, NULL)) {\n+\t\tstrbuf_release(&sb);\n+\t\treturn 0;\n+\t}\n+\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int is_excluded_from_virtualfilesystem(const char *pathname, int pathlen, int dtype)\n+{\n+\tif (!core_virtualfilesystem)\n+\t\treturn -1;\n+\n+\tif (dtype != DT_REG && dtype != DT_DIR && dtype != DT_LNK)\n+\t\tdie(_(\"is_excluded_from_virtualfilesystem passed unhandled dtype\"));\n+\n+\tif (dtype == DT_REG || dtype == DT_LNK) {\n+\t\tint ret = is_included_in_virtualfilesystem(pathname, pathlen);\n+\t\tif (ret > 0)\n+\t\t\treturn 0;\n+\t\tif (ret == 0)\n+\t\t\treturn 1;\n+\t\treturn ret;\n+\t}\n+\n+\tif (dtype == DT_DIR) {\n+\t\tint ret = is_included_in_virtualfilesystem(pathname, pathlen);\n+\t\tif (ret > 0)\n+\t\t\treturn 0;\n+\n+\t\tif (!parent_directory_hashmap.tablesize && virtual_filesystem_data.len)\n+\t\t\tinitialize_parent_directory_hashmap(&parent_directory_hashmap, &virtual_filesystem_data);\n+\t\tif (!parent_directory_hashmap.tablesize)\n+\t\t\treturn -1;\n+\n+\t\treturn check_directory_hashmap(&parent_directory_hashmap, pathname, pathlen);\n+\t}\n+\n+\treturn -1;\n+}\n+\n+/*\n+ * Update the CE_SKIP_WORKTREE bits based on the virtual file system.\n+ */\n+void apply_virtualfilesystem(struct index_state *istate)\n+{\n+\tchar *buf, *entry;\n+\tint i;\n+\n+\tif (!git_config_get_virtualfilesystem())\n+\t\treturn;\n+\n+\tif (!virtual_filesystem_data.len)\n+\t\tget_virtual_filesystem_data(&virtual_filesystem_data);\n+\n+\t/* set CE_SKIP_WORKTREE bit on all entries */\n+\tfor (i = 0; i < istate->cache_nr; i++)\n+\t\tistate->cache[i]->ce_flags |= CE_SKIP_WORKTREE;\n+\n+\t/* clear CE_SKIP_WORKTREE bit for everything in the virtual file system */\n+\tentry = buf = virtual_filesystem_data.buf;\n+\tfor (i = 0; i < virtual_filesystem_data.len; i++) {\n+\t\tif (buf[i] == '\\0') {\n+\t\t\tint pos, len;\n+\n+\t\t\tlen = buf + i - entry;\n+\n+\t\t\t/* look for a directory wild card (ie \"dir1/\") */\n+\t\t\tif (buf[i - 1] == '/') {\n+\t\t\t\tif (ignore_case)\n+\t\t\t\t\tadjust_dirname_case(istate, entry);\n+\t\t\t\tpos = index_name_pos(istate, entry, len - 1);\n+\t\t\t\tif (pos < 0) {\n+\t\t\t\t\tpos = -pos - 1;\n+\t\t\t\t\twhile (pos < istate->cache_nr && !fspathncmp(istate->cache[pos]->name, entry, len)) {\n+\t\t\t\t\t\tistate->cache[pos]->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t\t\t\t\tpos++;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\tif (ignore_case) {\n+\t\t\t\t\tstruct cache_entry *ce = index_file_exists(istate, entry, len, ignore_case);\n+\t\t\t\t\tif (ce)\n+\t\t\t\t\t\tce->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t\t\t} else {\n+\t\t\t\t\tint pos = index_name_pos(istate, entry, len);\n+\t\t\t\t\tif (pos >= 0)\n+\t\t\t\t\t\tistate->cache[pos]->ce_flags &= ~CE_SKIP_WORKTREE;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tentry += len + 1;\n+\t\t}\n+\t}\n+}\n+\n+/*\n+ * Free the virtual file system data structures.\n+ */\n+void free_virtualfilesystem(void) {\n+\thashmap_free(&virtual_filesystem_hashmap, 1);\n+\thashmap_free(&parent_directory_hashmap, 1);\n+\tstrbuf_release(&virtual_filesystem_data);\n+}\ndiff --git a/virtualfilesystem.h b/virtualfilesystem.h\nnew file mode 100644\nindex 0000000000..5e8c5b096d\n--- /dev/null\n+++ b/virtualfilesystem.h\n@@ -0,0 +1,25 @@\n+#ifndef VIRTUALFILESYSTEM_H\n+#define VIRTUALFILESYSTEM_H\n+\n+/*\n+ * Update the CE_SKIP_WORKTREE bits based on the virtual file system.\n+ */\n+void apply_virtualfilesystem(struct index_state *istate);\n+\n+/*\n+ * Return 1 if the requested item is found in the virtual file system,\n+ * 0 for not found and -1 for undecided.\n+ */\n+int is_included_in_virtualfilesystem(const char *pathname, int pathlen);\n+\n+/*\n+ * Return 1 for exclude, 0 for include and -1 for undecided.\n+ */\n+int is_excluded_from_virtualfilesystem(const char *pathname, int pathlen, int dtype);\n+\n+/*\n+ * Free the virtual file system data structures.\n+ */\n+void free_virtualfilesystem(void);\n+\n+#endif\n\nbase-commit: 4ede3d42dfb57f9a41ac96a1f216c62eb7566cc2\n-- \n2.9.2.gvfs.2.27918.g0990287eef\n\n"},{"id":"362042","messageId":"xmqqsh0nyqx9.fsf@gitster-ct.c.googlers.com","threadId":"49719","inReplyTo":"20181030191608.18716-1-peartben@gmail.com","subject":"Re: [RFC v1] Add virtual file system settings and hook proc","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-30T23:07:30Z","receivedAt":"2018-10-30T23:07:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> diff --git a/config.c b/config.c\n> index 4051e38823..96e05ee0f1 100644\n> --- a/config.c\n> +++ b/config.c\n> ...\n> @@ -2307,6 +2311,37 @@ int git_config_get_index_threads(void)\n> \treturn 0; /* auto */\n> }\n> \n> +int git_config_get_virtualfilesystem(void)\n> +{\n> +\tif (git_config_get_pathname(\"core.virtualfilesystem\", &core_virtualfilesystem))\n> +\t\tcore_virtualfilesystem = getenv(\"GIT_TEST_VIRTUALFILESYSTEM\");\n> +\n> +\tif (core_virtualfilesystem && !*core_virtualfilesystem)\n> +\t\tcore_virtualfilesystem = NULL;\n> +\n> +\tif (core_virtualfilesystem) {\n> +\t\t/*\n> +\t\t * Some git commands spawn helpers and redirect the index to a different\n> +\t\t * location. These include \"difftool -d\" and the sequencer\n> +\t\t * (i.e. `git rebase -i`, `git cherry-pick` and `git revert`) and others.\n> +\t\t * In those instances we don't want to update their temporary index with\n> +\t\t * our virtualization data.\n> +\t\t */\n> +\t\tchar *default_index_file = xstrfmt(\"%s/%s\", the_repository->gitdir, \"index\");\n> +\t\tint should_run_hook = !strcmp(default_index_file, the_repository->index_file);\n> +\n> +\t\tfree(default_index_file);\n> +\t\tif (should_run_hook) {\n> +\t\t\t/* virtual file system relies on the sparse checkout logic so force it on */\n> +\t\t\tcore_apply_sparse_checkout = 1;\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t\tcore_virtualfilesystem = NULL;\n\nIt would be a small leak if this came from config_get_pathname(),\nbut if it came from $GIT_TEST_VFS env, we cannot free it X-<.\n\nA helper function called *_get_X() that does not return X as its\nreturn value or updating the location pointed by its *dst parameter,\nand instead only stores its finding in a global variable feels\nsomewhat odd. It smells more like \"find out\", \"probe\", \"check\",\netc.\n\n> diff --git a/dir.c b/dir.c\n> index 47c2fca8dc..3097b0e446 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -21,6 +21,7 @@\n> #include \"ewah/ewok.h\"\n> #include \"fsmonitor.h\"\n> #include \"submodule-config.h\"\n> +#include \"virtualfilesystem.h\"\n> \n> /*\n> * Tells read_directory_recursive how a file or directory should be treated.\n> @@ -1109,6 +1110,18 @@ int is_excluded_from_list(const char *pathname,\n> \t\t\t struct exclude_list *el, struct index_state *istate)\n> {\n> \tstruct exclude *exclude;\n> +\n> +\t/*\n> +\t * The virtual file system data is used to prevent git from traversing\n> +\t * any part of the tree that is not in the virtual file system. Return\n> +\t * 1 to exclude the entry if it is not found in the virtual file system,\n> +\t * else fall through to the regular excludes logic as it may further exclude.\n> +\t */\n\nThis comment will sit better immediately in front of the call to \"is\nexcluded from vfs?\" helper function.\n\n> +\tif (*dtype == DT_UNKNOWN)\n> +\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n\nWe try to defer paying cost to determine unknown *dtype as late as\npossible by having this call in last_exclude_matching_from_list(),\nand not here. If we are doing this, we probably should update the\ncallpaths that call last_exclude_matching_from_list() to make the\ncaller responsible for doing get_dtype() and drop the lazy finding\nof dtype from the callee. Alternatively, the new \"is excluded from\nvfs\" helper can learn to do the lazy get_dtype() just like the\nexisting last_exclude_matching_from_list() does. I suspect the\nlatter may be simpler.\n\n> +\tif (is_excluded_from_virtualfilesystem(pathname, pathlen, *dtype) > 0)\n> +\t\treturn 1;\n> +\n> \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n> \t\t\t\t\t\t dtype, el, istate);\n> \tif (exclude)\n> @@ -1324,8 +1337,20 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n> int is_excluded(struct dir_struct *dir, struct index_state *istate,\n> \t\tconst char *pathname, int *dtype_p)\n> {\n> -\tstruct exclude *exclude =\n> -\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n> +\tstruct exclude *exclude;\n> +\n> +\t/*\n> +\t * The virtual file system data is used to prevent git from traversing\n> +\t * any part of the tree that is not in the virtual file system. Return\n> +\t * 1 to exclude the entry if it is not found in the virtual file system,\n> +\t * else fall through to the regular excludes logic as it may further exclude.\n> +\t */\n> +\tif (*dtype_p == DT_UNKNOWN)\n> +\t\t*dtype_p = get_dtype(NULL, istate, pathname, strlen(pathname));\n\nExactly the same comment as above.\n\n> +\tif (is_excluded_from_virtualfilesystem(pathname, strlen(pathname), *dtype_p) > 0)\n> +\t\treturn 1;\n> +\n> +\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n> \tif (exclude)\n> \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n> \treturn 0;\n> @@ -1678,6 +1703,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n> \tif (dtype != DT_DIR && has_path_in_index)\n> \t\treturn path_none;\n> \n> +\tif (is_excluded_from_virtualfilesystem(path->buf, path->len, dtype) > 0)\n> +\t\treturn path_excluded;\n> +\n> \t/*\n> \t * When we are looking at a directory P in the working tree,\n> \t * there are three cases:\n> @@ -2018,6 +2046,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n> \t\t/* add the path to the appropriate result list */\n> \t\tswitch (state) {\n> \t\tcase path_excluded:\n> +\t\t\tif (is_excluded_from_virtualfilesystem(path.buf, path.len, DT_DIR) > 0)\n> +\t\t\t\tbreak;\n> \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n> \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n> \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n\nI am kind-of surprised that the \"damage\" to dir.c need to support\nthis is so isolated and small ;-)\n\n> ...\n> +if test \"$1\" != 1\n> +then\n> +\techo \"Unsupported core.virtualfilesystem hook version.\" >&2\n> +\texit 1\n> +fi\n> +\n> +#find -type f -printf '%P\\0'\n> +find -type d -printf '%P/\\0'\n\nI am not reading (hence not commenting on) tests in this review\nmessage yet.\n\n> diff --git a/unpack-trees.c b/unpack-trees.c\n> index 7570df481b..ee3cda2e94 100644\n> --- a/unpack-trees.c\n> +++ b/unpack-trees.c\n> @@ -18,6 +18,7 @@\n> #include \"fsmonitor.h\"\n> #include \"object-store.h\"\n> #include \"fetch-object.h\"\n> +#include \"virtualfilesystem.h\"\n> \n> /*\n> * Error messages expected by scripts out of plumbing commands such as\n> @@ -1363,6 +1364,14 @@ static int clear_ce_flags_1(struct index_state *istate,\n> \t\t\tcontinue;\n> \t\t}\n> \n> +\t\t/* if it's not in the virtual file system, exit early */\n> +\t\tif (core_virtualfilesystem) {\n> +\t\t\tif (is_included_in_virtualfilesystem(ce->name, ce->ce_namelen) > 0)\n> +\t\t\t\tce->ce_flags &= ~clear_mask;\n> +\t\t\tcache++;\n> +\t\t\tcontinue;\n> +\t\t}\n\nEarlier we saw \"is it excluded?\" and now we have \"is it included?\"\nThey have different function signature (i.e. \"included?\" does not\nneed to know the type of the entry), and I am guessing that for the\npurpose of this particular patch that may be sufficient, but I have\nto wonder if in the longer term we'd be better off to keep the\ninterface to these two functions similar. Also, I wonder if we need\nboth---I see below that these \"is included/excluded?\" helpers are\nallowed to say \"I dunno\", so that may be the reason why we cannot\nsimply say \"included is the same as !excluded\".\n\n"},{"id":"362120","messageId":"CACsJy8DbiVZYmY11Nt4c_+egSi5tz0iVq7rNv2BiVdyJ4htgvw@mail.gmail.com","threadId":"49719","inReplyTo":"20181030191608.18716-1-peartben@gmail.com","subject":"Re: [RFC v1] Add virtual file system settings and hook proc","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-10-31T19:11:47Z","receivedAt":"2018-10-31T19:12:16Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"not really a review, just a couple quick notes..\n\nOn Tue, Oct 30, 2018 at 9:40 PM Ben Peart wrote:\n>\n> From: Ben Peart \n>\n> On index load, clear/set the skip worktree bits based on the virtual\n> file system data. Use virtual file system data to update skip-worktree\n> bit in unpack-trees. Use virtual file system data to exclude files and\n> folders not explicitly requested.\n>\n> Signed-off-by: Ben Peart \n> ---\n>\n> We have taken several steps to make git perform well on very large repos.\n> Some of those steps include: improving underlying algorithms, utilizing\n> multi-threading where possible, and simplifying the behavior of some commands.\n> These changes typically benefit all git repos to varying degrees. While\n> these optimizations all help, they are insufficient to provide adequate\n> performance on the very large repos we often work with.\n>\n> To make git perform well on the very largest repos, we had to make more\n> significant changes. The biggest performance win by far is the work we have\n> done to make git operations O(modified) instead of O(size of repo). This\n> takes advantage of the fact that the number of files a developer has modified\n> is a tiny fraction of the overall repo size.\n>\n> We accomplished this by utilizing the existing internal logic for the skip\n> worktree bit and excludes to tell git to ignore all files and folders other\n> than those that have been modified. This logic is driven by an external\n> process that monitors writes to the repo and communicates the list of files\n> and folders with changes to git via the virtual file system hook in this patch.\n>\n> The external process maintains a list of files and folders that have been\n> modified. When git runs, it requests the list of files and folders that\n> have been modified via the virtual file system hook. Git then sets/clears\n> the skip-worktree bit on the cache entries and builds a hashmap of the\n> modified files/folders that is used by the excludes logic to avoid scanning\n> the entire repo looking for changes and untracked files.\n>\n> With this system, we have been able to make local git command performance on\n> extremely large repos (millions of files, 1/2 million folders) entirely\n> manageable (30 second checkout, 3.5 seconds status, 4 second add, 7 second\n> commit, etc).\n>\n> Our desire is to eliminate all custom patches in our fork of git. To that\n> end, I'm submitting this as an RFC to see how much interest there is and how\n> much willingness to take this type of change into git.\n\nMost of these paragraphs (perhaps except the last one) should be part\nof the commit message. You describe briefly what the patch does but\nit's even more important to say why you want to do it.\n\n> +core.virtualFilesystem::\n> + If set, the value of this variable is used as a command which\n> + will identify all files and directories that are present in\n> + the working directory. Git will only track and update files\n> + listed in the virtual file system. Using the virtual file system\n> + will supersede the sparse-checkout settings which will be ignored.\n> + See the \"virtual file system\" section of linkgit:githooks[6].\n\nIt sounds like \"virtual file system\" is just one of the use cases for\nthis feature, which is more about a dynamic source of sparse-checkout\nbits. Perhaps name the config key with something along sparse checkout\ninstead of naming it after a use case.\n\nThis is a hook. I notice we start to avoid adding real hooks and just\nadd config keys instead. Eventually we should have config-based hooks,\nbut if we're going to add more like this, I think these should be in a\nseparate section, hook.virtualFileSystem or something.\n\nI don't think the superseding makes sense. There's no reason this\ncould not be used in combination with $GIT_DIR/info/sparse-checkout.\nIf you don't want both, disable the other.\n\nOne last note. Since this is related to filesystem. Shouldn't it be\npart of fsmonitor (the protocol, not the implementation)? Then\nwatchman user could use it to.\n-- \nDuy\n"},{"id":"362127","messageId":"dbc2eb4f-842e-f49a-256f-3a140d801bb0@gmail.com","threadId":"49719","inReplyTo":"xmqqsh0nyqx9.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC v1] Add virtual file system settings and hook proc","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-10-31T20:12:53Z","receivedAt":"2018-10-31T20:12:58Z","isPatch":false,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/30/2018 7:07 PM, Junio C Hamano wrote:\n> Ben Peart writes:\n> \n>> diff --git a/config.c b/config.c\n>> index 4051e38823..96e05ee0f1 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> ...\n>> @@ -2307,6 +2311,37 @@ int git_config_get_index_threads(void)\n>> \treturn 0; /* auto */\n>> }\n>> \n>> +int git_config_get_virtualfilesystem(void)\n>> +{\n>> +\tif (git_config_get_pathname(\"core.virtualfilesystem\", &core_virtualfilesystem))\n>> +\t\tcore_virtualfilesystem = getenv(\"GIT_TEST_VIRTUALFILESYSTEM\");\n>> +\n>> +\tif (core_virtualfilesystem && !*core_virtualfilesystem)\n>> +\t\tcore_virtualfilesystem = NULL;\n>> +\n>> +\tif (core_virtualfilesystem) {\n>> +\t\t/*\n>> +\t\t * Some git commands spawn helpers and redirect the index to a different\n>> +\t\t * location. These include \"difftool -d\" and the sequencer\n>> +\t\t * (i.e. `git rebase -i`, `git cherry-pick` and `git revert`) and others.\n>> +\t\t * In those instances we don't want to update their temporary index with\n>> +\t\t * our virtualization data.\n>> +\t\t */\n>> +\t\tchar *default_index_file = xstrfmt(\"%s/%s\", the_repository->gitdir, \"index\");\n>> +\t\tint should_run_hook = !strcmp(default_index_file, the_repository->index_file);\n>> +\n>> +\t\tfree(default_index_file);\n>> +\t\tif (should_run_hook) {\n>> +\t\t\t/* virtual file system relies on the sparse checkout logic so force it on */\n>> +\t\t\tcore_apply_sparse_checkout = 1;\n>> +\t\t\treturn 1;\n>> +\t\t}\n>> +\t\tcore_virtualfilesystem = NULL;\n> \n> It would be a small leak if this came from config_get_pathname(),\n> but if it came from $GIT_TEST_VFS env, we cannot free it X-<.\n> \n> A helper function called *_get_X() that does not return X as its\n> return value or updating the location pointed by its *dst parameter,\n> and instead only stores its finding in a global variable feels\n> somewhat odd. It smells more like \"find out\", \"probe\", \"check\",\n> etc.\n> \n\nI agree. Frankly, I think it should just return whether it should be \nused or not (bool) and the hook name should be fixed. I got push back \nwhen I did that for fsmonitor so I made this the same in an effort to \nhead off that same feedback.\n\n>> diff --git a/dir.c b/dir.c\n>> index 47c2fca8dc..3097b0e446 100644\n>> --- a/dir.c\n>> +++ b/dir.c\n>> @@ -21,6 +21,7 @@\n>> #include \"ewah/ewok.h\"\n>> #include \"fsmonitor.h\"\n>> #include \"submodule-config.h\"\n>> +#include \"virtualfilesystem.h\"\n>> \n>> /*\n>> * Tells read_directory_recursive how a file or directory should be treated.\n>> @@ -1109,6 +1110,18 @@ int is_excluded_from_list(const char *pathname,\n>> \t\t\t struct exclude_list *el, struct index_state *istate)\n>> {\n>> \tstruct exclude *exclude;\n>> +\n>> +\t/*\n>> +\t * The virtual file system data is used to prevent git from traversing\n>> +\t * any part of the tree that is not in the virtual file system. Return\n>> +\t * 1 to exclude the entry if it is not found in the virtual file system,\n>> +\t * else fall through to the regular excludes logic as it may further exclude.\n>> +\t */\n> \n> This comment will sit better immediately in front of the call to \"is\n> excluded from vfs?\" helper function.\n> \n>> +\tif (*dtype == DT_UNKNOWN)\n>> +\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n> \n> We try to defer paying cost to determine unknown *dtype as late as\n> possible by having this call in last_exclude_matching_from_list(),\n> and not here. If we are doing this, we probably should update the\n> callpaths that call last_exclude_matching_from_list() to make the\n> caller responsible for doing get_dtype() and drop the lazy finding\n> of dtype from the callee. Alternatively, the new \"is excluded from\n> vfs\" helper can learn to do the lazy get_dtype() just like the\n> existing last_exclude_matching_from_list() does. I suspect the\n> latter may be simpler.\n\nIn is_excluded_from_virtualfilesystem() dtype can't be lazy because it \nis always needed (which is why I test and die if it isn't known). I \nconsidered doing the test/call to get_dtype() within \nis_excluded_from_virtualfilesystem() but didn't like making it dependent \non istate just so I could move the get_dtype() call in one level. It is \nfunctionally identical so I can easily move it in if that is preferred.\n\n> \n>> +\tif (is_excluded_from_virtualfilesystem(pathname, pathlen, *dtype) > 0)\n>> +\t\treturn 1;\n>> +\n>> \texclude = last_exclude_matching_from_list(pathname, pathlen, basename,\n>> \t\t\t\t\t\t dtype, el, istate);\n>> \tif (exclude)\n>> @@ -1324,8 +1337,20 @@ struct exclude *last_exclude_matching(struct dir_struct *dir,\n>> int is_excluded(struct dir_struct *dir, struct index_state *istate,\n>> \t\tconst char *pathname, int *dtype_p)\n>> {\n>> -\tstruct exclude *exclude =\n>> -\t\tlast_exclude_matching(dir, istate, pathname, dtype_p);\n>> +\tstruct exclude *exclude;\n>> +\n>> +\t/*\n>> +\t * The virtual file system data is used to prevent git from traversing\n>> +\t * any part of the tree that is not in the virtual file system. Return\n>> +\t * 1 to exclude the entry if it is not found in the virtual file system,\n>> +\t * else fall through to the regular excludes logic as it may further exclude.\n>> +\t */\n>> +\tif (*dtype_p == DT_UNKNOWN)\n>> +\t\t*dtype_p = get_dtype(NULL, istate, pathname, strlen(pathname));\n> \n> Exactly the same comment as above.\n> \n>> +\tif (is_excluded_from_virtualfilesystem(pathname, strlen(pathname), *dtype_p) > 0)\n>> +\t\treturn 1;\n>> +\n>> +\texclude = last_exclude_matching(dir, istate, pathname, dtype_p);\n>> \tif (exclude)\n>> \t\treturn exclude->flags & EXC_FLAG_NEGATIVE ? 0 : 1;\n>> \treturn 0;\n>> @@ -1678,6 +1703,9 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n>> \tif (dtype != DT_DIR && has_path_in_index)\n>> \t\treturn path_none;\n>> \n>> +\tif (is_excluded_from_virtualfilesystem(path->buf, path->len, dtype) > 0)\n>> +\t\treturn path_excluded;\n>> +\n>> \t/*\n>> \t * When we are looking at a directory P in the working tree,\n>> \t * there are three cases:\n>> @@ -2018,6 +2046,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n>> \t\t/* add the path to the appropriate result list */\n>> \t\tswitch (state) {\n>> \t\tcase path_excluded:\n>> +\t\t\tif (is_excluded_from_virtualfilesystem(path.buf, path.len, DT_DIR) > 0)\n>> +\t\t\t\tbreak;\n>> \t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n>> \t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n>> \t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n> \n> I am kind-of surprised that the \"damage\" to dir.c need to support\n> this is so isolated and small ;-)\n> \n\nWhen I'm expecting to carry a patch in a fork I try to structure it to \nhave as small of a footprint as possible (hence moving most of the code \ninto a separate file). Keeps the merge conflicts to a minimum. ;-)\n\n>> ...\n>> +if test \"$1\" != 1\n>> +then\n>> +\techo \"Unsupported core.virtualfilesystem hook version.\" >&2\n>> +\texit 1\n>> +fi\n>> +\n>> +#find -type f -printf '%P\\0'\n>> +find -type d -printf '%P/\\0'\n> \n> I am not reading (hence not commenting on) tests in this review\n> message yet.\n> \n>> diff --git a/unpack-trees.c b/unpack-trees.c\n>> index 7570df481b..ee3cda2e94 100644\n>> --- a/unpack-trees.c\n>> +++ b/unpack-trees.c\n>> @@ -18,6 +18,7 @@\n>> #include \"fsmonitor.h\"\n>> #include \"object-store.h\"\n>> #include \"fetch-object.h\"\n>> +#include \"virtualfilesystem.h\"\n>> \n>> /*\n>> * Error messages expected by scripts out of plumbing commands such as\n>> @@ -1363,6 +1364,14 @@ static int clear_ce_flags_1(struct index_state *istate,\n>> \t\t\tcontinue;\n>> \t\t}\n>> \n>> +\t\t/* if it's not in the virtual file system, exit early */\n>> +\t\tif (core_virtualfilesystem) {\n>> +\t\t\tif (is_included_in_virtualfilesystem(ce->name, ce->ce_namelen) > 0)\n>> +\t\t\t\tce->ce_flags &= ~clear_mask;\n>> +\t\t\tcache++;\n>> +\t\t\tcontinue;\n>> +\t\t}\n> \n> Earlier we saw \"is it excluded?\" and now we have \"is it included?\"\n> They have different function signature (i.e. \"included?\" does not\n> need to know the type of the entry), and I am guessing that for the\n> purpose of this particular patch that may be sufficient, but I have\n> to wonder if in the longer term we'd be better off to keep the\n> interface to these two functions similar. Also, I wonder if we need\n> both---I see below that these \"is included/excluded?\" helpers are\n> allowed to say \"I dunno\", so that may be the reason why we cannot\n> simply say \"included is the same as !excluded\".\n> \n\nMost of the logic _is_ just inverted (which is why is_excluded...() can \ncall is_included...() to do most of the work) but there are differences \nas well. The differences are mostly about how parent directories are \nhandled. I tried to write it to share as much of the logic as possible \nbut there may be something I've missed.\n\n\nYour comments are all feedback on the code - how it was implemented, \nstyle, etc. Any thoughts on whether this is something we could/should \nmerge into master (after any necessary cleanup)? Would anyone else find \nthis interesting/helpful?\n"},{"id":"362130","messageId":"1f7efd07-4881-daa7-cd1d-145bbf3ffcc8@gmail.com","threadId":"49719","inReplyTo":"CACsJy8DbiVZYmY11Nt4c_+egSi5tz0iVq7rNv2BiVdyJ4htgvw@mail.gmail.com","subject":"Re: [RFC v1] Add virtual file system settings and hook proc","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2018-10-31T20:53:41Z","receivedAt":"2018-10-31T20:53:47Z","isPatch":false,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 10/31/2018 3:11 PM, Duy Nguyen wrote:\n> not really a review, just a couple quick notes..\n> \n\nPerfect! As an RFC, I'm more looking for high level thoughts/notes than \na style/syntax code review.\n\n> On Tue, Oct 30, 2018 at 9:40 PM Ben Peart wrote:\n>>\n>> From: Ben Peart \n>>\n>> On index load, clear/set the skip worktree bits based on the virtual\n>> file system data. Use virtual file system data to update skip-worktree\n>> bit in unpack-trees. Use virtual file system data to exclude files and\n>> folders not explicitly requested.\n>>\n>> Signed-off-by: Ben Peart \n>> ---\n>>\n>> We have taken several steps to make git perform well on very large repos.\n>> Some of those steps include: improving underlying algorithms, utilizing\n>> multi-threading where possible, and simplifying the behavior of some commands.\n>> These changes typically benefit all git repos to varying degrees. While\n>> these optimizations all help, they are insufficient to provide adequate\n>> performance on the very large repos we often work with.\n>>\n>> To make git perform well on the very largest repos, we had to make more\n>> significant changes. The biggest performance win by far is the work we have\n>> done to make git operations O(modified) instead of O(size of repo). This\n>> takes advantage of the fact that the number of files a developer has modified\n>> is a tiny fraction of the overall repo size.\n>>\n>> We accomplished this by utilizing the existing internal logic for the skip\n>> worktree bit and excludes to tell git to ignore all files and folders other\n>> than those that have been modified. This logic is driven by an external\n>> process that monitors writes to the repo and communicates the list of files\n>> and folders with changes to git via the virtual file system hook in this patch.\n>>\n>> The external process maintains a list of files and folders that have been\n>> modified. When git runs, it requests the list of files and folders that\n>> have been modified via the virtual file system hook. Git then sets/clears\n>> the skip-worktree bit on the cache entries and builds a hashmap of the\n>> modified files/folders that is used by the excludes logic to avoid scanning\n>> the entire repo looking for changes and untracked files.\n>>\n>> With this system, we have been able to make local git command performance on\n>> extremely large repos (millions of files, 1/2 million folders) entirely\n>> manageable (30 second checkout, 3.5 seconds status, 4 second add, 7 second\n>> commit, etc).\n>>\n>> Our desire is to eliminate all custom patches in our fork of git. To that\n>> end, I'm submitting this as an RFC to see how much interest there is and how\n>> much willingness to take this type of change into git.\n> \n> Most of these paragraphs (perhaps except the last one) should be part\n> of the commit message. You describe briefly what the patch does but\n> it's even more important to say why you want to do it.\n> \n>> +core.virtualFilesystem::\n>> + If set, the value of this variable is used as a command which\n>> + will identify all files and directories that are present in\n>> + the working directory. Git will only track and update files\n>> + listed in the virtual file system. Using the virtual file system\n>> + will supersede the sparse-checkout settings which will be ignored.\n>> + See the \"virtual file system\" section of linkgit:githooks[6].\n> \n> It sounds like \"virtual file system\" is just one of the use cases for\n> this feature, which is more about a dynamic source of sparse-checkout\n> bits. Perhaps name the config key with something along sparse checkout\n> instead of naming it after a use case.\n\nIt's more than a dynamic sparse-checkout because the same list is also \nused to exclude any file/folder not listed. That means any file not \nlisted won't ever be updated by git (like in 'checkout' for example) so \n'stale' files could be left in the working directory. It also means git \nwon't find new/untracked files unless they are specifically added to the \nlist.\n\n> \n> This is a hook. I notice we start to avoid adding real hooks and just\n> add config keys instead. Eventually we should have config-based hooks,\n> but if we're going to add more like this, I think these should be in a\n> separate section, hook.virtualFileSystem or something.\n> \n\nThat is a great idea. I don't personally like specifying the hook as \nthe 'flag' for whether a feature should be used. I'd rather have it be \na bool (enable the feature? true/false) and 1) either have the hook name \nhard coded (like most existing hooks) or 2) as you suggest add a \nconsistent way to have config-based hooks. Config based hooks could \nalso help provide a consistent way to configure them using GIT_TEST_* \nenvironment variables for testing.\n\n> I don't think the superseding makes sense. There's no reason this\n> could not be used in combination with $GIT_DIR/info/sparse-checkout.\n> If you don't want both, disable the other.\n> \n> One last note. Since this is related to filesystem. Shouldn't it be\n> part of fsmonitor (the protocol, not the implementation)? Then\n> watchman user could use it to.\n> \n\nTo get this to work properly takes a lot more logic than exists in \nfsmonitor/watchman. The challenge is that 1) fsmonitor/watchman is \nfocused on \"what has changed since