{"thread":{"id":"55334","subject":"[PATCH 00/27] Sparse Index: API protections","startedAt":"2021-03-16T21:17:56Z","lastAt":"2021-04-15T02:42:36Z","messageCount":111,"participants":["Derrick Stolee via GitGitGadget","Elijah Newren","Derrick Stolee","Matheus Tavares Bernardino","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":27},"messages":[{"id":"419408","messageId":"pull.906.git.1615929435.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":null,"subject":"[PATCH 00/27] Sparse Index: API protections","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:48Z","receivedAt":"2021-03-16T21:17:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is the second patch series submission coming out of the sparse-index\nRFC [1].\n\n[1]\nhttps://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n\nThis is based on v3 of the format series [2].\n\n[2]\nhttps://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@gmail.com/\n\nThe point of this series is to insert protections for the consumers of the\nin-memory index. We mark certain regions of code as needing a full index, so\nwe call ensure_full_index() to expand a sparse index to a full one, if\nnecessary. These protections are inserted file-by-file in every loop over\nall cache entries. Well, \"most\" loops, because some are going to be handled\nin the very next series so I leave them out.\n\nMany callers use index_name_pos() to find a path by name. In these cases, we\ncan check if that position resolves to a sparse directory instance. In those\ncases, we just expand to a full index and run the search again.\n\nThe last few patches deal with the name-hash hashtable for doing O(1)\nlookups.\n\nThese protections don't do much right now, since the previous series created\nthe_repository->settings.command_requires_full_index to guard all index\nreads and writes to ensure the in-memory copy is full for commands that have\nnot been tested with the sparse index yet.\n\nHowever, after this series is complete, we now have a straight-forward plan\nfor making commands \"sparse aware\" one-by-one:\n\n 1. Disable settings.command_requires_full_index to allow an in-memory\n sparse-index.\n 2. Run versions of that command under a debugger, breaking on\n ensure_full_index().\n 3. Examine the call stack to determine the context of that expansion, then\n implement the proper behavior in those locations.\n 4. Add tests to ensure we are checking this logic in the presence of sparse\n directory entries.\n\nI will admit that mostly it is the writing of the test cases that takes the\nmost time in the conversions I've done so far.\n\nThanks, -Stolee\n\nDerrick Stolee (27):\n *: remove 'const' qualifier for struct index_state\n read-cache: expand on query into sparse-directory entry\n sparse-index: API protection strategy\n cache: move ensure_full_index() to cache.h\n add: ensure full index\n checkout-index: ensure full index\n checkout: ensure full index\n commit: ensure full index\n difftool: ensure full index\n fsck: ensure full index\n grep: ensure full index\n ls-files: ensure full index\n merge-index: ensure full index\n rm: ensure full index\n sparse-checkout: ensure full index\n update-index: ensure full index\n diff-lib: ensure full index\n dir: ensure full index\n entry: ensure full index\n merge-ort: ensure full index\n merge-recursive: ensure full index\n pathspec: ensure full index\n read-cache: ensure full index\n resolve-undo: ensure full index\n revision: ensure full index\n sparse-index: expand_to_path()\n name-hash: use expand_to_path()\n\n Documentation/technical/sparse-index.txt | 32 ++++++++++-\n attr.c | 14 ++---\n attr.h | 4 +-\n builtin/add.c | 1 +\n builtin/checkout-index.c | 1 +\n builtin/checkout.c | 2 +\n builtin/commit.c | 2 +\n builtin/difftool.c | 2 +\n builtin/fsck.c | 1 +\n builtin/grep.c | 1 +\n builtin/ls-files.c | 12 ++--\n builtin/merge-index.c | 3 +\n builtin/rm.c | 1 +\n builtin/stash.c | 1 +\n builtin/update-index.c | 1 +\n cache.h | 7 ++-\n convert.c | 26 ++++-----\n convert.h | 20 +++----\n diff-lib.c | 1 +\n dir.c | 13 +++--\n dir.h | 8 +--\n entry.c | 1 +\n merge-ort.c | 1 +\n merge-recursive.c | 3 +-\n name-hash.c | 10 ++++\n pathspec.c | 7 ++-\n pathspec.h | 6 +-\n read-cache.c | 33 +++++++++--\n resolve-undo.c | 2 +\n revision.c | 1 +\n sparse-index.c | 72 ++++++++++++++++++++++++\n sparse-index.h | 14 ++++-\n submodule.c | 6 +-\n submodule.h | 6 +-\n 34 files changed, 243 insertions(+), 72 deletions(-)\n\n\nbase-commit: 3db06ac46dd5c61e83d7fc4747615d616fdbbdda\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/906\n-- \ngitgitgadget\n"},{"id":"419409","messageId":"8e11e891701985729cd5ecd7737e59d5a18284ce.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 02/27] read-cache: expand on query into sparse-directory entry","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:50Z","receivedAt":"2021-03-16T21:17:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCallers to index_name_pos() or index_name_stage_pos() have a specific\npath in mind. If that happens to be a path with an ancestor being a\nsparse-directory entry, it can lead to unexpected results.\n\nIn the case that we did not find the requested path, check to see if the\nposition _before_ the inserted position is a sparse directory entry that\nmatches the initial segment of the input path (including the directory\nseparator at the end of the directory name). If so, then expand the\nindex to be a full index and search again. This expansion will only\nhappen once per index read.\n\nFuture enhancements could be more careful to expand only the necessary\nsparse directory entry, but then we would have a special \"not fully\nsparse, but also not fully expanded\" mode that could affect writing the\nindex to file. Since this only occurs if a specific file is requested\noutside of the sparse checkout definition, this is unlikely to be a\ncommon situation.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex dfec221d9128..b143277ed106 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -566,6 +566,27 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t}\n \t\tfirst = next+1;\n \t}\n+\n+\tif (istate->sparse_index &&\n+\t first > 0) {\n+\t\t/* Note: first <= istate->cache_nr */\n+\t\tstruct cache_entry *ce = istate->cache[first - 1];\n+\n+\t\t/*\n+\t\t * If we are in a sparse-index _and_ the entry before the\n+\t\t * insertion position is a sparse-directory entry that is\n+\t\t * an ancestor of 'name', then we need to expand the index\n+\t\t * and search again. This will only trigger once, because\n+\t\t * thereafter the index is fully expanded.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t ce_namelen(ce) < namelen &&\n+\t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n+\t\t\tensure_full_index(istate);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t}\n+\t}\n+\n \treturn -first-1;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"419410","messageId":"bbf19f8a2be599a3451469731eed2eada7d3456a.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 03/27] sparse-index: API protection strategy","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:51Z","receivedAt":"2021-03-16T21:17:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nEdit and expand the sparse-index design document with the plan for\nguarding index operations with ensure_full_index().\n\nNotably, the plan has changed to not have an expand_to_path() method in\nfavor of checking for a sparse-directory hit inside of the\nindex_path_pos() API.\n\nThe changes that follow this one will incrementally add\nensure_full_index() guards to iterations over all cache entries. Some\niterations over the cache entries are not protected due to a few\ncategories listed in the document. Since these are not being modified,\nhere is a short list of the files and methods that will not receive\nthese guards:\n\nLooking for non-zero stage:\n* builtin/add.c:chmod_pathspec()\n* builtin/merge.c:count_unmerged_entries()\n* read-cache.c:unmerged_index()\n* rerere.c:check_one_conflict(), find_conflict(), rerere_remaining()\n* revision.c:prepare_show_merge()\n* sequencer.c:append_conflicts_hint()\n* wt-status.c:wt_status_collect_changes_initial()\n\nLooking for submodules:\n* builtin/submodule--helper.c:module_list_compute()\n* submodule.c: several methods\n* worktree.c:validate_no_submodules()\n\nPart of the index API:\n* name-hash.c: lazy init methods\n* preload-index.c:preload_thread(), preload_index()\n* read-cache.c: file format methods\n\nChecking for correct order of cache entries:\n* read-cache.c:check_ce_order()\n\nIgnores SKIP_WORKTREE entries or already aware:\n* unpack-trees.c:mark_new_skip_worktree()\n* wt-status.c:wt_status_check_sparse_checkout()\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/sparse-index.txt | 32 +++++++++++++++++++++---\n 1 file changed, 29 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/sparse-index.txt b/Documentation/technical/sparse-index.txt\nindex aa116406a016..7ab51bf6c441 100644\n--- a/Documentation/technical/sparse-index.txt\n+++ b/Documentation/technical/sparse-index.txt\n@@ -82,9 +82,35 @@ also introduce other features that have been considered for improving the\n index, as well.\n \n Next, consumers of the index will be guarded against operating on a\n-sparse-index by inserting calls to `ensure_full_index()` or\n-`expand_index_to_path()`. After these guards are in place, we can begin\n-leaving sparse-directory entries in the in-memory index structure.\n+sparse-index by inserting calls to `ensure_full_index()` before iterating\n+over all cache entries. If a specific path is requested, then those will\n+be protected from within the `index_file_exists()` and `index_name_pos()`\n+API calls: they will call `ensure_full_index()` if necessary.\n+\n+During a scan of the codebase, not every iteration of the cache entries\n+needs an `ensure_full_index()` check. The basic reasons include:\n+\n+1. The loop is scanning for entries with non-zero stage. These entries\n+ are not collapsed into a sparse-directory entry.\n+\n+2. The loop is scanning for submodules. These entries are not collapsed\n+ into a sparse-directory entry.\n+\n+3. The loop is part of the index API, especially around reading or\n+ writing the format.\n+\n+4. The loop is checking for correct order of cache entries and that is\n+ correct if and only if the sparse-directory entries are in the correct\n+ location.\n+\n+5. The loop ignores entries with the `SKIP_WORKTREE` bit set, or is\n+ otherwise already aware of sparse directory entries.\n+\n+6. The sparse-index is disabled at this point when using the split-index\n+ feature, so no effort is made to protect the split-index API.\n+\n+After these guards are in place, we can begin leaving sparse-directory\n+entries in the in-memory index structure.\n \n Even after inserting these guards, we will keep expanding sparse-indexes\n for most Git commands using the `command_requires_full_index` repository\n-- \ngitgitgadget\n\n"},{"id":"419411","messageId":"7e4079c48eb2ad06a849eda8f4bd6cb14493ffdb.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 04/27] cache: move ensure_full_index() to cache.h","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:52Z","receivedAt":"2021-03-16T21:17:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSoon we will insert ensure_full_index() calls across the codebase.\nInstead of also adding include statements for sparse-index.h, let's just\nuse the fact that anything that cares about the index already has\ncache.h in its includes.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 1 +\n sparse-index.h | 1 -\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex f244e6ac90fd..9c6d254e2bae 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -350,6 +350,7 @@ void add_name_hash(struct index_state *istate, struct cache_entry *ce);\n void remove_name_hash(struct index_state *istate, struct cache_entry *ce);\n void free_name_hash(struct index_state *istate);\n \n+void ensure_full_index(struct index_state *istate);\n \n /* Cache entry creation and cleanup */\n \ndiff --git a/sparse-index.h b/sparse-index.h\nindex 39dcc859735e..0268f38753c0 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,6 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-void ensure_full_index(struct index_state *istate);\n int convert_to_sparse(struct index_state *istate);\n \n struct repository;\n-- \ngitgitgadget\n\n"},{"id":"419412","messageId":"628e16dd3fc7be96d40a7b3201573a71ae7fbbd6.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 01/27] *: remove 'const' qualifier for struct index_state","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:49Z","receivedAt":"2021-03-16T21:17:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSeveral methods specify that they take a 'struct index_state' pointer\nwith the 'const' qualifier because they intend to only query the data,\nnot change it. However, we will be introducing a step very low in the\nmethod stack that might modify a sparse-index to become a full index in\nthe case that our queries venture inside a sparse-directory entry.\n\nThis change only removes the 'const' qualifiers that are necessary for\nthe following change which will actually modify the implementation of\nindex_name_stage_pos().\n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 +++++++-------\n attr.h | 4 ++--\n builtin/ls-files.c | 10 +++++-----\n cache.h | 6 +++---\n convert.c | 26 +++++++++++++-------------\n convert.h | 20 ++++++++++----------\n dir.c | 12 ++++++------\n dir.h | 8 ++++----\n merge-recursive.c | 2 +-\n pathspec.c | 6 +++---\n pathspec.h | 6 +++---\n read-cache.c | 10 +++++-----\n submodule.c | 6 +++---\n submodule.h | 6 +++---\n 14 files changed, 68 insertions(+), 68 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 4ef85d668b54..8de553293ee4 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -718,7 +718,7 @@ static struct attr_stack *read_attr_from_file(const char *path, int macro_ok)\n \treturn res;\n }\n \n-static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n+static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t int macro_ok)\n {\n@@ -748,7 +748,7 @@ static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n \treturn res;\n }\n \n-static struct attr_stack *read_attr(const struct index_state *istate,\n+static struct attr_stack *read_attr(struct index_state *istate,\n \t\t\t\t const char *path, int macro_ok)\n {\n \tstruct attr_stack *res = NULL;\n@@ -840,7 +840,7 @@ static void push_stack(struct attr_stack **attr_stack_p,\n \t}\n }\n \n-static void bootstrap_attr_stack(const struct index_state *istate,\n+static void bootstrap_attr_stack(struct index_state *istate,\n \t\t\t\t struct attr_stack **stack)\n {\n \tstruct attr_stack *e;\n@@ -878,7 +878,7 @@ static void bootstrap_attr_stack(const struct index_state *istate,\n \tpush_stack(stack, e, NULL, 0);\n }\n \n-static void prepare_attr_stack(const struct index_state *istate,\n+static void prepare_attr_stack(struct index_state *istate,\n \t\t\t const char *path, int dirlen,\n \t\t\t struct attr_stack **stack)\n {\n@@ -1078,7 +1078,7 @@ static void determine_macros(struct all_attrs_item *all_attrs,\n * If check->check_nr is non-zero, only attributes in check[] are collected.\n * Otherwise all attributes are collected.\n */\n-static void collect_some_attrs(const struct index_state *istate,\n+static void collect_some_attrs(struct index_state *istate,\n \t\t\t const char *path,\n \t\t\t struct attr_check *check)\n {\n@@ -1107,7 +1107,7 @@ static void collect_some_attrs(const struct index_state *istate,\n \tfill(path, pathlen, basename_offset, check->stack, check->all_attrs, rem);\n }\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path,\n \t\t struct attr_check *check)\n {\n@@ -1124,7 +1124,7 @@ void git_check_attr(const struct index_state *istate,\n \t}\n }\n \n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check)\n {\n \tint i;\ndiff --git a/attr.h b/attr.h\nindex 404548f028a8..3732505edae8 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -190,14 +190,14 @@ void attr_check_free(struct attr_check *check);\n */\n const char *git_attr_name(const struct git_attr *);\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n /*\n * Retrieve all attributes that apply to the specified path.\n * check holds the attributes and their values.\n */\n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n enum git_attr_direction {\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 13bcc2d84735..76a0d35c1a01 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -57,7 +57,7 @@ static const char *tag_modified = \"\";\n static const char *tag_skip_worktree = \"\";\n static const char *tag_resolve_undo = \"\";\n \n-static void write_eolinfo(const struct index_state *istate,\n+static void write_eolinfo(struct index_state *istate,\n \t\t\t const struct cache_entry *ce, const char *path)\n {\n \tif (show_eol) {\n@@ -122,7 +122,7 @@ static void print_debug(const struct cache_entry *ce)\n \t}\n }\n \n-static void show_dir_entry(const struct index_state *istate,\n+static void show_dir_entry(struct index_state *istate,\n \t\t\t const char *tag, struct dir_entry *ent)\n {\n \tint len = max_prefix_len;\n@@ -139,7 +139,7 @@ static void show_dir_entry(const struct index_state *istate,\n \twrite_name(ent->name);\n }\n \n-static void show_other_files(const struct index_state *istate,\n+static void show_other_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -152,7 +152,7 @@ static void show_other_files(const struct index_state *istate,\n \t}\n }\n \n-static void show_killed_files(const struct index_state *istate,\n+static void show_killed_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -254,7 +254,7 @@ static void show_ce(struct repository *repo, struct dir_struct *dir,\n \t}\n }\n \n-static void show_ru_info(const struct index_state *istate)\n+static void show_ru_info(struct index_state *istate)\n {\n \tstruct string_list_item *item;\n \ndiff --git a/cache.h b/cache.h\nindex 4ca6cd7f782c..f244e6ac90fd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -800,7 +800,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n * index_name_pos(&index, \"f\", 1) -> -3\n * index_name_pos(&index, \"g\", 1) -> -5\n */\n-int index_name_pos(const struct index_state *, const char *name, int namelen);\n+int index_name_pos(struct index_state *, const char *name, int namelen);\n \n /*\n * Some functions return the negative complement of an insert position when a\n@@ -850,8 +850,8 @@ int add_file_to_index(struct index_state *, const char *path, int flags);\n int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n-int index_name_is_other(const struct index_state *, const char *, int);\n-void *read_blob_data_from_index(const struct index_state *, const char *, unsigned long *);\n+int index_name_is_other(struct index_state *, const char *, int);\n+void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/convert.c b/convert.c\nindex ee360c2f07ce..20b19abffa36 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -138,7 +138,7 @@ static const char *gather_convert_stats_ascii(const char *data, unsigned long si\n \t}\n }\n \n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n@@ -222,7 +222,7 @@ static void check_global_conv_flags_eol(const char *path,\n \t}\n }\n \n-static int has_crlf_in_index(const struct index_state *istate, const char *path)\n+static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n \tunsigned long sz;\n \tvoid *data;\n@@ -496,7 +496,7 @@ static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n \treturn 1;\n }\n \n-static int crlf_to_git(const struct index_state *istate,\n+static int crlf_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n \t\t enum crlf_action crlf_action, int conv_flags)\n@@ -1307,7 +1307,7 @@ struct conv_attrs {\n \n static struct attr_check *check;\n \n-static void convert_attrs(const struct index_state *istate,\n+static void convert_attrs(struct index_state *istate,\n \t\t\t struct conv_attrs *ca, const char *path)\n {\n \tstruct attr_check_item *ccheck = NULL;\n@@ -1369,7 +1369,7 @@ void reset_parsed_attributes(void)\n \tuser_convert_tail = NULL;\n }\n \n-int would_convert_to_git_filter_fd(const struct index_state *istate, const char *path)\n+int would_convert_to_git_filter_fd(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1388,7 +1388,7 @@ int would_convert_to_git_filter_fd(const struct index_state *istate, const char\n \treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL, NULL);\n }\n \n-const char *get_convert_attr_ascii(const struct index_state *istate, const char *path)\n+const char *get_convert_attr_ascii(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1414,7 +1414,7 @@ const char *get_convert_attr_ascii(const struct index_state *istate, const char\n \treturn \"\";\n }\n \n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags)\n {\n@@ -1448,7 +1448,7 @@ int convert_to_git(const struct index_state *istate,\n \treturn ret | ident_to_git(src, len, dst, ca.ident);\n }\n \n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n \t\t\t int conv_flags)\n {\n@@ -1466,7 +1466,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tident_to_git(dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const struct index_state *istate,\n+static int convert_to_working_tree_internal(struct index_state *istate,\n \t\t\t\t\t const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing,\n@@ -1510,7 +1510,7 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \treturn ret | ret_filter;\n }\n \n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n@@ -1519,7 +1519,7 @@ int async_convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);\n }\n \n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta)\n@@ -1527,7 +1527,7 @@ int convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);\n }\n \n-int renormalize_buffer(const struct index_state *istate, const char *path,\n+int renormalize_buffer(struct index_state *istate, const char *path,\n \t\t const char *src, size_t len, struct strbuf *dst)\n {\n \tint ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);\n@@ -1964,7 +1964,7 @@ static struct stream_filter *ident_filter(const struct object_id *oid)\n * Note that you would be crazy to set CRLF, smudge/clean or ident to a\n * large binary blob you would want us not to slurp into the memory!\n */\n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/convert.h b/convert.h\nindex e29d1026a686..0f7c8a1f0446 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -65,41 +65,41 @@ struct checkout_metadata {\n \n extern enum eol core_eol;\n extern char *check_roundtrip_encoding;\n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path);\n const char *get_wt_convert_stats_ascii(const char *path);\n-const char *get_convert_attr_ascii(const struct index_state *istate,\n+const char *get_convert_attr_ascii(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /* returns 1 if *dst was used */\n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags);\n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta);\n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco);\n int async_query_available_blobs(const char *cmd,\n \t\t\t\tstruct string_list *available_paths);\n-int renormalize_buffer(const struct index_state *istate,\n+int renormalize_buffer(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst);\n-static inline int would_convert_to_git(const struct index_state *istate,\n+static inline int would_convert_to_git(struct index_state *istate,\n \t\t\t\t const char *path)\n {\n \treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd,\n \t\t\t struct strbuf *dst,\n \t\t\t int conv_flags);\n-int would_convert_to_git_filter_fd(const struct index_state *istate,\n+int would_convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /*\n@@ -133,7 +133,7 @@ void reset_parsed_attributes(void);\n \n struct stream_filter; /* opaque */\n \n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *);\n void free_stream_filter(struct stream_filter *);\ndiff --git a/dir.c b/dir.c\nindex fd8aa7c40faa..5b00dfb5b144 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -306,7 +306,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,\n * [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match.\n * [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match.\n */\n-static int match_pathspec_item(const struct index_state *istate,\n+static int match_pathspec_item(struct index_state *istate,\n \t\t\t const struct pathspec_item *item, int prefix,\n \t\t\t const char *name, int namelen, unsigned flags)\n {\n@@ -429,7 +429,7 @@ static int match_pathspec_item(const struct index_state *istate,\n * pathspec did not match any names, which could indicate that the\n * user mistyped the nth pathspec.\n */\n-static int do_match_pathspec(const struct index_state *istate,\n+static int do_match_pathspec(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *name, int namelen,\n \t\t\t int prefix, char *seen,\n@@ -500,7 +500,7 @@ static int do_match_pathspec(const struct index_state *istate,\n \treturn retval;\n }\n \n-static int match_pathspec_with_flags(const struct index_state *istate,\n+static int match_pathspec_with_flags(struct index_state *istate,\n \t\t\t\t const struct pathspec *ps,\n \t\t\t\t const char *name, int namelen,\n \t\t\t\t int prefix, char *seen, unsigned flags)\n@@ -516,7 +516,7 @@ static int match_pathspec_with_flags(const struct index_state *istate,\n \treturn negative ? 0 : positive;\n }\n \n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *ps,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir)\n@@ -529,7 +529,7 @@ int match_pathspec(const struct index_state *istate,\n /**\n * Check if a submodule is a superset of the pathspec\n */\n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen)\n@@ -892,7 +892,7 @@ void add_pattern(const char *string, const char *base,\n \tadd_pattern_to_hashsets(pl, pattern);\n }\n \n-static int read_skip_worktree_file_from_index(const struct index_state *istate,\n+static int read_skip_worktree_file_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t size_t *size_out, char **data_out,\n \t\t\t\t\t struct oid_stat *oid_stat)\ndiff --git a/dir.h b/dir.h\nindex facfae47402a..51cb0e217247 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -354,7 +354,7 @@ int count_slashes(const char *s);\n int simple_length(const char *match);\n int no_wildcard(const char *string);\n char *common_prefix(const struct pathspec *pathspec);\n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *pathspec,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir);\n@@ -492,12 +492,12 @@ int git_fnmatch(const struct pathspec_item *item,\n \t\tconst char *pattern, const char *string,\n \t\tint prefix);\n \n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen);\n \n-static inline int ce_path_match(const struct index_state *istate,\n+static inline int ce_path_match(struct index_state *istate,\n \t\t\t\tconst struct cache_entry *ce,\n \t\t\t\tconst struct pathspec *pathspec,\n \t\t\t\tchar *seen)\n@@ -506,7 +506,7 @@ static inline int ce_path_match(const struct index_state *istate,\n \t\t\t S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n }\n \n-static inline int dir_path_match(const struct index_state *istate,\n+static inline int dir_path_match(struct index_state *istate,\n \t\t\t\t const struct dir_entry *ent,\n \t\t\t\t const struct pathspec *pathspec,\n \t\t\t\t int prefix, char *seen)\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 3d9207455b3a..b8de7a704eae 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -2987,7 +2987,7 @@ static int blob_unchanged(struct merge_options *opt,\n \tstruct strbuf obuf = STRBUF_INIT;\n \tstruct strbuf abuf = STRBUF_INIT;\n \tint ret = 0; /* assume changed for safety */\n-\tconst struct index_state *idx = opt->repo->index;\n+\tstruct index_state *idx = opt->repo->index;\n \n \tif (a->mode != o->mode)\n \t\treturn 0;\ndiff --git a/pathspec.c b/pathspec.c\nindex 7a229d8d22f2..b6e333965cb4 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -20,7 +20,7 @@\n * to use find_pathspecs_matching_against_index() instead.\n */\n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen)\n {\n \tint num_unmatched = 0, i;\n@@ -51,7 +51,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n * given pathspecs achieves against all items in the index.\n */\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate)\n+\t\t\t\t\t struct index_state *istate)\n {\n \tchar *seen = xcalloc(pathspec->nr, 1);\n \tadd_pathspec_matches_against_index(pathspec, istate, seen);\n@@ -702,7 +702,7 @@ void clear_pathspec(struct pathspec *pathspec)\n \tpathspec->nr = 0;\n }\n \n-int match_pathspec_attrs(const struct index_state *istate,\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item)\n {\ndiff --git a/pathspec.h b/pathspec.h\nindex 454ce364fac7..2ccc8080b6cc 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -150,11 +150,11 @@ static inline int ps_strcmp(const struct pathspec_item *item,\n }\n \n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen);\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate);\n-int match_pathspec_attrs(const struct index_state *istate,\n+\t\t\t\t\t struct index_state *istate);\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex b9c08773466c..dfec221d9128 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -548,7 +548,7 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(const struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n {\n \tint first, last;\n \n@@ -569,7 +569,7 @@ static int index_name_stage_pos(const struct index_state *istate, const char *na\n \treturn -first-1;\n }\n \n-int index_name_pos(const struct index_state *istate, const char *name, int namelen)\n+int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n \treturn index_name_stage_pos(istate, name, namelen, 0);\n }\n@@ -3380,8 +3380,8 @@ int repo_read_index_unmerged(struct repository *repo)\n * We helpfully remove a trailing \"/\" from directories so that\n * the output of read_directory can be used as-is.\n */\n-int index_name_is_other(const struct index_state *istate, const char *name,\n-\t\tint namelen)\n+int index_name_is_other(struct index_state *istate, const char *name,\n+\t\t\tint namelen)\n {\n \tint pos;\n \tif (namelen && name[namelen - 1] == '/')\n@@ -3399,7 +3399,7 @@ int index_name_is_other(const struct index_state *istate, const char *name,\n \treturn 1;\n }\n \n-void *read_blob_data_from_index(const struct index_state *istate,\n+void *read_blob_data_from_index(struct index_state *istate,\n \t\t\t\tconst char *path, unsigned long *size)\n {\n \tint pos, len;\ndiff --git a/submodule.c b/submodule.c\nindex 9767ba9893cc..83809a4f7bd2 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -33,7 +33,7 @@ static struct oid_array ref_tips_after_fetch;\n * will be disabled because we can't guess what might be configured in\n * .gitmodules unless the user resolves the conflict.\n */\n-int is_gitmodules_unmerged(const struct index_state *istate)\n+int is_gitmodules_unmerged(struct index_state *istate)\n {\n \tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n \tif (pos < 0) { /* .gitmodules not found or isn't merged */\n@@ -301,7 +301,7 @@ int is_submodule_populated_gently(const char *path, int *return_error_code)\n /*\n * Dies if the provided 'prefix' corresponds to an unpopulated submodule\n */\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix)\n {\n \tint i, prefixlen;\n@@ -331,7 +331,7 @@ void die_in_unpopulated_submodule(const struct index_state *istate,\n /*\n * Dies if any paths in the provided pathspec descends into a submodule\n */\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps)\n {\n \tint i, j;\ndiff --git a/submodule.h b/submodule.h\nindex 4ac6e31cf1f7..84640c49c114 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -39,7 +39,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n-int is_gitmodules_unmerged(const struct index_state *istate);\n+int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n@@ -60,9 +60,9 @@ int is_submodule_active(struct repository *repo, const char *path);\n * Otherwise the return error code is the same as of resolve_gitdir_gently.\n */\n int is_submodule_populated_gently(const char *path, int *return_error_code);\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix);\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps);\n enum submodule_update_type parse_submodule_update_type(const char *value);\n int parse_submodule_update_strategy(const char *value,\n-- \ngitgitgadget\n\n"},{"id":"419413","messageId":"1288f02177d2275937fa4f0928b437756cdaf712.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 13/27] merge-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:01Z","receivedAt":"2021-03-16T21:18:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge-index.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/merge-index.c b/builtin/merge-index.c\nindex 38ea6ad6ca25..c1c4409b536d 100644\n--- a/builtin/merge-index.c\n+++ b/builtin/merge-index.c\n@@ -58,6 +58,7 @@ static void merge_one_path(const char *path)\n static void merge_all(void)\n {\n \tint i;\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_stage(ce))\n@@ -80,6 +81,8 @@ int cmd_merge_index(int argc, const char **argv, const char *prefix)\n \n \tread_cache();\n \n+\tensure_full_index(&the_index);\n+\n \ti = 1;\n \tif (!strcmp(argv[i], \"-o\")) {\n \t\tone_shot = 1;\n-- \ngitgitgadget\n\n"},{"id":"419415","messageId":"00c8a7e1d119fa3b403227eb415200bba01f4bda.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 12/27] ls-files: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:00Z","receivedAt":"2021-03-16T21:18:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid missing files.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/ls-files.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 76a0d35c1a01..8fbef2b14f29 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -317,6 +317,7 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n+\tensure_full_index(repo->index);\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -495,6 +496,7 @@ void overlay_tree_on_index(struct index_state *istate,\n \t\tdie(\"bad tree-ish %s\", tree_name);\n \n \t/* Hoist the unmerged entries up to stage #3 to make room */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_stage(ce))\n-- \ngitgitgadget\n\n"},{"id":"419414","messageId":"a59e45c4ae8f8a1b99e6ddabaec438e281a44057.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 08/27] commit: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:56Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThese two loops iterate over all cache entries, so ensure that a sparse\nindex is expanded to a full index before we do so.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 739110c5a7f6..9ea9e7b53390 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -251,6 +251,7 @@ static int list_paths(struct string_list *list, const char *with_tree,\n \t\tfree(max_prefix);\n \t}\n \n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tstruct string_list_item *item;\n@@ -931,6 +932,7 @@ static int prepare_to_commit(const char *index_file, const char *prefix,\n \t\tif (get_oid(parent, &oid)) {\n \t\t\tint i, ita_nr = 0;\n \n+\t\t\tensure_full_index(&the_index);\n \t\t\tfor (i = 0; i < active_nr; i++)\n \t\t\t\tif (ce_intent_to_add(active_cache[i]))\n \t\t\t\t\tita_nr++;\n-- \ngitgitgadget\n\n"},{"id":"419416","messageId":"142df1ab8e3e8ea5e213f2477e271e60e5b62f84.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 05/27] add: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:53Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ea762a41e3a2..ab2ef4a63530 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -141,6 +141,7 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n {\n \tint i, retval = 0;\n \n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"419417","messageId":"1bb7b6117e41ecf84afc03a29727d65606601be4.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 07/27] checkout: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:55Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries in the checkout builtin, ensure\nthat we have a full index to avoid any unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 0e6639052001..2e64569d2e13 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -368,6 +368,7 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -512,6 +513,7 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t * Make sure all pathspecs participated in locating the paths\n \t * to be checked out.\n \t */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++)\n \t\tif (opts->overlay_mode)\n \t\t\tmark_ce_for_checkout_overlay(active_cache[pos],\n-- \ngitgitgadget\n\n"},{"id":"419418","messageId":"83040e7558f3effac91e76352f259a5a7d367ed8.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 09/27] difftool: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:57Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index has\nbeen expanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/difftool.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 6e18e623fddf..1d62f77efa61 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -584,6 +584,8 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\tsetenv(\"GIT_DIFFTOOL_DIRDIFF\", \"true\", 1);\n \trc = run_command_v_opt(helper_argv, flags);\n \n+\tensure_full_index(&wtindex);\n+\n \t/*\n \t * If the diff includes working copy files and those\n \t * files were modified during the diff, then the changes\n-- \ngitgitgadget\n\n"},{"id":"419419","messageId":"92ccd31dd343c20bad4a6b8d89c50559209e7236.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 15/27] sparse-checkout: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:03Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen adjusting the sparse-checkout definition, ensure that a sparse\nindex is expanded to a full one to avoid unexpected behavior. This is a\ntop candidate for later integration with the sparse-index, but requires\ncareful testing.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/stash.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex ba774cce674f..b3d4fe814b6b 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1350,6 +1350,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\tint i;\n \t\tchar *ps_matched = xcalloc(ps->nr, 1);\n \n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++)\n \t\t\tce_path_match(&the_index, active_cache[i], ps,\n \t\t\t\t ps_matched);\n-- \ngitgitgadget\n\n"},{"id":"419420","messageId":"d6816560b32f020d6c38f5e9ad36be532685279c.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 14/27] rm: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:02Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4858631e0f02..99125ccb8424 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -293,6 +293,7 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \n \tseen = xcalloc(pathspec.nr, 1);\n \n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n-- \ngitgitgadget\n\n"},{"id":"419421","messageId":"988f7bd2d7364970a3c192bb229a21ea77803f26.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 10/27] fsck: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:58Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen verifying all blobs reachable from the index, ensure that a sparse\nindex has been expanded to a full one to avoid missing some blobs.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fsck.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 821e7798c706..524d3fca28f6 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -883,6 +883,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tverify_index_checksum = 1;\n \t\tverify_ce_order = 1;\n \t\tread_cache();\n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++) {\n \t\t\tunsigned int mode;\n \t\t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"419424","messageId":"abe189051a0c6e4cb083ff3ab8a240fd375b2581.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 11/27] grep: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:59Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one so we do not miss blobs to scan. Later, this can\nintegrate more carefully with sparse indexes with proper testing.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/grep.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4e91a253ac3b..652d6d60f644 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -504,6 +504,7 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n+\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-- \ngitgitgadget\n\n"},{"id":"419425","messageId":"40cfc59f695a6069277d4744a12c9f607e9aa35f.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 27/27] name-hash: use expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:15Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA sparse-index loads the name-hash data for its entries, including the\nsparse-directory entries. If a caller asks for a path that is contained\nwithin a sparse-directory entry, we need to expand to a full index and\nrecalculate the name hash table before returning the result. Insert\ncalls to expand_to_path() to protect against this case.\n\nSigned-off-by: Derrick Stolee \n---\n name-hash.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 4e03fac9bb12..75c159e06eeb 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -8,6 +8,7 @@\n #include \"cache.h\"\n #include \"thread-utils.h\"\n #include \"trace2.h\"\n+#include \"sparse-index.h\"\n \n struct dir_entry {\n \tstruct hashmap_entry ent;\n@@ -109,6 +110,12 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)\n \tif (ce->ce_flags & CE_HASHED)\n \t\treturn;\n \tce->ce_flags |= CE_HASHED;\n+\n+\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\tadd_dir_entry(istate, ce);\n+\t\treturn;\n+\t}\n+\n \thashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n \thashmap_add(&istate->name_hash, &ce->ent);\n \n@@ -680,6 +687,7 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n \tstruct dir_entry *dir;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, 0);\n \tdir = find_dir_entry(istate, name, namelen);\n \treturn dir && dir->nr;\n }\n@@ -690,6 +698,7 @@ void adjust_dirname_case(struct index_state *istate, char *name)\n \tconst char *ptr = startPtr;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, strlen(name), 0);\n \twhile (*ptr) {\n \t\twhile (*ptr && *ptr != '/')\n \t\t\tptr++;\n@@ -713,6 +722,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n \tunsigned int hash = memihash(name, namelen);\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, icase);\n \n \tce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL,\n \t\t\t\t\t struct cache_entry, ent);\n-- \ngitgitgadget\n"},{"id":"419429","messageId":"796449758a08ad9f2038f16c7d157edf27ca3cc4.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 16/27] update-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:04Z","receivedAt":"2021-03-16T21:18:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/update-index.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 79087bccea4b..8eb654ddc754 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -745,6 +745,7 @@ static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n \t\tstruct cache_entry *old = NULL;\n-- \ngitgitgadget\n\n"},{"id":"419422","messageId":"bfa0164cc3c167e383cdb5405526202432ae624e.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 06/27] checkout-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:16:54Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore we iterate over all cache entries, ensure that the index is not\nsparse. This loop in checkout_all() might be safe to iterate over a\nsparse index, but let's put this protection here until it can be\ncarefully tested.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout-index.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 023e49e271c2..ba31a036f193 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -119,6 +119,7 @@ static void checkout_all(const char *prefix, int prefix_length)\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tif (ce_stage(ce) != checkout_stage\n-- \ngitgitgadget\n\n"},{"id":"419423","messageId":"8ce1f80985a441b430f8e86b18586f60137d0805.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 18/27] dir: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:06Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/dir.c b/dir.c\nindex 5b00dfb5b144..1c9f61566e66 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3533,6 +3533,7 @@ static void connect_wt_gitdir_in_nested(const char *sub_worktree,\n \tif (repo_read_index(&subrepo) < 0)\n \t\tdie(_(\"index file corrupt in repo %s\"), subrepo.gitdir);\n \n+\tensure_full_index(subrepo.index);\n \tfor (i = 0; i < subrepo.index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = subrepo.index->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"419426","messageId":"45bbed6150a2a9f9e9446edc9a613f027da73957.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 17/27] diff-lib: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:05Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n diff-lib.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex b73cc1859a49..41d6fcec1a81 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -102,6 +102,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \tif (diff_unmerged_stage < 0)\n \t\tdiff_unmerged_stage = 2;\n+\n \tentries = istate->cache_nr;\n \tfor (i = 0; i < entries; i++) {\n \t\tunsigned int oldmode, newmode;\n-- \ngitgitgadget\n\n"},{"id":"419427","messageId":"1690d33c11e0008c437eadfcef80de0dd870a86e.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 26/27] sparse-index: expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:14Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome users of the index API have a specific path they are looking for,\nbut choose to use index_file_exists() to rely on the name-hash hashtable\ninstead of doing binary search with index_name_pos(). These users only\nneed to know a yes/no answer, not a position within the cache array.\n\nWhen the index is sparse, the name-hash hash table does not contain the\nfull list of paths within sparse directories. It _does_ contain the\ndirectory names for the sparse-directory entries.\n\nCreate a helper function, expand_to_path(), for intended use with the\nname-hash hashtable functions. The integration with name-hash.c will\nfollow in a later change.\n\nThe solution here is to use ensure_full_index() when we determine that\nthe requested path is within a sparse directory entry. This will\npopulate the name-hash hashtable as the index is recomputed from\nscratch.\n\nThere may be cases where the caller is trying to find an untracked path\nthat is not in the index but also is not within a sparse directory\nentry. We want to minimize the overhead for these requests. If we used\nindex_name_pos() to find the insertion order of the path, then we could\ndetermine from that position if a sparse-directory exists. (In fact,\njust calling index_name_pos() in that case would lead to expanding the\nindex to a full index.) However, this takes O(log N) time where N is the\nnumber of cache entries.\n\nTo keep the performance of this call based mostly on the input string,\nuse index_file_exists() to look for the ancestors of the path. Using the\nheuristic that a sparse directory is likely to have a small number of\nparent directories, we start from the bottom and build up. Use a string\nbuffer to allow mutating the path name to terminate after each slash for\neach hashset test.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n sparse-index.h | 13 +++++++++\n 2 files changed, 85 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 56313e805d9d..e5712d98d8b9 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -291,3 +291,75 @@ void ensure_full_index(struct index_state *istate)\n \n \ttrace2_region_leave(\"index\", \"ensure_full_index\", istate->repo);\n }\n+\n+/*\n+ * This static global helps avoid infinite recursion between\n+ * expand_to_path() and index_file_exists().\n+ */\n+static int in_expand_to_path = 0;\n+\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase)\n+{\n+\tstruct strbuf path_mutable = STRBUF_INIT;\n+\tsize_t substr_len;\n+\n+\t/* prevent extra recursion */\n+\tif (in_expand_to_path)\n+\t\treturn;\n+\n+\tif (!istate || !istate->sparse_index)\n+\t\treturn;\n+\n+\tif (!istate->repo)\n+\t\tistate->repo = the_repository;\n+\n+\tin_expand_to_path = 1;\n+\n+\t/*\n+\t * We only need to actually expand a region if the\n+\t * following are both true:\n+\t *\n+\t * 1. 'path' is not already in the index.\n+\t * 2. Some parent directory of 'path' is a sparse directory.\n+\t */\n+\n+\tif (index_file_exists(istate, path, pathlen, icase))\n+\t\tgoto cleanup;\n+\n+\tstrbuf_add(&path_mutable, path, pathlen);\n+\tstrbuf_addch(&path_mutable, '/');\n+\n+\t/* Check the name hash for all parent directories */\n+\tsubstr_len = 0;\n+\twhile (substr_len < pathlen) {\n+\t\tchar temp;\n+\t\tchar *replace = strchr(path_mutable.buf + substr_len, '/');\n+\n+\t\tif (!replace)\n+\t\t\tbreak;\n+\n+\t\t/* replace the character _after_ the slash */\n+\t\treplace++;\n+\t\ttemp = *replace;\n+\t\t*replace = '\\0';\n+\t\tif (index_file_exists(istate, path_mutable.buf,\n+\t\t\t\t path_mutable.len, icase)) {\n+\t\t\t/*\n+\t\t\t * We found a parent directory in the name-hash\n+\t\t\t * hashtable, which means that this entry could\n+\t\t\t * exist within a sparse-directory entry. Expand\n+\t\t\t * accordingly.\n+\t\t\t */\n+\t\t\tensure_full_index(istate);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t*replace = temp;\n+\t\tsubstr_len = replace - path_mutable.buf;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&path_mutable);\n+\tin_expand_to_path = 0;\n+}\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 0268f38753c0..1115a0d7dd98 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -4,6 +4,19 @@\n struct index_state;\n int convert_to_sparse(struct index_state *istate);\n \n+/*\n+ * Some places in the codebase expect to search for a specific path.\n+ * This path might be outside of the sparse-checkout definition, in\n+ * which case a sparse-index may not contain a path for that index.\n+ *\n+ * Given an index and a path, check to see if a leading directory for\n+ * 'path' exists in the index as a sparse directory. In that case,\n+ * expand that sparse directory to a full range of cache entries and\n+ * populate the index accordingly.\n+ */\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase);\n+\n struct repository;\n int set_sparse_index_config(struct repository *repo, int enable);\n \n-- \ngitgitgadget\n\n"},{"id":"419428","messageId":"81519782d4b233790ddf773b088db1e2cd260c4d.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 23/27] read-cache: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:11Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex b143277ed106..7449f31381cf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1576,6 +1576,7 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -2493,6 +2494,7 @@ int repo_index_has_changes(struct repository *repo,\n \t\tdiff_flush(&opt);\n \t\treturn opt.flags.has_changes != 0;\n \t} else {\n+\t\tensure_full_index(istate);\n \t\tfor (i = 0; sb && i < istate->cache_nr; i++) {\n \t\t\tif (i)\n \t\t\t\tstrbuf_addch(sb, ' ');\n-- \ngitgitgadget\n\n"},{"id":"419430","messageId":"39e8bea0c1cadaf2a59760bea13025415569476f.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 24/27] resolve-undo: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:12Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n resolve-undo.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 236320f179cb..3ccd3f66f80f 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -172,6 +172,7 @@ void unmerge_marked_index(struct index_state *istate)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (ce->ce_flags & CE_MATCHED)\n@@ -186,6 +187,7 @@ void unmerge_index(struct index_state *istate, const struct pathspec *pathspec)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_path_match(istate, ce, pathspec, NULL))\n-- \ngitgitgadget\n\n"},{"id":"419431","messageId":"57d59825627f10c4383c2c4ee8ee449e2c50f865.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 21/27] merge-recursive: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:09Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n merge-recursive.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex b8de7a704eae..7954c7829d27 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -522,6 +522,7 @@ static struct string_list *get_unmerged(struct index_state *istate)\n \n \tunmerged->strdup_strings = 1;\n \n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct string_list_item *item;\n \t\tstruct stage_data *e;\n-- \ngitgitgadget\n\n"},{"id":"419432","messageId":"6e0b452f44c1b8f8a5f9c0478b05a7d9248f69ad.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 19/27] entry: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:07Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n entry.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/entry.c b/entry.c\nindex 7b9f43716f76..e55a2af867cf 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -412,6 +412,7 @@ static void mark_colliding_entries(const struct checkout *state,\n \n \tce->ce_flags |= CE_MATCHED;\n \n+\tensure_full_index(state->istate);\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"419433","messageId":"6abb1813ae109f4be991aac7ab8897d35110aa36.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 25/27] revision: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:13Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all index entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior. This case could\nbe integrated later by ensuring that we walk the tree in the\nsparse-directory entry, but the current behavior is only expecting\nblobs. Save this integration for later when it can be properly tested.\n\nSigned-off-by: Derrick Stolee \n---\n revision.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/revision.c b/revision.c\nindex b78733f5089b..1eee55934c08 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1680,6 +1680,7 @@ static void do_add_index_objects_to_pending(struct rev_info *revs,\n {\n \tint i;\n \n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"419435","messageId":"fb4c7038b7468c77c4d3c5181f0cb0de85d1149f.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 20/27] merge-ort: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:08Z","receivedAt":"2021-03-16T21:18:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid unexpected behavior.\n\nThis is a top candidate for updating later with a proper integration\nwith the sparse index, since we will likely enable the ORT strategy by\ndefault when the sparse index is enabled. This appears to be the only\nplace where the ORT strategy interacts with the index in such a global\nway, so that integration should be clear once the ORT strategy and the\nsparse index topics stabilize.\n\nSigned-off-by: Derrick Stolee \n---\n merge-ort.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex 603d30c52170..9f737212555d 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -3112,6 +3112,7 @@ static int record_conflicted_index_entries(struct merge_options *opt,\n \tif (strmap_empty(conflicted))\n \t\treturn 0;\n \n+\tensure_full_index(index);\n \toriginal_cache_nr = index->cache_nr;\n \n \t/* Put every entry from paths into plist, then sort */\n-- \ngitgitgadget\n\n"},{"id":"419434","messageId":"f4c470686b2764a6046c8a8b80f18c882ad8899b.1615929436.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH 22/27] pathspec: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-03-16T21:17:10Z","receivedAt":"2021-03-16T21:18:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex b6e333965cb4..08736ab368d9 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -36,6 +36,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\t\tnum_unmatched++;\n \tif (!num_unmatched)\n \t\treturn;\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tce_path_match(istate, ce, pathspec, seen);\n-- \ngitgitgadget\n\n"},{"id":"419483","messageId":"CABPp-BHyCDJH-78XGdW5TxAOPjDE669LL21yTY3NdmCP7iSMRA@mail.gmail.com","threadId":"55334","inReplyTo":"142df1ab8e3e8ea5e213f2477e271e60e5b62f84.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/27] add: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-17T17:35:27Z","receivedAt":"2021-03-17T17:36:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before iterating over all cache entries, ensure that a sparse index is\n> expanded to a full index to avoid unexpected behavior.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/add.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/builtin/add.c b/builtin/add.c\n> index ea762a41e3a2..ab2ef4a63530 100644\n> --- a/builtin/add.c\n> +++ b/builtin/add.c\n> @@ -141,6 +141,7 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n> {\n> int i, retval = 0;\n>\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> struct cache_entry *ce = active_cache[i];\n\nI'm not so sure about this one; from git-add(1):\n\n --renormalize\n Apply the \"clean\" process freshly to all tracked files to forcibly\n add them again to the index. This is useful after changing\n core.autocrlf configuration or the text attribute in order to\n correct files added with wrong CRLF/LF line endings. This option\n implies -u.\n\n... to \"all tracked files\". Should that be interpreted as all files\nwithin the sparsity paths, especially considering that we're trying to\nenable folks to work within the set of files of interest to them? If\nit doesn't imply that, wouldn't users want an extra option to be able\nto behave that way? And if we add an option, why are we adding a\nspecial option for people wanting to behave sparsely in a\nsparse-index-cone-mode-sparse-checkout rather than for the special\ncase of folks wanting to behave densely in such a setup?\n\nThe code below already has the following check below:\n\n if (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n continue; /* do not touch non blobs */\n\nsuggesting that it'd correctly skip over the sparse directories, so I\nthink this one just isn't needed.\n\nHowever, on a tangent, that S_ISLNK looks rather suspicious to me.\nWhy would we renormalize symlinks? I mean, sure, symlinks aren't\nlikely to have CRLF/LF characters in them, but if they did, wouldn't\nit be wrong to renormalize them?\n"},{"id":"419488","messageId":"CABPp-BF=-2qD9YodpnhYP49EnKEV=MOFDMUnB-agd-d69jVeQA@mail.gmail.com","threadId":"55334","inReplyTo":"bfa0164cc3c167e383cdb5405526202432ae624e.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-17T17:50:22Z","receivedAt":"2021-03-17T17:51:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before we iterate over all cache entries, ensure that the index is not\n> sparse. This loop in checkout_all() might be safe to iterate over a\n> sparse index, but let's put this protection here until it can be\n> carefully tested.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/checkout-index.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\n> index 023e49e271c2..ba31a036f193 100644\n> --- a/builtin/checkout-index.c\n> +++ b/builtin/checkout-index.c\n> @@ -119,6 +119,7 @@ static void checkout_all(const char *prefix, int prefix_length)\n> int i, errs = 0;\n> struct cache_entry *last_ce = NULL;\n>\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr ; i++) {\n> struct cache_entry *ce = active_cache[i];\n> if (ce_stage(ce) != checkout_stage\n\nWith the caveat in the commit message, this change looks okay, but\ncheckout-index may be buggy regardless of the presence of\nensure_full_index(). If ensure_full_index() really is needed here\nbecause it needs to operate on all SKIP_WORKTREE paths and not just\nleading directories, that's because it's writing all those\nSKIP_WORKTREE entries to the working tree. When it writes them to the\nworking tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\na bit of a pickle...\n\nMight be nice to add a\n/* TODO: audit if this is needed; if it is, we may have other bugs... */\nor something like that. But then again, perhaps you're considering\nall uses of ensure_full_index() to be need-to-be-reaudited codepaths?\nIf so, and we determine we really do need one and want to keep it\nindefinitely, will we mark those with a comment about why it's\nconsidered correct?\n\nI just want a way to know what still needs to be audited and what\ndoesn't without doing a lot of history spelunking...\n"},{"id":"419495","messageId":"CABPp-BGfkPscpro=W5vcRHD5cV6pddYpvHLLzMWjL9WLaxyu3w@mail.gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/27] Sparse Index: API protections","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-17T18:03:24Z","receivedAt":"2021-03-17T18:04:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> Here is the second patch series submission coming out of the sparse-index\n> RFC [1].\n>\n> [1]\n> https://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n>\n> This is based on v3 of the format series [2].\n>\n> [2]\n> https://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@gmail.com/\n>\n> The point of this series is to insert protections for the consumers of the\n> in-memory index. We mark certain regions of code as needing a full index, so\n> we call ensure_full_index() to expand a sparse index to a full one, if\n> necessary. These protections are inserted file-by-file in every loop over\n> all cache entries. Well, \"most\" loops, because some are going to be handled\n> in the very next series so I leave them out.\n>\n> Many callers use index_name_pos() to find a path by name. In these cases, we\n> can check if that position resolves to a sparse directory instance. In those\n> cases, we just expand to a full index and run the search again.\n>\n> The last few patches deal with the name-hash hashtable for doing O(1)\n> lookups.\n>\n> These protections don't do much right now, since the previous series created\n> the_repository->settings.command_requires_full_index to guard all index\n> reads and writes to ensure the in-memory copy is full for commands that have\n> not been tested with the sparse index yet.\n>\n> However, after this series is complete, we now have a straight-forward plan\n> for making commands \"sparse aware\" one-by-one:\n>\n> 1. Disable settings.command_requires_full_index to allow an in-memory\n> sparse-index.\n> 2. Run versions of that command under a debugger, breaking on\n> ensure_full_index().\n> 3. Examine the call stack to determine the context of that expansion, then\n> implement the proper behavior in those locations.\n> 4. Add tests to ensure we are checking this logic in the presence of sparse\n> directory entries.\n\nI started reading the series, then noticed I didn't like the first few\nadditions of ensure_full_index(). The first was because I thought it\njust wasn't needed as per a few lines of code later, but the more\nimportant one that stuck out to me was another where if the\nensure_full_index() call is needed to avoid the code blowing up, then\nthe code has a good chance that it is doing something inherently wrong\nin a sparse-checkout/sparse-index anyway.\n\nSo I guess that brings me to the question I asked in 07/27 -- is the\npresence of ensure_full_index() a note that the code needs to be later\naudited and tweaked to work with sparse-indexes? If so, then good,\nthis series may make sense. However, will that always be the case?\nIf we think some of these will be left in the code, is there a plan to\nannotate each one that has been audited and determined that it needs\nto stay? If not, then each ensure_full_index() might or might not\nhave been audited for correctness and it becomes a pain to know what's\nleft to fix. If the plan is that these are to be audited, and to be\nmarked if they are truly deemed necessary, then the series makes sense\nto me. If not, then I'm confused about something with the series and\nneed some help with the goals and plans.\n\nIf I'm confused about the goals and the plans, then my reviews will\nprobably be less than helpful, so I'll suspend reading the series\nuntil I understand the plan a little better.\n"},{"id":"419544","messageId":"08ffff48-7b9c-7113-1a5a-557f3efff26f@gmail.com","threadId":"55334","inReplyTo":"CABPp-BF=-2qD9YodpnhYP49EnKEV=MOFDMUnB-agd-d69jVeQA@mail.gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-17T20:05:13Z","receivedAt":"2021-03-17T20:06:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/17/2021 1:50 PM, Elijah Newren wrote:\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n> With the caveat in the commit message, this change looks okay, but\n> checkout-index may be buggy regardless of the presence of\n> ensure_full_index(). If ensure_full_index() really is needed here\n> because it needs to operate on all SKIP_WORKTREE paths and not just\n> leading directories, that's because it's writing all those\n> SKIP_WORKTREE entries to the working tree. When it writes them to the\n> working tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\n> a bit of a pickle...\n\nPerhaps I'm unclear in my intentions with this series: _every_\ninsertion of ensure_full_index() is intended to be audited with\ntests in the future. Some might need behavior change, and others\nwill not. In this series, I'm just putting in the protections so\nwe don't accidentally trigger unexpected behavior.\n\nSince tests take time to write and review, I was hoping that these\ninsertions were minimal enough to get us to a safe place where we\ncan remove the guards carefully.\n\nSo with that in mind...\n\n> Might be nice to add a\n> /* TODO: audit if this is needed; if it is, we may have other bugs... */\n> or something like that. But then again, perhaps you're considering\n> all uses of ensure_full_index() to be need-to-be-reaudited codepaths?\n> If so, and we determine we really do need one and want to keep it\n> indefinitely, will we mark those with a comment about why it's\n> considered correct?\n> \n> I just want a way to know what still needs to be audited and what\n> doesn't without doing a lot of history spelunking...\n\n...every insertion \"needs to be audited\" in the future. That's a\nbig part of the next \"phases\" in the implementation plan.\n\nAs you suggest, it might be a good idea to add a comment to every\ninsertion, to mark it as un-audited, such as:\n\n\t/* TODO: test if ensure_full_index() is necessary */\n\nWe can come back later to delete the comment if it truly is\nnecessary (and add tests to guarantee correct behavior). We can\nalso remove the comment _and_ the call by modifying the loop\nbehavior to do the right thing in some cases.\n\nThanks,\n-Stolee\n"},{"id":"419552","messageId":"CAHd-oW4DMAjQ_Gjbzkx6csFRAK1iKHsTfuJLniOjWehwrnHdVQ@mail.gmail.com","threadId":"55334","inReplyTo":"CABPp-BHyCDJH-78XGdW5TxAOPjDE669LL21yTY3NdmCP7iSMRA@mail.gmail.com","subject":"Re: [PATCH 05/27] add: ensure full index","fromName":"Matheus Tavares Bernardino","fromEmail":"matheus.bernardino@usp.br","sentAt":"2021-03-17T20:35:17Z","receivedAt":"2021-03-17T20:36:27Z","isPatch":true,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"On Wed, Mar 17, 2021 at 2:36 PM Elijah Newren wrote:\n>\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > From: Derrick Stolee \n> >\n> > Before iterating over all cache entries, ensure that a sparse index is\n> > expanded to a full index to avoid unexpected behavior.\n> >\n> > Signed-off-by: Derrick Stolee \n> > ---\n> > builtin/add.c | 1 +\n> > 1 file changed, 1 insertion(+)\n> >\n> > diff --git a/builtin/add.c b/builtin/add.c\n> > index ea762a41e3a2..ab2ef4a63530 100644\n> > --- a/builtin/add.c\n> > +++ b/builtin/add.c\n> > @@ -141,6 +141,7 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n> > {\n> > int i, retval = 0;\n> >\n> > + ensure_full_index(&the_index);\n> > for (i = 0; i < active_nr; i++) {\n> > struct cache_entry *ce = active_cache[i];\n>\n> I'm not so sure about this one; from git-add(1):\n>\n> --renormalize\n> Apply the \"clean\" process freshly to all tracked files to forcibly\n> add them again to the index. This is useful after changing\n> core.autocrlf configuration or the text attribute in order to\n> correct files added with wrong CRLF/LF line endings. This option\n> implies -u.\n>\n> ... to \"all tracked files\". Should that be interpreted as all files\n> within the sparsity paths, especially considering that we're trying to\n> enable folks to work within the set of files of interest to them?\n\nI had the same question when working on the add/rm sparse-checkout\nseries. As seen at [1], --renormalize and --chmod are, currently, the\nonly options in git-add that do not honor the sparsity patterns and do\nupdate SKIP_WORKTREE paths too.\n\nBut is this by design or possibly just an oversight? In my series I\nended up making both options honor the sparsity rules, with a warning\nwhen requested to update any sparse path. (If we are going with this\napproach, perhaps should we also amend the docs to remove any\nambiguity as for what \"all tracked files\" means in this case?)\n\n> The code below already has the following check below:\n>\n> if (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n> continue; /* do not touch non blobs */\n>\n> suggesting that it'd correctly skip over the sparse directories, so I\n> think this one just isn't needed.\n\nBut if sparse index is not enabled, it does not skip over the sparse\nfiles, right? So isn't the ensure_full_index() call here (and in\nchmod_pathspec()) important to be consistent with the case when sparse\nindex is not in use? Or maybe Stolee could rebase this on top of my\nseries, where both these places already skip the sparse paths?\n(Assuming that's the behavior we are looking for. If not, I can also\nfix my series.)\n\n[1]: https://lore.kernel.org/git/d93c3f51465a3e409819db63bd81802e7ef20ea9.1615588108.git.matheus.bernardino@usp.br/\n"},{"id":"419554","messageId":"862197d9-ca71-f682-f03f-62e45ea921ee@gmail.com","threadId":"55334","inReplyTo":"CAHd-oW4DMAjQ_Gjbzkx6csFRAK1iKHsTfuJLniOjWehwrnHdVQ@mail.gmail.com","subject":"Re: [PATCH 05/27] add: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-17T20:55:33Z","receivedAt":"2021-03-17T20:56:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/17/2021 4:35 PM, Matheus Tavares Bernardino wrote:\n> On Wed, Mar 17, 2021 at 2:36 PM Elijah Newren wrote:\n>>\n>> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n>> wrote:>> I'm not so sure about this one; from git-add(1):\n>>\n>> --renormalize\n>> Apply the \"clean\" process freshly to all tracked files to forcibly\n>> add them again to the index. This is useful after changing\n>> core.autocrlf configuration or the text attribute in order to\n>> correct files added with wrong CRLF/LF line endings. This option\n>> implies -u.\n>>\n>> ... to \"all tracked files\". Should that be interpreted as all files\n>> within the sparsity paths, especially considering that we're trying to\n>> enable folks to work within the set of files of interest to them?\n\nI wrote in reply to another similar comment that I'm being overly\ncautious in creating these protections. When I can come back later\nwith careful tests, we can ensure that everything behaves properly.\n\n> I had the same question when working on the add/rm sparse-checkout\n> series. As seen at [1], --renormalize and --chmod are, currently, the\n> only options in git-add that do not honor the sparsity patterns and do\n> update SKIP_WORKTREE paths too.\n> \n> But is this by design or possibly just an oversight? In my series I\n> ended up making both options honor the sparsity rules, with a warning\n> when requested to update any sparse path. (If we are going with this\n> approach, perhaps should we also amend the docs to remove any\n> ambiguity as for what \"all tracked files\" means in this case?)\n\nI'd be happy to see a resolution here, and it can happen in parallel\nto what I'm doing here.\n\n>> The code below already has the following check below:\n>>\n>> if (!S_ISREG(ce->ce_mode) && !S_ISLNK(ce->ce_mode))\n>> continue; /* do not touch non blobs */\n>>\n>> suggesting that it'd correctly skip over the sparse directories, so I\n>> think this one just isn't needed.\n> \n> But if sparse index is not enabled, it does not skip over the sparse\n> files, right? So isn't the ensure_full_index() call here (and in\n> chmod_pathspec()) important to be consistent with the case when sparse\n> index is not in use?\n\nThe tests I am adding in t1092-sparse-checkout-compatibility.sh are\nfocused on ensuring the same behavior across sparse-checkouts with\nand without the sparse-index, and also a full checkout (when possible).\n\nSince the sparse-index requires cone-mode sparse-checkout (currently),\nand the sparse files to be within a directory (or no index change\nhappens), the tests you created in t3705-add-sparse-checkout.sh don't\nseem to apply. I'll need to find some important scenarios to duplicate\nin t1092 without the full depth of t3705.\n\n> Or maybe Stolee could rebase this on top of my\n> series, where both these places already skip the sparse paths?\n> (Assuming that's the behavior we are looking for. If not, I can also> fix my series.)\n\nI think they can proceed in parallel and then continue more carefully\nin the future. The series _after_ this one makes 'git status' and\n'git add' work with the sparse-index, so at that point we will be\nremoving these protections at the right time, with tests. In addition,\nthose tests will ensure that we don't expand the sparse-index unless\nabsolutely necessary.\n\nIf that work collides with your approach, then I'll rebase onto a\nmerge of that topic and this one.\n\n> [1]: https://lore.kernel.org/git/d93c3f51465a3e409819db63bd81802e7ef20ea9.1615588108.git.matheus.bernardino@usp.br/\n \n(I will go take a look over here. I've been ignoring the thread for\ntoo long.)\n\nThanks,\n-Stolee\n"},{"id":"419572","messageId":"CABPp-BH-c8gzrkOFFNb=8b8R+X+VRXsziKoE_RtcR4mh6zjR4g@mail.gmail.com","threadId":"55334","inReplyTo":"08ffff48-7b9c-7113-1a5a-557f3efff26f@gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-17T21:10:37Z","receivedAt":"2021-03-17T21:11:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 17, 2021 at 1:05 PM Derrick Stolee wrote:\n>\n> On 3/17/2021 1:50 PM, Elijah Newren wrote:\n> > On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> > wrote:\n> > With the caveat in the commit message, this change looks okay, but\n> > checkout-index may be buggy regardless of the presence of\n> > ensure_full_index(). If ensure_full_index() really is needed here\n> > because it needs to operate on all SKIP_WORKTREE paths and not just\n> > leading directories, that's because it's writing all those\n> > SKIP_WORKTREE entries to the working tree. When it writes them to the\n> > working tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\n> > a bit of a pickle...\n>\n> Perhaps I'm unclear in my intentions with this series: _every_\n> insertion of ensure_full_index() is intended to be audited with\n> tests in the future. Some might need behavior change, and others\n> will not. In this series, I'm just putting in the protections so\n> we don't accidentally trigger unexpected behavior.\n\nI think this may be part of my qualms -- what do you mean by not\naccidentally triggering unexpected behavior? In particular, does your\nstatement imply that whatever behavior you get after putting in\nensure_full_index() is \"expected\"? I think I'm reading that\nimplication into it, and objecting that the behavior with the\nensure_full_index() still isn't expected. You've only removed a\ncertain class of unexpected behavior, namely code that wasn't written\nto expect tree entries that suddenly gets them. You haven't handled\nthe class of \"user wants to work with a subset of files, why are all\nthese unrelated files being munged/updated/computed/shown/etc.\"\nunexpected behavior.\n\nI'm worrying that expectations are being set up such that working with\njust a small section of the code will be unusably hard. There may be\nseveral commands/flags where it could make sense to operate on either\n(a) all files in the repo or (b) just on files within your sparse\npaths. If, though, folks interpret operate-on-all-files as the\n\"normal\" mode (and history suggests they will), then people start\nadding all kinds of --no-do-this-sparsely flags to each command, and\nthen users who want sparse operation have to remember to type such a\nflag with each and every command they ever run -- despite having taken\nat least three steps already to get a sparse-index.\n\nI believe the extended discussions (for _months_!) on just grep & rm,\nplus watching a --sparse patch being floated just in the last day for\nls-files suggest to me that this is a _very_ likely outcome and I'm\nworried about it.\n\n> Since tests take time to write and review, I was hoping that these\n> insertions were minimal enough to get us to a safe place where we\n> can remove the guards carefully.\n>\n> So with that in mind...\n>\n> > Might be nice to add a\n> > /* TODO: audit if this is needed; if it is, we may have other bugs... */\n> > or something like that. But then again, perhaps you're considering\n> > all uses of ensure_full_index() to be need-to-be-reaudited codepaths?\n> > If so, and we determine we really do need one and want to keep it\n> > indefinitely, will we mark those with a comment about why it's\n> > considered correct?\n> >\n> > I just want a way to know what still needs to be audited and what\n> > doesn't without doing a lot of history spelunking...\n>\n> ...every insertion \"needs to be audited\" in the future. That's a\n> big part of the next \"phases\" in the implementation plan.\n>\n> As you suggest, it might be a good idea to add a comment to every\n> insertion, to mark it as un-audited, such as:\n>\n> /* TODO: test if ensure_full_index() is necessary */\n>\n> We can come back later to delete the comment if it truly is\n> necessary (and add tests to guarantee correct behavior). We can\n> also remove the comment _and_ the call by modifying the loop\n> behavior to do the right thing in some cases.\n\nIf it's \"needs to be audited for both performance reasons (can we\noperate on fewer entries as an invisible doesn't-change-results\noptimization) and correctness reasons (should we operate on fewer\nentries and given a modified result within a sparse-index because\nusers would expect that, but maybe provide a special flag for the\nusers who want to operate on all files in the repo)\" and there's also\nan agreement that either audited or unaudited ones will be marked (or\nboth), then great, I'm happy. If not, can we discuss which part of my\nperformance/correctness/marking we aren't in agreement on?\n\n\nThanks,\nElijah\n"},{"id":"419599","messageId":"5c886fd7-710d-ac4a-c63a-c1d000c29126@gmail.com","threadId":"55334","inReplyTo":"CABPp-BH-c8gzrkOFFNb=8b8R+X+VRXsziKoE_RtcR4mh6zjR4g@mail.gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-17T21:33:55Z","receivedAt":"2021-03-17T21:35:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/17/2021 5:10 PM, Elijah Newren wrote:\n> On Wed, Mar 17, 2021 at 1:05 PM Derrick Stolee wrote:\n>>\n>> On 3/17/2021 1:50 PM, Elijah Newren wrote:\n>>> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n>>> wrote:\n>>> With the caveat in the commit message, this change looks okay, but\n>>> checkout-index may be buggy regardless of the presence of\n>>> ensure_full_index(). If ensure_full_index() really is needed here\n>>> because it needs to operate on all SKIP_WORKTREE paths and not just\n>>> leading directories, that's because it's writing all those\n>>> SKIP_WORKTREE entries to the working tree. When it writes them to the\n>>> working tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\n>>> a bit of a pickle...\n>>\n>> Perhaps I'm unclear in my intentions with this series: _every_\n>> insertion of ensure_full_index() is intended to be audited with\n>> tests in the future. Some might need behavior change, and others\n>> will not. In this series, I'm just putting in the protections so\n>> we don't accidentally trigger unexpected behavior.\n> \n> I think this may be part of my qualms -- what do you mean by not\n> accidentally triggering unexpected behavior? In particular, does your\n> statement imply that whatever behavior you get after putting in\n> ensure_full_index() is \"expected\"? I think I'm reading that\n> implication into it, and objecting that the behavior with the\n> ensure_full_index() still isn't expected. You've only removed a\n> certain class of unexpected behavior, namely code that wasn't written\n> to expect tree entries that suddenly gets them. You haven't handled\n> the class of \"user wants to work with a subset of files, why are all\n> these unrelated files being munged/updated/computed/shown/etc.\"\n> unexpected behavior.\n\nMy intention is to ensure that (at this moment) choosing to use\nthe on-disk sparse-index format does not alter Git's end-to-end\nbehavior.\n\nI want to avoid as much as possible a state where enabling the\nsparse-index can start changing how Git commands behave, perhaps\nin destructive ways.\n\nBy adding these checks, we ensure the in-memory data structure\nmatches whatever a full index would have created, and then the\nbehavior matches what Git would do there. It might not be the\n\"correct\" behavior, but it is _consistent_.\n\n> I'm worrying that expectations are being set up such that working with\n> just a small section of the code will be unusably hard. There may be\n> several commands/flags where it could make sense to operate on either\n> (a) all files in the repo or (b) just on files within your sparse\n> paths. If, though, folks interpret operate-on-all-files as the\n> \"normal\" mode (and history suggests they will), then people start\n> adding all kinds of --no-do-this-sparsely flags to each command, and\n> then users who want sparse operation have to remember to type such a\n> flag with each and every command they ever run -- despite having taken\n> at least three steps already to get a sparse-index.\n> \n> I believe the extended discussions (for _months_!) on just grep & rm,\n> plus watching a --sparse patch being floated just in the last day for\n> ls-files suggest to me that this is a _very_ likely outcome and I'm\n> worried about it.\n\nIt's these behavior changes that I would like to delay as much as\npossible and focus on the format and making commands fast that don't\nneed a change in behavior.\n\n(Yes, there will be exceptions, like when \"git add\" specifically\nadds a file that is in a directory that should be out of the cone,\nbut the user added it anyway. Atypical behavior like that can be\nslow for now.)\n\n>> Since tests take time to write and review, I was hoping that these\n>> insertions were minimal enough to get us to a safe place where we\n>> can remove the guards carefully.\n>>\n>> So with that in mind...\n>>\n>>> Might be nice to add a\n>>> /* TODO: audit if this is needed; if it is, we may have other bugs... */\n>>> or something like that. But then again, perhaps you're considering\n>>> all uses of ensure_full_index() to be need-to-be-reaudited codepaths?\n>>> If so, and we determine we really do need one and want to keep it\n>>> indefinitely, will we mark those with a comment about why it's\n>>> considered correct?\n>>>\n>>> I just want a way to know what still needs to be audited and what\n>>> doesn't without doing a lot of history spelunking...\n>>\n>> ...every insertion \"needs to be audited\" in the future. That's a\n>> big part of the next \"phases\" in the implementation plan.\n>>\n>> As you suggest, it might be a good idea to add a comment to every\n>> insertion, to mark it as un-audited, such as:\n>>\n>> /* TODO: test if ensure_full_index() is necessary */\n>>\n>> We can come back later to delete the comment if it truly is\n>> necessary (and add tests to guarantee correct behavior). We can\n>> also remove the comment _and_ the call by modifying the loop\n>> behavior to do the right thing in some cases.\n> \n> If it's \"needs to be audited for both performance reasons (can we\n> operate on fewer entries as an invisible doesn't-change-results\n> optimization) and correctness reasons (should we operate on fewer\n> entries and given a modified result within a sparse-index because\n> users would expect that, but maybe provide a special flag for the\n> users who want to operate on all files in the repo)\" and there's also\n> an agreement that either audited or unaudited ones will be marked (or\n> both), then great, I'm happy. If not, can we discuss which part of my\n> performance/correctness/marking we aren't in agreement on?\n\nI will mark all of the ones I'm inserting. My hope is to eventually\nremove it entirely except for when disabling the sparse-index. That\nis likely too far out to really hope for, but it is the direction I\nam trying to go.\n\nAs I indicate that we should carefully test each of these instances\nwhere ensure_full_index() _might_ be necessary before removing them,\nit is even more important to test the scenarios where the behavior\nchanges from a full index with sparse-checkout. Preferably, we just\nchange the behavior under sparse-checkout and then the sparse-index\ncan match that (see \"test_sparse_match\" in t1092).\n\nThanks,\n-Stolee\n"},{"id":"419606","messageId":"CABPp-BFaNOY8xPCTBzU+5p7FtWuC0kazh6DpQ1oKYVJNKw+UXg@mail.gmail.com","threadId":"55334","inReplyTo":"5c886fd7-710d-ac4a-c63a-c1d000c29126@gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-17T22:36:46Z","receivedAt":"2021-03-17T22:37:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 17, 2021 at 2:33 PM Derrick Stolee wrote:\n>\n> On 3/17/2021 5:10 PM, Elijah Newren wrote:\n> > On Wed, Mar 17, 2021 at 1:05 PM Derrick Stolee wrote:\n> >>\n> >> On 3/17/2021 1:50 PM, Elijah Newren wrote:\n> >>> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> >>> wrote:\n> >>> With the caveat in the commit message, this change looks okay, but\n> >>> checkout-index may be buggy regardless of the presence of\n> >>> ensure_full_index(). If ensure_full_index() really is needed here\n> >>> because it needs to operate on all SKIP_WORKTREE paths and not just\n> >>> leading directories, that's because it's writing all those\n> >>> SKIP_WORKTREE entries to the working tree. When it writes them to the\n> >>> working tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\n> >>> a bit of a pickle...\n> >>\n> >> Perhaps I'm unclear in my intentions with this series: _every_\n> >> insertion of ensure_full_index() is intended to be audited with\n> >> tests in the future. Some might need behavior change, and others\n> >> will not. In this series, I'm just putting in the protections so\n> >> we don't accidentally trigger unexpected behavior.\n> >\n> > I think this may be part of my qualms -- what do you mean by not\n> > accidentally triggering unexpected behavior? In particular, does your\n> > statement imply that whatever behavior you get after putting in\n> > ensure_full_index() is \"expected\"? I think I'm reading that\n> > implication into it, and objecting that the behavior with the\n> > ensure_full_index() still isn't expected. You've only removed a\n> > certain class of unexpected behavior, namely code that wasn't written\n> > to expect tree entries that suddenly gets them. You haven't handled\n> > the class of \"user wants to work with a subset of files, why are all\n> > these unrelated files being munged/updated/computed/shown/etc.\"\n> > unexpected behavior.\n>\n> My intention is to ensure that (at this moment) choosing to use\n> the on-disk sparse-index format does not alter Git's end-to-end\n> behavior.\n>\n> I want to avoid as much as possible a state where enabling the\n> sparse-index can start changing how Git commands behave, perhaps\n> in destructive ways.\n>\n> By adding these checks, we ensure the in-memory data structure\n> matches whatever a full index would have created, and then the\n> behavior matches what Git would do there. It might not be the\n> \"correct\" behavior, but it is _consistent_.\n\nThat sounds good. Could this be included in\nDocumentation/technical/sparse-index.txt? There's only an oblique\nreference to it when talking about grep and rm, which incidentally\nwere already updated by Matheus. Perhaps also a reference to the\nwarning in Documentation/git-sparse-checkout.txt would be worthwhile.\n\n> > I'm worrying that expectations are being set up such that working with\n> > just a small section of the code will be unusably hard. There may be\n> > several commands/flags where it could make sense to operate on either\n> > (a) all files in the repo or (b) just on files within your sparse\n> > paths. If, though, folks interpret operate-on-all-files as the\n> > \"normal\" mode (and history suggests they will), then people start\n> > adding all kinds of --no-do-this-sparsely flags to each command, and\n> > then users who want sparse operation have to remember to type such a\n> > flag with each and every command they ever run -- despite having taken\n> > at least three steps already to get a sparse-index.\n> >\n> > I believe the extended discussions (for _months_!) on just grep & rm,\n> > plus watching a --sparse patch being floated just in the last day for\n> > ls-files suggest to me that this is a _very_ likely outcome and I'm\n> > worried about it.\n>\n> It's these behavior changes that I would like to delay as much as\n> possible and focus on the format and making commands fast that don't\n> need a change in behavior.\n\nDelaying sounds great, just so long as that delay doesn't also cement\nthe behavior and confuse consistency for correctness.\n\nI still think we don't have correct behavior for sparse-checkouts in\nmany cases (I mean, when \"git reset --hard\" throws errors about not\nremoving files and then removes them, we've obviously got some\nproblems), but we had a decade long cementing of sorts with\nSKIP_WORKTREE and now a year-or-two long cementing since\nsparse-checkout was introduced and we never went through and cleaned\nup the commands. We should at some point, especially since we put the\nhuge scary warning in Documenation/git-sparse-checkout.txt expressly\nfor this purpose.\n\n(I would have started this sooner, but trying to feed merge-ort and\nkeep up with patch review already keeps me at less time on non-git\nprojects than I think is expected for me. Once merge-ort is done...)\n\n> (Yes, there will be exceptions, like when \"git add\" specifically\n> adds a file that is in a directory that should be out of the cone,\n> but the user added it anyway. Atypical behavior like that can be\n> slow for now.)\n\nI agree that there will be exceptions where we can't make the behavior\nbe fast, but I disagree with that specific example. \"git add\" should\njust give a warning message and not add any file outside the cone, for\nboth sparse-index and sparse-checkout. That can be done quickly.\n\nI know you want to delay the discussion of behavior fixes for specific\ncommands somewhat, but this particular discussion has already been\nongoing for about 4-12 months now (started at [1] and [2]) and it's\nreached a point where I've put my Reviewed-by on it; see [3] for the\ngit-add piece specifically.\n\n[1] https://lore.kernel.org/git/9f2135f90ffea7f4ccb226f506bf554deab324cc.1605205427.git.matheus.bernardino@usp.br/\n[2] https://lore.kernel.org/git/0b9b4c4b414a571877163667694afa3053bf8890.1585027716.git.matheus.bernardino@usp.br/\n[3] https://lore.kernel.org/git/66d5c71182274c78e1fcfe84e77deb17e4f0d7e6.1615588109.git.matheus.bernardino@usp.br/\n\n> >> Since tests take time to write and review, I was hoping that these\n> >> insertions were minimal enough to get us to a safe place where we\n> >> can remove the guards carefully.\n> >>\n> >> So with that in mind...\n> >>\n> >>> Might be nice to add a\n> >>> /* TODO: audit if this is needed; if it is, we may have other bugs... */\n> >>> or something like that. But then again, perhaps you're considering\n> >>> all uses of ensure_full_index() to be need-to-be-reaudited codepaths?\n> >>> If so, and we determine we really do need one and want to keep it\n> >>> indefinitely, will we mark those with a comment about why it's\n> >>> considered correct?\n> >>>\n> >>> I just want a way to know what still needs to be audited and what\n> >>> doesn't without doing a lot of history spelunking...\n> >>\n> >> ...every insertion \"needs to be audited\" in the future. That's a\n> >> big part of the next \"phases\" in the implementation plan.\n> >>\n> >> As you suggest, it might be a good idea to add a comment to every\n> >> insertion, to mark it as un-audited, such as:\n> >>\n> >> /* TODO: test if ensure_full_index() is necessary */\n> >>\n> >> We can come back later to delete the comment if it truly is\n> >> necessary (and add tests to guarantee correct behavior). We can\n> >> also remove the comment _and_ the call by modifying the loop\n> >> behavior to do the right thing in some cases.\n> >\n> > If it's \"needs to be audited for both performance reasons (can we\n> > operate on fewer entries as an invisible doesn't-change-results\n> > optimization) and correctness reasons (should we operate on fewer\n> > entries and given a modified result within a sparse-index because\n> > users would expect that, but maybe provide a special flag for the\n> > users who want to operate on all files in the repo)\" and there's also\n> > an agreement that either audited or unaudited ones will be marked (or\n> > both), then great, I'm happy. If not, can we discuss which part of my\n> > performance/correctness/marking we aren't in agreement on?\n>\n> I will mark all of the ones I'm inserting. My hope is to eventually\n> remove it entirely except for when disabling the sparse-index. That\n> is likely too far out to really hope for, but it is the direction I\n> am trying to go.\n>\n> As I indicate that we should carefully test each of these instances\n> where ensure_full_index() _might_ be necessary before removing them,\n> it is even more important to test the scenarios where the behavior\n> changes from a full index with sparse-checkout. Preferably, we just\n> change the behavior under sparse-checkout and then the sparse-index\n> can match that (see \"test_sparse_match\" in t1092).\n\nMakes sense. I agree that it'd be nice to have the two generally\nmatch, though I think we should be open to there being special cases\nthat differ. The only one I can think of right now is `git ls-files`\n(list the entries in the index) -- since there are tree entries in a\nsparse-index, ls-files would naturally show the tree entries.\nHowever, we can discuss that and any other cases -- if there are any\n-- later.\n"},{"id":"419611","messageId":"ba59413d-d32e-9e03-6c9e-c4b8a93ad3e5@gmail.com","threadId":"55334","inReplyTo":"CABPp-BFaNOY8xPCTBzU+5p7FtWuC0kazh6DpQ1oKYVJNKw+UXg@mail.gmail.com","subject":"Re: [PATCH 06/27] checkout-index: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-18T01:17:37Z","receivedAt":"2021-03-18T01:18:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/17/2021 6:36 PM, Elijah Newren wrote:\n> On Wed, Mar 17, 2021 at 2:33 PM Derrick Stolee wrote:\n>>\n>> On 3/17/2021 5:10 PM, Elijah Newren wrote:\n>>> On Wed, Mar 17, 2021 at 1:05 PM Derrick Stolee wrote:\n>>>>\n>>>> On 3/17/2021 1:50 PM, Elijah Newren wrote:\n>>>>> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n>>>>> wrote:\n>>>>> With the caveat in the commit message, this change looks okay, but\n>>>>> checkout-index may be buggy regardless of the presence of\n>>>>> ensure_full_index(). If ensure_full_index() really is needed here\n>>>>> because it needs to operate on all SKIP_WORKTREE paths and not just\n>>>>> leading directories, that's because it's writing all those\n>>>>> SKIP_WORKTREE entries to the working tree. When it writes them to the\n>>>>> working tree, is it clearing the SKIP_WORKTREE bit? If not, we're in\n>>>>> a bit of a pickle...\n>>>>\n>>>> Perhaps I'm unclear in my intentions with this series: _every_\n>>>> insertion of ensure_full_index() is intended to be audited with\n>>>> tests in the future. Some might need behavior change, and others\n>>>> will not. In this series, I'm just putting in the protections so\n>>>> we don't accidentally trigger unexpected behavior.\n>>>\n>>> I think this may be part of my qualms -- what do you mean by not\n>>> accidentally triggering unexpected behavior? In particular, does your\n>>> statement imply that whatever behavior you get after putting in\n>>> ensure_full_index() is \"expected\"? I think I'm reading that\n>>> implication into it, and objecting that the behavior with the\n>>> ensure_full_index() still isn't expected. You've only removed a\n>>> certain class of unexpected behavior, namely code that wasn't written\n>>> to expect tree entries that suddenly gets them. You haven't handled\n>>> the class of \"user wants to work with a subset of files, why are all\n>>> these unrelated files being munged/updated/computed/shown/etc.\"\n>>> unexpected behavior.\n>>\n>> My intention is to ensure that (at this moment) choosing to use\n>> the on-disk sparse-index format does not alter Git's end-to-end\n>> behavior.\n>>\n>> I want to avoid as much as possible a state where enabling the\n>> sparse-index can start changing how Git commands behave, perhaps\n>> in destructive ways.\n>>\n>> By adding these checks, we ensure the in-memory data structure\n>> matches whatever a full index would have created, and then the\n>> behavior matches what Git would do there. It might not be the\n>> \"correct\" behavior, but it is _consistent_.\n> \n> That sounds good. Could this be included in\n> Documentation/technical/sparse-index.txt? There's only an oblique\n> reference to it when talking about grep and rm, which incidentally\n> were already updated by Matheus. Perhaps also a reference to the\n> warning in Documentation/git-sparse-checkout.txt would be worthwhile.\n\nAs I was thinking about how to make this more clear, I realized\nthat an update to that doc early in this series would be wise.\nThanks.\n\n>>> I'm worrying that expectations are being set up such that working with\n>>> just a small section of the code will be unusably hard. There may be\n>>> several commands/flags where it could make sense to operate on either\n>>> (a) all files in the repo or (b) just on files within your sparse\n>>> paths. If, though, folks interpret operate-on-all-files as the\n>>> \"normal\" mode (and history suggests they will), then people start\n>>> adding all kinds of --no-do-this-sparsely flags to each command, and\n>>> then users who want sparse operation have to remember to type such a\n>>> flag with each and every command they ever run -- despite having taken\n>>> at least three steps already to get a sparse-index.\n>>>\n>>> I believe the extended discussions (for _months_!) on just grep & rm,\n>>> plus watching a --sparse patch being floated just in the last day for\n>>> ls-files suggest to me that this is a _very_ likely outcome and I'm\n>>> worried about it.\n>>\n>> It's these behavior changes that I would like to delay as much as\n>> possible and focus on the format and making commands fast that don't\n>> need a change in behavior.\n> \n> Delaying sounds great, just so long as that delay doesn't also cement\n> the behavior and confuse consistency for correctness.\n> \n> I still think we don't have correct behavior for sparse-checkouts in\n> many cases (I mean, when \"git reset --hard\" throws errors about not\n> removing files and then removes them, we've obviously got some\n> problems), but we had a decade long cementing of sorts with\n> SKIP_WORKTREE and now a year-or-two long cementing since\n> sparse-checkout was introduced and we never went through and cleaned\n> up the commands. We should at some point, especially since we put the\n> huge scary warning in Documenation/git-sparse-checkout.txt expressly\n> for this purpose.\n> \n> (I would have started this sooner, but trying to feed merge-ort and\n> keep up with patch review already keeps me at less time on non-git\n> projects than I think is expected for me. Once merge-ort is done...)\n> \n>> (Yes, there will be exceptions, like when \"git add\" specifically\n>> adds a file that is in a directory that should be out of the cone,\n>> but the user added it anyway. Atypical behavior like that can be\n>> slow for now.)\n> \n> I agree that there will be exceptions where we can't make the behavior\n> be fast, but I disagree with that specific example. \"git add\" should\n> just give a warning message and not add any file outside the cone, for\n> both sparse-index and sparse-checkout. That can be done quickly.\n\nThat would be great! I would love behavior changes that make the\nperformance work I'm interested in easier.\n\n> I know you want to delay the discussion of behavior fixes for specific\n> commands somewhat, but this particular discussion has already been\n> ongoing for about 4-12 months now (started at [1] and [2]) and it's\n> reached a point where I've put my Reviewed-by on it; see [3] for the\n> git-add piece specifically.\n> \n> [1] https://lore.kernel.org/git/9f2135f90ffea7f4ccb226f506bf554deab324cc.1605205427.git.matheus.bernardino@usp.br/\n> [2] https://lore.kernel.org/git/0b9b4c4b414a571877163667694afa3053bf8890.1585027716.git.matheus.bernardino@usp.br/\n> [3] https://lore.kernel.org/git/66d5c71182274c78e1fcfe84e77deb17e4f0d7e6.1615588109.git.matheus.bernardino@usp.br/\n\nI'm in full support of these ideas to change the behavior, when\npossible. I would love to see that those changes make it really\neasy to integrate the sparse-index into the commands.\n\n>>>> Since tests take time to write and review, I was hoping that these\n>>>> insertions were minimal enough to get us to a safe place where we\n>>>> can remove the guards carefully.\n>>>>\n>>>> So with that in mind...\n>>>>\n>>>>> Might be nice to add a\n>>>>> /* TODO: audit if this is needed; if it is, we may have other bugs... */\n>>>>> or something like that. But then again, perhaps you're considering\n>>>>> all uses of ensure_full_index() to be need-to-be-reaudited codepaths?\n>>>>> If so, and we determine we really do need one and want to keep it\n>>>>> indefinitely, will we mark those with a comment about why it's\n>>>>> considered correct?\n>>>>>\n>>>>> I just want a way to know what still needs to be audited and what\n>>>>> doesn't without doing a lot of history spelunking...\n>>>>\n>>>> ...every insertion \"needs to be audited\" in the future. That's a\n>>>> big part of the next \"phases\" in the implementation plan.\n>>>>\n>>>> As you suggest, it might be a good idea to add a comment to every\n>>>> insertion, to mark it as un-audited, such as:\n>>>>\n>>>> /* TODO: test if ensure_full_index() is necessary */\n>>>>\n>>>> We can come back later to delete the comment if it truly is\n>>>> necessary (and add tests to guarantee correct behavior). We can\n>>>> also remove the comment _and_ the call by modifying the loop\n>>>> behavior to do the right thing in some cases.\n>>>\n>>> If it's \"needs to be audited for both performance reasons (can we\n>>> operate on fewer entries as an invisible doesn't-change-results\n>>> optimization) and correctness reasons (should we operate on fewer\n>>> entries and given a modified result within a sparse-index because\n>>> users would expect that, but maybe provide a special flag for the\n>>> users who want to operate on all files in the repo)\" and there's also\n>>> an agreement that either audited or unaudited ones will be marked (or\n>>> both), then great, I'm happy. If not, can we discuss which part of my\n>>> performance/correctness/marking we aren't in agreement on?\n>>\n>> I will mark all of the ones I'm inserting. My hope is to eventually\n>> remove it entirely except for when disabling the sparse-index. That\n>> is likely too far out to really hope for, but it is the direction I\n>> am trying to go.\n>>\n>> As I indicate that we should carefully test each of these instances\n>> where ensure_full_index() _might_ be necessary before removing them,\n>> it is even more important to test the scenarios where the behavior\n>> changes from a full index with sparse-checkout. Preferably, we just\n>> change the behavior under sparse-checkout and then the sparse-index\n>> can match that (see \"test_sparse_match\" in t1092).\n> \n> Makes sense. I agree that it'd be nice to have the two generally\n> match, though I think we should be open to there being special cases\n> that differ. The only one I can think of right now is `git ls-files`\n> (list the entries in the index) -- since there are tree entries in a\n> sparse-index, ls-files would naturally show the tree entries.\n> However, we can discuss that and any other cases -- if there are any\n> -- later.\n\nHere, I at least want to make it very clear that the change is\nhappening and include updates to the documentation. This is a\ncase where it is unclear what the \"promise\" is to _external_\ntools that depend on ls-files (somehow). What is their intended\nuse of ls-files, and how do those expectations change when in a\nsparse-checkout?\n\nIt's questions like this where I at least want to be making a\nclear change that motivates why a change in behavior is merited,\nthen test that behavior to demonstrate that we expect that to be\nthe same in perpetuity.\n\nThanks,\n-Stolee\n"},{"id":"419614","messageId":"CABPp-BF4YVSNB1c=y11MZfTDAu=KHryNb5JQZ1kKRpigPALPxw@mail.gmail.com","threadId":"55334","inReplyTo":"92ccd31dd343c20bad4a6b8d89c50559209e7236.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 15/27] sparse-checkout: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-18T05:22:54Z","receivedAt":"2021-03-18T05:24:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> When adjusting the sparse-checkout definition, ensure that a sparse\n> index is expanded to a full one to avoid unexpected behavior. This is a\n> top candidate for later integration with the sparse-index, but requires\n> careful testing.\n\n\nI was going to comment on most of these collectively, but your commit\nmessage is completely divorced from the patch; it appears you either\nplace the wrong commit message here or copied it from the one for a\nsparse-checkout. The code modifies stash, though.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/stash.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/builtin/stash.c b/builtin/stash.c\n> index ba774cce674f..b3d4fe814b6b 100644\n> --- a/builtin/stash.c\n> +++ b/builtin/stash.c\n> @@ -1350,6 +1350,7 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n> int i;\n> char *ps_matched = xcalloc(ps->nr, 1);\n>\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++)\n> ce_path_match(&the_index, active_cache[i], ps,\n> ps_matched);\n> --\n> gitgitgadget\n"},{"id":"419615","messageId":"CABPp-BHsSpQieOGFPwXJ63qy1JhWcwu-bSvv1w7-yTZ6T4fx9Q@mail.gmail.com","threadId":"55334","inReplyTo":"45bbed6150a2a9f9e9446edc9a613f027da73957.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 17/27] diff-lib: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-18T05:24:04Z","receivedAt":"2021-03-18T05:25:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before iterating over all cache entries, ensure that a sparse index is\n> expanded to a full index to avoid unexpected behavior.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> diff-lib.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/diff-lib.c b/diff-lib.c\n> index b73cc1859a49..41d6fcec1a81 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -102,6 +102,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n>\n> if (diff_unmerged_stage < 0)\n> diff_unmerged_stage = 2;\n> +\n> entries = istate->cache_nr;\n> for (i = 0; i < entries; i++) {\n> unsigned int oldmode, newmode;\n\nI don't think adding a blank newline will ensure the index is expanded. ;-)\n"},{"id":"419616","messageId":"CABPp-BF93UaU7JnwxQiEp2wf1jffLfQax+z4_8Q3-6DxXPUgRg@mail.gmail.com","threadId":"55334","inReplyTo":"fb4c7038b7468c77c4d3c5181f0cb0de85d1149f.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 20/27] merge-ort: ensure full index","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-18T05:31:41Z","receivedAt":"2021-03-18T05:33:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Before iterating over all cache entries, ensure that a sparse index is\n> expanded to a full one to avoid unexpected behavior.\n>\n> This is a top candidate for updating later with a proper integration\n> with the sparse index, since we will likely enable the ORT strategy by\n> default when the sparse index is enabled. This appears to be the only\n\ns/appears to be/is/ :-)\n\n> place where the ORT strategy interacts with the index in such a global\n> way, so that integration should be clear once the ORT strategy and the\n> sparse index topics stabilize.\n\nRight, there is one more patch that will touch this function -- patch\n7 from the series that marks merge-ort stable over here:\nhttps://lore.kernel.org/git/pull.905.v2.git.1616016485.gitgitgadget@gmail.com/\n\nWhile I have more optimizations for merge-ort, none of them will touch\nthis function.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> merge-ort.c | 1 +\n> 1 file changed, 1 insertion(+)\n>\n> diff --git a/merge-ort.c b/merge-ort.c\n> index 603d30c52170..9f737212555d 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -3112,6 +3112,7 @@ static int record_conflicted_index_entries(struct merge_options *opt,\n> if (strmap_empty(conflicted))\n> return 0;\n>\n> + ensure_full_index(index);\n> original_cache_nr = index->cache_nr;\n>\n> /* Put every entry from paths into plist, then sort */\n> --\n> gitgitgadget\n"},{"id":"419618","messageId":"CABPp-BE4qc1-Y+gdZCxB=o+DEF0F_dH-=-YaGWJfO_T6LC6x2Q@mail.gmail.com","threadId":"55334","inReplyTo":"CABPp-BGfkPscpro=W5vcRHD5cV6pddYpvHLLzMWjL9WLaxyu3w@mail.gmail.com","subject":"Re: [PATCH 00/27] Sparse Index: API protections","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-03-18T06:32:28Z","receivedAt":"2021-03-18T06:33:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 17, 2021 at 11:03 AM Elijah Newren wrote:\n>\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n> >\n> > The point of this series is to insert protections for the consumers of the\n> > in-memory index. We mark certain regions of code as needing a full index, so\n> > we call ensure_full_index() to expand a sparse index to a full one, if\n> > necessary. These protections are inserted file-by-file in every loop over\n> > all cache entries. Well, \"most\" loops, because some are going to be handled\n> > in the very next series so I leave them out.\n...\n> > However, after this series is complete, we now have a straight-forward plan\n> > for making commands \"sparse aware\" one-by-one:\n> >\n> > 1. Disable settings.command_requires_full_index to allow an in-memory\n> > sparse-index.\n> > 2. Run versions of that command under a debugger, breaking on\n> > ensure_full_index().\n> > 3. Examine the call stack to determine the context of that expansion, then\n> > implement the proper behavior in those locations.\n> > 4. Add tests to ensure we are checking this logic in the presence of sparse\n> > directory entries.\n>\n...\n> If I'm confused about the goals and the plans, then my reviews will\n> probably be less than helpful, so I'll suspend reading the series\n> until I understand the plan a little better.\n\nThanks for patiently responding to all my other queries. I read\nthrough more of the series today. I inserted comments on a couple\nspecific patches, but most of patches 6-25 are all along the same\ntheme.\n\nI have some overall comments on those patches in 6-25 (none of which\nneed to be addressed in this series, but are meant as hopefully\nhelpful guides about future work):\n\nadd/rm: both of these were just above loops that had a\nskip-the-SKIP_WORKTREE entries, at least once Matheus' series is\nmerged (https://lore.kernel.org/git/cover.1615588108.git.matheus.bernardino@usp.br/).\nI think the upshot is just that these become easier to convert.\n\ncheckout/commit -- I think these follow the add/rm model and likewise\nbecome easy to convert.\n\nls-files - I commented elsewhere in this thread about how I think it'd\nmake sense to have it list the entries in the index, as always.\nObviously, that'd give different output than a full index, because\nthere are different entries present in the index when using a\nsparse-index vs. a full one.\n\nHowever, all of these, as you say, can wait until later.\n\n\nI also noted that patches 26 and 27 were based on ones from the RFC\nseries that I reviewed before, and I noticed you fixed an issue or two\nI flagged there, but you also made some other changes and it's too\nlate at night for my brain to continue thinking and try to compare\nthem; I may try again tomorrow.\n\nOther than patch 26 which I'm too tired to think through right now,\nand the patches I specifically commented on, the rest of the series\nlooks good to me.\n"},{"id":"419736","messageId":"xmqqwnu2lux2.fsf@gitster.g","threadId":"55334","inReplyTo":"628e16dd3fc7be96d40a7b3201573a71ae7fbbd6.1615929436.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/27] *: remove 'const' qualifier for struct index_state","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-19T21:01:45Z","receivedAt":"2021-03-19T21:02:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Several methods specify that they take a 'struct index_state' pointer\n> with the 'const' qualifier because they intend to only query the data,\n> not change it. However, we will be introducing a step very low in the\n> method stack that might modify a sparse-index to become a full index in\n> the case that our queries venture inside a sparse-directory entry.\n>\n> This change only removes the 'const' qualifiers that are necessary for\n> the following change which will actually modify the implementation of\n> index_name_stage_pos().\n\nThis step has a bit of interaction with Matheus's \"git add/rm\" work\nin sparse checkout (mt/add-rm-in-sparse-checkout), which I believe\nis still in a bit of flux. I didn't check potential conflicts the\nremainder of the series may have with other in-flight topics.\n\nSo, I may throw review comments at the patches in this topic as if\nthey are standalone, but please do not be upset if it didn't appear\nin the 'seen' topic.\n\nThanks.\n"},{"id":"419757","messageId":"b98a1080-2481-508e-bf3e-50ea665d8e21@gmail.com","threadId":"55334","inReplyTo":"xmqqwnu2lux2.fsf@gitster.g","subject":"Re: [PATCH 01/27] *: remove 'const' qualifier for struct index_state","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-20T01:45:07Z","receivedAt":"2021-03-20T01:46:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/19/2021 5:01 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> Several methods specify that they take a 'struct index_state' pointer\n>> with the 'const' qualifier because they intend to only query the data,\n>> not change it. However, we will be introducing a step very low in the\n>> method stack that might modify a sparse-index to become a full index in\n>> the case that our queries venture inside a sparse-directory entry.\n>>\n>> This change only removes the 'const' qualifiers that are necessary for\n>> the following change which will actually modify the implementation of\n>> index_name_stage_pos().\n> \n> This step has a bit of interaction with Matheus's \"git add/rm\" work\n> in sparse checkout (mt/add-rm-in-sparse-checkout), which I believe\n> is still in a bit of flux. I didn't check potential conflicts the\n> remainder of the series may have with other in-flight topics.\n> \n> So, I may throw review comments at the patches in this topic as if\n> they are standalone, but please do not be upset if it didn't appear\n> in the 'seen' topic.\n\nThanks for the point. I need to reset my expectations about how\nmuch I am stomping on other topics in flight. I rarely work so\nwidely across the source tree.\n\nI could split this patch out on its own so it can fit into a\nnarrow window between other topics that might collide with it,\nthen rebase the rest of the series on top after things settle.\n\nThanks,\n-Stolee\n"},{"id":"419759","messageId":"xmqq8s6ilhgs.fsf@gitster.g","threadId":"55334","inReplyTo":"xmqqwnu2lux2.fsf@gitster.g","subject":"Re: [PATCH 01/27] *: remove 'const' qualifier for struct index_state","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-20T01:52:19Z","receivedAt":"2021-03-20T01:53:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> From: Derrick Stolee \n>>\n>> Several methods specify that they take a 'struct index_state' pointer\n>> with the 'const' qualifier because they intend to only query the data,\n>> not change it. However, we will be introducing a step very low in the\n>> method stack that might modify a sparse-index to become a full index in\n>> the case that our queries venture inside a sparse-directory entry.\n>>\n>> This change only removes the 'const' qualifiers that are necessary for\n>> the following change which will actually modify the implementation of\n>> index_name_stage_pos().\n>\n> This step has a bit of interaction with Matheus's \"git add/rm\" work\n> in sparse checkout (mt/add-rm-in-sparse-checkout), which I believe\n> is still in a bit of flux. I didn't check potential conflicts the\n> remainder of the series may have with other in-flight topics.\n>\n> So, I may throw review comments at the patches in this topic as if\n> they are standalone, but please do not be upset if it didn't appear\n> in the 'seen' topic.\n\nTonight's pushout will have this topic in 'seen', but I consider the\nbranch a series of trial merges (there are other first-time topics).\nThe result seems to compile for me, but other than that, I have not\nmuch confidence in conflict resolution. Please give them an extra\nset of eyeballs.\n\nThanks.\n"},{"id":"420002","messageId":"64fe05be-b4be-5aac-af1b-0db03db49cfe@gmail.com","threadId":"55334","inReplyTo":"CABPp-BF4YVSNB1c=y11MZfTDAu=KHryNb5JQZ1kKRpigPALPxw@mail.gmail.com","subject":"Re: [PATCH 15/27] sparse-checkout: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-23T13:13:40Z","receivedAt":"2021-03-23T13:14:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/18/2021 1:22 AM, Elijah Newren wrote:\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> When adjusting the sparse-checkout definition, ensure that a sparse\n>> index is expanded to a full one to avoid unexpected behavior. This is a\n>> top candidate for later integration with the sparse-index, but requires\n>> careful testing.\n> \n> \n> I was going to comment on most of these collectively, but your commit\n> message is completely divorced from the patch; it appears you either\n> place the wrong commit message here or copied it from the one for a\n> sparse-checkout. The code modifies stash, though.\n\nThanks for pointing this out. Must have been a bad squash during one\nof many rebases.\n\n-Stolee\n"},{"id":"420003","messageId":"ef818ccc-f502-b474-c126-7fd65d16db49@gmail.com","threadId":"55334","inReplyTo":"CABPp-BHsSpQieOGFPwXJ63qy1JhWcwu-bSvv1w7-yTZ6T4fx9Q@mail.gmail.com","subject":"Re: [PATCH 17/27] diff-lib: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-23T13:15:51Z","receivedAt":"2021-03-23T13:17:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/18/2021 1:24 AM, Elijah Newren wrote:\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> Before iterating over all cache entries, ensure that a sparse index is\n>> expanded to a full index to avoid unexpected behavior.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> diff-lib.c | 1 +\n>> 1 file changed, 1 insertion(+)\n>>\n>> diff --git a/diff-lib.c b/diff-lib.c\n>> index b73cc1859a49..41d6fcec1a81 100644\n>> --- a/diff-lib.c\n>> +++ b/diff-lib.c\n>> @@ -102,6 +102,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n>>\n>> if (diff_unmerged_stage < 0)\n>> diff_unmerged_stage = 2;\n>> +\n>> entries = istate->cache_nr;\n>> for (i = 0; i < entries; i++) {\n>> unsigned int oldmode, newmode;\n> \n> I don't think adding a blank newline will ensure the index is expanded. ;-)\n\nOops! This is one where I think we determined the loop doesn't\nneed the guard, but I didn't remove the newline and hence the\npatch wasn't dropped.\n\nThanks,\n-Stolee\n"},{"id":"420004","messageId":"e445a124-4715-a304-a006-08c2d1ad86ad@gmail.com","threadId":"55334","inReplyTo":"CABPp-BF93UaU7JnwxQiEp2wf1jffLfQax+z4_8Q3-6DxXPUgRg@mail.gmail.com","subject":"Re: [PATCH 20/27] merge-ort: ensure full index","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-23T13:26:07Z","receivedAt":"2021-03-23T13:27:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/18/2021 1:31 AM, Elijah Newren wrote:\n> On Tue, Mar 16, 2021 at 2:17 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee \n>>\n>> Before iterating over all cache entries, ensure that a sparse index is\n>> expanded to a full one to avoid unexpected behavior.\n>>\n>> This is a top candidate for updating later with a proper integration\n>> with the sparse index, since we will likely enable the ORT strategy by\n>> default when the sparse index is enabled. This appears to be the only\n> \n> s/appears to be/is/ :-)\n> \n>> place where the ORT strategy interacts with the index in such a global\n>> way, so that integration should be clear once the ORT strategy and the\n>> sparse index topics stabilize.\n> \n> Right, there is one more patch that will touch this function -- patch\n> 7 from the series that marks merge-ort stable over here:\n> https://lore.kernel.org/git/pull.905.v2.git.1616016485.gitgitgadget@gmail.com/\n> \n> While I have more optimizations for merge-ort, none of them will touch\n> this function.\n\nYes, I noticed that that was one of the conflicts introduced with\nthis series.\n\nI took a closer look at that patch and method, and it seems like\nit only cares about paths with SKIP_WORKTREE _and_ is conflicted.\nIf it has a conflict, then I believe we won't collapse those\nentries to a sparse-directory entry.\n\nI think I will drop this patch for now, and rely on the fact that\nI plan to enable merge-ORT directly if sparse-index is enabled.\nAt that time, I will carefully test this logic.\n\nFor now, the 'command_requires_full_index' guard will be\nsufficient to preserve behavior.\n\nThanks,\n-Stolee\n"},{"id":"420628","messageId":"9cf7775f-189d-8350-ee7a-0bd4444175aa@gmail.com","threadId":"55334","inReplyTo":"xmqq8s6ilhgs.fsf@gitster.g","subject":"Re: [PATCH 01/27] *: remove 'const' qualifier for struct index_state","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-03-30T16:53:59Z","receivedAt":"2021-03-30T16:54:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/19/2021 9:52 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> From: Derrick Stolee \n>>>\n>>> Several methods specify that they take a 'struct index_state' pointer\n>>> with the 'const' qualifier because they intend to only query the data,\n>>> not change it. However, we will be introducing a step very low in the\n>>> method stack that might modify a sparse-index to become a full index in\n>>> the case that our queries venture inside a sparse-directory entry.\n>>>\n>>> This change only removes the 'const' qualifiers that are necessary for\n>>> the following change which will actually modify the implementation of\n>>> index_name_stage_pos().\n>>\n>> This step has a bit of interaction with Matheus's \"git add/rm\" work\n>> in sparse checkout (mt/add-rm-in-sparse-checkout), which I believe\n>> is still in a bit of flux. I didn't check potential conflicts the\n>> remainder of the series may have with other in-flight topics.\n>>\n>> So, I may throw review comments at the patches in this topic as if\n>> they are standalone, but please do not be upset if it didn't appear\n>> in the 'seen' topic.\n> \n> Tonight's pushout will have this topic in 'seen', but I consider the\n> branch a series of trial merges (there are other first-time topics).\n> The result seems to compile for me, but other than that, I have not\n> much confidence in conflict resolution. Please give them an extra\n> set of eyeballs.\n\nI looked at your merge, and recreated and equivalent one in my own\nmerging. Thanks for working through all those details. I trust that\nyour rerere data will remember those changes and help avoid trouble\nwhen replaying new versions.\n\nOtherwise, I could create a new version of this patch with the\nintention of playing directly on top of mt/add-rm-in-sparse-checkout,\nthen keep future versions of this series on top of a merge of that\nseries and ds/sparse-index.\n\nI have a v2 of this series ready to send, but I'll give it a day to\nsee how things shake out on ds/sparse-index and if you have another\nsuggestion for this conflict.\n\nThanks,\n-Stolee\n"},{"id":"420768","messageId":"7484e085e3426fbcc696e8122de512ab3a846c09.1617241802.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 01/25] sparse-index: API protection strategy","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:38Z","receivedAt":"2021-04-01T01:51:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nEdit and expand the sparse-index design document with the plan for\nguarding index operations with ensure_full_index().\n\nNotably, the plan has changed to not have an expand_to_path() method in\nfavor of checking for a sparse-directory hit inside of the\nindex_path_pos() API.\n\nThe changes that follow this one will incrementally add\nensure_full_index() guards to iterations over all cache entries. Some\niterations over the cache entries are not protected due to a few\ncategories listed in the document. Since these are not being modified,\nhere is a short list of the files and methods that will not receive\nthese guards:\n\nLooking for non-zero stage:\n* builtin/add.c:chmod_pathspec()\n* builtin/merge.c:count_unmerged_entries()\n* merge-ort.c:record_conflicted_index_entries()\n* read-cache.c:unmerged_index()\n* rerere.c:check_one_conflict(), find_conflict(), rerere_remaining()\n* revision.c:prepare_show_merge()\n* sequencer.c:append_conflicts_hint()\n* wt-status.c:wt_status_collect_changes_initial()\n\nLooking for submodules:\n* builtin/submodule--helper.c:module_list_compute()\n* submodule.c: several methods\n* worktree.c:validate_no_submodules()\n\nPart of the index API:\n* name-hash.c: lazy init methods\n* preload-index.c:preload_thread(), preload_index()\n* read-cache.c: file format methods\n\nChecking for correct order of cache entries:\n* read-cache.c:check_ce_order()\n\nIgnores SKIP_WORKTREE entries or already aware:\n* unpack-trees.c:mark_new_skip_worktree()\n* wt-status.c:wt_status_check_sparse_checkout()\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/sparse-index.txt | 37 ++++++++++++++++++++++--\n 1 file changed, 35 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/sparse-index.txt b/Documentation/technical/sparse-index.txt\nindex 8d3d80804604..3b24c1a219f8 100644\n--- a/Documentation/technical/sparse-index.txt\n+++ b/Documentation/technical/sparse-index.txt\n@@ -85,8 +85,41 @@ index, as well.\n \n Next, consumers of the index will be guarded against operating on a\n sparse-index by inserting calls to `ensure_full_index()` or\n-`expand_index_to_path()`. After these guards are in place, we can begin\n-leaving sparse-directory entries in the in-memory index structure.\n+`expand_index_to_path()`. If a specific path is requested, then those will\n+be protected from within the `index_file_exists()` and `index_name_pos()`\n+API calls: they will call `ensure_full_index()` if necessary. The\n+intention here is to preserve existing behavior when interacting with a\n+sparse-checkout. We don't want a change to happen by accident, without\n+tests. Many of these locations may not need any change before removing the\n+guards, but we should not do so without tests to ensure the expected\n+behavior happens.\n+\n+It may be desirable to _change_ the behavior of some commands in the\n+presence of a sparse index or more generally in any sparse-checkout\n+scenario. In such cases, these should be carefully communicated and\n+tested. No such behavior changes are intended during this phase.\n+\n+During a scan of the codebase, not every iteration of the cache entries\n+needs an `ensure_full_index()` check. The basic reasons include:\n+\n+1. The loop is scanning for entries with non-zero stage. These entries\n+ are not collapsed into a sparse-directory entry.\n+\n+2. The loop is scanning for submodules. These entries are not collapsed\n+ into a sparse-directory entry.\n+\n+3. The loop is part of the index API, especially around reading or\n+ writing the format.\n+\n+4. The loop is checking for correct order of cache entries and that is\n+ correct if and only if the sparse-directory entries are in the correct\n+ location.\n+\n+5. The loop ignores entries with the `SKIP_WORKTREE` bit set, or is\n+ otherwise already aware of sparse directory entries.\n+\n+6. The sparse-index is disabled at this point when using the split-index\n+ feature, so no effort is made to protect the split-index API.\n \n Even after inserting these guards, we will keep expanding sparse-indexes\n for most Git commands using the `command_requires_full_index` repository\n-- \ngitgitgadget\n\n"},{"id":"420769","messageId":"737d27e18d64d7151447eca90e74bf58451cc9a6.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 03/25] read-cache: expand on query into sparse-directory entry","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:40Z","receivedAt":"2021-04-01T01:51:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCallers to index_name_pos() or index_name_stage_pos() have a specific\npath in mind. If that happens to be a path with an ancestor being a\nsparse-directory entry, it can lead to unexpected results.\n\nIn the case that we did not find the requested path, check to see if the\nposition _before_ the inserted position is a sparse directory entry that\nmatches the initial segment of the input path (including the directory\nseparator at the end of the directory name). If so, then expand the\nindex to be a full index and search again. This expansion will only\nhappen once per index read.\n\nFuture enhancements could be more careful to expand only the necessary\nsparse directory entry, but then we would have a special \"not fully\nsparse, but also not fully expanded\" mode that could affect writing the\nindex to file. Since this only occurs if a specific file is requested\noutside of the sparse checkout definition, this is unlikely to be a\ncommon situation.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3ad94578095e..3698bc7bf77d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -567,6 +567,27 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t}\n \t\tfirst = next+1;\n \t}\n+\n+\tif (istate->sparse_index &&\n+\t first > 0) {\n+\t\t/* Note: first <= istate->cache_nr */\n+\t\tstruct cache_entry *ce = istate->cache[first - 1];\n+\n+\t\t/*\n+\t\t * If we are in a sparse-index _and_ the entry before the\n+\t\t * insertion position is a sparse-directory entry that is\n+\t\t * an ancestor of 'name', then we need to expand the index\n+\t\t * and search again. This will only trigger once, because\n+\t\t * thereafter the index is fully expanded.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t ce_namelen(ce) < namelen &&\n+\t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n+\t\t\tensure_full_index(istate);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t}\n+\t}\n+\n \treturn -first-1;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"420773","messageId":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.git.1615929435.gitgitgadget@gmail.com","subject":"[PATCH v2 00/25] Sparse Index: API protections","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:37Z","receivedAt":"2021-04-01T01:51:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is the second patch series submission coming out of the sparse-index\nRFC [1].\n\n[1]\nhttps://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n\nThis is based on v3 of the format series [2].\n\n[2]\nhttps://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@gmail.com/\n\nThe point of this series is to insert protections for the consumers of the\nin-memory index to avoid unintended behavior change when using a sparse\nindex versus a full one.\n\nWe mark certain regions of code as needing a full index, so we call\nensure_full_index() to expand a sparse index to a full one, if necessary.\nThese protections are inserted file-by-file in every loop over all cache\nentries. Well, \"most\" loops, because some are going to be handled in the\nvery next series so I leave them out.\n\nMany callers use index_name_pos() to find a path by name. In these cases, we\ncan check if that position resolves to a sparse directory instance. In those\ncases, we just expand to a full index and run the search again.\n\nThe last few patches deal with the name-hash hashtable for doing O(1)\nlookups.\n\nThese protections don't do much right now, since the previous series created\nthe_repository->settings.command_requires_full_index to guard all index\nreads and writes to ensure the in-memory copy is full for commands that have\nnot been tested with the sparse index yet.\n\nHowever, after this series is complete, we now have a straight-forward plan\nfor making commands \"sparse aware\" one-by-one:\n\n 1. Disable settings.command_requires_full_index to allow an in-memory\n sparse-index.\n 2. Run versions of that command under a debugger, breaking on\n ensure_full_index().\n 3. Examine the call stack to determine the context of that expansion, then\n implement the proper behavior in those locations.\n 4. Add tests to ensure we are checking this logic in the presence of sparse\n directory entries.\n\nI will admit that mostly it is the writing of the test cases that takes the\nmost time in the conversions I've done so far.\n\n\nUpdates in v2\n=============\n\n * Rebased onto v5 of ds/sparse-index\n * Updated the technical doc to describe how these protections are guards to\n keep behavior consistent between a sparse-index and a full index. Whether\n or not that behavior is \"correct\" can be interrogated later.\n * Calls to ensure_full_index() are marked with a TODO comment saying these\n calls should be audited later (with tests).\n * Fixed an incorrectly squashed commit message.\n * Dropped the diff-lib.c commit because it was erroneously included in v2.\n * Dropped the merge-ort.c commit because of conflicts with work in flight\n and a quick audit that it is not needed.\n * I reviewed the merge of this topic with mt/add-rm-in-sparse-checkout and\n found it equivalent to what I would have done.\n\nThanks, -Stolee\n\nDerrick Stolee (25):\n sparse-index: API protection strategy\n *: remove 'const' qualifier for struct index_state\n read-cache: expand on query into sparse-directory entry\n cache: move ensure_full_index() to cache.h\n add: ensure full index\n checkout-index: ensure full index\n checkout: ensure full index\n commit: ensure full index\n difftool: ensure full index\n fsck: ensure full index\n grep: ensure full index\n ls-files: ensure full index\n merge-index: ensure full index\n rm: ensure full index\n stash: ensure full index\n update-index: ensure full index\n dir: ensure full index\n entry: ensure full index\n merge-recursive: ensure full index\n pathspec: ensure full index\n read-cache: ensure full index\n resolve-undo: ensure full index\n revision: ensure full index\n sparse-index: expand_to_path()\n name-hash: use expand_to_path()\n\n Documentation/technical/sparse-index.txt | 37 +++++++++++-\n attr.c | 14 ++---\n attr.h | 4 +-\n builtin/add.c | 2 +\n builtin/checkout-index.c | 2 +\n builtin/checkout.c | 5 ++\n builtin/commit.c | 4 ++\n builtin/difftool.c | 3 +\n builtin/fsck.c | 2 +\n builtin/grep.c | 2 +\n builtin/ls-files.c | 14 +++--\n builtin/merge-index.c | 5 ++\n builtin/rm.c | 2 +\n builtin/stash.c | 2 +\n builtin/update-index.c | 2 +\n cache.h | 7 ++-\n convert.c | 26 ++++-----\n convert.h | 20 +++----\n dir.c | 14 +++--\n dir.h | 8 +--\n entry.c | 2 +\n merge-recursive.c | 4 +-\n name-hash.c | 10 ++++\n pathspec.c | 8 ++-\n pathspec.h | 6 +-\n read-cache.c | 35 ++++++++++--\n resolve-undo.c | 4 ++\n revision.c | 2 +\n sparse-index.c | 72 ++++++++++++++++++++++++\n sparse-index.h | 14 ++++-\n submodule.c | 6 +-\n submodule.h | 6 +-\n 32 files changed, 273 insertions(+), 71 deletions(-)\n\n\nbase-commit: 66602733cc95d9a53594520cd8b28d3338e258ea\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/906\n\nRange-diff vs v1:\n\n 3: bbf19f8a2be5 ! 1: 7484e085e342 sparse-index: API protection strategy\n @@ Commit message\n Looking for non-zero stage:\n * builtin/add.c:chmod_pathspec()\n * builtin/merge.c:count_unmerged_entries()\n + * merge-ort.c:record_conflicted_index_entries()\n * read-cache.c:unmerged_index()\n * rerere.c:check_one_conflict(), find_conflict(), rerere_remaining()\n * revision.c:prepare_show_merge()\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## Documentation/technical/sparse-index.txt ##\n -@@ Documentation/technical/sparse-index.txt: also introduce other features that have been considered for improving the\n - index, as well.\n +@@ Documentation/technical/sparse-index.txt: index, as well.\n \n Next, consumers of the index will be guarded against operating on a\n --sparse-index by inserting calls to `ensure_full_index()` or\n + sparse-index by inserting calls to `ensure_full_index()` or\n -`expand_index_to_path()`. After these guards are in place, we can begin\n -leaving sparse-directory entries in the in-memory index structure.\n -+sparse-index by inserting calls to `ensure_full_index()` before iterating\n -+over all cache entries. If a specific path is requested, then those will\n ++`expand_index_to_path()`. If a specific path is requested, then those will\n +be protected from within the `index_file_exists()` and `index_name_pos()`\n -+API calls: they will call `ensure_full_index()` if necessary.\n ++API calls: they will call `ensure_full_index()` if necessary. The\n ++intention here is to preserve existing behavior when interacting with a\n ++sparse-checkout. We don't want a change to happen by accident, without\n ++tests. Many of these locations may not need any change before removing the\n ++guards, but we should not do so without tests to ensure the expected\n ++behavior happens.\n ++\n ++It may be desirable to _change_ the behavior of some commands in the\n ++presence of a sparse index or more generally in any sparse-checkout\n ++scenario. In such cases, these should be carefully communicated and\n ++tested. No such behavior changes are intended during this phase.\n +\n +During a scan of the codebase, not every iteration of the cache entries\n +needs an `ensure_full_index()` check. The basic reasons include:\n @@ Documentation/technical/sparse-index.txt: also introduce other features that hav\n +\n +6. The sparse-index is disabled at this point when using the split-index\n + feature, so no effort is made to protect the split-index API.\n -+\n -+After these guards are in place, we can begin leaving sparse-directory\n -+entries in the in-memory index structure.\n \n Even after inserting these guards, we will keep expanding sparse-indexes\n for most Git commands using the `command_requires_full_index` repository\n 1: 628e16dd3fc7 = 2: 098b2c9ef352 *: remove 'const' qualifier for struct index_state\n 2: 8e11e8917019 = 3: 737d27e18d64 read-cache: expand on query into sparse-directory entry\n 4: 7e4079c48eb2 = 4: db5c100f3e2b cache: move ensure_full_index() to cache.h\n 5: 142df1ab8e3e ! 5: 4a5fc2eb5a9f add: ensure full index\n @@ builtin/add.c: static int renormalize_tracked_files(const struct pathspec *paths\n {\n \tint i, retval = 0;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n 6: bfa0164cc3c1 ! 6: 11c38f7277c5 checkout-index: ensure full index\n @@ builtin/checkout-index.c: static void checkout_all(const char *prefix, int prefi\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n 7: 1bb7b6117e41 ! 7: fd04adbb3f79 checkout: ensure full index\n @@ builtin/checkout.c: static int checkout_worktree(const struct checkout_opts *opt\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n ++\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n @@ builtin/checkout.c: static int checkout_paths(const struct checkout_opts *opts,\n \t * Make sure all pathspecs participated in locating the paths\n \t * to be checked out.\n \t */\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++)\n \t\tif (opts->overlay_mode)\n 8: a59e45c4ae8f ! 8: 65704f39edc9 commit: ensure full index\n @@ builtin/commit.c: static int list_paths(struct string_list *list, const char *wi\n \t\tfree(max_prefix);\n \t}\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n @@ builtin/commit.c: static int prepare_to_commit(const char *index_file, const cha\n \t\tif (get_oid(parent, &oid)) {\n \t\t\tint i, ita_nr = 0;\n \n ++\t\t\t/* TODO: audit for interaction with sparse-index. */\n +\t\t\tensure_full_index(&the_index);\n \t\t\tfor (i = 0; i < active_nr; i++)\n \t\t\t\tif (ce_intent_to_add(active_cache[i]))\n 9: 83040e7558f3 ! 9: 739f3fe9edf2 difftool: ensure full index\n @@ builtin/difftool.c: static int run_dir_diff(const char *extcmd, int symlinks, co\n \t\tsetenv(\"GIT_DIFFTOOL_DIRDIFF\", \"true\", 1);\n \trc = run_command_v_opt(helper_argv, flags);\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&wtindex);\n +\n \t/*\n 10: 988f7bd2d736 ! 10: 779a86ad1ec4 fsck: ensure full index\n @@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tverify_index_checksum = 1;\n \t\tverify_ce_order = 1;\n \t\tread_cache();\n ++\t\t/* TODO: audit for interaction with sparse-index. */\n +\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++) {\n \t\t\tunsigned int mode;\n 11: abe189051a0c ! 11: 8c0d377054fa grep: ensure full index\n @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n 12: 00c8a7e1d119 ! 12: beaa1467cabb ls-files: ensure full index\n @@ builtin/ls-files.c: static void show_files(struct repository *repo, struct dir_s\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(repo->index);\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n @@ builtin/ls-files.c: void overlay_tree_on_index(struct index_state *istate,\n \t\tdie(\"bad tree-ish %s\", tree_name);\n \n \t/* Hoist the unmerged entries up to stage #3 to make room */\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n 13: 1288f02177d2 ! 13: 73684141fcff merge-index: ensure full index\n @@ builtin/merge-index.c: static void merge_one_path(const char *path)\n static void merge_all(void)\n {\n \tint i;\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n @@ builtin/merge-index.c: int cmd_merge_index(int argc, const char **argv, const ch\n \n \tread_cache();\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n +\n \ti = 1;\n 14: d6816560b32f ! 14: 6ea81a49c6b5 rm: ensure full index\n @@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)\n \n \tseen = xcalloc(pathspec.nr, 1);\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n 15: 92ccd31dd343 ! 15: 49ca5ed05c8d sparse-checkout: ensure full index\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - sparse-checkout: ensure full index\n + stash: ensure full index\n \n - When adjusting the sparse-checkout definition, ensure that a sparse\n - index is expanded to a full one to avoid unexpected behavior. This is a\n - top candidate for later integration with the sparse-index, but requires\n - careful testing.\n + Before iterating over all cache entries, ensure that a sparse index is\n + expanded to a full index to avoid unexpected behavior.\n \n Signed-off-by: Derrick Stolee \n \n @@ builtin/stash.c: static int do_push_stash(const struct pathspec *ps, const char\n \t\tint i;\n \t\tchar *ps_matched = xcalloc(ps->nr, 1);\n \n ++\t\t/* TODO: audit for interaction with sparse-index. */\n +\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++)\n \t\t\tce_path_match(&the_index, active_cache[i], ps,\n 16: 796449758a08 ! 16: 9c4bb187c15d update-index: ensure full index\n @@ builtin/update-index.c: static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n 17: 45bbed6150a2 < -: ------------ diff-lib: ensure full index\n 18: 8ce1f80985a4 ! 17: fae4c078c3ef dir: ensure full index\n @@ dir.c: static void connect_wt_gitdir_in_nested(const char *sub_worktree,\n \tif (repo_read_index(&subrepo) < 0)\n \t\tdie(_(\"index file corrupt in repo %s\"), subrepo.gitdir);\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(subrepo.index);\n \tfor (i = 0; i < subrepo.index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = subrepo.index->cache[i];\n 19: 6e0b452f44c1 ! 18: 2b9180ee77d3 entry: ensure full index\n @@ entry.c: static void mark_colliding_entries(const struct checkout *state,\n \n \tce->ce_flags |= CE_MATCHED;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(state->istate);\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n 20: fb4c7038b746 < -: ------------ merge-ort: ensure full index\n 21: 57d59825627f ! 19: 1e3f6085a405 merge-recursive: ensure full index\n @@ merge-recursive.c: static struct string_list *get_unmerged(struct index_state *i\n \n \tunmerged->strdup_strings = 1;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct string_list_item *item;\n 22: f4c470686b27 ! 20: e62a597a9725 pathspec: ensure full index\n @@ pathspec.c: void add_pathspec_matches_against_index(const struct pathspec *paths\n \t\t\tnum_unmatched++;\n \tif (!num_unmatched)\n \t\treturn;\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n 23: 81519782d4b2 ! 21: ebfffdbdd6ad read-cache: ensure full index\n @@ read-cache.c: int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n @@ read-cache.c: int repo_index_has_changes(struct repository *repo,\n \t\tdiff_flush(&opt);\n \t\treturn opt.flags.has_changes != 0;\n \t} else {\n ++\t\t/* TODO: audit for interaction with sparse-index. */\n +\t\tensure_full_index(istate);\n \t\tfor (i = 0; sb && i < istate->cache_nr; i++) {\n \t\t\tif (i)\n 24: 39e8bea0c1ca ! 22: 495b07a87973 resolve-undo: ensure full index\n @@ resolve-undo.c: void unmerge_marked_index(struct index_state *istate)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n @@ resolve-undo.c: void unmerge_index(struct index_state *istate, const struct path\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n 25: 6abb1813ae10 ! 23: 3144114d1a75 revision: ensure full index\n @@ revision.c: static void do_add_index_objects_to_pending(struct rev_info *revs,\n {\n \tint i;\n \n ++\t/* TODO: audit for interaction with sparse-index. */\n +\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n 26: 1690d33c11e0 = 24: d52c72b4a7b9 sparse-index: expand_to_path()\n 27: 40cfc59f695a = 25: 7e2d3fae9a2a name-hash: use expand_to_path()\n\n-- \ngitgitgadget\n"},{"id":"420770","messageId":"db5c100f3e2bc9841e922acca916555eeba194c5.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 04/25] cache: move ensure_full_index() to cache.h","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:41Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSoon we will insert ensure_full_index() calls across the codebase.\nInstead of also adding include statements for sparse-index.h, let's just\nuse the fact that anything that cares about the index already has\ncache.h in its includes.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 1 +\n sparse-index.h | 1 -\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 5006278c13ca..b7e20e9778db 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -350,6 +350,7 @@ void add_name_hash(struct index_state *istate, struct cache_entry *ce);\n void remove_name_hash(struct index_state *istate, struct cache_entry *ce);\n void free_name_hash(struct index_state *istate);\n \n+void ensure_full_index(struct index_state *istate);\n \n /* Cache entry creation and cleanup */\n \ndiff --git a/sparse-index.h b/sparse-index.h\nindex 39dcc859735e..0268f38753c0 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,6 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-void ensure_full_index(struct index_state *istate);\n int convert_to_sparse(struct index_state *istate);\n \n struct repository;\n-- \ngitgitgadget\n\n"},{"id":"420771","messageId":"4a5fc2eb5a9fab5276ae502a7fd6e35309d6f17b.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 05/25] add: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:42Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ea762a41e3a2..afccf2fd5543 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -141,6 +141,8 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n {\n \tint i, retval = 0;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"420772","messageId":"098b2c9ef3523d9052cc3834871c0078f9616edf.1617241802.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 02/25] *: remove 'const' qualifier for struct index_state","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:39Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSeveral methods specify that they take a 'struct index_state' pointer\nwith the 'const' qualifier because they intend to only query the data,\nnot change it. However, we will be introducing a step very low in the\nmethod stack that might modify a sparse-index to become a full index in\nthe case that our queries venture inside a sparse-directory entry.\n\nThis change only removes the 'const' qualifiers that are necessary for\nthe following change which will actually modify the implementation of\nindex_name_stage_pos().\n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 +++++++-------\n attr.h | 4 ++--\n builtin/ls-files.c | 10 +++++-----\n cache.h | 6 +++---\n convert.c | 26 +++++++++++++-------------\n convert.h | 20 ++++++++++----------\n dir.c | 12 ++++++------\n dir.h | 8 ++++----\n merge-recursive.c | 2 +-\n pathspec.c | 6 +++---\n pathspec.h | 6 +++---\n read-cache.c | 10 +++++-----\n submodule.c | 6 +++---\n submodule.h | 6 +++---\n 14 files changed, 68 insertions(+), 68 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 4ef85d668b54..8de553293ee4 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -718,7 +718,7 @@ static struct attr_stack *read_attr_from_file(const char *path, int macro_ok)\n \treturn res;\n }\n \n-static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n+static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t int macro_ok)\n {\n@@ -748,7 +748,7 @@ static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n \treturn res;\n }\n \n-static struct attr_stack *read_attr(const struct index_state *istate,\n+static struct attr_stack *read_attr(struct index_state *istate,\n \t\t\t\t const char *path, int macro_ok)\n {\n \tstruct attr_stack *res = NULL;\n@@ -840,7 +840,7 @@ static void push_stack(struct attr_stack **attr_stack_p,\n \t}\n }\n \n-static void bootstrap_attr_stack(const struct index_state *istate,\n+static void bootstrap_attr_stack(struct index_state *istate,\n \t\t\t\t struct attr_stack **stack)\n {\n \tstruct attr_stack *e;\n@@ -878,7 +878,7 @@ static void bootstrap_attr_stack(const struct index_state *istate,\n \tpush_stack(stack, e, NULL, 0);\n }\n \n-static void prepare_attr_stack(const struct index_state *istate,\n+static void prepare_attr_stack(struct index_state *istate,\n \t\t\t const char *path, int dirlen,\n \t\t\t struct attr_stack **stack)\n {\n@@ -1078,7 +1078,7 @@ static void determine_macros(struct all_attrs_item *all_attrs,\n * If check->check_nr is non-zero, only attributes in check[] are collected.\n * Otherwise all attributes are collected.\n */\n-static void collect_some_attrs(const struct index_state *istate,\n+static void collect_some_attrs(struct index_state *istate,\n \t\t\t const char *path,\n \t\t\t struct attr_check *check)\n {\n@@ -1107,7 +1107,7 @@ static void collect_some_attrs(const struct index_state *istate,\n \tfill(path, pathlen, basename_offset, check->stack, check->all_attrs, rem);\n }\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path,\n \t\t struct attr_check *check)\n {\n@@ -1124,7 +1124,7 @@ void git_check_attr(const struct index_state *istate,\n \t}\n }\n \n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check)\n {\n \tint i;\ndiff --git a/attr.h b/attr.h\nindex 404548f028a8..3732505edae8 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -190,14 +190,14 @@ void attr_check_free(struct attr_check *check);\n */\n const char *git_attr_name(const struct git_attr *);\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n /*\n * Retrieve all attributes that apply to the specified path.\n * check holds the attributes and their values.\n */\n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n enum git_attr_direction {\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..4f9ed1fb29b7 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -57,7 +57,7 @@ static const char *tag_modified = \"\";\n static const char *tag_skip_worktree = \"\";\n static const char *tag_resolve_undo = \"\";\n \n-static void write_eolinfo(const struct index_state *istate,\n+static void write_eolinfo(struct index_state *istate,\n \t\t\t const struct cache_entry *ce, const char *path)\n {\n \tif (show_eol) {\n@@ -122,7 +122,7 @@ static void print_debug(const struct cache_entry *ce)\n \t}\n }\n \n-static void show_dir_entry(const struct index_state *istate,\n+static void show_dir_entry(struct index_state *istate,\n \t\t\t const char *tag, struct dir_entry *ent)\n {\n \tint len = max_prefix_len;\n@@ -139,7 +139,7 @@ static void show_dir_entry(const struct index_state *istate,\n \twrite_name(ent->name);\n }\n \n-static void show_other_files(const struct index_state *istate,\n+static void show_other_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -152,7 +152,7 @@ static void show_other_files(const struct index_state *istate,\n \t}\n }\n \n-static void show_killed_files(const struct index_state *istate,\n+static void show_killed_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -254,7 +254,7 @@ static void show_ce(struct repository *repo, struct dir_struct *dir,\n \t}\n }\n \n-static void show_ru_info(const struct index_state *istate)\n+static void show_ru_info(struct index_state *istate)\n {\n \tstruct string_list_item *item;\n \ndiff --git a/cache.h b/cache.h\nindex 8aede373aeb3..5006278c13ca 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -800,7 +800,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n * index_name_pos(&index, \"f\", 1) -> -3\n * index_name_pos(&index, \"g\", 1) -> -5\n */\n-int index_name_pos(const struct index_state *, const char *name, int namelen);\n+int index_name_pos(struct index_state *, const char *name, int namelen);\n \n /*\n * Some functions return the negative complement of an insert position when a\n@@ -850,8 +850,8 @@ int add_file_to_index(struct index_state *, const char *path, int flags);\n int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n-int index_name_is_other(const struct index_state *, const char *, int);\n-void *read_blob_data_from_index(const struct index_state *, const char *, unsigned long *);\n+int index_name_is_other(struct index_state *, const char *, int);\n+void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/convert.c b/convert.c\nindex ee360c2f07ce..20b19abffa36 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -138,7 +138,7 @@ static const char *gather_convert_stats_ascii(const char *data, unsigned long si\n \t}\n }\n \n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n@@ -222,7 +222,7 @@ static void check_global_conv_flags_eol(const char *path,\n \t}\n }\n \n-static int has_crlf_in_index(const struct index_state *istate, const char *path)\n+static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n \tunsigned long sz;\n \tvoid *data;\n@@ -496,7 +496,7 @@ static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n \treturn 1;\n }\n \n-static int crlf_to_git(const struct index_state *istate,\n+static int crlf_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n \t\t enum crlf_action crlf_action, int conv_flags)\n@@ -1307,7 +1307,7 @@ struct conv_attrs {\n \n static struct attr_check *check;\n \n-static void convert_attrs(const struct index_state *istate,\n+static void convert_attrs(struct index_state *istate,\n \t\t\t struct conv_attrs *ca, const char *path)\n {\n \tstruct attr_check_item *ccheck = NULL;\n@@ -1369,7 +1369,7 @@ void reset_parsed_attributes(void)\n \tuser_convert_tail = NULL;\n }\n \n-int would_convert_to_git_filter_fd(const struct index_state *istate, const char *path)\n+int would_convert_to_git_filter_fd(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1388,7 +1388,7 @@ int would_convert_to_git_filter_fd(const struct index_state *istate, const char\n \treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL, NULL);\n }\n \n-const char *get_convert_attr_ascii(const struct index_state *istate, const char *path)\n+const char *get_convert_attr_ascii(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1414,7 +1414,7 @@ const char *get_convert_attr_ascii(const struct index_state *istate, const char\n \treturn \"\";\n }\n \n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags)\n {\n@@ -1448,7 +1448,7 @@ int convert_to_git(const struct index_state *istate,\n \treturn ret | ident_to_git(src, len, dst, ca.ident);\n }\n \n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n \t\t\t int conv_flags)\n {\n@@ -1466,7 +1466,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tident_to_git(dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const struct index_state *istate,\n+static int convert_to_working_tree_internal(struct index_state *istate,\n \t\t\t\t\t const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing,\n@@ -1510,7 +1510,7 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \treturn ret | ret_filter;\n }\n \n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n@@ -1519,7 +1519,7 @@ int async_convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);\n }\n \n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta)\n@@ -1527,7 +1527,7 @@ int convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);\n }\n \n-int renormalize_buffer(const struct index_state *istate, const char *path,\n+int renormalize_buffer(struct index_state *istate, const char *path,\n \t\t const char *src, size_t len, struct strbuf *dst)\n {\n \tint ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);\n@@ -1964,7 +1964,7 @@ static struct stream_filter *ident_filter(const struct object_id *oid)\n * Note that you would be crazy to set CRLF, smudge/clean or ident to a\n * large binary blob you would want us not to slurp into the memory!\n */\n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/convert.h b/convert.h\nindex e29d1026a686..0f7c8a1f0446 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -65,41 +65,41 @@ struct checkout_metadata {\n \n extern enum eol core_eol;\n extern char *check_roundtrip_encoding;\n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path);\n const char *get_wt_convert_stats_ascii(const char *path);\n-const char *get_convert_attr_ascii(const struct index_state *istate,\n+const char *get_convert_attr_ascii(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /* returns 1 if *dst was used */\n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags);\n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta);\n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco);\n int async_query_available_blobs(const char *cmd,\n \t\t\t\tstruct string_list *available_paths);\n-int renormalize_buffer(const struct index_state *istate,\n+int renormalize_buffer(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst);\n-static inline int would_convert_to_git(const struct index_state *istate,\n+static inline int would_convert_to_git(struct index_state *istate,\n \t\t\t\t const char *path)\n {\n \treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd,\n \t\t\t struct strbuf *dst,\n \t\t\t int conv_flags);\n-int would_convert_to_git_filter_fd(const struct index_state *istate,\n+int would_convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /*\n@@ -133,7 +133,7 @@ void reset_parsed_attributes(void);\n \n struct stream_filter; /* opaque */\n \n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *);\n void free_stream_filter(struct stream_filter *);\ndiff --git a/dir.c b/dir.c\nindex fd8aa7c40faa..5b00dfb5b144 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -306,7 +306,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,\n * [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match.\n * [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match.\n */\n-static int match_pathspec_item(const struct index_state *istate,\n+static int match_pathspec_item(struct index_state *istate,\n \t\t\t const struct pathspec_item *item, int prefix,\n \t\t\t const char *name, int namelen, unsigned flags)\n {\n@@ -429,7 +429,7 @@ static int match_pathspec_item(const struct index_state *istate,\n * pathspec did not match any names, which could indicate that the\n * user mistyped the nth pathspec.\n */\n-static int do_match_pathspec(const struct index_state *istate,\n+static int do_match_pathspec(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *name, int namelen,\n \t\t\t int prefix, char *seen,\n@@ -500,7 +500,7 @@ static int do_match_pathspec(const struct index_state *istate,\n \treturn retval;\n }\n \n-static int match_pathspec_with_flags(const struct index_state *istate,\n+static int match_pathspec_with_flags(struct index_state *istate,\n \t\t\t\t const struct pathspec *ps,\n \t\t\t\t const char *name, int namelen,\n \t\t\t\t int prefix, char *seen, unsigned flags)\n@@ -516,7 +516,7 @@ static int match_pathspec_with_flags(const struct index_state *istate,\n \treturn negative ? 0 : positive;\n }\n \n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *ps,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir)\n@@ -529,7 +529,7 @@ int match_pathspec(const struct index_state *istate,\n /**\n * Check if a submodule is a superset of the pathspec\n */\n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen)\n@@ -892,7 +892,7 @@ void add_pattern(const char *string, const char *base,\n \tadd_pattern_to_hashsets(pl, pattern);\n }\n \n-static int read_skip_worktree_file_from_index(const struct index_state *istate,\n+static int read_skip_worktree_file_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t size_t *size_out, char **data_out,\n \t\t\t\t\t struct oid_stat *oid_stat)\ndiff --git a/dir.h b/dir.h\nindex facfae47402a..51cb0e217247 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -354,7 +354,7 @@ int count_slashes(const char *s);\n int simple_length(const char *match);\n int no_wildcard(const char *string);\n char *common_prefix(const struct pathspec *pathspec);\n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *pathspec,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir);\n@@ -492,12 +492,12 @@ int git_fnmatch(const struct pathspec_item *item,\n \t\tconst char *pattern, const char *string,\n \t\tint prefix);\n \n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen);\n \n-static inline int ce_path_match(const struct index_state *istate,\n+static inline int ce_path_match(struct index_state *istate,\n \t\t\t\tconst struct cache_entry *ce,\n \t\t\t\tconst struct pathspec *pathspec,\n \t\t\t\tchar *seen)\n@@ -506,7 +506,7 @@ static inline int ce_path_match(const struct index_state *istate,\n \t\t\t S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n }\n \n-static inline int dir_path_match(const struct index_state *istate,\n+static inline int dir_path_match(struct index_state *istate,\n \t\t\t\t const struct dir_entry *ent,\n \t\t\t\t const struct pathspec *pathspec,\n \t\t\t\t int prefix, char *seen)\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 3d9207455b3a..b8de7a704eae 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -2987,7 +2987,7 @@ static int blob_unchanged(struct merge_options *opt,\n \tstruct strbuf obuf = STRBUF_INIT;\n \tstruct strbuf abuf = STRBUF_INIT;\n \tint ret = 0; /* assume changed for safety */\n-\tconst struct index_state *idx = opt->repo->index;\n+\tstruct index_state *idx = opt->repo->index;\n \n \tif (a->mode != o->mode)\n \t\treturn 0;\ndiff --git a/pathspec.c b/pathspec.c\nindex 7a229d8d22f2..b6e333965cb4 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -20,7 +20,7 @@\n * to use find_pathspecs_matching_against_index() instead.\n */\n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen)\n {\n \tint num_unmatched = 0, i;\n@@ -51,7 +51,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n * given pathspecs achieves against all items in the index.\n */\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate)\n+\t\t\t\t\t struct index_state *istate)\n {\n \tchar *seen = xcalloc(pathspec->nr, 1);\n \tadd_pathspec_matches_against_index(pathspec, istate, seen);\n@@ -702,7 +702,7 @@ void clear_pathspec(struct pathspec *pathspec)\n \tpathspec->nr = 0;\n }\n \n-int match_pathspec_attrs(const struct index_state *istate,\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item)\n {\ndiff --git a/pathspec.h b/pathspec.h\nindex 454ce364fac7..2ccc8080b6cc 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -150,11 +150,11 @@ static inline int ps_strcmp(const struct pathspec_item *item,\n }\n \n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen);\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate);\n-int match_pathspec_attrs(const struct index_state *istate,\n+\t\t\t\t\t struct index_state *istate);\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2410e6e0df13..3ad94578095e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -549,7 +549,7 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(const struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n {\n \tint first, last;\n \n@@ -570,7 +570,7 @@ static int index_name_stage_pos(const struct index_state *istate, const char *na\n \treturn -first-1;\n }\n \n-int index_name_pos(const struct index_state *istate, const char *name, int namelen)\n+int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n \treturn index_name_stage_pos(istate, name, namelen, 0);\n }\n@@ -3389,8 +3389,8 @@ int repo_read_index_unmerged(struct repository *repo)\n * We helpfully remove a trailing \"/\" from directories so that\n * the output of read_directory can be used as-is.\n */\n-int index_name_is_other(const struct index_state *istate, const char *name,\n-\t\tint namelen)\n+int index_name_is_other(struct index_state *istate, const char *name,\n+\t\t\tint namelen)\n {\n \tint pos;\n \tif (namelen && name[namelen - 1] == '/')\n@@ -3408,7 +3408,7 @@ int index_name_is_other(const struct index_state *istate, const char *name,\n \treturn 1;\n }\n \n-void *read_blob_data_from_index(const struct index_state *istate,\n+void *read_blob_data_from_index(struct index_state *istate,\n \t\t\t\tconst char *path, unsigned long *size)\n {\n \tint pos, len;\ndiff --git a/submodule.c b/submodule.c\nindex 9767ba9893cc..83809a4f7bd2 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -33,7 +33,7 @@ static struct oid_array ref_tips_after_fetch;\n * will be disabled because we can't guess what might be configured in\n * .gitmodules unless the user resolves the conflict.\n */\n-int is_gitmodules_unmerged(const struct index_state *istate)\n+int is_gitmodules_unmerged(struct index_state *istate)\n {\n \tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n \tif (pos < 0) { /* .gitmodules not found or isn't merged */\n@@ -301,7 +301,7 @@ int is_submodule_populated_gently(const char *path, int *return_error_code)\n /*\n * Dies if the provided 'prefix' corresponds to an unpopulated submodule\n */\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix)\n {\n \tint i, prefixlen;\n@@ -331,7 +331,7 @@ void die_in_unpopulated_submodule(const struct index_state *istate,\n /*\n * Dies if any paths in the provided pathspec descends into a submodule\n */\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps)\n {\n \tint i, j;\ndiff --git a/submodule.h b/submodule.h\nindex 4ac6e31cf1f7..84640c49c114 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -39,7 +39,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n-int is_gitmodules_unmerged(const struct index_state *istate);\n+int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n@@ -60,9 +60,9 @@ int is_submodule_active(struct repository *repo, const char *path);\n * Otherwise the return error code is the same as of resolve_gitdir_gently.\n */\n int is_submodule_populated_gently(const char *path, int *return_error_code);\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix);\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps);\n enum submodule_update_type parse_submodule_update_type(const char *value);\n int parse_submodule_update_strategy(const char *value,\n-- \ngitgitgadget\n\n"},{"id":"420774","messageId":"779a86ad1ec46f53ae38b17cb53df347b01576fe.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 10/25] fsck: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:47Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen verifying all blobs reachable from the index, ensure that a sparse\nindex has been expanded to a full one to avoid missing some blobs.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fsck.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 821e7798c706..4d7f5c63ce0d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -883,6 +883,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tverify_index_checksum = 1;\n \t\tverify_ce_order = 1;\n \t\tread_cache();\n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++) {\n \t\t\tunsigned int mode;\n \t\t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"420775","messageId":"11c38f7277c5217b13065ee4c53f6f94e1c6a7a9.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 06/25] checkout-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:43Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore we iterate over all cache entries, ensure that the index is not\nsparse. This loop in checkout_all() might be safe to iterate over a\nsparse index, but let's put this protection here until it can be\ncarefully tested.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 023e49e271c2..2c2936a9dae0 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -119,6 +119,8 @@ static void checkout_all(const char *prefix, int prefix_length)\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tif (ce_stage(ce) != checkout_stage\n-- \ngitgitgadget\n\n"},{"id":"420776","messageId":"65704f39edc971a2f866d6a526d04ea2ed1927ae.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 08/25] commit: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:45Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThese two loops iterate over all cache entries, so ensure that a sparse\nindex is expanded to a full index before we do so.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 739110c5a7f6..cf0c36d1dcb2 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -251,6 +251,8 @@ static int list_paths(struct string_list *list, const char *with_tree,\n \t\tfree(max_prefix);\n \t}\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tstruct string_list_item *item;\n@@ -931,6 +933,8 @@ static int prepare_to_commit(const char *index_file, const char *prefix,\n \t\tif (get_oid(parent, &oid)) {\n \t\t\tint i, ita_nr = 0;\n \n+\t\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\t\tensure_full_index(&the_index);\n \t\t\tfor (i = 0; i < active_nr; i++)\n \t\t\t\tif (ce_intent_to_add(active_cache[i]))\n \t\t\t\t\tita_nr++;\n-- \ngitgitgadget\n\n"},{"id":"420777","messageId":"8c0d377054fa126f6b7426f8d87cd9bd57f5328c.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 11/25] grep: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:48Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one so we do not miss blobs to scan. Later, this can\nintegrate more carefully with sparse indexes with proper testing.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/grep.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4e91a253ac3b..c2d40414e975 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -504,6 +504,8 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-- \ngitgitgadget\n\n"},{"id":"420778","messageId":"fd04adbb3f79b3dc2e7f36fb4ddf852ba3165f5b.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 07/25] checkout: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:44Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries in the checkout builtin, ensure\nthat we have a full index to avoid any unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 0e6639052001..d0dbe63ea119 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -368,6 +368,9 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n+\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -512,6 +515,8 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t * Make sure all pathspecs participated in locating the paths\n \t * to be checked out.\n \t */\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++)\n \t\tif (opts->overlay_mode)\n \t\t\tmark_ce_for_checkout_overlay(active_cache[pos],\n-- \ngitgitgadget\n\n"},{"id":"420779","messageId":"fae4c078c3ef8f65dc6673969642a3b35a254ba6.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 17/25] dir: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:54Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 5b00dfb5b144..166238e79f52 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3533,6 +3533,8 @@ static void connect_wt_gitdir_in_nested(const char *sub_worktree,\n \tif (repo_read_index(&subrepo) < 0)\n \t\tdie(_(\"index file corrupt in repo %s\"), subrepo.gitdir);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(subrepo.index);\n \tfor (i = 0; i < subrepo.index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = subrepo.index->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"420780","messageId":"beaa1467cabbdd27b2d8aba4538bfe7232faa116.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 12/25] ls-files: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:49Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid missing files.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/ls-files.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 4f9ed1fb29b7..a0b4e54d1149 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -317,6 +317,8 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(repo->index);\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -494,6 +496,8 @@ void overlay_tree_on_index(struct index_state *istate,\n \t\tdie(\"bad tree-ish %s\", tree_name);\n \n \t/* Hoist the unmerged entries up to stage #3 to make room */\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_stage(ce))\n-- \ngitgitgadget\n\n"},{"id":"420782","messageId":"49ca5ed05c8d0e9333494ebb08ed65cb167dc96e.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 15/25] stash: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:52Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/stash.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex ba774cce674f..6fb7178ef2fa 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1350,6 +1350,8 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\tint i;\n \t\tchar *ps_matched = xcalloc(ps->nr, 1);\n \n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++)\n \t\t\tce_path_match(&the_index, active_cache[i], ps,\n \t\t\t\t ps_matched);\n-- \ngitgitgadget\n\n"},{"id":"420783","messageId":"6ea81a49c6b51e98f223b59df6e8e73eac894bbe.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 14/25] rm: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:51Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4858631e0f02..5559a0b453a3 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -293,6 +293,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \n \tseen = xcalloc(pathspec.nr, 1);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n-- \ngitgitgadget\n\n"},{"id":"420789","messageId":"739f3fe9edf276555cfe153e6f165b0af68a72ba.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 09/25] difftool: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:46Z","receivedAt":"2021-04-01T01:51:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index has\nbeen expanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/difftool.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 6e18e623fddf..32c914dde6a0 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -584,6 +584,9 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\tsetenv(\"GIT_DIFFTOOL_DIRDIFF\", \"true\", 1);\n \trc = run_command_v_opt(helper_argv, flags);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&wtindex);\n+\n \t/*\n \t * If the diff includes working copy files and those\n \t * files were modified during the diff, then the changes\n-- \ngitgitgadget\n\n"},{"id":"420781","messageId":"1e3f6085a405aa39692f3100790a596cc7b6fbfd.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 19/25] merge-recursive: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:56Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n merge-recursive.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex b8de7a704eae..91d8597728c1 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -522,6 +522,8 @@ static struct string_list *get_unmerged(struct index_state *istate)\n \n \tunmerged->strdup_strings = 1;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct string_list_item *item;\n \t\tstruct stage_data *e;\n-- \ngitgitgadget\n\n"},{"id":"420784","messageId":"e62a597a972582828356d2d270554bec52e8ff96.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 20/25] pathspec: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:57Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex b6e333965cb4..d67688bab74b 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -36,6 +36,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\t\tnum_unmatched++;\n \tif (!num_unmatched)\n \t\treturn;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tce_path_match(istate, ce, pathspec, seen);\n-- \ngitgitgadget\n\n"},{"id":"420785","messageId":"2b9180ee77d30c797f58f7e50ff9174e70797069.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 18/25] entry: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:55Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n entry.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/entry.c b/entry.c\nindex 7b9f43716f76..891e4ba2b45a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -412,6 +412,8 @@ static void mark_colliding_entries(const struct checkout *state,\n \n \tce->ce_flags |= CE_MATCHED;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(state->istate);\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"420786","messageId":"73684141fcff4d9c0df45e4bbc8d37729bd14ffb.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 13/25] merge-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:50Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge-index.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/merge-index.c b/builtin/merge-index.c\nindex 38ea6ad6ca25..c0383fe9df9a 100644\n--- a/builtin/merge-index.c\n+++ b/builtin/merge-index.c\n@@ -58,6 +58,8 @@ static void merge_one_path(const char *path)\n static void merge_all(void)\n {\n \tint i;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_stage(ce))\n@@ -80,6 +82,9 @@ int cmd_merge_index(int argc, const char **argv, const char *prefix)\n \n \tread_cache();\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n+\n \ti = 1;\n \tif (!strcmp(argv[i], \"-o\")) {\n \t\tone_shot = 1;\n-- \ngitgitgadget\n\n"},{"id":"420787","messageId":"ebfffdbdd6ad7fcefc47a3581a335df96f930539.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 21/25] read-cache: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:58Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3698bc7bf77d..a9dcf0ab4f78 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1577,6 +1577,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -2498,6 +2500,8 @@ int repo_index_has_changes(struct repository *repo,\n \t\tdiff_flush(&opt);\n \t\treturn opt.flags.has_changes != 0;\n \t} else {\n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(istate);\n \t\tfor (i = 0; sb && i < istate->cache_nr; i++) {\n \t\t\tif (i)\n \t\t\t\tstrbuf_addch(sb, ' ');\n-- \ngitgitgadget\n\n"},{"id":"420788","messageId":"3144114d1a75a8515fc35ea9625a7d088652b178.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 23/25] revision: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:50:00Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all index entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior. This case could\nbe integrated later by ensuring that we walk the tree in the\nsparse-directory entry, but the current behavior is only expecting\nblobs. Save this integration for later when it can be properly tested.\n\nSigned-off-by: Derrick Stolee \n---\n revision.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex b78733f5089b..b72e0ac1bdca 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1680,6 +1680,8 @@ static void do_add_index_objects_to_pending(struct rev_info *revs,\n {\n \tint i;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"420790","messageId":"495b07a87973f32ad81e513b2be04a1b1b92bdc8.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 22/25] resolve-undo: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:59Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n resolve-undo.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 236320f179cb..ee36ca58b135 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -172,6 +172,8 @@ void unmerge_marked_index(struct index_state *istate)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (ce->ce_flags & CE_MATCHED)\n@@ -186,6 +188,8 @@ void unmerge_index(struct index_state *istate, const struct pathspec *pathspec)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_path_match(istate, ce, pathspec, NULL))\n-- \ngitgitgadget\n\n"},{"id":"420791","messageId":"d52c72b4a7b9a3ae10bd2daae85c1bd1bafcc2f3.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 24/25] sparse-index: expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:50:01Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome users of the index API have a specific path they are looking for,\nbut choose to use index_file_exists() to rely on the name-hash hashtable\ninstead of doing binary search with index_name_pos(). These users only\nneed to know a yes/no answer, not a position within the cache array.\n\nWhen the index is sparse, the name-hash hash table does not contain the\nfull list of paths within sparse directories. It _does_ contain the\ndirectory names for the sparse-directory entries.\n\nCreate a helper function, expand_to_path(), for intended use with the\nname-hash hashtable functions. The integration with name-hash.c will\nfollow in a later change.\n\nThe solution here is to use ensure_full_index() when we determine that\nthe requested path is within a sparse directory entry. This will\npopulate the name-hash hashtable as the index is recomputed from\nscratch.\n\nThere may be cases where the caller is trying to find an untracked path\nthat is not in the index but also is not within a sparse directory\nentry. We want to minimize the overhead for these requests. If we used\nindex_name_pos() to find the insertion order of the path, then we could\ndetermine from that position if a sparse-directory exists. (In fact,\njust calling index_name_pos() in that case would lead to expanding the\nindex to a full index.) However, this takes O(log N) time where N is the\nnumber of cache entries.\n\nTo keep the performance of this call based mostly on the input string,\nuse index_file_exists() to look for the ancestors of the path. Using the\nheuristic that a sparse directory is likely to have a small number of\nparent directories, we start from the bottom and build up. Use a string\nbuffer to allow mutating the path name to terminate after each slash for\neach hashset test.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n sparse-index.h | 13 +++++++++\n 2 files changed, 85 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 95ea17174da3..8a1223041296 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -283,3 +283,75 @@ void ensure_full_index(struct index_state *istate)\n \n \ttrace2_region_leave(\"index\", \"ensure_full_index\", istate->repo);\n }\n+\n+/*\n+ * This static global helps avoid infinite recursion between\n+ * expand_to_path() and index_file_exists().\n+ */\n+static int in_expand_to_path = 0;\n+\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase)\n+{\n+\tstruct strbuf path_mutable = STRBUF_INIT;\n+\tsize_t substr_len;\n+\n+\t/* prevent extra recursion */\n+\tif (in_expand_to_path)\n+\t\treturn;\n+\n+\tif (!istate || !istate->sparse_index)\n+\t\treturn;\n+\n+\tif (!istate->repo)\n+\t\tistate->repo = the_repository;\n+\n+\tin_expand_to_path = 1;\n+\n+\t/*\n+\t * We only need to actually expand a region if the\n+\t * following are both true:\n+\t *\n+\t * 1. 'path' is not already in the index.\n+\t * 2. Some parent directory of 'path' is a sparse directory.\n+\t */\n+\n+\tif (index_file_exists(istate, path, pathlen, icase))\n+\t\tgoto cleanup;\n+\n+\tstrbuf_add(&path_mutable, path, pathlen);\n+\tstrbuf_addch(&path_mutable, '/');\n+\n+\t/* Check the name hash for all parent directories */\n+\tsubstr_len = 0;\n+\twhile (substr_len < pathlen) {\n+\t\tchar temp;\n+\t\tchar *replace = strchr(path_mutable.buf + substr_len, '/');\n+\n+\t\tif (!replace)\n+\t\t\tbreak;\n+\n+\t\t/* replace the character _after_ the slash */\n+\t\treplace++;\n+\t\ttemp = *replace;\n+\t\t*replace = '\\0';\n+\t\tif (index_file_exists(istate, path_mutable.buf,\n+\t\t\t\t path_mutable.len, icase)) {\n+\t\t\t/*\n+\t\t\t * We found a parent directory in the name-hash\n+\t\t\t * hashtable, which means that this entry could\n+\t\t\t * exist within a sparse-directory entry. Expand\n+\t\t\t * accordingly.\n+\t\t\t */\n+\t\t\tensure_full_index(istate);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t*replace = temp;\n+\t\tsubstr_len = replace - path_mutable.buf;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&path_mutable);\n+\tin_expand_to_path = 0;\n+}\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 0268f38753c0..1115a0d7dd98 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -4,6 +4,19 @@\n struct index_state;\n int convert_to_sparse(struct index_state *istate);\n \n+/*\n+ * Some places in the codebase expect to search for a specific path.\n+ * This path might be outside of the sparse-checkout definition, in\n+ * which case a sparse-index may not contain a path for that index.\n+ *\n+ * Given an index and a path, check to see if a leading directory for\n+ * 'path' exists in the index as a sparse directory. In that case,\n+ * expand that sparse directory to a full range of cache entries and\n+ * populate the index accordingly.\n+ */\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase);\n+\n struct repository;\n int set_sparse_index_config(struct repository *repo, int enable);\n \n-- \ngitgitgadget\n\n"},{"id":"420792","messageId":"7e2d3fae9a2a0574d2f881d0bcd4931a2cad6b45.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 25/25] name-hash: use expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:50:02Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA sparse-index loads the name-hash data for its entries, including the\nsparse-directory entries. If a caller asks for a path that is contained\nwithin a sparse-directory entry, we need to expand to a full index and\nrecalculate the name hash table before returning the result. Insert\ncalls to expand_to_path() to protect against this case.\n\nSigned-off-by: Derrick Stolee \n---\n name-hash.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 4e03fac9bb12..75c159e06eeb 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -8,6 +8,7 @@\n #include \"cache.h\"\n #include \"thread-utils.h\"\n #include \"trace2.h\"\n+#include \"sparse-index.h\"\n \n struct dir_entry {\n \tstruct hashmap_entry ent;\n@@ -109,6 +110,12 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)\n \tif (ce->ce_flags & CE_HASHED)\n \t\treturn;\n \tce->ce_flags |= CE_HASHED;\n+\n+\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\tadd_dir_entry(istate, ce);\n+\t\treturn;\n+\t}\n+\n \thashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n \thashmap_add(&istate->name_hash, &ce->ent);\n \n@@ -680,6 +687,7 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n \tstruct dir_entry *dir;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, 0);\n \tdir = find_dir_entry(istate, name, namelen);\n \treturn dir && dir->nr;\n }\n@@ -690,6 +698,7 @@ void adjust_dirname_case(struct index_state *istate, char *name)\n \tconst char *ptr = startPtr;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, strlen(name), 0);\n \twhile (*ptr) {\n \t\twhile (*ptr && *ptr != '/')\n \t\t\tptr++;\n@@ -713,6 +722,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n \tunsigned int hash = memihash(name, namelen);\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, icase);\n \n \tce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL,\n \t\t\t\t\t struct cache_entry, ent);\n-- \ngitgitgadget\n"},{"id":"420793","messageId":"9c4bb187c15da527f2a1692cafce7ec844a94ae1.1617241803.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v2 16/25] update-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-01T01:49:53Z","receivedAt":"2021-04-01T01:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/update-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 79087bccea4b..f1f16f2de526 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -745,6 +745,8 @@ static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n \t\tstruct cache_entry *old = NULL;\n-- \ngitgitgadget\n\n"},{"id":"420798","messageId":"xmqqv99632lh.fsf@gitster.g","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/25] Sparse Index: API protections","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-01T07:07:06Z","receivedAt":"2021-04-01T07:07:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Here is the second patch series submission coming out of the sparse-index\n> RFC [1].\n\nQueued. There were a few conflicts with new codepaths and I've\ntouched up a few of them (Alban's merge-strategies-in-c moves some\ncode that needs to dehydrate the sparse index entries from\nbuiltin/merge-index.c to a new file, Elijah's ort has \"const\"\npointer to the istate that you need to be writable), but there may\nbe more that I missed. Please check the tip of 'seen' when it gets\npushed out.\n\nThanks.\n"},{"id":"420835","messageId":"9137b585-0184-3bc2-0d65-5e04a3469e92@gmail.com","threadId":"55334","inReplyTo":"xmqqv99632lh.fsf@gitster.g","subject":"Re: [PATCH v2 00/25] Sparse Index: API protections","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-01T13:32:04Z","receivedAt":"2021-04-01T18:10:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2021 3:07 AM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> Here is the second patch series submission coming out of the sparse-index\n>> RFC [1].\n> \n> Queued. There were a few conflicts with new codepaths and I've\n> touched up a few of them (Alban's merge-strategies-in-c moves some\n> code that needs to dehydrate the sparse index entries from\n> builtin/merge-index.c to a new file, Elijah's ort has \"const\"\n> pointer to the istate that you need to be writable), but there may\n> be more that I missed. Please check the tip of 'seen' when it gets\n> pushed out.\n\nThank you for dealing with these complicated merges. I promise\nthat the series after this one are much more focused in their\ncode footprint. Thus, they should be easier to merge with\nongoing topics.\n\nI validated your merge by rebasing the rest of my current work\nin this area onto 'seen'. The tests for 'git status' and 'git\nadd' with sparse-indexes adds to the confidence of the merge.\n\nI did see a small conflict with mt/add-rm-in-sparse-checkout\nin one of my future patches because of an adjacent edit. I'll\nbe sure to base my next submission on a merge with that topic.\n\nThanks,\n-Stolee\n"},{"id":"421005","messageId":"CABPp-BE5xLMsVdamAb=MgiBnXKQrYDPJXa1TWutDK+nVYRPyRQ@mail.gmail.com","threadId":"55334","inReplyTo":"d52c72b4a7b9a3ae10bd2daae85c1bd1bafcc2f3.1617241803.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 24/25] sparse-index: expand_to_path()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-05T19:32:23Z","receivedAt":"2021-04-05T19:32:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 31, 2021 at 6:50 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> Some users of the index API have a specific path they are looking for,\n> but choose to use index_file_exists() to rely on the name-hash hashtable\n> instead of doing binary search with index_name_pos(). These users only\n> need to know a yes/no answer, not a position within the cache array.\n>\n> When the index is sparse, the name-hash hash table does not contain the\n> full list of paths within sparse directories. It _does_ contain the\n> directory names for the sparse-directory entries.\n>\n> Create a helper function, expand_to_path(), for intended use with the\n> name-hash hashtable functions. The integration with name-hash.c will\n> follow in a later change.\n>\n> The solution here is to use ensure_full_index() when we determine that\n> the requested path is within a sparse directory entry. This will\n> populate the name-hash hashtable as the index is recomputed from\n> scratch.\n>\n> There may be cases where the caller is trying to find an untracked path\n> that is not in the index but also is not within a sparse directory\n> entry. We want to minimize the overhead for these requests. If we used\n> index_name_pos() to find the insertion order of the path, then we could\n> determine from that position if a sparse-directory exists. (In fact,\n> just calling index_name_pos() in that case would lead to expanding the\n> index to a full index.) However, this takes O(log N) time where N is the\n> number of cache entries.\n>\n> To keep the performance of this call based mostly on the input string,\n> use index_file_exists() to look for the ancestors of the path. Using the\n> heuristic that a sparse directory is likely to have a small number of\n> parent directories, we start from the bottom and build up. Use a string\n> buffer to allow mutating the path name to terminate after each slash for\n> each hashset test.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> sparse-index.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n> sparse-index.h | 13 +++++++++\n> 2 files changed, 85 insertions(+)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index 95ea17174da3..8a1223041296 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -283,3 +283,75 @@ void ensure_full_index(struct index_state *istate)\n>\n> trace2_region_leave(\"index\", \"ensure_full_index\", istate->repo);\n> }\n> +\n> +/*\n> + * This static global helps avoid infinite recursion between\n> + * expand_to_path() and index_file_exists().\n> + */\n> +static int in_expand_to_path = 0;\n> +\n> +void expand_to_path(struct index_state *istate,\n> + const char *path, size_t pathlen, int icase)\n> +{\n> + struct strbuf path_mutable = STRBUF_INIT;\n> + size_t substr_len;\n> +\n> + /* prevent extra recursion */\n> + if (in_expand_to_path)\n> + return;\n> +\n> + if (!istate || !istate->sparse_index)\n> + return;\n> +\n> + if (!istate->repo)\n> + istate->repo = the_repository;\n> +\n> + in_expand_to_path = 1;\n> +\n> + /*\n> + * We only need to actually expand a region if the\n> + * following are both true:\n> + *\n> + * 1. 'path' is not already in the index.\n> + * 2. Some parent directory of 'path' is a sparse directory.\n> + */\n> +\n> + if (index_file_exists(istate, path, pathlen, icase))\n> + goto cleanup;\n> +\n> + strbuf_add(&path_mutable, path, pathlen);\n> + strbuf_addch(&path_mutable, '/');\n> +\n> + /* Check the name hash for all parent directories */\n> + substr_len = 0;\n> + while (substr_len < pathlen) {\n> + char temp;\n> + char *replace = strchr(path_mutable.buf + substr_len, '/');\n> +\n> + if (!replace)\n> + break;\n> +\n> + /* replace the character _after_ the slash */\n> + replace++;\n> + temp = *replace;\n> + *replace = '\\0';\n> + if (index_file_exists(istate, path_mutable.buf,\n> + path_mutable.len, icase)) {\n> + /*\n> + * We found a parent directory in the name-hash\n> + * hashtable, which means that this entry could\n> + * exist within a sparse-directory entry. Expand\n> + * accordingly.\n\n\"this entry\" is a bit unclear; it might be worth referring to the\n\"path\" variable instead. I think it'd also help to explain the\nreasoning for using the '/' character, because while it's clear when\nlooking at the series, just running across it in the code it might be\neasy to forget or miss. Perhaps:\n\n * We found a parent directory in the name-hash\n * hashtable, because only sparse directory entries\n * have a trailing '/' character. Since \"path\" wasn't\n * in the index, perhaps it exists within this\n * sparse-directory. Expand accordingly.\n\n> + */\n> + ensure_full_index(istate);\n> + break;\n> + }\n> +\n> + *replace = temp;\n> + substr_len = replace - path_mutable.buf;\n> + }\n> +\n> +cleanup:\n> + strbuf_release(&path_mutable);\n> + in_expand_to_path = 0;\n> +}\n> diff --git a/sparse-index.h b/sparse-index.h\n> index 0268f38753c0..1115a0d7dd98 100644\n> --- a/sparse-index.h\n> +++ b/sparse-index.h\n> @@ -4,6 +4,19 @@\n> struct index_state;\n> int convert_to_sparse(struct index_state *istate);\n>\n> +/*\n> + * Some places in the codebase expect to search for a specific path.\n> + * This path might be outside of the sparse-checkout definition, in\n> + * which case a sparse-index may not contain a path for that index.\n> + *\n> + * Given an index and a path, check to see if a leading directory for\n> + * 'path' exists in the index as a sparse directory. In that case,\n> + * expand that sparse directory to a full range of cache entries and\n> + * populate the index accordingly.\n> + */\n> +void expand_to_path(struct index_state *istate,\n> + const char *path, size_t pathlen, int icase);\n> +\n> struct repository;\n> int set_sparse_index_config(struct repository *repo, int enable);\n>\n> --\n> gitgitgadget\n"},{"id":"421006","messageId":"CABPp-BHvMa5Vf-fpmd7mK8awTU2WK_Gr7Mc8PA-r+NHxqzNu6w@mail.gmail.com","threadId":"55334","inReplyTo":"7e2d3fae9a2a0574d2f881d0bcd4931a2cad6b45.1617241803.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 25/25] name-hash: use expand_to_path()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-05T19:53:20Z","receivedAt":"2021-04-05T19:53:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 31, 2021 at 6:50 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> From: Derrick Stolee \n>\n> A sparse-index loads the name-hash data for its entries, including the\n> sparse-directory entries. If a caller asks for a path that is contained\n> within a sparse-directory entry, we need to expand to a full index and\n> recalculate the name hash table before returning the result. Insert\n> calls to expand_to_path() to protect against this case.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> name-hash.c | 10 ++++++++++\n> 1 file changed, 10 insertions(+)\n>\n> diff --git a/name-hash.c b/name-hash.c\n> index 4e03fac9bb12..75c159e06eeb 100644\n> --- a/name-hash.c\n> +++ b/name-hash.c\n> @@ -8,6 +8,7 @@\n> #include \"cache.h\"\n> #include \"thread-utils.h\"\n> #include \"trace2.h\"\n> +#include \"sparse-index.h\"\n>\n> struct dir_entry {\n> struct hashmap_entry ent;\n> @@ -109,6 +110,12 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)\n> if (ce->ce_flags & CE_HASHED)\n> return;\n> ce->ce_flags |= CE_HASHED;\n> +\n> + if (S_ISSPARSEDIR(ce->ce_mode)) {\n> + add_dir_entry(istate, ce);\n> + return;\n> + }\n\nI don't see how this relates to the commit message. Was this meant to\nbe included in another commit? If it is intended to be included here,\ncould an explanation be added?\n\n> +\n> hashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n> hashmap_add(&istate->name_hash, &ce->ent);\n>\n> @@ -680,6 +687,7 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n> struct dir_entry *dir;\n>\n> lazy_init_name_hash(istate);\n> + expand_to_path(istate, name, namelen, 0);\n> dir = find_dir_entry(istate, name, namelen);\n> return dir && dir->nr;\n> }\n> @@ -690,6 +698,7 @@ void adjust_dirname_case(struct index_state *istate, char *name)\n> const char *ptr = startPtr;\n>\n> lazy_init_name_hash(istate);\n> + expand_to_path(istate, name, strlen(name), 0);\n> while (*ptr) {\n> while (*ptr && *ptr != '/')\n> ptr++;\n> @@ -713,6 +722,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n> unsigned int hash = memihash(name, namelen);\n>\n> lazy_init_name_hash(istate);\n> + expand_to_path(istate, name, namelen, icase);\n>\n> ce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL,\n> struct cache_entry, ent);\n> --\n> gitgitgadget\n"},{"id":"421007","messageId":"CABPp-BGrycN2QDgueVG5+fokNkyd6wXowGw3R+-0U0jTFp9DUg@mail.gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/25] Sparse Index: API protections","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-05T19:55:31Z","receivedAt":"2021-04-05T19:55:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Mar 31, 2021 at 6:50 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> Here is the second patch series submission coming out of the sparse-index\n> RFC [1].\n>\n> [1]\n> https://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n>\n> This is based on v3 of the format series [2].\n>\n> [2]\n> https://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@gmail.com/\n>\n> The point of this series is to insert protections for the consumers of the\n> in-memory index to avoid unintended behavior change when using a sparse\n> index versus a full one.\n>\n> We mark certain regions of code as needing a full index, so we call\n> ensure_full_index() to expand a sparse index to a full one, if necessary.\n> These protections are inserted file-by-file in every loop over all cache\n> entries. Well, \"most\" loops, because some are going to be handled in the\n> very next series so I leave them out.\n>\n> Many callers use index_name_pos() to find a path by name. In these cases, we\n> can check if that position resolves to a sparse directory instance. In those\n> cases, we just expand to a full index and run the search again.\n>\n> The last few patches deal with the name-hash hashtable for doing O(1)\n> lookups.\n>\n> These protections don't do much right now, since the previous series created\n> the_repository->settings.command_requires_full_index to guard all index\n> reads and writes to ensure the in-memory copy is full for commands that have\n> not been tested with the sparse index yet.\n>\n> However, after this series is complete, we now have a straight-forward plan\n> for making commands \"sparse aware\" one-by-one:\n>\n> 1. Disable settings.command_requires_full_index to allow an in-memory\n> sparse-index.\n> 2. Run versions of that command under a debugger, breaking on\n> ensure_full_index().\n> 3. Examine the call stack to determine the context of that expansion, then\n> implement the proper behavior in those locations.\n> 4. Add tests to ensure we are checking this logic in the presence of sparse\n> directory entries.\n>\n> I will admit that mostly it is the writing of the test cases that takes the\n> most time in the conversions I've done so far.\n>\n>\n> Updates in v2\n> =============\n>\n> * Rebased onto v5 of ds/sparse-index\n> * Updated the technical doc to describe how these protections are guards to\n> keep behavior consistent between a sparse-index and a full index. Whether\n> or not that behavior is \"correct\" can be interrogated later.\n> * Calls to ensure_full_index() are marked with a TODO comment saying these\n> calls should be audited later (with tests).\n> * Fixed an incorrectly squashed commit message.\n> * Dropped the diff-lib.c commit because it was erroneously included in v2.\n> * Dropped the merge-ort.c commit because of conflicts with work in flight\n> and a quick audit that it is not needed.\n> * I reviewed the merge of this topic with mt/add-rm-in-sparse-checkout and\n> found it equivalent to what I would have done.\n\nYou addressed most my concerns moving from the RFC to v1, and this\nversion addresses most of my remaining comments/questions/suggestions.\nI only had two minor comments on the last two patches. This series is\nlooking pretty good to me.\n\n> Thanks, -Stolee\n>\n> Derrick Stolee (25):\n> sparse-index: API protection strategy\n> *: remove 'const' qualifier for struct index_state\n> read-cache: expand on query into sparse-directory entry\n> cache: move ensure_full_index() to cache.h\n> add: ensure full index\n> checkout-index: ensure full index\n> checkout: ensure full index\n> commit: ensure full index\n> difftool: ensure full index\n> fsck: ensure full index\n> grep: ensure full index\n> ls-files: ensure full index\n> merge-index: ensure full index\n> rm: ensure full index\n> stash: ensure full index\n> update-index: ensure full index\n> dir: ensure full index\n> entry: ensure full index\n> merge-recursive: ensure full index\n> pathspec: ensure full index\n> read-cache: ensure full index\n> resolve-undo: ensure full index\n> revision: ensure full index\n> sparse-index: expand_to_path()\n> name-hash: use expand_to_path()\n>\n> Documentation/technical/sparse-index.txt | 37 +++++++++++-\n> attr.c | 14 ++---\n> attr.h | 4 +-\n> builtin/add.c | 2 +\n> builtin/checkout-index.c | 2 +\n> builtin/checkout.c | 5 ++\n> builtin/commit.c | 4 ++\n> builtin/difftool.c | 3 +\n> builtin/fsck.c | 2 +\n> builtin/grep.c | 2 +\n> builtin/ls-files.c | 14 +++--\n> builtin/merge-index.c | 5 ++\n> builtin/rm.c | 2 +\n> builtin/stash.c | 2 +\n> builtin/update-index.c | 2 +\n> cache.h | 7 ++-\n> convert.c | 26 ++++-----\n> convert.h | 20 +++----\n> dir.c | 14 +++--\n> dir.h | 8 +--\n> entry.c | 2 +\n> merge-recursive.c | 4 +-\n> name-hash.c | 10 ++++\n> pathspec.c | 8 ++-\n> pathspec.h | 6 +-\n> read-cache.c | 35 ++++++++++--\n> resolve-undo.c | 4 ++\n> revision.c | 2 +\n> sparse-index.c | 72 ++++++++++++++++++++++++\n> sparse-index.h | 14 ++++-\n> submodule.c | 6 +-\n> submodule.h | 6 +-\n> 32 files changed, 273 insertions(+), 71 deletions(-)\n>\n>\n> base-commit: 66602733cc95d9a53594520cd8b28d3338e258ea\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v2\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v2\n> Pull-Request: https://github.com/gitgitgadget/git/pull/906\n>\n> Range-diff vs v1:\n>\n> 3: bbf19f8a2be5 ! 1: 7484e085e342 sparse-index: API protection strategy\n> @@ Commit message\n> Looking for non-zero stage:\n> * builtin/add.c:chmod_pathspec()\n> * builtin/merge.c:count_unmerged_entries()\n> + * merge-ort.c:record_conflicted_index_entries()\n> * read-cache.c:unmerged_index()\n> * rerere.c:check_one_conflict(), find_conflict(), rerere_remaining()\n> * revision.c:prepare_show_merge()\n> @@ Commit message\n> Signed-off-by: Derrick Stolee \n>\n> ## Documentation/technical/sparse-index.txt ##\n> -@@ Documentation/technical/sparse-index.txt: also introduce other features that have been considered for improving the\n> - index, as well.\n> +@@ Documentation/technical/sparse-index.txt: index, as well.\n>\n> Next, consumers of the index will be guarded against operating on a\n> --sparse-index by inserting calls to `ensure_full_index()` or\n> + sparse-index by inserting calls to `ensure_full_index()` or\n> -`expand_index_to_path()`. After these guards are in place, we can begin\n> -leaving sparse-directory entries in the in-memory index structure.\n> -+sparse-index by inserting calls to `ensure_full_index()` before iterating\n> -+over all cache entries. If a specific path is requested, then those will\n> ++`expand_index_to_path()`. If a specific path is requested, then those will\n> +be protected from within the `index_file_exists()` and `index_name_pos()`\n> -+API calls: they will call `ensure_full_index()` if necessary.\n> ++API calls: they will call `ensure_full_index()` if necessary. The\n> ++intention here is to preserve existing behavior when interacting with a\n> ++sparse-checkout. We don't want a change to happen by accident, without\n> ++tests. Many of these locations may not need any change before removing the\n> ++guards, but we should not do so without tests to ensure the expected\n> ++behavior happens.\n> ++\n> ++It may be desirable to _change_ the behavior of some commands in the\n> ++presence of a sparse index or more generally in any sparse-checkout\n> ++scenario. In such cases, these should be carefully communicated and\n> ++tested. No such behavior changes are intended during this phase.\n> +\n> +During a scan of the codebase, not every iteration of the cache entries\n> +needs an `ensure_full_index()` check. The basic reasons include:\n> @@ Documentation/technical/sparse-index.txt: also introduce other features that hav\n> +\n> +6. The sparse-index is disabled at this point when using the split-index\n> + feature, so no effort is made to protect the split-index API.\n> -+\n> -+After these guards are in place, we can begin leaving sparse-directory\n> -+entries in the in-memory index structure.\n>\n> Even after inserting these guards, we will keep expanding sparse-indexes\n> for most Git commands using the `command_requires_full_index` repository\n> 1: 628e16dd3fc7 = 2: 098b2c9ef352 *: remove 'const' qualifier for struct index_state\n> 2: 8e11e8917019 = 3: 737d27e18d64 read-cache: expand on query into sparse-directory entry\n> 4: 7e4079c48eb2 = 4: db5c100f3e2b cache: move ensure_full_index() to cache.h\n> 5: 142df1ab8e3e ! 5: 4a5fc2eb5a9f add: ensure full index\n> @@ builtin/add.c: static int renormalize_tracked_files(const struct pathspec *paths\n> {\n> int i, retval = 0;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> struct cache_entry *ce = active_cache[i];\n> 6: bfa0164cc3c1 ! 6: 11c38f7277c5 checkout-index: ensure full index\n> @@ builtin/checkout-index.c: static void checkout_all(const char *prefix, int prefi\n> int i, errs = 0;\n> struct cache_entry *last_ce = NULL;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr ; i++) {\n> struct cache_entry *ce = active_cache[i];\n> 7: 1bb7b6117e41 ! 7: fd04adbb3f79 checkout: ensure full index\n> @@ builtin/checkout.c: static int checkout_worktree(const struct checkout_opts *opt\n> NULL);\n>\n> enable_delayed_checkout(&state);\n> ++\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (pos = 0; pos < active_nr; pos++) {\n> struct cache_entry *ce = active_cache[pos];\n> @@ builtin/checkout.c: static int checkout_paths(const struct checkout_opts *opts,\n> * Make sure all pathspecs participated in locating the paths\n> * to be checked out.\n> */\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (pos = 0; pos < active_nr; pos++)\n> if (opts->overlay_mode)\n> 8: a59e45c4ae8f ! 8: 65704f39edc9 commit: ensure full index\n> @@ builtin/commit.c: static int list_paths(struct string_list *list, const char *wi\n> free(max_prefix);\n> }\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> const struct cache_entry *ce = active_cache[i];\n> @@ builtin/commit.c: static int prepare_to_commit(const char *index_file, const cha\n> if (get_oid(parent, &oid)) {\n> int i, ita_nr = 0;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++)\n> if (ce_intent_to_add(active_cache[i]))\n> 9: 83040e7558f3 ! 9: 739f3fe9edf2 difftool: ensure full index\n> @@ builtin/difftool.c: static int run_dir_diff(const char *extcmd, int symlinks, co\n> setenv(\"GIT_DIFFTOOL_DIRDIFF\", \"true\", 1);\n> rc = run_command_v_opt(helper_argv, flags);\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&wtindex);\n> +\n> /*\n> 10: 988f7bd2d736 ! 10: 779a86ad1ec4 fsck: ensure full index\n> @@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)\n> verify_index_checksum = 1;\n> verify_ce_order = 1;\n> read_cache();\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> unsigned int mode;\n> 11: abe189051a0c ! 11: 8c0d377054fa grep: ensure full index\n> @@ builtin/grep.c: static int grep_cache(struct grep_opt *opt,\n> if (repo_read_index(repo) < 0)\n> die(_(\"index file corrupt\"));\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(repo->index);\n> for (nr = 0; nr < repo->index->cache_nr; nr++) {\n> const struct cache_entry *ce = repo->index->cache[nr];\n> 12: 00c8a7e1d119 ! 12: beaa1467cabb ls-files: ensure full index\n> @@ builtin/ls-files.c: static void show_files(struct repository *repo, struct dir_s\n>\n> if (!(show_cached || show_stage || show_deleted || show_modified))\n> return;\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(repo->index);\n> for (i = 0; i < repo->index->cache_nr; i++) {\n> const struct cache_entry *ce = repo->index->cache[i];\n> @@ builtin/ls-files.c: void overlay_tree_on_index(struct index_state *istate,\n> die(\"bad tree-ish %s\", tree_name);\n>\n> /* Hoist the unmerged entries up to stage #3 to make room */\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> struct cache_entry *ce = istate->cache[i];\n> 13: 1288f02177d2 ! 13: 73684141fcff merge-index: ensure full index\n> @@ builtin/merge-index.c: static void merge_one_path(const char *path)\n> static void merge_all(void)\n> {\n> int i;\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> const struct cache_entry *ce = active_cache[i];\n> @@ builtin/merge-index.c: int cmd_merge_index(int argc, const char **argv, const ch\n>\n> read_cache();\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> +\n> i = 1;\n> 14: d6816560b32f ! 14: 6ea81a49c6b5 rm: ensure full index\n> @@ builtin/rm.c: int cmd_rm(int argc, const char **argv, const char *prefix)\n>\n> seen = xcalloc(pathspec.nr, 1);\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++) {\n> const struct cache_entry *ce = active_cache[i];\n> 15: 92ccd31dd343 ! 15: 49ca5ed05c8d sparse-checkout: ensure full index\n> @@ Metadata\n> Author: Derrick Stolee \n>\n> ## Commit message ##\n> - sparse-checkout: ensure full index\n> + stash: ensure full index\n>\n> - When adjusting the sparse-checkout definition, ensure that a sparse\n> - index is expanded to a full one to avoid unexpected behavior. This is a\n> - top candidate for later integration with the sparse-index, but requires\n> - careful testing.\n> + Before iterating over all cache entries, ensure that a sparse index is\n> + expanded to a full index to avoid unexpected behavior.\n>\n> Signed-off-by: Derrick Stolee \n>\n> @@ builtin/stash.c: static int do_push_stash(const struct pathspec *ps, const char\n> int i;\n> char *ps_matched = xcalloc(ps->nr, 1);\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (i = 0; i < active_nr; i++)\n> ce_path_match(&the_index, active_cache[i], ps,\n> 16: 796449758a08 ! 16: 9c4bb187c15d update-index: ensure full index\n> @@ builtin/update-index.c: static int do_reupdate(int ac, const char **av,\n> */\n> has_head = 0;\n> redo:\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(&the_index);\n> for (pos = 0; pos < active_nr; pos++) {\n> const struct cache_entry *ce = active_cache[pos];\n> 17: 45bbed6150a2 < -: ------------ diff-lib: ensure full index\n> 18: 8ce1f80985a4 ! 17: fae4c078c3ef dir: ensure full index\n> @@ dir.c: static void connect_wt_gitdir_in_nested(const char *sub_worktree,\n> if (repo_read_index(&subrepo) < 0)\n> die(_(\"index file corrupt in repo %s\"), subrepo.gitdir);\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(subrepo.index);\n> for (i = 0; i < subrepo.index->cache_nr; i++) {\n> const struct cache_entry *ce = subrepo.index->cache[i];\n> 19: 6e0b452f44c1 ! 18: 2b9180ee77d3 entry: ensure full index\n> @@ entry.c: static void mark_colliding_entries(const struct checkout *state,\n>\n> ce->ce_flags |= CE_MATCHED;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(state->istate);\n> for (i = 0; i < state->istate->cache_nr; i++) {\n> struct cache_entry *dup = state->istate->cache[i];\n> 20: fb4c7038b746 < -: ------------ merge-ort: ensure full index\n> 21: 57d59825627f ! 19: 1e3f6085a405 merge-recursive: ensure full index\n> @@ merge-recursive.c: static struct string_list *get_unmerged(struct index_state *i\n>\n> unmerged->strdup_strings = 1;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> struct string_list_item *item;\n> 22: f4c470686b27 ! 20: e62a597a9725 pathspec: ensure full index\n> @@ pathspec.c: void add_pathspec_matches_against_index(const struct pathspec *paths\n> num_unmatched++;\n> if (!num_unmatched)\n> return;\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> const struct cache_entry *ce = istate->cache[i];\n> 23: 81519782d4b2 ! 21: ebfffdbdd6ad read-cache: ensure full index\n> @@ read-cache.c: int refresh_index(struct index_state *istate, unsigned int flags,\n> */\n> preload_index(istate, pathspec, 0);\n> trace2_region_enter(\"index\", \"refresh\", NULL);\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> struct cache_entry *ce, *new_entry;\n> @@ read-cache.c: int repo_index_has_changes(struct repository *repo,\n> diff_flush(&opt);\n> return opt.flags.has_changes != 0;\n> } else {\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; sb && i < istate->cache_nr; i++) {\n> if (i)\n> 24: 39e8bea0c1ca ! 22: 495b07a87973 resolve-undo: ensure full index\n> @@ resolve-undo.c: void unmerge_marked_index(struct index_state *istate)\n> if (!istate->resolve_undo)\n> return;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> const struct cache_entry *ce = istate->cache[i];\n> @@ resolve-undo.c: void unmerge_index(struct index_state *istate, const struct path\n> if (!istate->resolve_undo)\n> return;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> const struct cache_entry *ce = istate->cache[i];\n> 25: 6abb1813ae10 ! 23: 3144114d1a75 revision: ensure full index\n> @@ revision.c: static void do_add_index_objects_to_pending(struct rev_info *revs,\n> {\n> int i;\n>\n> ++ /* TODO: audit for interaction with sparse-index. */\n> + ensure_full_index(istate);\n> for (i = 0; i < istate->cache_nr; i++) {\n> struct cache_entry *ce = istate->cache[i];\n> 26: 1690d33c11e0 = 24: d52c72b4a7b9 sparse-index: expand_to_path()\n> 27: 40cfc59f695a = 25: 7e2d3fae9a2a name-hash: use expand_to_path()\n>\n> --\n> gitgitgadget\n"},{"id":"421057","messageId":"4768fac6-1d8d-4865-76a2-19f110f66219@gmail.com","threadId":"55334","inReplyTo":"CABPp-BE5xLMsVdamAb=MgiBnXKQrYDPJXa1TWutDK+nVYRPyRQ@mail.gmail.com","subject":"Re: [PATCH v2 24/25] sparse-index: expand_to_path()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-06T11:46:17Z","receivedAt":"2021-04-06T11:46:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/5/2021 3:32 PM, Elijah Newren wrote:\n> On Wed, Mar 31, 2021 at 6:50 PM Derrick Stolee via GitGitGadget\n> wrote:\n>>\n>> From: Derrick Stolee >> + if (index_file_exists(istate, path_mutable.buf,\n>> + path_mutable.len, icase)) {\n>> + /*\n>> + * We found a parent directory in the name-hash\n>> + * hashtable, which means that this entry could\n>> + * exist within a sparse-directory entry. Expand\n>> + * accordingly.\n> \n> \"this entry\" is a bit unclear; it might be worth referring to the\n> \"path\" variable instead. I think it'd also help to explain the\n> reasoning for using the '/' character, because while it's clear when\n> looking at the series, just running across it in the code it might be\n> easy to forget or miss. Perhaps:\n> \n> * We found a parent directory in the name-hash\n> * hashtable, because only sparse directory entries\n> * have a trailing '/' character. Since \"path\" wasn't\n> * in the index, perhaps it exists within this\n> * sparse-directory. Expand accordingly.\n\nA good recommendation. Thanks.\n\n-Stolee\n"},{"id":"421785","messageId":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v2.git.1617241802.gitgitgadget@gmail.com","subject":"[PATCH v3 00/26] Sparse Index: API protections","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:51Z","receivedAt":"2021-04-12T21:08:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Here is the second patch series submission coming out of the sparse-index\nRFC [1].\n\n[1]\nhttps://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n\nThis is based on ds/sparse-index.\n\nThe point of this series is to insert protections for the consumers of the\nin-memory index to avoid unintended behavior change when using a sparse\nindex versus a full one.\n\nWe mark certain regions of code as needing a full index, so we call\nensure_full_index() to expand a sparse index to a full one, if necessary.\nThese protections are inserted file-by-file in every loop over all cache\nentries. Well, \"most\" loops, because some are going to be handled in the\nvery next series so I leave them out.\n\nMany callers use index_name_pos() to find a path by name. In these cases, we\ncan check if that position resolves to a sparse directory instance. In those\ncases, we just expand to a full index and run the search again.\n\nThe last few patches deal with the name-hash hashtable for doing O(1)\nlookups.\n\nThese protections don't do much right now, since the previous series created\nthe_repository->settings.command_requires_full_index to guard all index\nreads and writes to ensure the in-memory copy is full for commands that have\nnot been tested with the sparse index yet.\n\nHowever, after this series is complete, we now have a straight-forward plan\nfor making commands \"sparse aware\" one-by-one:\n\n 1. Disable settings.command_requires_full_index to allow an in-memory\n sparse-index.\n 2. Run versions of that command under a debugger, breaking on\n ensure_full_index().\n 3. Examine the call stack to determine the context of that expansion, then\n implement the proper behavior in those locations.\n 4. Add tests to ensure we are checking this logic in the presence of sparse\n directory entries.\n\nI will admit that mostly it is the writing of the test cases that takes the\nmost time in the conversions I've done so far.\n\n\nUpdates in v3\n=============\n\n * I updated based on Elijah's feedback.\n * One new patch splits out a change that Elijah (rightfully) pointed out\n did not belong with the patch it was originally in.\n\nI gave it time to see if any other comments came in, but it looks like\nreview stabilized. I probably waited a bit longer than I should have.\n\n\nUpdates in v2\n=============\n\n * Rebased onto v5 of ds/sparse-index\n * Updated the technical doc to describe how these protections are guards to\n keep behavior consistent between a sparse-index and a full index. Whether\n or not that behavior is \"correct\" can be interrogated later.\n * Calls to ensure_full_index() are marked with a TODO comment saying these\n calls should be audited later (with tests).\n * Fixed an incorrectly squashed commit message.\n * Dropped the diff-lib.c commit because it was erroneously included in v2.\n * Dropped the merge-ort.c commit because of conflicts with work in flight\n and a quick audit that it is not needed.\n * I reviewed the merge of this topic with mt/add-rm-in-sparse-checkout and\n found it equivalent to what I would have done.\n\nThanks, -Stolee\n\nDerrick Stolee (26):\n sparse-index: API protection strategy\n *: remove 'const' qualifier for struct index_state\n read-cache: expand on query into sparse-directory entry\n cache: move ensure_full_index() to cache.h\n add: ensure full index\n checkout-index: ensure full index\n checkout: ensure full index\n commit: ensure full index\n difftool: ensure full index\n fsck: ensure full index\n grep: ensure full index\n ls-files: ensure full index\n merge-index: ensure full index\n rm: ensure full index\n stash: ensure full index\n update-index: ensure full index\n dir: ensure full index\n entry: ensure full index\n merge-recursive: ensure full index\n pathspec: ensure full index\n read-cache: ensure full index\n resolve-undo: ensure full index\n revision: ensure full index\n name-hash: don't add directories to name_hash\n sparse-index: expand_to_path()\n name-hash: use expand_to_path()\n\n Documentation/technical/sparse-index.txt | 37 +++++++++++-\n attr.c | 14 ++---\n attr.h | 4 +-\n builtin/add.c | 2 +\n builtin/checkout-index.c | 2 +\n builtin/checkout.c | 5 ++\n builtin/commit.c | 4 ++\n builtin/difftool.c | 3 +\n builtin/fsck.c | 2 +\n builtin/grep.c | 2 +\n builtin/ls-files.c | 14 +++--\n builtin/merge-index.c | 5 ++\n builtin/rm.c | 2 +\n builtin/stash.c | 2 +\n builtin/update-index.c | 2 +\n cache.h | 7 ++-\n convert.c | 26 ++++-----\n convert.h | 20 +++----\n dir.c | 14 +++--\n dir.h | 8 +--\n entry.c | 2 +\n merge-recursive.c | 4 +-\n name-hash.c | 11 +++-\n pathspec.c | 8 ++-\n pathspec.h | 6 +-\n read-cache.c | 35 ++++++++++--\n resolve-undo.c | 4 ++\n revision.c | 2 +\n sparse-index.c | 73 ++++++++++++++++++++++++\n sparse-index.h | 14 ++++-\n submodule.c | 6 +-\n submodule.h | 6 +-\n 32 files changed, 273 insertions(+), 73 deletions(-)\n\n\nbase-commit: c9e40ae8ec41c5566e5849a87c969fa81ef49fcd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/906\n\nRange-diff vs v2:\n\n 1: 7484e085e342 = 1: 4731f610ba6e sparse-index: API protection strategy\n 2: 098b2c9ef352 = 2: d3a92538edb6 *: remove 'const' qualifier for struct index_state\n 3: 737d27e18d64 = 3: f4b77aa18b93 read-cache: expand on query into sparse-directory entry\n 4: db5c100f3e2b = 4: da17774a53c5 cache: move ensure_full_index() to cache.h\n 5: 4a5fc2eb5a9f = 5: b59c9f482828 add: ensure full index\n 6: 11c38f7277c5 = 6: 0082855b5961 checkout-index: ensure full index\n 7: fd04adbb3f79 = 7: e2ac527143ff checkout: ensure full index\n 8: 65704f39edc9 = 8: 1a3b51fd3c4b commit: ensure full index\n 9: 739f3fe9edf2 = 9: 8c61d40dfe01 difftool: ensure full index\n 10: 779a86ad1ec4 = 10: 45b603379422 fsck: ensure full index\n 11: 8c0d377054fa = 11: 97124e9fdc7f grep: ensure full index\n 12: beaa1467cabb = 12: b00e214515e8 ls-files: ensure full index\n 13: 73684141fcff = 13: 6497f2ce225b merge-index: ensure full index\n 14: 6ea81a49c6b5 = 14: 175f3bc6b336 rm: ensure full index\n 15: 49ca5ed05c8d = 15: daa77e84e0e2 stash: ensure full index\n 16: 9c4bb187c15d = 16: 8c5336964d9b update-index: ensure full index\n 17: fae4c078c3ef = 17: 08a62c23c8f7 dir: ensure full index\n 18: 2b9180ee77d3 = 18: 825ebceee508 entry: ensure full index\n 19: 1e3f6085a405 = 19: 3673db517235 merge-recursive: ensure full index\n 20: e62a597a9725 = 20: 4d3f6de29a63 pathspec: ensure full index\n 21: ebfffdbdd6ad = 21: bda9cab15966 read-cache: ensure full index\n 22: 495b07a87973 = 22: 38f295a41ec1 resolve-undo: ensure full index\n 23: 3144114d1a75 = 23: f928e104f0d3 revision: ensure full index\n -: ------------ > 24: 5fd83dcf2747 name-hash: don't add directories to name_hash\n 24: d52c72b4a7b9 ! 25: 335fec3676a0 sparse-index: expand_to_path()\n @@ sparse-index.c: void ensure_full_index(struct index_state *istate)\n +\t\t\t\t path_mutable.len, icase)) {\n +\t\t\t/*\n +\t\t\t * We found a parent directory in the name-hash\n -+\t\t\t * hashtable, which means that this entry could\n -+\t\t\t * exist within a sparse-directory entry. Expand\n -+\t\t\t * accordingly.\n ++\t\t\t * hashtable, because only sparse directory entries\n ++\t\t\t * have a trailing '/' character. Since \"path\" wasn't\n ++\t\t\t * in the index, perhaps it exists within this\n ++\t\t\t * sparse-directory. Expand accordingly.\n +\t\t\t */\n +\t\t\tensure_full_index(istate);\n +\t\t\tbreak;\n 25: 7e2d3fae9a2a ! 26: 1f3af8a886e5 name-hash: use expand_to_path()\n @@ name-hash.c\n \n struct dir_entry {\n \tstruct hashmap_entry ent;\n -@@ name-hash.c: static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)\n - \tif (ce->ce_flags & CE_HASHED)\n - \t\treturn;\n - \tce->ce_flags |= CE_HASHED;\n -+\n -+\tif (S_ISSPARSEDIR(ce->ce_mode)) {\n -+\t\tadd_dir_entry(istate, ce);\n -+\t\treturn;\n -+\t}\n -+\n - \thashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n - \thashmap_add(&istate->name_hash, &ce->ent);\n - \n @@ name-hash.c: int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n \tstruct dir_entry *dir;\n \n\n-- \ngitgitgadget\n"},{"id":"421786","messageId":"4731f610ba6e70a271126e513f8df46d4e0cc5c6.1618261697.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 01/26] sparse-index: API protection strategy","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:52Z","receivedAt":"2021-04-12T21:08:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nEdit and expand the sparse-index design document with the plan for\nguarding index operations with ensure_full_index().\n\nNotably, the plan has changed to not have an expand_to_path() method in\nfavor of checking for a sparse-directory hit inside of the\nindex_path_pos() API.\n\nThe changes that follow this one will incrementally add\nensure_full_index() guards to iterations over all cache entries. Some\niterations over the cache entries are not protected due to a few\ncategories listed in the document. Since these are not being modified,\nhere is a short list of the files and methods that will not receive\nthese guards:\n\nLooking for non-zero stage:\n* builtin/add.c:chmod_pathspec()\n* builtin/merge.c:count_unmerged_entries()\n* merge-ort.c:record_conflicted_index_entries()\n* read-cache.c:unmerged_index()\n* rerere.c:check_one_conflict(), find_conflict(), rerere_remaining()\n* revision.c:prepare_show_merge()\n* sequencer.c:append_conflicts_hint()\n* wt-status.c:wt_status_collect_changes_initial()\n\nLooking for submodules:\n* builtin/submodule--helper.c:module_list_compute()\n* submodule.c: several methods\n* worktree.c:validate_no_submodules()\n\nPart of the index API:\n* name-hash.c: lazy init methods\n* preload-index.c:preload_thread(), preload_index()\n* read-cache.c: file format methods\n\nChecking for correct order of cache entries:\n* read-cache.c:check_ce_order()\n\nIgnores SKIP_WORKTREE entries or already aware:\n* unpack-trees.c:mark_new_skip_worktree()\n* wt-status.c:wt_status_check_sparse_checkout()\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/sparse-index.txt | 37 ++++++++++++++++++++++--\n 1 file changed, 35 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/sparse-index.txt b/Documentation/technical/sparse-index.txt\nindex 8d3d80804604..3b24c1a219f8 100644\n--- a/Documentation/technical/sparse-index.txt\n+++ b/Documentation/technical/sparse-index.txt\n@@ -85,8 +85,41 @@ index, as well.\n \n Next, consumers of the index will be guarded against operating on a\n sparse-index by inserting calls to `ensure_full_index()` or\n-`expand_index_to_path()`. After these guards are in place, we can begin\n-leaving sparse-directory entries in the in-memory index structure.\n+`expand_index_to_path()`. If a specific path is requested, then those will\n+be protected from within the `index_file_exists()` and `index_name_pos()`\n+API calls: they will call `ensure_full_index()` if necessary. The\n+intention here is to preserve existing behavior when interacting with a\n+sparse-checkout. We don't want a change to happen by accident, without\n+tests. Many of these locations may not need any change before removing the\n+guards, but we should not do so without tests to ensure the expected\n+behavior happens.\n+\n+It may be desirable to _change_ the behavior of some commands in the\n+presence of a sparse index or more generally in any sparse-checkout\n+scenario. In such cases, these should be carefully communicated and\n+tested. No such behavior changes are intended during this phase.\n+\n+During a scan of the codebase, not every iteration of the cache entries\n+needs an `ensure_full_index()` check. The basic reasons include:\n+\n+1. The loop is scanning for entries with non-zero stage. These entries\n+ are not collapsed into a sparse-directory entry.\n+\n+2. The loop is scanning for submodules. These entries are not collapsed\n+ into a sparse-directory entry.\n+\n+3. The loop is part of the index API, especially around reading or\n+ writing the format.\n+\n+4. The loop is checking for correct order of cache entries and that is\n+ correct if and only if the sparse-directory entries are in the correct\n+ location.\n+\n+5. The loop ignores entries with the `SKIP_WORKTREE` bit set, or is\n+ otherwise already aware of sparse directory entries.\n+\n+6. The sparse-index is disabled at this point when using the split-index\n+ feature, so no effort is made to protect the split-index API.\n \n Even after inserting these guards, we will keep expanding sparse-indexes\n for most Git commands using the `command_requires_full_index` repository\n-- \ngitgitgadget\n\n"},{"id":"421787","messageId":"f4b77aa18b93f4b3355cbb947fe7145a52d29e5c.1618261697.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 03/26] read-cache: expand on query into sparse-directory entry","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:54Z","receivedAt":"2021-04-12T21:08:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCallers to index_name_pos() or index_name_stage_pos() have a specific\npath in mind. If that happens to be a path with an ancestor being a\nsparse-directory entry, it can lead to unexpected results.\n\nIn the case that we did not find the requested path, check to see if the\nposition _before_ the inserted position is a sparse directory entry that\nmatches the initial segment of the input path (including the directory\nseparator at the end of the directory name). If so, then expand the\nindex to be a full index and search again. This expansion will only\nhappen once per index read.\n\nFuture enhancements could be more careful to expand only the necessary\nsparse directory entry, but then we would have a special \"not fully\nsparse, but also not fully expanded\" mode that could affect writing the\nindex to file. Since this only occurs if a specific file is requested\noutside of the sparse checkout definition, this is unlikely to be a\ncommon situation.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 21 +++++++++++++++++++++\n 1 file changed, 21 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3ad94578095e..3698bc7bf77d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -567,6 +567,27 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in\n \t\t}\n \t\tfirst = next+1;\n \t}\n+\n+\tif (istate->sparse_index &&\n+\t first > 0) {\n+\t\t/* Note: first <= istate->cache_nr */\n+\t\tstruct cache_entry *ce = istate->cache[first - 1];\n+\n+\t\t/*\n+\t\t * If we are in a sparse-index _and_ the entry before the\n+\t\t * insertion position is a sparse-directory entry that is\n+\t\t * an ancestor of 'name', then we need to expand the index\n+\t\t * and search again. This will only trigger once, because\n+\t\t * thereafter the index is fully expanded.\n+\t\t */\n+\t\tif (S_ISSPARSEDIR(ce->ce_mode) &&\n+\t\t ce_namelen(ce) < namelen &&\n+\t\t !strncmp(name, ce->name, ce_namelen(ce))) {\n+\t\t\tensure_full_index(istate);\n+\t\t\treturn index_name_stage_pos(istate, name, namelen, stage);\n+\t\t}\n+\t}\n+\n \treturn -first-1;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"421789","messageId":"d3a92538edb6efb1f0901307a128e3a49aaa21c3.1618261697.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 02/26] *: remove 'const' qualifier for struct index_state","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:53Z","receivedAt":"2021-04-12T21:08:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSeveral methods specify that they take a 'struct index_state' pointer\nwith the 'const' qualifier because they intend to only query the data,\nnot change it. However, we will be introducing a step very low in the\nmethod stack that might modify a sparse-index to become a full index in\nthe case that our queries venture inside a sparse-directory entry.\n\nThis change only removes the 'const' qualifiers that are necessary for\nthe following change which will actually modify the implementation of\nindex_name_stage_pos().\n\nSigned-off-by: Derrick Stolee \n---\n attr.c | 14 +++++++-------\n attr.h | 4 ++--\n builtin/ls-files.c | 10 +++++-----\n cache.h | 6 +++---\n convert.c | 26 +++++++++++++-------------\n convert.h | 20 ++++++++++----------\n dir.c | 12 ++++++------\n dir.h | 8 ++++----\n merge-recursive.c | 2 +-\n pathspec.c | 6 +++---\n pathspec.h | 6 +++---\n read-cache.c | 10 +++++-----\n submodule.c | 6 +++---\n submodule.h | 6 +++---\n 14 files changed, 68 insertions(+), 68 deletions(-)\n\ndiff --git a/attr.c b/attr.c\nindex 4ef85d668b54..8de553293ee4 100644\n--- a/attr.c\n+++ b/attr.c\n@@ -718,7 +718,7 @@ static struct attr_stack *read_attr_from_file(const char *path, int macro_ok)\n \treturn res;\n }\n \n-static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n+static struct attr_stack *read_attr_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t int macro_ok)\n {\n@@ -748,7 +748,7 @@ static struct attr_stack *read_attr_from_index(const struct index_state *istate,\n \treturn res;\n }\n \n-static struct attr_stack *read_attr(const struct index_state *istate,\n+static struct attr_stack *read_attr(struct index_state *istate,\n \t\t\t\t const char *path, int macro_ok)\n {\n \tstruct attr_stack *res = NULL;\n@@ -840,7 +840,7 @@ static void push_stack(struct attr_stack **attr_stack_p,\n \t}\n }\n \n-static void bootstrap_attr_stack(const struct index_state *istate,\n+static void bootstrap_attr_stack(struct index_state *istate,\n \t\t\t\t struct attr_stack **stack)\n {\n \tstruct attr_stack *e;\n@@ -878,7 +878,7 @@ static void bootstrap_attr_stack(const struct index_state *istate,\n \tpush_stack(stack, e, NULL, 0);\n }\n \n-static void prepare_attr_stack(const struct index_state *istate,\n+static void prepare_attr_stack(struct index_state *istate,\n \t\t\t const char *path, int dirlen,\n \t\t\t struct attr_stack **stack)\n {\n@@ -1078,7 +1078,7 @@ static void determine_macros(struct all_attrs_item *all_attrs,\n * If check->check_nr is non-zero, only attributes in check[] are collected.\n * Otherwise all attributes are collected.\n */\n-static void collect_some_attrs(const struct index_state *istate,\n+static void collect_some_attrs(struct index_state *istate,\n \t\t\t const char *path,\n \t\t\t struct attr_check *check)\n {\n@@ -1107,7 +1107,7 @@ static void collect_some_attrs(const struct index_state *istate,\n \tfill(path, pathlen, basename_offset, check->stack, check->all_attrs, rem);\n }\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path,\n \t\t struct attr_check *check)\n {\n@@ -1124,7 +1124,7 @@ void git_check_attr(const struct index_state *istate,\n \t}\n }\n \n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check)\n {\n \tint i;\ndiff --git a/attr.h b/attr.h\nindex 404548f028a8..3732505edae8 100644\n--- a/attr.h\n+++ b/attr.h\n@@ -190,14 +190,14 @@ void attr_check_free(struct attr_check *check);\n */\n const char *git_attr_name(const struct git_attr *);\n \n-void git_check_attr(const struct index_state *istate,\n+void git_check_attr(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n /*\n * Retrieve all attributes that apply to the specified path.\n * check holds the attributes and their values.\n */\n-void git_all_attrs(const struct index_state *istate,\n+void git_all_attrs(struct index_state *istate,\n \t\t const char *path, struct attr_check *check);\n \n enum git_attr_direction {\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..4f9ed1fb29b7 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -57,7 +57,7 @@ static const char *tag_modified = \"\";\n static const char *tag_skip_worktree = \"\";\n static const char *tag_resolve_undo = \"\";\n \n-static void write_eolinfo(const struct index_state *istate,\n+static void write_eolinfo(struct index_state *istate,\n \t\t\t const struct cache_entry *ce, const char *path)\n {\n \tif (show_eol) {\n@@ -122,7 +122,7 @@ static void print_debug(const struct cache_entry *ce)\n \t}\n }\n \n-static void show_dir_entry(const struct index_state *istate,\n+static void show_dir_entry(struct index_state *istate,\n \t\t\t const char *tag, struct dir_entry *ent)\n {\n \tint len = max_prefix_len;\n@@ -139,7 +139,7 @@ static void show_dir_entry(const struct index_state *istate,\n \twrite_name(ent->name);\n }\n \n-static void show_other_files(const struct index_state *istate,\n+static void show_other_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -152,7 +152,7 @@ static void show_other_files(const struct index_state *istate,\n \t}\n }\n \n-static void show_killed_files(const struct index_state *istate,\n+static void show_killed_files(struct index_state *istate,\n \t\t\t const struct dir_struct *dir)\n {\n \tint i;\n@@ -254,7 +254,7 @@ static void show_ce(struct repository *repo, struct dir_struct *dir,\n \t}\n }\n \n-static void show_ru_info(const struct index_state *istate)\n+static void show_ru_info(struct index_state *istate)\n {\n \tstruct string_list_item *item;\n \ndiff --git a/cache.h b/cache.h\nindex 8aede373aeb3..5006278c13ca 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -800,7 +800,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n * index_name_pos(&index, \"f\", 1) -> -3\n * index_name_pos(&index, \"g\", 1) -> -5\n */\n-int index_name_pos(const struct index_state *, const char *name, int namelen);\n+int index_name_pos(struct index_state *, const char *name, int namelen);\n \n /*\n * Some functions return the negative complement of an insert position when a\n@@ -850,8 +850,8 @@ int add_file_to_index(struct index_state *, const char *path, int flags);\n int chmod_index_entry(struct index_state *, struct cache_entry *ce, char flip);\n int ce_same_name(const struct cache_entry *a, const struct cache_entry *b);\n void set_object_name_for_intent_to_add_entry(struct cache_entry *ce);\n-int index_name_is_other(const struct index_state *, const char *, int);\n-void *read_blob_data_from_index(const struct index_state *, const char *, unsigned long *);\n+int index_name_is_other(struct index_state *, const char *, int);\n+void *read_blob_data_from_index(struct index_state *, const char *, unsigned long *);\n \n /* do stat comparison even if CE_VALID is true */\n #define CE_MATCH_IGNORE_VALID\t\t01\ndiff --git a/convert.c b/convert.c\nindex ee360c2f07ce..20b19abffa36 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -138,7 +138,7 @@ static const char *gather_convert_stats_ascii(const char *data, unsigned long si\n \t}\n }\n \n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path)\n {\n \tconst char *ret;\n@@ -222,7 +222,7 @@ static void check_global_conv_flags_eol(const char *path,\n \t}\n }\n \n-static int has_crlf_in_index(const struct index_state *istate, const char *path)\n+static int has_crlf_in_index(struct index_state *istate, const char *path)\n {\n \tunsigned long sz;\n \tvoid *data;\n@@ -496,7 +496,7 @@ static int encode_to_worktree(const char *path, const char *src, size_t src_len,\n \treturn 1;\n }\n \n-static int crlf_to_git(const struct index_state *istate,\n+static int crlf_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *buf,\n \t\t enum crlf_action crlf_action, int conv_flags)\n@@ -1307,7 +1307,7 @@ struct conv_attrs {\n \n static struct attr_check *check;\n \n-static void convert_attrs(const struct index_state *istate,\n+static void convert_attrs(struct index_state *istate,\n \t\t\t struct conv_attrs *ca, const char *path)\n {\n \tstruct attr_check_item *ccheck = NULL;\n@@ -1369,7 +1369,7 @@ void reset_parsed_attributes(void)\n \tuser_convert_tail = NULL;\n }\n \n-int would_convert_to_git_filter_fd(const struct index_state *istate, const char *path)\n+int would_convert_to_git_filter_fd(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1388,7 +1388,7 @@ int would_convert_to_git_filter_fd(const struct index_state *istate, const char\n \treturn apply_filter(path, NULL, 0, -1, NULL, ca.drv, CAP_CLEAN, NULL, NULL);\n }\n \n-const char *get_convert_attr_ascii(const struct index_state *istate, const char *path)\n+const char *get_convert_attr_ascii(struct index_state *istate, const char *path)\n {\n \tstruct conv_attrs ca;\n \n@@ -1414,7 +1414,7 @@ const char *get_convert_attr_ascii(const struct index_state *istate, const char\n \treturn \"\";\n }\n \n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags)\n {\n@@ -1448,7 +1448,7 @@ int convert_to_git(const struct index_state *istate,\n \treturn ret | ident_to_git(src, len, dst, ca.ident);\n }\n \n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd, struct strbuf *dst,\n \t\t\t int conv_flags)\n {\n@@ -1466,7 +1466,7 @@ void convert_to_git_filter_fd(const struct index_state *istate,\n \tident_to_git(dst->buf, dst->len, dst, ca.ident);\n }\n \n-static int convert_to_working_tree_internal(const struct index_state *istate,\n+static int convert_to_working_tree_internal(struct index_state *istate,\n \t\t\t\t\t const char *path, const char *src,\n \t\t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t\t int normalizing,\n@@ -1510,7 +1510,7 @@ static int convert_to_working_tree_internal(const struct index_state *istate,\n \treturn ret | ret_filter;\n }\n \n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n@@ -1519,7 +1519,7 @@ int async_convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, dco);\n }\n \n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta)\n@@ -1527,7 +1527,7 @@ int convert_to_working_tree(const struct index_state *istate,\n \treturn convert_to_working_tree_internal(istate, path, src, len, dst, 0, meta, NULL);\n }\n \n-int renormalize_buffer(const struct index_state *istate, const char *path,\n+int renormalize_buffer(struct index_state *istate, const char *path,\n \t\t const char *src, size_t len, struct strbuf *dst)\n {\n \tint ret = convert_to_working_tree_internal(istate, path, src, len, dst, 1, NULL, NULL);\n@@ -1964,7 +1964,7 @@ static struct stream_filter *ident_filter(const struct object_id *oid)\n * Note that you would be crazy to set CRLF, smudge/clean or ident to a\n * large binary blob you would want us not to slurp into the memory!\n */\n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *oid)\n {\ndiff --git a/convert.h b/convert.h\nindex e29d1026a686..0f7c8a1f0446 100644\n--- a/convert.h\n+++ b/convert.h\n@@ -65,41 +65,41 @@ struct checkout_metadata {\n \n extern enum eol core_eol;\n extern char *check_roundtrip_encoding;\n-const char *get_cached_convert_stats_ascii(const struct index_state *istate,\n+const char *get_cached_convert_stats_ascii(struct index_state *istate,\n \t\t\t\t\t const char *path);\n const char *get_wt_convert_stats_ascii(const char *path);\n-const char *get_convert_attr_ascii(const struct index_state *istate,\n+const char *get_convert_attr_ascii(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /* returns 1 if *dst was used */\n-int convert_to_git(const struct index_state *istate,\n+int convert_to_git(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst, int conv_flags);\n-int convert_to_working_tree(const struct index_state *istate,\n+int convert_to_working_tree(struct index_state *istate,\n \t\t\t const char *path, const char *src,\n \t\t\t size_t len, struct strbuf *dst,\n \t\t\t const struct checkout_metadata *meta);\n-int async_convert_to_working_tree(const struct index_state *istate,\n+int async_convert_to_working_tree(struct index_state *istate,\n \t\t\t\t const char *path, const char *src,\n \t\t\t\t size_t len, struct strbuf *dst,\n \t\t\t\t const struct checkout_metadata *meta,\n \t\t\t\t void *dco);\n int async_query_available_blobs(const char *cmd,\n \t\t\t\tstruct string_list *available_paths);\n-int renormalize_buffer(const struct index_state *istate,\n+int renormalize_buffer(struct index_state *istate,\n \t\t const char *path, const char *src, size_t len,\n \t\t struct strbuf *dst);\n-static inline int would_convert_to_git(const struct index_state *istate,\n+static inline int would_convert_to_git(struct index_state *istate,\n \t\t\t\t const char *path)\n {\n \treturn convert_to_git(istate, path, NULL, 0, NULL, 0);\n }\n /* Precondition: would_convert_to_git_filter_fd(path) == true */\n-void convert_to_git_filter_fd(const struct index_state *istate,\n+void convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t const char *path, int fd,\n \t\t\t struct strbuf *dst,\n \t\t\t int conv_flags);\n-int would_convert_to_git_filter_fd(const struct index_state *istate,\n+int would_convert_to_git_filter_fd(struct index_state *istate,\n \t\t\t\t const char *path);\n \n /*\n@@ -133,7 +133,7 @@ void reset_parsed_attributes(void);\n \n struct stream_filter; /* opaque */\n \n-struct stream_filter *get_stream_filter(const struct index_state *istate,\n+struct stream_filter *get_stream_filter(struct index_state *istate,\n \t\t\t\t\tconst char *path,\n \t\t\t\t\tconst struct object_id *);\n void free_stream_filter(struct stream_filter *);\ndiff --git a/dir.c b/dir.c\nindex fd8aa7c40faa..5b00dfb5b144 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -306,7 +306,7 @@ static int do_read_blob(const struct object_id *oid, struct oid_stat *oid_stat,\n * [1] Only if DO_MATCH_DIRECTORY is passed; otherwise, this is NOT a match.\n * [2] Only if DO_MATCH_LEADING_PATHSPEC is passed; otherwise, not a match.\n */\n-static int match_pathspec_item(const struct index_state *istate,\n+static int match_pathspec_item(struct index_state *istate,\n \t\t\t const struct pathspec_item *item, int prefix,\n \t\t\t const char *name, int namelen, unsigned flags)\n {\n@@ -429,7 +429,7 @@ static int match_pathspec_item(const struct index_state *istate,\n * pathspec did not match any names, which could indicate that the\n * user mistyped the nth pathspec.\n */\n-static int do_match_pathspec(const struct index_state *istate,\n+static int do_match_pathspec(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *name, int namelen,\n \t\t\t int prefix, char *seen,\n@@ -500,7 +500,7 @@ static int do_match_pathspec(const struct index_state *istate,\n \treturn retval;\n }\n \n-static int match_pathspec_with_flags(const struct index_state *istate,\n+static int match_pathspec_with_flags(struct index_state *istate,\n \t\t\t\t const struct pathspec *ps,\n \t\t\t\t const char *name, int namelen,\n \t\t\t\t int prefix, char *seen, unsigned flags)\n@@ -516,7 +516,7 @@ static int match_pathspec_with_flags(const struct index_state *istate,\n \treturn negative ? 0 : positive;\n }\n \n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *ps,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir)\n@@ -529,7 +529,7 @@ int match_pathspec(const struct index_state *istate,\n /**\n * Check if a submodule is a superset of the pathspec\n */\n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen)\n@@ -892,7 +892,7 @@ void add_pattern(const char *string, const char *base,\n \tadd_pattern_to_hashsets(pl, pattern);\n }\n \n-static int read_skip_worktree_file_from_index(const struct index_state *istate,\n+static int read_skip_worktree_file_from_index(struct index_state *istate,\n \t\t\t\t\t const char *path,\n \t\t\t\t\t size_t *size_out, char **data_out,\n \t\t\t\t\t struct oid_stat *oid_stat)\ndiff --git a/dir.h b/dir.h\nindex facfae47402a..51cb0e217247 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -354,7 +354,7 @@ int count_slashes(const char *s);\n int simple_length(const char *match);\n int no_wildcard(const char *string);\n char *common_prefix(const struct pathspec *pathspec);\n-int match_pathspec(const struct index_state *istate,\n+int match_pathspec(struct index_state *istate,\n \t\t const struct pathspec *pathspec,\n \t\t const char *name, int namelen,\n \t\t int prefix, char *seen, int is_dir);\n@@ -492,12 +492,12 @@ int git_fnmatch(const struct pathspec_item *item,\n \t\tconst char *pattern, const char *string,\n \t\tint prefix);\n \n-int submodule_path_match(const struct index_state *istate,\n+int submodule_path_match(struct index_state *istate,\n \t\t\t const struct pathspec *ps,\n \t\t\t const char *submodule_name,\n \t\t\t char *seen);\n \n-static inline int ce_path_match(const struct index_state *istate,\n+static inline int ce_path_match(struct index_state *istate,\n \t\t\t\tconst struct cache_entry *ce,\n \t\t\t\tconst struct pathspec *pathspec,\n \t\t\t\tchar *seen)\n@@ -506,7 +506,7 @@ static inline int ce_path_match(const struct index_state *istate,\n \t\t\t S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode));\n }\n \n-static inline int dir_path_match(const struct index_state *istate,\n+static inline int dir_path_match(struct index_state *istate,\n \t\t\t\t const struct dir_entry *ent,\n \t\t\t\t const struct pathspec *pathspec,\n \t\t\t\t int prefix, char *seen)\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 3d9207455b3a..b8de7a704eae 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -2987,7 +2987,7 @@ static int blob_unchanged(struct merge_options *opt,\n \tstruct strbuf obuf = STRBUF_INIT;\n \tstruct strbuf abuf = STRBUF_INIT;\n \tint ret = 0; /* assume changed for safety */\n-\tconst struct index_state *idx = opt->repo->index;\n+\tstruct index_state *idx = opt->repo->index;\n \n \tif (a->mode != o->mode)\n \t\treturn 0;\ndiff --git a/pathspec.c b/pathspec.c\nindex 7a229d8d22f2..b6e333965cb4 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -20,7 +20,7 @@\n * to use find_pathspecs_matching_against_index() instead.\n */\n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen)\n {\n \tint num_unmatched = 0, i;\n@@ -51,7 +51,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n * given pathspecs achieves against all items in the index.\n */\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate)\n+\t\t\t\t\t struct index_state *istate)\n {\n \tchar *seen = xcalloc(pathspec->nr, 1);\n \tadd_pathspec_matches_against_index(pathspec, istate, seen);\n@@ -702,7 +702,7 @@ void clear_pathspec(struct pathspec *pathspec)\n \tpathspec->nr = 0;\n }\n \n-int match_pathspec_attrs(const struct index_state *istate,\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item)\n {\ndiff --git a/pathspec.h b/pathspec.h\nindex 454ce364fac7..2ccc8080b6cc 100644\n--- a/pathspec.h\n+++ b/pathspec.h\n@@ -150,11 +150,11 @@ static inline int ps_strcmp(const struct pathspec_item *item,\n }\n \n void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\tconst struct index_state *istate,\n+\t\t\t\t\tstruct index_state *istate,\n \t\t\t\t\tchar *seen);\n char *find_pathspecs_matching_against_index(const struct pathspec *pathspec,\n-\t\t\t\t\t const struct index_state *istate);\n-int match_pathspec_attrs(const struct index_state *istate,\n+\t\t\t\t\t struct index_state *istate);\n+int match_pathspec_attrs(struct index_state *istate,\n \t\t\t const char *name, int namelen,\n \t\t\t const struct pathspec_item *item);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2410e6e0df13..3ad94578095e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -549,7 +549,7 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char\n \treturn 0;\n }\n \n-static int index_name_stage_pos(const struct index_state *istate, const char *name, int namelen, int stage)\n+static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage)\n {\n \tint first, last;\n \n@@ -570,7 +570,7 @@ static int index_name_stage_pos(const struct index_state *istate, const char *na\n \treturn -first-1;\n }\n \n-int index_name_pos(const struct index_state *istate, const char *name, int namelen)\n+int index_name_pos(struct index_state *istate, const char *name, int namelen)\n {\n \treturn index_name_stage_pos(istate, name, namelen, 0);\n }\n@@ -3389,8 +3389,8 @@ int repo_read_index_unmerged(struct repository *repo)\n * We helpfully remove a trailing \"/\" from directories so that\n * the output of read_directory can be used as-is.\n */\n-int index_name_is_other(const struct index_state *istate, const char *name,\n-\t\tint namelen)\n+int index_name_is_other(struct index_state *istate, const char *name,\n+\t\t\tint namelen)\n {\n \tint pos;\n \tif (namelen && name[namelen - 1] == '/')\n@@ -3408,7 +3408,7 @@ int index_name_is_other(const struct index_state *istate, const char *name,\n \treturn 1;\n }\n \n-void *read_blob_data_from_index(const struct index_state *istate,\n+void *read_blob_data_from_index(struct index_state *istate,\n \t\t\t\tconst char *path, unsigned long *size)\n {\n \tint pos, len;\ndiff --git a/submodule.c b/submodule.c\nindex 9767ba9893cc..83809a4f7bd2 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -33,7 +33,7 @@ static struct oid_array ref_tips_after_fetch;\n * will be disabled because we can't guess what might be configured in\n * .gitmodules unless the user resolves the conflict.\n */\n-int is_gitmodules_unmerged(const struct index_state *istate)\n+int is_gitmodules_unmerged(struct index_state *istate)\n {\n \tint pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));\n \tif (pos < 0) { /* .gitmodules not found or isn't merged */\n@@ -301,7 +301,7 @@ int is_submodule_populated_gently(const char *path, int *return_error_code)\n /*\n * Dies if the provided 'prefix' corresponds to an unpopulated submodule\n */\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix)\n {\n \tint i, prefixlen;\n@@ -331,7 +331,7 @@ void die_in_unpopulated_submodule(const struct index_state *istate,\n /*\n * Dies if any paths in the provided pathspec descends into a submodule\n */\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps)\n {\n \tint i, j;\ndiff --git a/submodule.h b/submodule.h\nindex 4ac6e31cf1f7..84640c49c114 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -39,7 +39,7 @@ struct submodule_update_strategy {\n };\n #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL}\n \n-int is_gitmodules_unmerged(const struct index_state *istate);\n+int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n int update_path_in_gitmodules(const char *oldpath, const char *newpath);\n@@ -60,9 +60,9 @@ int is_submodule_active(struct repository *repo, const char *path);\n * Otherwise the return error code is the same as of resolve_gitdir_gently.\n */\n int is_submodule_populated_gently(const char *path, int *return_error_code);\n-void die_in_unpopulated_submodule(const struct index_state *istate,\n+void die_in_unpopulated_submodule(struct index_state *istate,\n \t\t\t\t const char *prefix);\n-void die_path_inside_submodule(const struct index_state *istate,\n+void die_path_inside_submodule(struct index_state *istate,\n \t\t\t const struct pathspec *ps);\n enum submodule_update_type parse_submodule_update_type(const char *value);\n int parse_submodule_update_strategy(const char *value,\n-- \ngitgitgadget\n\n"},{"id":"421788","messageId":"1a3b51fd3c4bdfc50fa390d04c49b49acc47b90a.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 08/26] commit: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:59Z","receivedAt":"2021-04-12T21:08:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThese two loops iterate over all cache entries, so ensure that a sparse\nindex is expanded to a full index before we do so.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 739110c5a7f6..cf0c36d1dcb2 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -251,6 +251,8 @@ static int list_paths(struct string_list *list, const char *with_tree,\n \t\tfree(max_prefix);\n \t}\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tstruct string_list_item *item;\n@@ -931,6 +933,8 @@ static int prepare_to_commit(const char *index_file, const char *prefix,\n \t\tif (get_oid(parent, &oid)) {\n \t\t\tint i, ita_nr = 0;\n \n+\t\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\t\tensure_full_index(&the_index);\n \t\t\tfor (i = 0; i < active_nr; i++)\n \t\t\t\tif (ce_intent_to_add(active_cache[i]))\n \t\t\t\t\tita_nr++;\n-- \ngitgitgadget\n\n"},{"id":"421790","messageId":"0082855b59617f6f04f36024d79e99c4d8f303b7.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 06/26] checkout-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:57Z","receivedAt":"2021-04-12T21:08:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore we iterate over all cache entries, ensure that the index is not\nsparse. This loop in checkout_all() might be safe to iterate over a\nsparse index, but let's put this protection here until it can be\ncarefully tested.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/checkout-index.c b/builtin/checkout-index.c\nindex 023e49e271c2..2c2936a9dae0 100644\n--- a/builtin/checkout-index.c\n+++ b/builtin/checkout-index.c\n@@ -119,6 +119,8 @@ static void checkout_all(const char *prefix, int prefix_length)\n \tint i, errs = 0;\n \tstruct cache_entry *last_ce = NULL;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr ; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \t\tif (ce_stage(ce) != checkout_stage\n-- \ngitgitgadget\n\n"},{"id":"421791","messageId":"e2ac527143ffbfc1b480778c2ced786021c97af4.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 07/26] checkout: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:58Z","receivedAt":"2021-04-12T21:08:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries in the checkout builtin, ensure\nthat we have a full index to avoid any unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/checkout.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 0e6639052001..d0dbe63ea119 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -368,6 +368,9 @@ static int checkout_worktree(const struct checkout_opts *opts,\n \t\t\t NULL);\n \n \tenable_delayed_checkout(&state);\n+\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tstruct cache_entry *ce = active_cache[pos];\n \t\tif (ce->ce_flags & CE_MATCHED) {\n@@ -512,6 +515,8 @@ static int checkout_paths(const struct checkout_opts *opts,\n \t * Make sure all pathspecs participated in locating the paths\n \t * to be checked out.\n \t */\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++)\n \t\tif (opts->overlay_mode)\n \t\t\tmark_ce_for_checkout_overlay(active_cache[pos],\n-- \ngitgitgadget\n\n"},{"id":"421792","messageId":"b59c9f482828c54cfa8ccc2022cb72b60833daf5.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 05/26] add: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:56Z","receivedAt":"2021-04-12T21:08:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/add.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/add.c b/builtin/add.c\nindex ea762a41e3a2..afccf2fd5543 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -141,6 +141,8 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags)\n {\n \tint i, retval = 0;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tstruct cache_entry *ce = active_cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"421793","messageId":"da17774a53c547374aaf23e6f016d9e54d096f53.1618261697.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 04/26] cache: move ensure_full_index() to cache.h","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:07:55Z","receivedAt":"2021-04-12T21:08:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSoon we will insert ensure_full_index() calls across the codebase.\nInstead of also adding include statements for sparse-index.h, let's just\nuse the fact that anything that cares about the index already has\ncache.h in its includes.\n\nSigned-off-by: Derrick Stolee \n---\n cache.h | 1 +\n sparse-index.h | 1 -\n 2 files changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 5006278c13ca..b7e20e9778db 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -350,6 +350,7 @@ void add_name_hash(struct index_state *istate, struct cache_entry *ce);\n void remove_name_hash(struct index_state *istate, struct cache_entry *ce);\n void free_name_hash(struct index_state *istate);\n \n+void ensure_full_index(struct index_state *istate);\n \n /* Cache entry creation and cleanup */\n \ndiff --git a/sparse-index.h b/sparse-index.h\nindex 39dcc859735e..0268f38753c0 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -2,7 +2,6 @@\n #define SPARSE_INDEX_H__\n \n struct index_state;\n-void ensure_full_index(struct index_state *istate);\n int convert_to_sparse(struct index_state *istate);\n \n struct repository;\n-- \ngitgitgadget\n\n"},{"id":"421794","messageId":"8c61d40dfe01d89d248ff9208e955801c1a18c18.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 09/26] difftool: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:00Z","receivedAt":"2021-04-12T21:08:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index has\nbeen expanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/difftool.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/difftool.c b/builtin/difftool.c\nindex 6e18e623fddf..32c914dde6a0 100644\n--- a/builtin/difftool.c\n+++ b/builtin/difftool.c\n@@ -584,6 +584,9 @@ static int run_dir_diff(const char *extcmd, int symlinks, const char *prefix,\n \t\tsetenv(\"GIT_DIFFTOOL_DIRDIFF\", \"true\", 1);\n \trc = run_command_v_opt(helper_argv, flags);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&wtindex);\n+\n \t/*\n \t * If the diff includes working copy files and those\n \t * files were modified during the diff, then the changes\n-- \ngitgitgadget\n\n"},{"id":"421795","messageId":"b00e214515e842f0a4fdb6a6493dc83f3ee17398.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 12/26] ls-files: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:03Z","receivedAt":"2021-04-12T21:08:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid missing files.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/ls-files.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 4f9ed1fb29b7..a0b4e54d1149 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -317,6 +317,8 @@ static void show_files(struct repository *repo, struct dir_struct *dir)\n \n \tif (!(show_cached || show_stage || show_deleted || show_modified))\n \t\treturn;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(repo->index);\n \tfor (i = 0; i < repo->index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[i];\n \t\tstruct stat st;\n@@ -494,6 +496,8 @@ void overlay_tree_on_index(struct index_state *istate,\n \t\tdie(\"bad tree-ish %s\", tree_name);\n \n \t/* Hoist the unmerged entries up to stage #3 to make room */\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_stage(ce))\n-- \ngitgitgadget\n\n"},{"id":"421796","messageId":"6497f2ce225bba4f3706be3e5970f815b767b72d.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 13/26] merge-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:04Z","receivedAt":"2021-04-12T21:08:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/merge-index.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/merge-index.c b/builtin/merge-index.c\nindex 38ea6ad6ca25..c0383fe9df9a 100644\n--- a/builtin/merge-index.c\n+++ b/builtin/merge-index.c\n@@ -58,6 +58,8 @@ static void merge_one_path(const char *path)\n static void merge_all(void)\n {\n \tint i;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_stage(ce))\n@@ -80,6 +82,9 @@ int cmd_merge_index(int argc, const char **argv, const char *prefix)\n \n \tread_cache();\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n+\n \ti = 1;\n \tif (!strcmp(argv[i], \"-o\")) {\n \t\tone_shot = 1;\n-- \ngitgitgadget\n\n"},{"id":"421797","messageId":"175f3bc6b336539700d50b1906fd2a33c0bed906.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 14/26] rm: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:05Z","receivedAt":"2021-04-12T21:08:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rm.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/rm.c b/builtin/rm.c\nindex 4858631e0f02..5559a0b453a3 100644\n--- a/builtin/rm.c\n+++ b/builtin/rm.c\n@@ -293,6 +293,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix)\n \n \tseen = xcalloc(pathspec.nr, 1);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (i = 0; i < active_nr; i++) {\n \t\tconst struct cache_entry *ce = active_cache[i];\n \t\tif (!ce_path_match(&the_index, ce, &pathspec, seen))\n-- \ngitgitgadget\n\n"},{"id":"421798","messageId":"daa77e84e0e23ec675aef6805e7c2df7dedbb6bb.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 15/26] stash: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:06Z","receivedAt":"2021-04-12T21:08:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/stash.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex ba774cce674f..6fb7178ef2fa 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -1350,6 +1350,8 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q\n \t\tint i;\n \t\tchar *ps_matched = xcalloc(ps->nr, 1);\n \n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++)\n \t\t\tce_path_match(&the_index, active_cache[i], ps,\n \t\t\t\t ps_matched);\n-- \ngitgitgadget\n\n"},{"id":"421799","messageId":"8c5336964d9b513825a9f7ac983ff06cc5757d2f.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 16/26] update-index: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:07Z","receivedAt":"2021-04-12T21:08:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/update-index.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 79087bccea4b..f1f16f2de526 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -745,6 +745,8 @@ static int do_reupdate(int ac, const char **av,\n \t\t */\n \t\thas_head = 0;\n redo:\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(&the_index);\n \tfor (pos = 0; pos < active_nr; pos++) {\n \t\tconst struct cache_entry *ce = active_cache[pos];\n \t\tstruct cache_entry *old = NULL;\n-- \ngitgitgadget\n\n"},{"id":"421800","messageId":"08a62c23c8f77c9ea033bf4f7b8e164bdff8b7de.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 17/26] dir: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:08Z","receivedAt":"2021-04-12T21:08:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 5b00dfb5b144..166238e79f52 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3533,6 +3533,8 @@ static void connect_wt_gitdir_in_nested(const char *sub_worktree,\n \tif (repo_read_index(&subrepo) < 0)\n \t\tdie(_(\"index file corrupt in repo %s\"), subrepo.gitdir);\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(subrepo.index);\n \tfor (i = 0; i < subrepo.index->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = subrepo.index->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"421801","messageId":"825ebceee508bd5a00e5113bd43ccf82421ec5e2.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 18/26] entry: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:09Z","receivedAt":"2021-04-12T21:09:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n entry.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/entry.c b/entry.c\nindex 7b9f43716f76..891e4ba2b45a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -412,6 +412,8 @@ static void mark_colliding_entries(const struct checkout *state,\n \n \tce->ce_flags |= CE_MATCHED;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(state->istate);\n \tfor (i = 0; i < state->istate->cache_nr; i++) {\n \t\tstruct cache_entry *dup = state->istate->cache[i];\n \n-- \ngitgitgadget\n\n"},{"id":"421802","messageId":"3673db5172352152fcaa29feafae432530f36b36.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 19/26] merge-recursive: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:10Z","receivedAt":"2021-04-12T21:09:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n merge-recursive.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex b8de7a704eae..91d8597728c1 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -522,6 +522,8 @@ static struct string_list *get_unmerged(struct index_state *istate)\n \n \tunmerged->strdup_strings = 1;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct string_list_item *item;\n \t\tstruct stage_data *e;\n-- \ngitgitgadget\n\n"},{"id":"421803","messageId":"5fd83dcf2747dddc04a1386b7e57a48d3d9aa49b.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 24/26] name-hash: don't add directories to name_hash","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:15Z","receivedAt":"2021-04-12T21:09:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSparse directory entries represent a directory that is outside the\nsparse-checkout definition. These are not paths to blobs, so should not\nbe added to the name_hash table. Instead, they should be added to the\ndirectory hashtable when 'ignore_case' is true.\n\nAdd a condition to avoid placing sparse directories into the name_hash\nhashtable. This avoids filling the table with extra entries that will\nnever be queried.\n\nSigned-off-by: Derrick Stolee \n---\n name-hash.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 4e03fac9bb12..d08deaa2c9e7 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -109,8 +109,11 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)\n \tif (ce->ce_flags & CE_HASHED)\n \t\treturn;\n \tce->ce_flags |= CE_HASHED;\n-\thashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n-\thashmap_add(&istate->name_hash, &ce->ent);\n+\n+\tif (!S_ISSPARSEDIR(ce->ce_mode)) {\n+\t\thashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce)));\n+\t\thashmap_add(&istate->name_hash, &ce->ent);\n+\t}\n \n \tif (ignore_case)\n \t\tadd_dir_entry(istate, ce);\n-- \ngitgitgadget\n\n"},{"id":"421804","messageId":"f928e104f0d307b84477e4d37a701ecfe372b187.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 23/26] revision: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:14Z","receivedAt":"2021-04-12T21:09:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all index entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior. This case could\nbe integrated later by ensuring that we walk the tree in the\nsparse-directory entry, but the current behavior is only expecting\nblobs. Save this integration for later when it can be properly tested.\n\nSigned-off-by: Derrick Stolee \n---\n revision.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex b78733f5089b..b72e0ac1bdca 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1680,6 +1680,8 @@ static void do_add_index_objects_to_pending(struct rev_info *revs,\n {\n \tint i;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"421805","messageId":"38f295a41ec1bcbf0a67b0b1e12bd2590877f576.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 22/26] resolve-undo: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:13Z","receivedAt":"2021-04-12T21:09:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n resolve-undo.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 236320f179cb..ee36ca58b135 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -172,6 +172,8 @@ void unmerge_marked_index(struct index_state *istate)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (ce->ce_flags & CE_MATCHED)\n@@ -186,6 +188,8 @@ void unmerge_index(struct index_state *istate, const struct pathspec *pathspec)\n \tif (!istate->resolve_undo)\n \t\treturn;\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tif (!ce_path_match(istate, ce, pathspec, NULL))\n-- \ngitgitgadget\n\n"},{"id":"421806","messageId":"bda9cab159663cad9170527a826e204a88f91ac2.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 21/26] read-cache: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:12Z","receivedAt":"2021-04-12T21:09:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n read-cache.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3698bc7bf77d..a9dcf0ab4f78 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1577,6 +1577,8 @@ int refresh_index(struct index_state *istate, unsigned int flags,\n \t */\n \tpreload_index(istate, pathspec, 0);\n \ttrace2_region_enter(\"index\", \"refresh\", NULL);\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct cache_entry *ce, *new_entry;\n \t\tint cache_errno = 0;\n@@ -2498,6 +2500,8 @@ int repo_index_has_changes(struct repository *repo,\n \t\tdiff_flush(&opt);\n \t\treturn opt.flags.has_changes != 0;\n \t} else {\n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(istate);\n \t\tfor (i = 0; sb && i < istate->cache_nr; i++) {\n \t\t\tif (i)\n \t\t\t\tstrbuf_addch(sb, ' ');\n-- \ngitgitgadget\n\n"},{"id":"421807","messageId":"4d3f6de29a6393e46923bbaece69ed7e21a03101.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 20/26] pathspec: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:11Z","receivedAt":"2021-04-12T21:09:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full index to avoid unexpected behavior.\n\nSigned-off-by: Derrick Stolee \n---\n pathspec.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/pathspec.c b/pathspec.c\nindex b6e333965cb4..d67688bab74b 100644\n--- a/pathspec.c\n+++ b/pathspec.c\n@@ -36,6 +36,8 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec,\n \t\t\tnum_unmatched++;\n \tif (!num_unmatched)\n \t\treturn;\n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(istate);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tconst struct cache_entry *ce = istate->cache[i];\n \t\tce_path_match(istate, ce, pathspec, seen);\n-- \ngitgitgadget\n\n"},{"id":"421808","messageId":"1f3af8a886e5015a70a880dabd71f791d63c88df.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 26/26] name-hash: use expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:17Z","receivedAt":"2021-04-12T21:09:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA sparse-index loads the name-hash data for its entries, including the\nsparse-directory entries. If a caller asks for a path that is contained\nwithin a sparse-directory entry, we need to expand to a full index and\nrecalculate the name hash table before returning the result. Insert\ncalls to expand_to_path() to protect against this case.\n\nSigned-off-by: Derrick Stolee \n---\n name-hash.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex d08deaa2c9e7..383cf589969c 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -8,6 +8,7 @@\n #include \"cache.h\"\n #include \"thread-utils.h\"\n #include \"trace2.h\"\n+#include \"sparse-index.h\"\n \n struct dir_entry {\n \tstruct hashmap_entry ent;\n@@ -683,6 +684,7 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n \tstruct dir_entry *dir;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, 0);\n \tdir = find_dir_entry(istate, name, namelen);\n \treturn dir && dir->nr;\n }\n@@ -693,6 +695,7 @@ void adjust_dirname_case(struct index_state *istate, char *name)\n \tconst char *ptr = startPtr;\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, strlen(name), 0);\n \twhile (*ptr) {\n \t\twhile (*ptr && *ptr != '/')\n \t\t\tptr++;\n@@ -716,6 +719,7 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na\n \tunsigned int hash = memihash(name, namelen);\n \n \tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, icase);\n \n \tce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL,\n \t\t\t\t\t struct cache_entry, ent);\n-- \ngitgitgadget\n"},{"id":"421809","messageId":"335fec3676a0f66dc68582027970b0bad886bde7.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 25/26] sparse-index: expand_to_path()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:16Z","receivedAt":"2021-04-12T21:09:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome users of the index API have a specific path they are looking for,\nbut choose to use index_file_exists() to rely on the name-hash hashtable\ninstead of doing binary search with index_name_pos(). These users only\nneed to know a yes/no answer, not a position within the cache array.\n\nWhen the index is sparse, the name-hash hash table does not contain the\nfull list of paths within sparse directories. It _does_ contain the\ndirectory names for the sparse-directory entries.\n\nCreate a helper function, expand_to_path(), for intended use with the\nname-hash hashtable functions. The integration with name-hash.c will\nfollow in a later change.\n\nThe solution here is to use ensure_full_index() when we determine that\nthe requested path is within a sparse directory entry. This will\npopulate the name-hash hashtable as the index is recomputed from\nscratch.\n\nThere may be cases where the caller is trying to find an untracked path\nthat is not in the index but also is not within a sparse directory\nentry. We want to minimize the overhead for these requests. If we used\nindex_name_pos() to find the insertion order of the path, then we could\ndetermine from that position if a sparse-directory exists. (In fact,\njust calling index_name_pos() in that case would lead to expanding the\nindex to a full index.) However, this takes O(log N) time where N is the\nnumber of cache entries.\n\nTo keep the performance of this call based mostly on the input string,\nuse index_file_exists() to look for the ancestors of the path. Using the\nheuristic that a sparse directory is likely to have a small number of\nparent directories, we start from the bottom and build up. Use a string\nbuffer to allow mutating the path name to terminate after each slash for\neach hashset test.\n\nSigned-off-by: Derrick Stolee \n---\n sparse-index.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++\n sparse-index.h | 13 +++++++++\n 2 files changed, 86 insertions(+)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 95ea17174da3..6f21397e2ee0 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -283,3 +283,76 @@ void ensure_full_index(struct index_state *istate)\n \n \ttrace2_region_leave(\"index\", \"ensure_full_index\", istate->repo);\n }\n+\n+/*\n+ * This static global helps avoid infinite recursion between\n+ * expand_to_path() and index_file_exists().\n+ */\n+static int in_expand_to_path = 0;\n+\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase)\n+{\n+\tstruct strbuf path_mutable = STRBUF_INIT;\n+\tsize_t substr_len;\n+\n+\t/* prevent extra recursion */\n+\tif (in_expand_to_path)\n+\t\treturn;\n+\n+\tif (!istate || !istate->sparse_index)\n+\t\treturn;\n+\n+\tif (!istate->repo)\n+\t\tistate->repo = the_repository;\n+\n+\tin_expand_to_path = 1;\n+\n+\t/*\n+\t * We only need to actually expand a region if the\n+\t * following are both true:\n+\t *\n+\t * 1. 'path' is not already in the index.\n+\t * 2. Some parent directory of 'path' is a sparse directory.\n+\t */\n+\n+\tif (index_file_exists(istate, path, pathlen, icase))\n+\t\tgoto cleanup;\n+\n+\tstrbuf_add(&path_mutable, path, pathlen);\n+\tstrbuf_addch(&path_mutable, '/');\n+\n+\t/* Check the name hash for all parent directories */\n+\tsubstr_len = 0;\n+\twhile (substr_len < pathlen) {\n+\t\tchar temp;\n+\t\tchar *replace = strchr(path_mutable.buf + substr_len, '/');\n+\n+\t\tif (!replace)\n+\t\t\tbreak;\n+\n+\t\t/* replace the character _after_ the slash */\n+\t\treplace++;\n+\t\ttemp = *replace;\n+\t\t*replace = '\\0';\n+\t\tif (index_file_exists(istate, path_mutable.buf,\n+\t\t\t\t path_mutable.len, icase)) {\n+\t\t\t/*\n+\t\t\t * We found a parent directory in the name-hash\n+\t\t\t * hashtable, because only sparse directory entries\n+\t\t\t * have a trailing '/' character. Since \"path\" wasn't\n+\t\t\t * in the index, perhaps it exists within this\n+\t\t\t * sparse-directory. Expand accordingly.\n+\t\t\t */\n+\t\t\tensure_full_index(istate);\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t*replace = temp;\n+\t\tsubstr_len = replace - path_mutable.buf;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&path_mutable);\n+\tin_expand_to_path = 0;\n+}\ndiff --git a/sparse-index.h b/sparse-index.h\nindex 0268f38753c0..1115a0d7dd98 100644\n--- a/sparse-index.h\n+++ b/sparse-index.h\n@@ -4,6 +4,19 @@\n struct index_state;\n int convert_to_sparse(struct index_state *istate);\n \n+/*\n+ * Some places in the codebase expect to search for a specific path.\n+ * This path might be outside of the sparse-checkout definition, in\n+ * which case a sparse-index may not contain a path for that index.\n+ *\n+ * Given an index and a path, check to see if a leading directory for\n+ * 'path' exists in the index as a sparse directory. In that case,\n+ * expand that sparse directory to a full range of cache entries and\n+ * populate the index accordingly.\n+ */\n+void expand_to_path(struct index_state *istate,\n+\t\t const char *path, size_t pathlen, int icase);\n+\n struct repository;\n int set_sparse_index_config(struct repository *repo, int enable);\n \n-- \ngitgitgadget\n\n"},{"id":"421810","messageId":"45b603379422bcf7382ff905a97921e3ca5114a3.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 10/26] fsck: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:01Z","receivedAt":"2021-04-12T21:10:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen verifying all blobs reachable from the index, ensure that a sparse\nindex has been expanded to a full one to avoid missing some blobs.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fsck.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 821e7798c706..4d7f5c63ce0d 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -883,6 +883,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tverify_index_checksum = 1;\n \t\tverify_ce_order = 1;\n \t\tread_cache();\n+\t\t/* TODO: audit for interaction with sparse-index. */\n+\t\tensure_full_index(&the_index);\n \t\tfor (i = 0; i < active_nr; i++) {\n \t\t\tunsigned int mode;\n \t\t\tstruct blob *blob;\n-- \ngitgitgadget\n\n"},{"id":"421811","messageId":"97124e9fdc7f21d1f82162b8269551c023392fa3.1618261698.git.gitgitgadget@gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"[PATCH v3 11/26] grep: ensure full index","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-04-12T21:08:02Z","receivedAt":"2021-04-12T21:10:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nBefore iterating over all cache entries, ensure that a sparse index is\nexpanded to a full one so we do not miss blobs to scan. Later, this can\nintegrate more carefully with sparse indexes with proper testing.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/grep.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 4e91a253ac3b..c2d40414e975 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -504,6 +504,8 @@ static int grep_cache(struct grep_opt *opt,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(_(\"index file corrupt\"));\n \n+\t/* TODO: audit for interaction with sparse-index. */\n+\tensure_full_index(repo->index);\n \tfor (nr = 0; nr < repo->index->cache_nr; nr++) {\n \t\tconst struct cache_entry *ce = repo->index->cache[nr];\n \n-- \ngitgitgadget\n\n"},{"id":"421909","messageId":"CABPp-BGCpwguTJk3N-4Okrdpsio9Q7Hj1HSPPUZX1EZSiXkepQ@mail.gmail.com","threadId":"55334","inReplyTo":"pull.906.v3.git.1618261697.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/26] Sparse Index: API protections","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-13T16:02:04Z","receivedAt":"2021-04-13T16:02:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 12, 2021 at 2:08 PM Derrick Stolee via GitGitGadget\n wrote:\n>\n> Here is the second patch series submission coming out of the sparse-index\n> RFC [1].\n>\n> [1]\n> https://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/\n>\n> This is based on ds/sparse-index.\n>\n> The point of this series is to insert protections for the consumers of the\n> in-memory index to avoid unintended behavior change when using a sparse\n> index versus a full one.\n>\n> We mark certain regions of code as needing a full index, so we call\n> ensure_full_index() to expand a sparse index to a full one, if necessary.\n> These protections are inserted file-by-file in every loop over all cache\n> entries. Well, \"most\" loops, because some are going to be handled in the\n> very next series so I leave them out.\n>\n> Many callers use index_name_pos() to find a path by name. In these cases, we\n> can check if that position resolves to a sparse directory instance. In those\n> cases, we just expand to a full index and run the search again.\n>\n> The last few patches deal with the name-hash hashtable for doing O(1)\n> lookups.\n>\n> These protections don't do much right now, since the previous series created\n> the_repository->settings.command_requires_full_index to guard all index\n> reads and writes to ensure the in-memory copy is full for commands that have\n> not been tested with the sparse index yet.\n>\n> However, after this series is complete, we now have a straight-forward plan\n> for making commands \"sparse aware\" one-by-one:\n>\n> 1. Disable settings.command_requires_full_index to allow an in-memory\n> sparse-index.\n> 2. Run versions of that command under a debugger, breaking on\n> ensure_full_index().\n> 3. Examine the call stack to determine the context of that expansion, then\n> implement the proper behavior in those locations.\n> 4. Add tests to ensure we are checking this logic in the presence of sparse\n> directory entries.\n>\n> I will admit that mostly it is the writing of the test cases that takes the\n> most time in the conversions I've done so far.\n>\n>\n> Updates in v3\n> =============\n>\n> * I updated based on Elijah's feedback.\n> * One new patch splits out a change that Elijah (rightfully) pointed out\n> did not belong with the patch it was originally in.\n>\n> I gave it time to see if any other comments came in, but it looks like\n> review stabilized. I probably waited a bit longer than I should have.\n\nThis round looks good to me.\n\nReviewed-by: Elijah Newren \n"},{"id":"421983","messageId":"xmqqr1jcy4tl.fsf@gitster.g","threadId":"55334","inReplyTo":"CABPp-BGCpwguTJk3N-4Okrdpsio9Q7Hj1HSPPUZX1EZSiXkepQ@mail.gmail.com","subject":"Re: [PATCH v3 00/26] Sparse Index: API protections","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-14T20:44:06Z","receivedAt":"2021-04-14T20:44:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> This round looks good to me.\n>\n> Reviewed-by: Elijah Newren \n\nThanks; this kind of change inevitably would involve semantic\nconflicts with topics in flight, but we've seen Derrick works well\ntogether with others in such scenarios already, so let's run with\nthis version and see what happens.\n\nWill merge to 'next' until there are new issues found, by the end of\nthe week at the latest.\n\nThanks.\n"},{"id":"421999","messageId":"154a24ab-238f-2180-2e71-40f6ed5ceb5f@gmail.com","threadId":"55334","inReplyTo":"xmqqr1jcy4tl.fsf@gitster.g","subject":"Re: [PATCH v3 00/26] Sparse Index: API protections","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-15T02:42:24Z","receivedAt":"2021-04-15T02:42:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/14/2021 4:44 PM, Junio C Hamano wrote:\n> Elijah Newren writes:\n> \n>> This round looks good to me.\n>>\n>> Reviewed-by: Elijah Newren \n> \n> Thanks; this kind of change inevitably would involve semantic\n> conflicts with topics in flight, but we've seen Derrick works well\n> together with others in such scenarios already, so let's run with\n> this version and see what happens.\n\nThanks!\n\nSemantic conflicts like the one in mt/add-rm-sparse-checkout won't\nshow up until I start relaxing command_requires_full_index, so I'll\nneed to be careful at those points. But I'll keep a lookout for\nother issues contributors have when integrating across this change.\n\n> Will merge to 'next' until there are new issues found, by the end of\n> the week at the latest.\n\nThat timeline works for me.\n\n-Stolee\n"}]}