[PATCH 00/27] Sparse Index: API protections

Here is the second patch series submission coming out of the sparse-index RFC [1].

[1] https://lore.kernel.org/git/pull.847.git.1611596533.gitgitgadget@gmail.com/

This is based on v3 of the format series [2].

[2] https://lore.kernel.org/git/pull.883.v3.git.1615912983.gitgitgadget@gmail.com/

The point of this series is to insert protections for the consumers of the in-memory index. We mark certain regions of code as needing a full index, so we call ensure_full_index() to expand a sparse index to a full one, if necessary. These protections are inserted file-by-file in every loop over all cache entries. Well, "most" loops, because some are going to be handled in the very next series so I leave them out.

Many callers use index_name_pos() to find a path by name. In these cases, we can check if that position resolves to a sparse directory instance. In those cases, we just expand to a full index and run the search again.

The last few patches deal with the name-hash hashtable for doing O(1) lookups.

These protections don't do much right now, since the previous series created the_repository->settings.command_requires_full_index to guard all index reads and writes to ensure the in-memory copy is full for commands that have not been tested with the sparse index yet.

However, after this series is complete, we now have a straight-forward plan for making commands "sparse aware" one-by-one:

 1. Disable settings.command_requires_full_index to allow an in-memory
    sparse-index.
 2. Run versions of that command under a debugger, breaking on
    ensure_full_index().
 3. Examine the call stack to determine the context of that expansion, then
    implement the proper behavior in those locations.
 4. Add tests to ensure we are checking this logic in the presence of sparse
    directory entries.

I will admit that mostly it is the writing of the test cases that takes the most time in the conversions I've done so far.

Thanks, -Stolee

Derrick Stolee (27):
  *: remove 'const' qualifier for struct index_state
  read-cache: expand on query into sparse-directory entry
  sparse-index: API protection strategy
  cache: move ensure_full_index() to cache.h
  add: ensure full index
  checkout-index: ensure full index
  checkout: ensure full index
  commit: ensure full index
  difftool: ensure full index
  fsck: ensure full index
  grep: ensure full index
  ls-files: ensure full index
  merge-index: ensure full index
  rm: ensure full index
  sparse-checkout: ensure full index
  update-index: ensure full index
  diff-lib: ensure full index
  dir: ensure full index
  entry: ensure full index
  merge-ort: ensure full index
  merge-recursive: ensure full index
  pathspec: ensure full index
  read-cache: ensure full index
  resolve-undo: ensure full index
  revision: ensure full index
  sparse-index: expand_to_path()
  name-hash: use expand_to_path()

 Documentation/technical/sparse-index.txt | 32 ++++++++++-
 attr.c                                   | 14 ++---
 attr.h                                   |  4 +-
 builtin/add.c                            |  1 +
 builtin/checkout-index.c                 |  1 +
 builtin/checkout.c                       |  2 +
 builtin/commit.c                         |  2 +
 builtin/difftool.c                       |  2 +
 builtin/fsck.c                           |  1 +
 builtin/grep.c                           |  1 +
 builtin/ls-files.c                       | 12 ++--
 builtin/merge-index.c                    |  3 +
 builtin/rm.c                             |  1 +
 builtin/stash.c                          |  1 +
 builtin/update-index.c                   |  1 +
 cache.h                                  |  7 ++-
 convert.c                                | 26 ++++-----
 convert.h                                | 20 +++----
 diff-lib.c                               |  1 +
 dir.c                                    | 13 +++--
 dir.h                                    |  8 +--
 entry.c                                  |  1 +
 merge-ort.c                              |  1 +
 merge-recursive.c                        |  3 +-
 name-hash.c                              | 10 ++++
 pathspec.c                               |  7 ++-
 pathspec.h                               |  6 +-
 read-cache.c                             | 33 +++++++++--
 resolve-undo.c                           |  2 +
 revision.c                               |  1 +
 sparse-index.c                           | 72 ++++++++++++++++++++++++
 sparse-index.h                           | 14 ++++-
 submodule.c                              |  6 +-
 submodule.h                              |  6 +-
 34 files changed, 243 insertions(+), 72 deletions(-)

base-commit: 3db06ac46dd5c61e83d7fc4747615d616fdbbdda
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-906%2Fderrickstolee%2Fsparse-index%2Fprotections-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-906/derrickstolee/sparse-index/protections-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/906