Re: [PATCH v3 2/7] sparse-checkout: add basics of 'clean' command

>
> From: Derrick Stolee <stolee@gmail.com>
>
> When users change their sparse-checkout definitions to add new
> directories and remove old ones, there may be a few reasons why
> directories no longer in scope remain (ignored or excluded files still
> exist, Windows handles are still open, etc.). When these files still
> exist, the sparse index feature notices that a tracked, but sparse,
> directory still exists on disk and thus the index expands. This causes a
> performance hit _and_ the advice printed isn't very helpful. Using 'git
> clean' isn't enough (generally '-dfx' may be needed) but also this may
> not be sufficient.
>
> Add a new subcommand to 'git sparse-checkout' that removes these
> tracked-but-sparse directories.
>
> The implementation details provide a clear definition of what is happening,
> but it is difficult to describe this without including the internal
> implementation details. The core operation converts the index to a sparse
> index (in memory if not already on disk) and then deletes any directories in
> the worktree that correspond with a sparse directory entry in that sparse
> index.
>
> In the most common case, this means that a file will be removed if it is
> contained within a directory that is both tracked and outside of the
> sparse-checkout definition. However, there can be exceptions depending on
> the current state of the index:
>
>  * If the worktree has a modification to a tracked, sparse file, then that
>    file's parent directories will be expanded instead of represented as
>    sparse directories. Siblings of those parent directories may be
>    considered sparse.
>
>  * If the user staged a sparse file with "git add --sparse", then that file
>    loses the SKIP_WORKTREE bit until the sparse-checkout is reapplied. Until
>    then, that file's parent directories are not represented as sparse
>    directory entries and thus will not be removed. Siblings of those parent
>    directories may be considered sparse. (There may be other reasons why
>    the SKIP_WORKTREE bit was removed for a file and this impact on the
>    sparse directories will apply to those as well.)
>
>  * If the user has a merge conflict outside of the sparse-checkout
>    definition, then those conflict entries prevent the parent directories
>    from being represented as sparse directory entries and thus are not
>    removed.
>
>  * The cases above present reasons why certain _file conditions_ will impact
>    which _directories_ are considered sparse. The list of tracked
>    directories that are outside of the sparse-checkout definition but not
>    represented as a sparse directory further reduces the list of files that
>    will be removed.
>
> For these complicated reasons, the documentation details a potential list of
> files that will be "considered for removal" instead of defining the list
> concretely. The special cases can be handled by resolving conflicts,
> committing staged changes, and running 'git sparse-checkout reapply' to
> update the SKIP_WORKTREE bits as expected by the sparse-checkout definition.
>
> It is important to make clear that this operation will remove ignored and
> excluded files which would normally be ignored even by 'git clean -f' unless
> the '-x' or '-X' option is provided. This is the most extreme method for
> doing this, but it works when the sparse-checkout is in cone mode and is
> expected to rescope based on directories, not files.
>
> The current implementation always deletes these sparse directories
> without warning. This is unacceptable for a released version, but those
> features will be added in changes coming immediately after this one.
>
> Note that this will not remove an untracked directory (or any of its
> contents) if its parent is a tracked directory within the sparse-checkout
> definition. This is required to prevent removing data created by tools that
> perform caching operations for editors or build tools.
>
> Thus, 'git sparse-checkout clean' is both more aggressive and more careful
> than 'git clean -fx':
>
>  * It is more aggressive because it will remove _tracked_ files within the
>    sparse directories.
>
>  * It is less aggressive because it will leave _untracked_ files that are
>    not contained in sparse directories.
>
> These special cases will be handled more explicitly in a future change that
> expands tests for the 'git sparse-checkout clean' command. We handle some of
> the modified, staged, and committed states including some impact on 'git
> status' after cleaning.

> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  Documentation/git-sparse-checkout.adoc |  19 ++++-
>  builtin/sparse-checkout.c              |  64 ++++++++++++++-
>  t/t1091-sparse-checkout-builtin.sh     | 103 +++++++++++++++++++++++++
>  3 files changed, 184 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/git-sparse-checkout.adoc b/Documentation/git-sparse-checkout.adoc
> index 529a8edd9c..baaebce746 100644
> --- a/Documentation/git-sparse-checkout.adoc
> +++ b/Documentation/git-sparse-checkout.adoc
> @@ -9,7 +9,7 @@ git-sparse-checkout - Reduce your working tree to a subset of tracked files
>  SYNOPSIS
>  --------
>  [verse]
> -'git sparse-checkout' (init | list | set | add | reapply | disable | check-rules) [<options>]
> +'git sparse-checkout' (init | list | set | add | reapply | disable | check-rules | clean) [<options>]
>
>
>  DESCRIPTION
> @@ -111,6 +111,23 @@ flags, with the same meaning as the flags from the `set` command, in order
>  to change which sparsity mode you are using without needing to also respecify
>  all sparsity paths.
>
> +'clean'::
> +       Opportunistically remove files outside of the sparse-checkout
> +       definition. This command requires cone mode to use recursive
> +       directory matches to determine which files should be removed. A
> +       file is considered for removal if it is contained within a tracked
> +       directory that is outside of the sparse-checkout definition.
> ++
> +Some special cases, such as merge conflicts or modified files outside of
> +the sparse-checkout definition could lead to keeping files that would
> +otherwise be removed. Resolve conflicts, stage modifications, and use
> +`git sparse-checkout reapply` in conjunction with `git sparse-checkout
> +clean` to resolve these cases.
> ++
> +This command can be used to be sure the sparse index works efficiently,
> +though it does not require enabling the sparse index feature via the
> +`index.sparse=true` configuration.

> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index ab3a105fff..bdb7b21e32 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -1050,5 +1050,108 @@ test_expect_success 'check-rules null termination' '
>         test_cmp expect actual
>  '
>
> +test_expect_success 'clean' '
> +       git -C repo sparse-checkout set --cone deep/deeper1 &&
> +       git -C repo sparse-checkout reapply &&
> +       mkdir repo/deep/deeper2 repo/folder1 &&
> +
> +       # Add untracked files
> +       touch repo/deep/deeper2/file &&
> +       touch repo/folder1/file &&
> +
> +       cat >expect <<-\EOF &&
> +       Removing deep/deeper2/
> +       Removing folder1/
> +       EOF
> +
> +       git -C repo sparse-checkout clean >out &&
> +       test_cmp expect out &&
> +
> +       test_path_is_missing repo/deep/deeper2 &&
> +       test_path_is_missing repo/folder1
> +'
> +
> +test_expect_success 'clean with sparse file states' '
> +       test_when_finished git reset --hard &&
> +       git -C repo sparse-checkout set --cone deep/deeper1 &&
> +       mkdir repo/folder2 &&
> +
> +       # create an untracked file and a modified file
> +       touch repo/folder2/file &&
> +       echo dirty >repo/folder2/a &&
> +
> +       # First clean/reapply pass will do nothing.
> +       git -C repo sparse-checkout clean >out &&
> +       test_must_be_empty out &&
> +       test_path_exists repo/folder2/a &&
> +       test_path_exists repo/folder2/file &&
> +
> +       git -C repo sparse-checkout reapply 2>err &&
> +       test_grep folder2 err &&
> +       test_path_exists repo/folder2/a &&
> +       test_path_exists repo/folder2/file &&
> +
> +       # Now, stage the change to the tracked file.
> +       git -C repo add --sparse folder2/a &&
> +
> +       # Clean will continue not doing anything.
> +       git -C repo sparse-checkout clean >out &&
> +       test_line_count = 0 out &&
> +       test_path_exists repo/folder2/a &&
> +       test_path_exists repo/folder2/file &&
> +
> +       # But we can reapply to remove the staged change.
> +       git -C repo sparse-checkout reapply 2>err &&
> +       test_grep folder2 err &&
> +       test_path_is_missing repo/folder2/a &&
> +       test_path_exists repo/folder2/file &&
> +
> +       # We can clean now.
> +       cat >expect <<-\EOF &&
> +       Removing folder2/
> +       EOF
> +       git -C repo sparse-checkout clean >out &&
> +       test_cmp expect out &&
> +       test_path_is_missing repo/folder2 &&
> +
> +       # At the moment, the file is staged.
> +       cat >expect <<-\EOF &&
> +       M  folder2/a
> +       EOF
> +
> +       git -C repo status -s >out &&
> +       test_cmp expect out &&
> +
> +       # Reapply persists the modified state.
> +       git -C repo sparse-checkout reapply &&
> +       cat >expect <<-\EOF &&
> +       M  folder2/a
> +       EOF
> +       git -C repo status -s >out &&
> +       test_cmp expect out &&
> +
> +       # Committing the change leads to resolved status.
> +       git -C repo commit -m "modified" &&
> +       git -C repo status -s >out &&
> +       test_must_be_empty out &&
> +
> +       # Repeat, but this time commit before reapplying.
> +       mkdir repo/folder2/ &&
> +       echo dirtier >repo/folder2/a &&
> +       git -C repo add --sparse folder2/a &&
> +       git -C repo sparse-checkout clean >out &&
> +       test_must_be_empty out &&
> +       test_path_exists repo/folder2/a &&
> +
> +       # Committing without reapplying makes it look like a deletion
> +       # due to no skip-worktree bit.
> +       git -C repo commit -m "dirtier" &&
> +       git -C repo status -s >out &&
> +       test_must_be_empty out &&
> +
> +       git -C repo sparse-checkout reapply &&
> +       git -C repo status -s >out &&
> +       test_must_be_empty out
> +'
>
>  test_done
> --
> gitgitgadget