git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/3] sparse-checkout: add 'clean' command

From
Derrick Stolee <stolee@gmail.com>
Date
Jul 9, 2025, 14:39 UTC
Message-ID
<2503c79c-68f3-4ed5-bbfd-3a7af07a89cc@gmail.com>
In-Reply-To
<xmqqa55etm5g.fsf@gitster.g>
On 7/8/2025 5:20 PM, Junio C Hamano wrote:
Show 24 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <stolee@gmail.com>
>>
>> When users change their sparse-checkout definitions to add new
>> directories and remove old ones, there may be a few reasons why
>> directories no longer in scope remain (ignored or excluded files still
>> exist, Windows handles are still open, etc.). When these files still
>> exist, the sparse index feature notices that a tracked, but sparse,
>> directory still exists on disk and thus the index expands. This causes a
>> performance hit _and_ the advice printed isn't very helpful. Using 'git
>> clean' isn't enough (generally '-dfx' may be needed) but also this may
>> not be sufficient.
>>
>> Add a new subcommand to 'git sparse-checkout' that removes these
>> tracked-but-sparse directories, including any excluded or ignored files
> 
> Are excluded files and ignored files form two separate sets, or are
> they one and the same?  Do files that users forgot to add (e.g. new
> source file that would not match any patterns listed in .gitignore)
> and object files left over from the previous compilation (most
> likely match *.o in .gitignore) treated the same way for the purpose
> of determining if the directory that is no longer in the cone can be
> removed?
I think of them as separate in my head because:
* .gitignore is committed to the repo, and is common to all users of
  the repo.
* .git/info/exclude is custom to each user, so users are choosing to
  ignore extra files that are atypical from most users.

In the monorepo I'm thinking about, .gitignore files are rather small because all build output has already been redirected out of the worktree for performance reasons. Thus, _most_ users don't have this problem. However, some users add extra excludes for things like vim files and those get leftover, causing invisible (to 'git status') pain.

Show 18 quoted lines
>> underneath. This is the most extreme method for doing this, but it works
>> when the sparse-checkout is in cone mode and is expected to rescope
>> based on directories, not files.
>>
>> Be sure to add a --dry-run option so users can predict what will be
>> deleted. In general, output the directories that are being removed so
>> users can know what was removed.
> 
> Hmph.  It would be safer to show not just the directories but which
> excluded files are about to be lost, wouldn't it, especially when
> the user is trying to play safe and see what potential damage they
> are looking at?
> > Also even though ignored files are "ignored and expendable", nobody
> marks their temporary file as "ignored but precious" (yet), so "it
> is listed in .gitignore so we can safely remove it" may not be a
> safe assumption for us to be making (yet).  Shouldn't we at least be
> listing these ignored files in --dry-run output, next to those files
> that the user may have forgotten to add?

I considered this, but mostly behind a potential --verbose option to list the files that are leftover. Much of the design here is that these _directories_ are out of scope, skipping over any details about the contained files, so I thought this directory-based output would communicate enough information.

A curious user may want to know "why are these directories still around?" and the more verbose output would assist.

Show 17 quoted lines
>> Note that untracked directories remain. Further, directories that
>> contain staged changes are not deleted. This is a detail that is partly
>> hidden by the implementation which relies on collapsing the index to a
>> sparse index in-memory and only deleting directories that are listed as
>> sparse in the index. If a staged change exists, then that entry is not
>> stored as a sparse tree entry and thus remains on-disk until committed
>> or reset.
> 
> Removing untracked directories is a job for "clean -d", so it makes
> sense for this new command not to touch them.  Not losing changes
> that have already been added is just a bad as losing new files that
> the user forgot to add, so it does make sense not to remove them.
> 
> I wonder if we need "-x" and/or "-X" options "clean" has (and
> perhaps "-d" that is a no-op, as the whole point of this subcommand
> is about removing directories from the working tree) to control its
> operation a bit finer-grained way.
I'm of two minds here.

My first inclination is "we already have 'git clean' for fine-grained control of removing ignored/excluded files".

My second inclination is "'git clean' would remove these ignored files even when they are within the sparse-checkout, so that's too big of a hammer".

There are a lot of ways to filter the files that would be removed, but I think that in this case most users are wanting a one-command way to get their sparse-checkout into a better state.

I'm not making any final statements here. I appreciate all of the thoughts around which options should be default and which should be hidden behind options.

Thanks, -Stolee

Previous: Junio C HamanoNext: Junio C Hamano
Message 9 of 69 in “sparse-checkout: add 'clean' command”
  1. 0/3 sparse-checkout: add 'clean' commandDerrick Stolee via GitGitGadget, Jul 8, 2025
  2. 1/3 sparse-checkout: remove use of the_repositoryDerrick Stolee via GitGitGadget, Jul 8, 2025
  3. Elijah NewrenJul 8, 2025
  4. Junio C HamanoJul 8, 2025
  5. 2/3 sparse-checkout: add 'clean' commandDerrick Stolee via GitGitGadget, Jul 8, 2025
  6. Patrick SteinhardtJul 8, 2025
  7. Junio C HamanoJul 8, 2025
  8. Junio C HamanoJul 8, 2025
  9. Derrick StoleeJul 9, 2025
  10. Junio C HamanoJul 9, 2025
  11. Elijah NewrenJul 8, 2025
  12. Derrick StoleeJul 9, 2025
  13. Elijah NewrenJul 9, 2025
  14. Derrick StoleeJul 15, 2025
  15. Elijah NewrenJul 15, 2025
  16. 3/3 sparse-index: point users to new 'clean' actionDerrick Stolee via GitGitGadget, Jul 8, 2025
  17. Elijah NewrenJul 8, 2025
  18. Patrick SteinhardtJul 8, 2025
  19. Elijah NewrenJul 8, 2025
  20. Elijah NewrenJul 8, 2025
  21. Junio C HamanoJul 8, 2025
  22. Derrick StoleeJul 9, 2025
  23. 0/8 sparse-checkout: add 'clean' commandDerrick Stolee via GitGitGadget, Jul 17, 2025
  24. 1/8 sparse-checkout: remove use of the_repositoryDerrick Stolee via GitGitGadget, Jul 17, 2025
  25. 2/8 sparse-checkout: add basics of 'clean' commandDerrick Stolee via GitGitGadget, Jul 17, 2025
  26. Elijah NewrenAug 5, 2025
  27. Derrick StoleeSep 11, 2025
  28. 3/8 sparse-checkout: match some 'clean' behaviorDerrick Stolee via GitGitGadget, Jul 17, 2025
  29. Elijah NewrenAug 5, 2025
  30. Derrick StoleeSep 11, 2025
  31. 4/8 dir: add generic "walk all files" helperDerrick Stolee via GitGitGadget, Jul 17, 2025
  32. Elijah NewrenAug 5, 2025
  33. 5/8 sparse-checkout: add --verbose option to 'clean'Derrick Stolee via GitGitGadget, Jul 17, 2025
  34. Elijah NewrenAug 5, 2025
  35. Derrick StoleeSep 11, 2025
  36. 6/8 sparse-index: point users to new 'clean' actionDerrick Stolee via GitGitGadget, Jul 17, 2025
  37. 7/8 t: expand tests around sparse merges and cleanDerrick Stolee via GitGitGadget, Jul 17, 2025
  38. 8/8 sparse-checkout: make 'clean' clear more filesDerrick Stolee via GitGitGadget, Jul 17, 2025
  39. Elijah NewrenAug 6, 2025
  40. Derrick StoleeSep 11, 2025
  41. Derrick StoleeSep 11, 2025
  42. Junio C HamanoAug 28, 2025
  43. Elijah NewrenAug 29, 2025
  44. Junio C HamanoAug 29, 2025
  45. Junio C HamanoAug 29, 2025
  46. Derrick StoleeAug 30, 2025
  47. 0/7 sparse-checkout: add 'clean' commandDerrick Stolee via GitGitGadget, Sep 12, 2025
  48. 1/7 sparse-checkout: remove use of the_repositoryDerrick Stolee via GitGitGadget, Sep 12, 2025
  49. 2/7 sparse-checkout: add basics of 'clean' commandDerrick Stolee via GitGitGadget, Sep 12, 2025
  50. Elijah NewrenOct 7, 2025
  51. Derrick StoleeOct 20, 2025
  52. 3/7 sparse-checkout: match some 'clean' behaviorDerrick Stolee via GitGitGadget, Sep 12, 2025
  53. 4/7 dir: add generic "walk all files" helperDerrick Stolee via GitGitGadget, Sep 12, 2025
  54. 5/7 sparse-checkout: add --verbose option to 'clean'Derrick Stolee via GitGitGadget, Sep 12, 2025
  55. Derrick StoleeSep 15, 2025
  56. Junio C HamanoSep 15, 2025
  57. Derrick StoleeSep 16, 2025
  58. 6/7 sparse-index: point users to new 'clean' actionDerrick Stolee via GitGitGadget, Sep 12, 2025
  59. Elijah NewrenOct 7, 2025
  60. Derrick StoleeOct 20, 2025
  61. 7/7 t: expand tests around sparse merges and cleanDerrick Stolee via GitGitGadget, Sep 12, 2025
  62. Junio C HamanoSep 12, 2025
  63. Derrick StoleeSep 26, 2025
  64. Elijah NewrenSep 26, 2025
  65. Elijah NewrenOct 7, 2025
  66. Derrick StoleeOct 20, 2025
  67. 8/8 sparse-index: improve advice message instructionsDerrick Stolee, Oct 20, 2025
  68. Junio C HamanoOct 20, 2025
  69. Elijah NewrenOct 24, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.