git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Mar 27, 2023, 07:49 UTC
Message-ID
<8afb1988-df49-d759-ead8-1bd384f09c82@gmx.de>
In-Reply-To
<CAPbDM0d5Xe=KTLECDCJYtpzmBt2wqzZrMWVcbiisM+abopenig@mail.gmail.com>
Hi Josh,
On Sat, 4 Mar 2023, Josh Smeaton wrote:
> git write-tree and then git diff-index reports thousands of files as
> `deleted` when core.fsmonitor is true, even though the files are still
> present on the file system.

I suspect that the issue is a missing `git update-index --refresh`. While `git diff` refreshes the index automatically, `git diff-index` does not (when you use low-level commands like `diff-index` you need to do such things explicitly).

If that does not fix your problem, I encourage you to distill the issue into a simple MCVE (https://stackoverflow.com/help/mcve), preferably in the form of a patch, say, to t7527 (see https://github.com/git/git/blob/v2.40.0/t/t7527-builtin-fsmonitor.sh).

Ciao, Johannes

Show 162 quoted lines
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> ```
> git config core.fsmonitor true
> touch new-file.txt
> git add new-file.txt
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> ```
>
> What did you expect to happen? (Expected behavior)
>
> I expected that no files would be reported as having changes.
>
> What happened instead? (Actual behavior)
>
> Thousands of unrelated files were reported as `deleted`.
>
> ```
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/__init__.py
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py
> :100644 000000 d8edaad92120781998e811277f966551b12fc823
> 0000000000000000000000000000000000000000 D
> src/tests/site/test_models.py
>
>  ... thousands of other files
> ```
>
> What's different between what you expected and what actually happened?
>
> Thousands of unrelated files were reported as having been deleted.
>
>
> Anything else you want to add:
>
> Myself and a few co-workers all ran into this issue in the last few days. It
> presented itself when pre-commit hooks were run (specifically with the
> pre-commit
> tool from github pre-commit).
>
> When committing changes, the pre-commit tool runs the equivalent of:
>
> `git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --`
>
> It then stashes and unstashes those changes, causing the files to be
> removed from
> the working tree.
>
>
> ```
> $ git config core.fsmonitor true
> $ git checkout -b test-branch
> Switched to a new branch 'test-branch'
>
> $ echo "test 3" >> newfile.txt
> $ git add .
> $ git status
> On branch test-branch
> Changes to be committed:
>   (use "git restore --staged <file>..." to unstage)
> new file:   newfile.txt
>
> $ git commit -a -m "will it fail?"
> [WARNING] Unstaged files detected.
> [INFO] Stashing unstaged files to
> /Users/josh/.cache/pre-commit/patch1677888420-10129.
> trim trailing whitespace.................................................Passed
> check python ast.....................................(no files to check)Skipped
> check for case conflicts.................................................Passed
> [INFO] Restored changes from
> /Users/josh/.cache/pre-commit/patch1677888420-10129.
> [test-branch 7c899582c19] will it fail?
>  1 file changed, 1 insertion(+)
>  create mode 100644 newfile.txt
>
> $ git status
> On branch test-branch
> Changes not staged for commit:
>   (use "git add/rm <file>..." to update what will be committed)
>   (use "git restore <file>..." to discard changes in working directory)
> deleted:    src/tests/__init__.py
> deleted:    src/tests/site/__init__.py
> deleted:    src/tests/site/test_models.py
>         ... lots more files ...
>
> no changes added to commit (use "git add" and/or "git commit -a")
>
> $ git status | wc -l
>    15298
>
> $ git restore .
>
> $ git status
>
> On branch test-branch
> nothing to commit, working tree clean
> ```
>
> It can be reproduced, in our repo, without using pre-commit at all:
>
> ```
>
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/__init__.py
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py
> :100644 000000 d8edaad92120781998e811277f966551b12fc823
> 0000000000000000000000000000000000000000 D
> src/tests/site/test_models.py
>
>  ... thousands of other files
> ```
>
> Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the
> correct thing:
>
> ```
> $ git update-index --really-refresh --no-fsmonitor
> warning: core.fsmonitor is set; remove it if you really want to
> disable fsmonitor
>
> $ git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> > no output
> ```
>
> The problem persists when running `update-index` without disabling fsmonitor.
>
> I have reported this issue to the pre-commit project, but it looks like an issue
> with fsmonitor + write-tree.
>
> See: pre-commit issues on github #2795
>
>
>
> [System Info]
> git version:
> git version 2.39.2 (others reproduced with 2.39.1)
> cpu: arm64
> no commit associated with this build
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37
> PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64
> compiler info: clang: 14.0.0 (clang-1400.0.29.202)
> libc info: no libc information available
> $SHELL (typically, interactive shell): /bin/zsh
>
>
> [Enabled Hooks]
> pre-commit
>
Previous: Josh Smeaton
Message 2 of 2 in “Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled”
  1. Josh SmeatonMar 4, 2023
  2. Johannes SchindelinMar 27, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.