# Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled 2 messages from 2023-03-04 to 2023-03-27. Participants: Josh Smeaton, Johannes Schindelin. Thread: https://gitlist.dev/t/59339 ## Josh Smeaton, 2023-03-04 01:55 Subject: Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled Message-ID: URL: https://gitlist.dev/e/CAPbDM0d5Xe%3DKTLECDCJYtpzmBt2wqzZrMWVcbiisM%2Babopenig%40mail.gmail.com ``` git write-tree and then git diff-index reports thousands of files as `deleted` when core.fsmonitor is true, even though the files are still present on the file system. What did you do before the bug happened? (Steps to reproduce your issue) ˋˋˋ git config core.fsmonitor true touch new-file.txt git add new-file.txt git diff-index --ignore-submodules --exit-code --no-color --no-ext-diff $(git write-tree) -- ˋˋˋ What did you expect to happen? (Expected behavior) I expected that no files would be reported as having changes. What happened instead? (Actual behavior) Thousands of unrelated files were reported as `deleted`. ˋˋˋ git diff-index --ignore-submodules --exit-code --no-color --no-ext-diff $(git write-tree) -- :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 D src/tests/__init__.py :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 D src/tests/site/__init__.py :100644 000000 d8edaad92120781998e811277f966551b12fc823 0000000000000000000000000000000000000000 D src/tests/site/test_models.py ... thousands of other files ˋˋˋ What's different between what you expected and what actually happened? Thousands of unrelated files were reported as having been deleted. Anything else you want to add: Myself and a few co-workers all ran into this issue in the last few days. It presented itself when pre-commit hooks were run (specifically with the pre-commit tool from github pre-commit). When committing changes, the pre-commit tool runs the equivalent of: `git diff-index --ignore-submodules --exit-code --no-color --no-ext-diff $(git write-tree) --` It then stashes and unstashes those changes, causing the files to be removed from the working tree. ˋˋˋ $ git config core.fsmonitor true $ git checkout -b test-branch Switched to a new branch 'test-branch' $ echo "test 3" >> newfile.txt $ git add . $ git status On branch test-branch Changes to be committed: (use "git restore --staged ..." to unstage) new file: newfile.txt $ git commit -a -m "will it fail?" [WARNING] Unstaged files detected. [INFO] Stashing unstaged files to /Users/josh/.cache/pre-commit/patch1677888420-10129. trim trailing whitespace.................................................Passed check python ast.....................................(no files to check)Skipped check for case conflicts.................................................Passed [INFO] Restored changes from /Users/josh/.cache/pre-commit/patch1677888420-10129. [test-branch 7c899582c19] will it fail? 1 file changed, 1 insertion(+) create mode 100644 newfile.txt $ git status On branch test-branch Changes not staged for commit: (use "git add/rm ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) deleted: src/tests/__init__.py deleted: src/tests/site/__init__.py deleted: src/tests/site/test_models.py ... lots more files ... no changes added to commit (use "git add" and/or "git commit -a") $ git status | wc -l 15298 $ git restore . $ git status On branch test-branch nothing to commit, working tree clean ˋˋˋ It can be reproduced, in our repo, without using pre-commit at all: ˋˋˋ git diff-index --ignore-submodules --exit-code --no-color --no-ext-diff $(git write-tree) -- :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 D src/tests/__init__.py :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0000000000000000000000000000000000000000 D src/tests/site/__init__.py :100644 000000 d8edaad92120781998e811277f966551b12fc823 0000000000000000000000000000000000000000 D src/tests/site/test_models.py ... thousands of other files ˋˋˋ Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the correct thing: ˋˋˋ $ git update-index --really-refresh --no-fsmonitor warning: core.fsmonitor is set; remove it if you really want to disable fsmonitor $ git diff-index --ignore-submodules --exit-code --no-color --no-ext-diff $(git write-tree) -- > no output ˋˋˋ The problem persists when running `update-index` without disabling fsmonitor. I have reported this issue to the pre-commit project, but it looks like an issue with fsmonitor + write-tree. See: pre-commit issues on github #2795 [System Info] git version: git version 2.39.2 (others reproduced with 2.39.1) cpu: arm64 no commit associated with this build sizeof-long: 8 sizeof-size_t: 8 shell-path: /bin/sh feature: fsmonitor--daemon uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64 compiler info: clang: 14.0.0 (clang-1400.0.29.202) libc info: no libc information available $SHELL (typically, interactive shell): /bin/zsh [Enabled Hooks] pre-commit ``` ## Johannes Schindelin, 2023-03-27 07:49 Subject: Re: Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled Message-ID: <8afb1988-df49-d759-ead8-1bd384f09c82@gmx.de> URL: https://gitlist.dev/e/8afb1988-df49-d759-ead8-1bd384f09c82%40gmx.de In-Reply-To: ``` Hi Josh, On Sat, 4 Mar 2023, Josh Smeaton wrote: > git write-tree and then git diff-index reports thousands of files as > `deleted` when core.fsmonitor is true, even though the files are still > present on the file system. I suspect that the issue is a missing `git update-index --refresh`. While `git diff` refreshes the index automatically, `git diff-index` does not (when you use low-level commands like `diff-index` you need to do such things explicitly). If that does not fix your problem, I encourage you to distill the issue into a simple MCVE (https://stackoverflow.com/help/mcve), preferably in the form of a patch, say, to t7527 (see https://github.com/git/git/blob/v2.40.0/t/t7527-builtin-fsmonitor.sh). Ciao, Johannes > > What did you do before the bug happened? (Steps to reproduce your issue) > > ˋˋˋ > git config core.fsmonitor true > touch new-file.txt > git add new-file.txt > git diff-index --ignore-submodules --exit-code --no-color > --no-ext-diff $(git write-tree) -- > ˋˋˋ > > What did you expect to happen? (Expected behavior) > > I expected that no files would be reported as having changes. > > What happened instead? (Actual behavior) > > Thousands of unrelated files were reported as `deleted`. > > ˋˋˋ > git diff-index --ignore-submodules --exit-code --no-color > --no-ext-diff $(git write-tree) -- > :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > 0000000000000000000000000000000000000000 D src/tests/__init__.py > :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > 0000000000000000000000000000000000000000 D src/tests/site/__init__.py > :100644 000000 d8edaad92120781998e811277f966551b12fc823 > 0000000000000000000000000000000000000000 D > src/tests/site/test_models.py > > ... thousands of other files > ˋˋˋ > > What's different between what you expected and what actually happened? > > Thousands of unrelated files were reported as having been deleted. > > > Anything else you want to add: > > Myself and a few co-workers all ran into this issue in the last few days. It > presented itself when pre-commit hooks were run (specifically with the > pre-commit > tool from github pre-commit). > > When committing changes, the pre-commit tool runs the equivalent of: > > `git diff-index --ignore-submodules --exit-code --no-color > --no-ext-diff $(git write-tree) --` > > It then stashes and unstashes those changes, causing the files to be > removed from > the working tree. > > > ˋˋˋ > $ git config core.fsmonitor true > $ git checkout -b test-branch > Switched to a new branch 'test-branch' > > $ echo "test 3" >> newfile.txt > $ git add . > $ git status > On branch test-branch > Changes to be committed: > (use "git restore --staged ..." to unstage) > new file: newfile.txt > > $ git commit -a -m "will it fail?" > [WARNING] Unstaged files detected. > [INFO] Stashing unstaged files to > /Users/josh/.cache/pre-commit/patch1677888420-10129. > trim trailing whitespace.................................................Passed > check python ast.....................................(no files to check)Skipped > check for case conflicts.................................................Passed > [INFO] Restored changes from > /Users/josh/.cache/pre-commit/patch1677888420-10129. > [test-branch 7c899582c19] will it fail? > 1 file changed, 1 insertion(+) > create mode 100644 newfile.txt > > $ git status > On branch test-branch > Changes not staged for commit: > (use "git add/rm ..." to update what will be committed) > (use "git restore ..." to discard changes in working directory) > deleted: src/tests/__init__.py > deleted: src/tests/site/__init__.py > deleted: src/tests/site/test_models.py > ... lots more files ... > > no changes added to commit (use "git add" and/or "git commit -a") > > $ git status | wc -l > 15298 > > $ git restore . > > $ git status > > On branch test-branch > nothing to commit, working tree clean > ˋˋˋ > > It can be reproduced, in our repo, without using pre-commit at all: > > ˋˋˋ > > git diff-index --ignore-submodules --exit-code --no-color > --no-ext-diff $(git write-tree) -- > :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > 0000000000000000000000000000000000000000 D src/tests/__init__.py > :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 > 0000000000000000000000000000000000000000 D src/tests/site/__init__.py > :100644 000000 d8edaad92120781998e811277f966551b12fc823 > 0000000000000000000000000000000000000000 D > src/tests/site/test_models.py > > ... thousands of other files > ˋˋˋ > > Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the > correct thing: > > ˋˋˋ > $ git update-index --really-refresh --no-fsmonitor > warning: core.fsmonitor is set; remove it if you really want to > disable fsmonitor > > $ git diff-index --ignore-submodules --exit-code --no-color > --no-ext-diff $(git write-tree) -- > > no output > ˋˋˋ > > The problem persists when running `update-index` without disabling fsmonitor. > > I have reported this issue to the pre-commit project, but it looks like an issue > with fsmonitor + write-tree. > > See: pre-commit issues on github #2795 > > > > [System Info] > git version: > git version 2.39.2 (others reproduced with 2.39.1) > cpu: arm64 > no commit associated with this build > sizeof-long: 8 > sizeof-size_t: 8 > shell-path: /bin/sh > feature: fsmonitor--daemon > uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37 > PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64 > compiler info: clang: 14.0.0 (clang-1400.0.29.202) > libc info: no libc information available > $SHELL (typically, interactive shell): /bin/zsh > > > [Enabled Hooks] > pre-commit > ```