{"thread":{"id":"59339","subject":"Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled","startedAt":"2023-03-04T01:59:35Z","lastAt":"2023-03-27T07:50:02Z","messageCount":2,"participants":["Josh Smeaton","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"472987","messageId":"CAPbDM0d5Xe=KTLECDCJYtpzmBt2wqzZrMWVcbiisM+abopenig@mail.gmail.com","threadId":"59339","inReplyTo":null,"subject":"Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled","fromName":"Josh Smeaton","fromEmail":"josh.smeaton@gmail.com","sentAt":"2023-03-04T01:55:13Z","receivedAt":"2023-03-04T01:59:35Z","isPatch":false,"sender":{"key":"josh.smeaton@gmail.com","avatar":null},"body":"git write-tree and then git diff-index reports thousands of files as `deleted`\nwhen core.fsmonitor is true, even though the files are still present on the\nfile system.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\n\n```\ngit config core.fsmonitor true\ntouch new-file.txt\ngit add new-file.txt\ngit diff-index --ignore-submodules --exit-code --no-color\n--no-ext-diff $(git write-tree) --\n```\n\nWhat did you expect to happen? (Expected behavior)\n\nI expected that no files would be reported as having changes.\n\nWhat happened instead? (Actual behavior)\n\nThousands of unrelated files were reported as `deleted`.\n\n```\ngit diff-index --ignore-submodules --exit-code --no-color\n--no-ext-diff $(git write-tree) --\n:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n0000000000000000000000000000000000000000 D src/tests/__init__.py\n:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n0000000000000000000000000000000000000000 D src/tests/site/__init__.py\n:100644 000000 d8edaad92120781998e811277f966551b12fc823\n0000000000000000000000000000000000000000 D\nsrc/tests/site/test_models.py\n\n ... thousands of other files\n```\n\nWhat's different between what you expected and what actually happened?\n\nThousands of unrelated files were reported as having been deleted.\n\n\nAnything else you want to add:\n\nMyself and a few co-workers all ran into this issue in the last few days. It\npresented itself when pre-commit hooks were run (specifically with the\npre-commit\ntool from github pre-commit).\n\nWhen committing changes, the pre-commit tool runs the equivalent of:\n\n`git diff-index --ignore-submodules --exit-code --no-color\n--no-ext-diff $(git write-tree) --`\n\nIt then stashes and unstashes those changes, causing the files to be\nremoved from\nthe working tree.\n\n\n```\n$ git config core.fsmonitor true\n$ git checkout -b test-branch\nSwitched to a new branch 'test-branch'\n\n$ echo \"test 3\" >> newfile.txt\n$ git add .\n$ git status\nOn branch test-branch\nChanges to be committed:\n  (use \"git restore --staged <file>...\" to unstage)\nnew file:   newfile.txt\n\n$ git commit -a -m \"will it fail?\"\n[WARNING] Unstaged files detected.\n[INFO] Stashing unstaged files to\n/Users/josh/.cache/pre-commit/patch1677888420-10129.\ntrim trailing whitespace.................................................Passed\ncheck python ast.....................................(no files to check)Skipped\ncheck for case conflicts.................................................Passed\n[INFO] Restored changes from\n/Users/josh/.cache/pre-commit/patch1677888420-10129.\n[test-branch 7c899582c19] will it fail?\n 1 file changed, 1 insertion(+)\n create mode 100644 newfile.txt\n\n$ git status\nOn branch test-branch\nChanges not staged for commit:\n  (use \"git add/rm <file>...\" to update what will be committed)\n  (use \"git restore <file>...\" to discard changes in working directory)\ndeleted:    src/tests/__init__.py\ndeleted:    src/tests/site/__init__.py\ndeleted:    src/tests/site/test_models.py\n        ... lots more files ...\n\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\n$ git status | wc -l\n   15298\n\n$ git restore .\n\n$ git status\n\nOn branch test-branch\nnothing to commit, working tree clean\n```\n\nIt can be reproduced, in our repo, without using pre-commit at all:\n\n```\n\ngit diff-index --ignore-submodules --exit-code --no-color\n--no-ext-diff $(git write-tree) --\n:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n0000000000000000000000000000000000000000 D src/tests/__init__.py\n:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n0000000000000000000000000000000000000000 D src/tests/site/__init__.py\n:100644 000000 d8edaad92120781998e811277f966551b12fc823\n0000000000000000000000000000000000000000 D\nsrc/tests/site/test_models.py\n\n ... thousands of other files\n```\n\nUsing `update-index` with the `--no-fsmonitor` flag prior to diff-index does the\ncorrect thing:\n\n```\n$ git update-index --really-refresh --no-fsmonitor\nwarning: core.fsmonitor is set; remove it if you really want to\ndisable fsmonitor\n\n$ git diff-index --ignore-submodules --exit-code --no-color\n--no-ext-diff $(git write-tree) --\n> no output\n```\n\nThe problem persists when running `update-index` without disabling fsmonitor.\n\nI have reported this issue to the pre-commit project, but it looks like an issue\nwith fsmonitor + write-tree.\n\nSee: pre-commit issues on github #2795\n\n\n\n[System Info]\ngit version:\ngit version 2.39.2 (others reproduced with 2.39.1)\ncpu: arm64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37\nPST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64\ncompiler info: clang: 14.0.0 (clang-1400.0.29.202)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/zsh\n\n\n[Enabled Hooks]\npre-commit\n"},{"id":"474216","messageId":"8afb1988-df49-d759-ead8-1bd384f09c82@gmx.de","threadId":"59339","inReplyTo":"CAPbDM0d5Xe=KTLECDCJYtpzmBt2wqzZrMWVcbiisM+abopenig@mail.gmail.com","subject":"Re: Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-03-27T07:49:53Z","receivedAt":"2023-03-27T07:50:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Josh,\n\nOn Sat, 4 Mar 2023, Josh Smeaton wrote:\n\n> git write-tree and then git diff-index reports thousands of files as\n> `deleted` when core.fsmonitor is true, even though the files are still\n> present on the file system.\n\nI suspect that the issue is a missing `git update-index --refresh`. While\n`git diff` refreshes the index automatically, `git diff-index` does not\n(when you use low-level commands like `diff-index` you need to do such\nthings explicitly).\n\nIf that does not fix your problem, I encourage you to distill the issue\ninto a simple MCVE (https://stackoverflow.com/help/mcve), preferably in\nthe form of a patch, say, to t7527 (see\nhttps://github.com/git/git/blob/v2.40.0/t/t7527-builtin-fsmonitor.sh).\n\nCiao,\nJohannes\n\n>\n> What did you do before the bug happened? (Steps to reproduce your issue)\n>\n> ```\n> git config core.fsmonitor true\n> touch new-file.txt\n> git add new-file.txt\n> git diff-index --ignore-submodules --exit-code --no-color\n> --no-ext-diff $(git write-tree) --\n> ```\n>\n> What did you expect to happen? (Expected behavior)\n>\n> I expected that no files would be reported as having changes.\n>\n> What happened instead? (Actual behavior)\n>\n> Thousands of unrelated files were reported as `deleted`.\n>\n> ```\n> git diff-index --ignore-submodules --exit-code --no-color\n> --no-ext-diff $(git write-tree) --\n> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n> 0000000000000000000000000000000000000000 D src/tests/__init__.py\n> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py\n> :100644 000000 d8edaad92120781998e811277f966551b12fc823\n> 0000000000000000000000000000000000000000 D\n> src/tests/site/test_models.py\n>\n>  ... thousands of other files\n> ```\n>\n> What's different between what you expected and what actually happened?\n>\n> Thousands of unrelated files were reported as having been deleted.\n>\n>\n> Anything else you want to add:\n>\n> Myself and a few co-workers all ran into this issue in the last few days. It\n> presented itself when pre-commit hooks were run (specifically with the\n> pre-commit\n> tool from github pre-commit).\n>\n> When committing changes, the pre-commit tool runs the equivalent of:\n>\n> `git diff-index --ignore-submodules --exit-code --no-color\n> --no-ext-diff $(git write-tree) --`\n>\n> It then stashes and unstashes those changes, causing the files to be\n> removed from\n> the working tree.\n>\n>\n> ```\n> $ git config core.fsmonitor true\n> $ git checkout -b test-branch\n> Switched to a new branch 'test-branch'\n>\n> $ echo \"test 3\" >> newfile.txt\n> $ git add .\n> $ git status\n> On branch test-branch\n> Changes to be committed:\n>   (use \"git restore --staged <file>...\" to unstage)\n> new file:   newfile.txt\n>\n> $ git commit -a -m \"will it fail?\"\n> [WARNING] Unstaged files detected.\n> [INFO] Stashing unstaged files to\n> /Users/josh/.cache/pre-commit/patch1677888420-10129.\n> trim trailing whitespace.................................................Passed\n> check python ast.....................................(no files to check)Skipped\n> check for case conflicts.................................................Passed\n> [INFO] Restored changes from\n> /Users/josh/.cache/pre-commit/patch1677888420-10129.\n> [test-branch 7c899582c19] will it fail?\n>  1 file changed, 1 insertion(+)\n>  create mode 100644 newfile.txt\n>\n> $ git status\n> On branch test-branch\n> Changes not staged for commit:\n>   (use \"git add/rm <file>...\" to update what will be committed)\n>   (use \"git restore <file>...\" to discard changes in working directory)\n> deleted:    src/tests/__init__.py\n> deleted:    src/tests/site/__init__.py\n> deleted:    src/tests/site/test_models.py\n>         ... lots more files ...\n>\n> no changes added to commit (use \"git add\" and/or \"git commit -a\")\n>\n> $ git status | wc -l\n>    15298\n>\n> $ git restore .\n>\n> $ git status\n>\n> On branch test-branch\n> nothing to commit, working tree clean\n> ```\n>\n> It can be reproduced, in our repo, without using pre-commit at all:\n>\n> ```\n>\n> git diff-index --ignore-submodules --exit-code --no-color\n> --no-ext-diff $(git write-tree) --\n> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n> 0000000000000000000000000000000000000000 D src/tests/__init__.py\n> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py\n> :100644 000000 d8edaad92120781998e811277f966551b12fc823\n> 0000000000000000000000000000000000000000 D\n> src/tests/site/test_models.py\n>\n>  ... thousands of other files\n> ```\n>\n> Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the\n> correct thing:\n>\n> ```\n> $ git update-index --really-refresh --no-fsmonitor\n> warning: core.fsmonitor is set; remove it if you really want to\n> disable fsmonitor\n>\n> $ git diff-index --ignore-submodules --exit-code --no-color\n> --no-ext-diff $(git write-tree) --\n> > no output\n> ```\n>\n> The problem persists when running `update-index` without disabling fsmonitor.\n>\n> I have reported this issue to the pre-commit project, but it looks like an issue\n> with fsmonitor + write-tree.\n>\n> See: pre-commit issues on github #2795\n>\n>\n>\n> [System Info]\n> git version:\n> git version 2.39.2 (others reproduced with 2.39.1)\n> cpu: arm64\n> no commit associated with this build\n> sizeof-long: 8\n> sizeof-size_t: 8\n> shell-path: /bin/sh\n> feature: fsmonitor--daemon\n> uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37\n> PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64\n> compiler info: clang: 14.0.0 (clang-1400.0.29.202)\n> libc info: no libc information available\n> $SHELL (typically, interactive shell): /bin/zsh\n>\n>\n> [Enabled Hooks]\n> pre-commit\n>\n"}]}