{"thread":{"id":"51026","subject":"bug: git pull may delete untracked files in submodule without notice","startedAt":"2019-05-03T10:02:39Z","lastAt":"2019-05-03T15:41:04Z","messageCount":3,"participants":["Christian Spanier","Duy Nguyen","Phillip Wood"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"374870","messageId":"2e321dbe-42a3-7516-52aa-4bc50a3c403e@boxie.eu","threadId":"51026","inReplyTo":null,"subject":"bug: git pull may delete untracked files in submodule without notice","fromName":"Christian Spanier","fromEmail":"cspanier@boxie.eu","sentAt":"2019-05-03T08:02:35Z","receivedAt":"2019-05-03T10:02:39Z","isPatch":false,"sender":{"key":"cspanier@boxie.eu","avatar":null},"body":"Hi,\n\nI found a bug where Git may delete untracked files without notice in \ncertain situations. This bug effects Git 2.21.0 both on Linux and Windows.\nIn summary this happens when git pull merges a commit that replaces a \nsubmodule folder with a symlink. Any files within the folder are deleted \nwithout notice.\nCheck out the script below for details.\n\nThis happend on some developer's machine and deleted a repository \ncontaining about 200GiB of files and tons of uncommited local scripts, \nlog files and whatever, just because some other dev accidentally \ncommited a temporary change.\n\nGreetings,\nChristian Spanier\n\n##### PREPARATION #####\n\n# New empty repository #1\nmkdir rep1\ncd rep1\ngit init --bare .\ncd ..\n\n# New empty repository #2\nmkdir rep2\ncd rep2\ngit init --bare .\ncd ..\n\n# Clone repository #1 and create initial commit\ngit clone rep1 clone_rep1_user1\ncd clone_rep1_user1\ntouch README\ngit add README\ngit commit -m \"initial commit\"\ngit push\ncd ..\n\n# Clone repository #2 and create initial commit\ngit clone rep2 clone_rep2\ncd clone_rep2\ntouch README\ngit add README\ngit commit -m \"initial commit\"\ngit push\ncd ..\n\n# Add repository #2 as a submodule to repository #1\ncd clone_rep1_user1\ngit submodule add ../rep2\ngit commit -m \"add submodule\"\ngit push\ncd ..\n\n# User 2 also clones repository #1 and #2 recursively\ngit clone --recursive rep1 clone_rep1_user2\n\n# User 2 starts working in his folder and adds an important local file \nwhich is\n# not yet committed inside the submodule folder.\ncd clone_rep1_user2/rep2\necho \"important work\" > uncommitted_file\ncd ../../\n\n# Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2 \nwith a\n# symbolic link to a different folder (for whatever reason, maybe a copy \nof an\n# older version or anything).\nmkdir rep2_alternative\ncd clone_rep1_user1\nmv rep2 ../rep2_backup\nln -s ../rep2_alternative rep2\n# On Windows this can be done with 'mklink /D rep2 ../rep2_alternative',\n# which requires admin privileges. The bug is not reproducible when\n# using a directory junction with 'mklink /D /J ...'.\n\n# He does some work on rep1 but then accidently adds the symbolic link \nto his\n# next commit and pushes the changes. Notice the typechange of rep2.\necho \"some\" > work\ngit status\n# On branch master\n# Your branch is up to date with 'origin/master'.\n#\n# Changes not staged for commit:\n#   (use \"git add <file>...\" to update what will be committed)\n#   (use \"git checkout -- <file>...\" to discard changes in working \ndirectory)\n#\n#         typechange: rep2\n#\n# Untracked files:\n#   (use \"git add <file>...\" to include in what will be committed)\n#\n#         work\n#\n# no changes added to commit (use \"git add\" and/or \"git commit -a\")\ngit add .\ngit commit -m \"do some work\"\ngit push\ncd ..\n\n# NOW THE BUG:\n\n# User 2 pulls the changes and loses his important work in\n# rep2/uncommitted_file because Git replaces the folder with a symlink\n# without checking for modified or uncommited files!\n# He should get an error in this case!\ncd clone_rep1_user2\ngit pull\ncat rep2/uncommitted_file\n# cat: rep2/uncommitted_file: Not a directory\n# \"important work\" in rep2/uncommitted_file is gone :(\n"},{"id":"374873","messageId":"CACsJy8ArJ9Lc1pA7SbJ1OE6hK+rprXpZViLDPUKPMcWG7Bre3A@mail.gmail.com","threadId":"51026","inReplyTo":"2e321dbe-42a3-7516-52aa-4bc50a3c403e@boxie.eu","subject":"Re: bug: git pull may delete untracked files in submodule without notice","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-05-03T10:28:48Z","receivedAt":"2019-05-03T10:29:17Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, May 3, 2019 at 5:25 PM Christian Spanier <cspanier@boxie.eu> wrote:\n>\n> Hi,\n>\n> I found a bug where Git may delete untracked files without notice in\n> certain situations. This bug effects Git 2.21.0 both on Linux and Windows.\n> In summary this happens when git pull merges a commit that replaces a\n> submodule folder with a symlink. Any files within the folder are deleted\n> without notice.\n\nMaybe it's not the same, maybe it is. But Phillip recently did some\nwork protecting untracked files and I think he touched a test case\nabout submodule. Adding him so he can check, if he has time.\n\n> Check out the script below for details.\n>\n> This happend on some developer's machine and deleted a repository\n> containing about 200GiB of files and tons of uncommited local scripts,\n> log files and whatever, just because some other dev accidentally\n> commited a temporary change.\n>\n> Greetings,\n> Christian Spanier\n>\n> ##### PREPARATION #####\n>\n> # New empty repository #1\n> mkdir rep1\n> cd rep1\n> git init --bare .\n> cd ..\n>\n> # New empty repository #2\n> mkdir rep2\n> cd rep2\n> git init --bare .\n> cd ..\n>\n> # Clone repository #1 and create initial commit\n> git clone rep1 clone_rep1_user1\n> cd clone_rep1_user1\n> touch README\n> git add README\n> git commit -m \"initial commit\"\n> git push\n> cd ..\n>\n> # Clone repository #2 and create initial commit\n> git clone rep2 clone_rep2\n> cd clone_rep2\n> touch README\n> git add README\n> git commit -m \"initial commit\"\n> git push\n> cd ..\n>\n> # Add repository #2 as a submodule to repository #1\n> cd clone_rep1_user1\n> git submodule add ../rep2\n> git commit -m \"add submodule\"\n> git push\n> cd ..\n>\n> # User 2 also clones repository #1 and #2 recursively\n> git clone --recursive rep1 clone_rep1_user2\n>\n> # User 2 starts working in his folder and adds an important local file\n> which is\n> # not yet committed inside the submodule folder.\n> cd clone_rep1_user2/rep2\n> echo \"important work\" > uncommitted_file\n> cd ../../\n>\n> # Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2\n> with a\n> # symbolic link to a different folder (for whatever reason, maybe a copy\n> of an\n> # older version or anything).\n> mkdir rep2_alternative\n> cd clone_rep1_user1\n> mv rep2 ../rep2_backup\n> ln -s ../rep2_alternative rep2\n> # On Windows this can be done with 'mklink /D rep2 ../rep2_alternative',\n> # which requires admin privileges. The bug is not reproducible when\n> # using a directory junction with 'mklink /D /J ...'.\n>\n> # He does some work on rep1 but then accidently adds the symbolic link\n> to his\n> # next commit and pushes the changes. Notice the typechange of rep2.\n> echo \"some\" > work\n> git status\n> # On branch master\n> # Your branch is up to date with 'origin/master'.\n> #\n> # Changes not staged for commit:\n> #   (use \"git add <file>...\" to update what will be committed)\n> #   (use \"git checkout -- <file>...\" to discard changes in working\n> directory)\n> #\n> #         typechange: rep2\n> #\n> # Untracked files:\n> #   (use \"git add <file>...\" to include in what will be committed)\n> #\n> #         work\n> #\n> # no changes added to commit (use \"git add\" and/or \"git commit -a\")\n> git add .\n> git commit -m \"do some work\"\n> git push\n> cd ..\n>\n> # NOW THE BUG:\n>\n> # User 2 pulls the changes and loses his important work in\n> # rep2/uncommitted_file because Git replaces the folder with a symlink\n> # without checking for modified or uncommited files!\n> # He should get an error in this case!\n> cd clone_rep1_user2\n> git pull\n> cat rep2/uncommitted_file\n> # cat: rep2/uncommitted_file: Not a directory\n> # \"important work\" in rep2/uncommitted_file is gone :(\n\n\n\n-- \nDuy\n"},{"id":"374884","messageId":"6d985c61-d06f-b794-8533-c7f29444b148@gmail.com","threadId":"51026","inReplyTo":"CACsJy8ArJ9Lc1pA7SbJ1OE6hK+rprXpZViLDPUKPMcWG7Bre3A@mail.gmail.com","subject":"Re: bug: git pull may delete untracked files in submodule without notice","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2019-05-03T15:10:01Z","receivedAt":"2019-05-03T15:41:04Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 03/05/2019 11:28, Duy Nguyen wrote:\n> On Fri, May 3, 2019 at 5:25 PM Christian Spanier <cspanier@boxie.eu> wrote:\n>>\n>> Hi,\n>>\n>> I found a bug where Git may delete untracked files without notice in\n>> certain situations. This bug effects Git 2.21.0 both on Linux and Windows.\n>> In summary this happens when git pull merges a commit that replaces a\n>> submodule folder with a symlink. Any files within the folder are deleted\n>> without notice.\n> \n> Maybe it's not the same, maybe it is. But Phillip recently did some\n> work protecting untracked files and I think he touched a test case\n> about submodule. Adding him so he can check, if he has time.\n\nThat was a different case. I've just tried adding an untracked file to \nthe submodule directory in the test that replaces a submodule with a \nfile (patch below hopefully thunderbird does not break it) and it still \npasses when running t1013-read-tree-submodule.sh (it's test 8) - so I \nthink there is something wrong with the detection of untracked files in \na submodule directory when we replace it with a file. Normally I'd ping \nStefan about submodules but he's not around at the moment.\n\nBest Wishes\n\nPhillip\n\ndiff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh\nindex 5b56b23166..618176ae54 100755\n--- a/t/lib-submodule-update.sh\n+++ b/t/lib-submodule-update.sh\n@@ -735,6 +735,7 @@ test_submodule_recursing_with_args_common() {\n                 reset_work_tree_to_interested add_sub1 &&\n                 (\n                         cd submodule_update &&\n+                       echo untracked>sub1/untracked &&\n                         git branch -t replace_sub1_with_file \norigin/replace_sub1_with_file &&\n                         $command replace_sub1_with_file &&\n                         test_superproject_content \norigin/replace_sub1_with_file &&\n\n\n> \n>> Check out the script below for details.\n>>\n>> This happend on some developer's machine and deleted a repository\n>> containing about 200GiB of files and tons of uncommited local scripts,\n>> log files and whatever, just because some other dev accidentally\n>> commited a temporary change.\n>>\n>> Greetings,\n>> Christian Spanier\n>>\n>> ##### PREPARATION #####\n>>\n>> # New empty repository #1\n>> mkdir rep1\n>> cd rep1\n>> git init --bare .\n>> cd ..\n>>\n>> # New empty repository #2\n>> mkdir rep2\n>> cd rep2\n>> git init --bare .\n>> cd ..\n>>\n>> # Clone repository #1 and create initial commit\n>> git clone rep1 clone_rep1_user1\n>> cd clone_rep1_user1\n>> touch README\n>> git add README\n>> git commit -m \"initial commit\"\n>> git push\n>> cd ..\n>>\n>> # Clone repository #2 and create initial commit\n>> git clone rep2 clone_rep2\n>> cd clone_rep2\n>> touch README\n>> git add README\n>> git commit -m \"initial commit\"\n>> git push\n>> cd ..\n>>\n>> # Add repository #2 as a submodule to repository #1\n>> cd clone_rep1_user1\n>> git submodule add ../rep2\n>> git commit -m \"add submodule\"\n>> git push\n>> cd ..\n>>\n>> # User 2 also clones repository #1 and #2 recursively\n>> git clone --recursive rep1 clone_rep1_user2\n>>\n>> # User 2 starts working in his folder and adds an important local file\n>> which is\n>> # not yet committed inside the submodule folder.\n>> cd clone_rep1_user2/rep2\n>> echo \"important work\" > uncommitted_file\n>> cd ../../\n>>\n>> # Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2\n>> with a\n>> # symbolic link to a different folder (for whatever reason, maybe a copy\n>> of an\n>> # older version or anything).\n>> mkdir rep2_alternative\n>> cd clone_rep1_user1\n>> mv rep2 ../rep2_backup\n>> ln -s ../rep2_alternative rep2\n>> # On Windows this can be done with 'mklink /D rep2 ../rep2_alternative',\n>> # which requires admin privileges. The bug is not reproducible when\n>> # using a directory junction with 'mklink /D /J ...'.\n>>\n>> # He does some work on rep1 but then accidently adds the symbolic link\n>> to his\n>> # next commit and pushes the changes. Notice the typechange of rep2.\n>> echo \"some\" > work\n>> git status\n>> # On branch master\n>> # Your branch is up to date with 'origin/master'.\n>> #\n>> # Changes not staged for commit:\n>> #   (use \"git add <file>...\" to update what will be committed)\n>> #   (use \"git checkout -- <file>...\" to discard changes in working\n>> directory)\n>> #\n>> #         typechange: rep2\n>> #\n>> # Untracked files:\n>> #   (use \"git add <file>...\" to include in what will be committed)\n>> #\n>> #         work\n>> #\n>> # no changes added to commit (use \"git add\" and/or \"git commit -a\")\n>> git add .\n>> git commit -m \"do some work\"\n>> git push\n>> cd ..\n>>\n>> # NOW THE BUG:\n>>\n>> # User 2 pulls the changes and loses his important work in\n>> # rep2/uncommitted_file because Git replaces the folder with a symlink\n>> # without checking for modified or uncommited files!\n>> # He should get an error in this case!\n>> cd clone_rep1_user2\n>> git pull\n>> cat rep2/uncommitted_file\n>> # cat: rep2/uncommitted_file: Not a directory\n>> # \"important work\" in rep2/uncommitted_file is gone :(\n> \n> \n> \n"}]}