threads / bug / 51026

bug: git pull may delete untracked files in submodule without notice

Subject: bug: git pull may delete untracked files in submodule without notice

## tl;dr

3 messages between May 3, 2019 and May 3, 2019.

replies: 2people: 3as markdown or json

Christian Spanier· May 3, 2019, 08:02 UTC · lore

Hi,

I found a bug where Git may delete untracked files without notice in certain situations. This bug effects Git 2.21.0 both on Linux and Windows. In summary this happens when git pull merges a commit that replaces a submodule folder with a symlink. Any files within the folder are deleted without notice. Check out the script below for details.

This happend on some developer's machine and deleted a repository containing about 200GiB of files and tons of uncommited local scripts, log files and whatever, just because some other dev accidentally commited a temporary change.

Greetings, Christian Spanier

##### PREPARATION #####

# New empty repository #1 mkdir rep1 cd rep1 git init --bare . cd ..

# New empty repository #2 mkdir rep2 cd rep2 git init --bare . cd ..

# Clone repository #1 and create initial commit git clone rep1 clone_rep1_user1 cd clone_rep1_user1 touch README git add README git commit -m "initial commit" git push cd ..

# Clone repository #2 and create initial commit git clone rep2 clone_rep2 cd clone_rep2 touch README git add README git commit -m "initial commit" git push cd ..

# Add repository #2 as a submodule to repository #1 cd clone_rep1_user1 git submodule add ../rep2 git commit -m "add submodule" git push cd ..

# User 2 also clones repository #1 and #2 recursively git clone --recursive rep1 clone_rep1_user2

# User 2 starts working in his folder and adds an important local file which is # not yet committed inside the submodule folder. cd clone_rep1_user2/rep2 echo "important work" > uncommitted_file cd ../../

# Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2 with a # symbolic link to a different folder (for whatever reason, maybe a copy of an # older version or anything). mkdir rep2_alternative cd clone_rep1_user1 mv rep2 ../rep2_backup ln -s ../rep2_alternative rep2 # On Windows this can be done with 'mklink /D rep2 ../rep2_alternative', # which requires admin privileges. The bug is not reproducible when # using a directory junction with 'mklink /D /J ...'.

# He does some work on rep1 but then accidently adds the symbolic link to his # next commit and pushes the changes. Notice the typechange of rep2. echo "some" > work git status # On branch master # Your branch is up to date with 'origin/master'. # # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # typechange: rep2 # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # work # # no changes added to commit (use "git add" and/or "git commit -a") git add . git commit -m "do some work" git push cd ..

# NOW THE BUG:

# User 2 pulls the changes and loses his important work in # rep2/uncommitted_file because Git replaces the folder with a symlink # without checking for modified or uncommited files! # He should get an error in this case! cd clone_rep1_user2 git pull cat rep2/uncommitted_file # cat: rep2/uncommitted_file: Not a directory # "important work" in rep2/uncommitted_file is gone :(

Duy Nguyen· May 3, 2019, 10:28 UTC · re: Christian Spanier · lore

Re: bug: git pull may delete untracked files in submodule without notice

On Fri, May 3, 2019 at 5:25 PM Christian Spanier <cspanier@boxie.eu> wrote:

Show 8 quoted lines

>
> Hi,
>
> I found a bug where Git may delete untracked files without notice in
> certain situations. This bug effects Git 2.21.0 both on Linux and Windows.
> In summary this happens when git pull merges a commit that replaces a
> submodule folder with a symlink. Any files within the folder are deleted
> without notice.

Maybe it's not the same, maybe it is. But Phillip recently did some work protecting untracked files and I think he touched a test case about submodule. Adding him so he can check, if he has time.

Show 109 quoted lines

> Check out the script below for details.
>
> This happend on some developer's machine and deleted a repository
> containing about 200GiB of files and tons of uncommited local scripts,
> log files and whatever, just because some other dev accidentally
> commited a temporary change.
>
> Greetings,
> Christian Spanier
>
> ##### PREPARATION #####
>
> # New empty repository #1
> mkdir rep1
> cd rep1
> git init --bare .
> cd ..
>
> # New empty repository #2
> mkdir rep2
> cd rep2
> git init --bare .
> cd ..
>
> # Clone repository #1 and create initial commit
> git clone rep1 clone_rep1_user1
> cd clone_rep1_user1
> touch README
> git add README
> git commit -m "initial commit"
> git push
> cd ..
>
> # Clone repository #2 and create initial commit
> git clone rep2 clone_rep2
> cd clone_rep2
> touch README
> git add README
> git commit -m "initial commit"
> git push
> cd ..
>
> # Add repository #2 as a submodule to repository #1
> cd clone_rep1_user1
> git submodule add ../rep2
> git commit -m "add submodule"
> git push
> cd ..
>
> # User 2 also clones repository #1 and #2 recursively
> git clone --recursive rep1 clone_rep1_user2
>
> # User 2 starts working in his folder and adds an important local file
> which is
> # not yet committed inside the submodule folder.
> cd clone_rep1_user2/rep2
> echo "important work" > uncommitted_file
> cd ../../
>
> # Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2
> with a
> # symbolic link to a different folder (for whatever reason, maybe a copy
> of an
> # older version or anything).
> mkdir rep2_alternative
> cd clone_rep1_user1
> mv rep2 ../rep2_backup
> ln -s ../rep2_alternative rep2
> # On Windows this can be done with 'mklink /D rep2 ../rep2_alternative',
> # which requires admin privileges. The bug is not reproducible when
> # using a directory junction with 'mklink /D /J ...'.
>
> # He does some work on rep1 but then accidently adds the symbolic link
> to his
> # next commit and pushes the changes. Notice the typechange of rep2.
> echo "some" > work
> git status
> # On branch master
> # Your branch is up to date with 'origin/master'.
> #
> # Changes not staged for commit:
> #   (use "git add <file>..." to update what will be committed)
> #   (use "git checkout -- <file>..." to discard changes in working
> directory)
> #
> #         typechange: rep2
> #
> # Untracked files:
> #   (use "git add <file>..." to include in what will be committed)
> #
> #         work
> #
> # no changes added to commit (use "git add" and/or "git commit -a")
> git add .
> git commit -m "do some work"
> git push
> cd ..
>
> # NOW THE BUG:
>
> # User 2 pulls the changes and loses his important work in
> # rep2/uncommitted_file because Git replaces the folder with a symlink
> # without checking for modified or uncommited files!
> # He should get an error in this case!
> cd clone_rep1_user2
> git pull
> cat rep2/uncommitted_file
> # cat: rep2/uncommitted_file: Not a directory
> # "important work" in rep2/uncommitted_file is gone :(

-- 
Duy

Phillip Wood· May 3, 2019, 15:10 UTC · re: Duy Nguyen · lore

Re: bug: git pull may delete untracked files in submodule without notice

On 03/05/2019 11:28, Duy Nguyen wrote:

Show 13 quoted lines

> On Fri, May 3, 2019 at 5:25 PM Christian Spanier <cspanier@boxie.eu> wrote:
>>
>> Hi,
>>
>> I found a bug where Git may delete untracked files without notice in
>> certain situations. This bug effects Git 2.21.0 both on Linux and Windows.
>> In summary this happens when git pull merges a commit that replaces a
>> submodule folder with a symlink. Any files within the folder are deleted
>> without notice.
> 
> Maybe it's not the same, maybe it is. But Phillip recently did some
> work protecting untracked files and I think he touched a test case
> about submodule. Adding him so he can check, if he has time.

That was a different case. I've just tried adding an untracked file to the submodule directory in the test that replaces a submodule with a file (patch below hopefully thunderbird does not break it) and it still passes when running t1013-read-tree-submodule.sh (it's test 8) - so I think there is something wrong with the detection of untracked files in a submodule directory when we replace it with a file. Normally I'd ping Stefan about submodules but he's not around at the moment.

Best Wishes

Phillip

diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
index 5b56b23166..618176ae54 100755
--- a/t/lib-submodule-update.sh
+++ b/t/lib-submodule-update.sh
@@ -735,6 +735,7 @@ test_submodule_recursing_with_args_common() {
                 reset_work_tree_to_interested add_sub1 &&
                 (
                         cd submodule_update &&
+                       echo untracked>sub1/untracked &&
                         git branch -t replace_sub1_with_file 
origin/replace_sub1_with_file &&
                         $command replace_sub1_with_file &&
                         test_superproject_content 
origin/replace_sub1_with_file &&


> 
>> Check out the script below for details.
>>
>> This happend on some developer's machine and deleted a repository
>> containing about 200GiB of files and tons of uncommited local scripts,
>> log files and whatever, just because some other dev accidentally
>> commited a temporary change.
>>
>> Greetings,
>> Christian Spanier
>>
>> ##### PREPARATION #####
>>
>> # New empty repository #1
>> mkdir rep1
>> cd rep1
>> git init --bare .
>> cd ..
>>
>> # New empty repository #2
>> mkdir rep2
>> cd rep2
>> git init --bare .
>> cd ..
>>
>> # Clone repository #1 and create initial commit
>> git clone rep1 clone_rep1_user1
>> cd clone_rep1_user1
>> touch README
>> git add README
>> git commit -m "initial commit"
>> git push
>> cd ..
>>
>> # Clone repository #2 and create initial commit
>> git clone rep2 clone_rep2
>> cd clone_rep2
>> touch README
>> git add README
>> git commit -m "initial commit"
>> git push
>> cd ..
>>
>> # Add repository #2 as a submodule to repository #1
>> cd clone_rep1_user1
>> git submodule add ../rep2
>> git commit -m "add submodule"
>> git push
>> cd ..
>>
>> # User 2 also clones repository #1 and #2 recursively
>> git clone --recursive rep1 clone_rep1_user2
>>
>> # User 2 starts working in his folder and adds an important local file
>> which is
>> # not yet committed inside the submodule folder.
>> cd clone_rep1_user2/rep2
>> echo "important work" > uncommitted_file
>> cd ../../
>>
>> # Meanwhile, user 1 temporarily switch out folder /clone_rep1_user1/rep2
>> with a
>> # symbolic link to a different folder (for whatever reason, maybe a copy
>> of an
>> # older version or anything).
>> mkdir rep2_alternative
>> cd clone_rep1_user1
>> mv rep2 ../rep2_backup
>> ln -s ../rep2_alternative rep2
>> # On Windows this can be done with 'mklink /D rep2 ../rep2_alternative',
>> # which requires admin privileges. The bug is not reproducible when
>> # using a directory junction with 'mklink /D /J ...'.
>>
>> # He does some work on rep1 but then accidently adds the symbolic link
>> to his
>> # next commit and pushes the changes. Notice the typechange of rep2.
>> echo "some" > work
>> git status
>> # On branch master
>> # Your branch is up to date with 'origin/master'.
>> #
>> # Changes not staged for commit:
>> #   (use "git add <file>..." to update what will be committed)
>> #   (use "git checkout -- <file>..." to discard changes in working
>> directory)
>> #
>> #         typechange: rep2
>> #
>> # Untracked files:
>> #   (use "git add <file>..." to include in what will be committed)
>> #
>> #         work
>> #
>> # no changes added to commit (use "git add" and/or "git commit -a")
>> git add .
>> git commit -m "do some work"
>> git push
>> cd ..
>>
>> # NOW THE BUG:
>>
>> # User 2 pulls the changes and loses his important work in
>> # rep2/uncommitted_file because Git replaces the folder with a symlink
>> # without checking for modified or uncommited files!
>> # He should get an error in this case!
>> cd clone_rep1_user2
>> git pull
>> cat rep2/uncommitted_file
>> # cat: rep2/uncommitted_file: Not a directory
>> # "important work" in rep2/uncommitted_file is gone :(
> 
> 
>