# BUG: git clean -d cannot remove files from read-only directories 5 messages from 2020-02-20 to 2020-02-21. Participants: Adam Milazzo, Junio C Hamano, Daniel Knittl-Frank, brian m. carlson. Thread: https://gitlist.dev/t/52849 ## Adam Milazzo, 2020-02-20 18:27 Subject: BUG: git clean -d cannot remove files from read-only directories Message-ID: URL: https://gitlist.dev/e/BL0PR2101MB10898F2F84B06B1A6B23228380130%40BL0PR2101MB1089.namprd21.prod.outlook.com ``` Repro steps: 1. git init 2. mkdir d 3. touch d/a 4. chmod -w d 5. git clean -fd Actual result: Git doesn't remove anything, saying "warning: failed to remove d/a". Expected result: Git should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem. Environment: git version 2.7.4 on GNU/Linux Why this is important: * This has a significant impact in real scenarios: the go language, when using modules, will download referenced modules and place them in read-only directories inside a cache directory, potentially inside the git repository. These cached modules can't be cleaned up by git clean. (Furthermore, git clean then returns a failure status code, which causes our build system to fail. I don't want to ignore the failure code.) * There is no simple and reliable way to find out which directories I would need to chmod u+w in order to allow git to delete their contents. I'd have to effectively replicate git's algorithm for determining tracked files. "git clean" is invoked at the root of the repository and the repository contains many projects worked on by different teams, so there's no fixed list of things I can manually chmod to make "git clean" work reliably. * It seems inconsistent for "git clean" to be able to remove read-only files but not files from read-only directories. ``` ## Junio C Hamano, 2020-02-20 18:32 Subject: Re: BUG: git clean -d cannot remove files from read-only directories Message-ID: URL: https://gitlist.dev/e/xmqqo8ttaz3p.fsf%40gitster-ct.c.googlers.com In-Reply-To: ``` Adam Milazzo writes: > Repro steps: > 1. git init > 2. mkdir d > 3. touch d/a > 4. chmod -w d > 5. git clean -fd > > Actual result: > Git doesn't remove anything, saying "warning: failed to remove d/a". > > Expected result: > Git should remove the subdirectory 'd' along with its > contents. Note that git can remove a read-only file (touch b; > chmod -w b; git clean -f) with no problem. It is how UNIX-like filesystem works, isn't it? As long as a directory D is writable/executable, any regular file inside it can be removed regardless of the perm-bits of the file. mkdir d touch d/a chmod -w d rm d/a would not let you remove the file d/a from d/, exactly because you cannot modify d/ (it is not writable). ``` ## Junio C Hamano, 2020-02-20 18:47 Subject: Re: BUG: git clean -d cannot remove files from read-only directories Message-ID: URL: https://gitlist.dev/e/xmqqk14hayea.fsf%40gitster-ct.c.googlers.com In-Reply-To: ``` Junio C Hamano writes: > Adam Milazzo writes: > >> Repro steps: >> 1. git init >> 2. mkdir d >> 3. touch d/a >> 4. chmod -w d >> 5. git clean -fd >> >> Actual result: >> Git doesn't remove anything, saying "warning: failed to remove d/a". >> >> Expected result: >> Git should remove the subdirectory 'd' along with its >> contents. Note that git can remove a read-only file (touch b; >> chmod -w b; git clean -f) with no problem. > > It is how UNIX-like filesystem works, isn't it? > > As long as a directory D is writable/executable, any regular file > inside it can be removed regardless of the perm-bits of the file. > > mkdir d > touch d/a > chmod -w d > rm d/a > > would not let you remove the file d/a from d/, exactly because you > cannot modify d/ (it is not writable). I realize that the above is probably a bit too terse for those who are not familiar with the UNIX/POSIX-like filesystems. A directory is a record of what files (and subdirectories) are in it. If you add a file to or remove a file from a directory, you are updating that record---so you need to be able to write to the directory. You do not have to be able to write (or even to read for that matter) to the file you are adding or removing to the directory, because adding or removing a file from a directory involves only updating the record kept in the directory about what files and subdirectories are in that directory (hence you need to be able to write to the directory---but perm bits on the files you are adding or removing do not play a role in this operation). A collorary is that modifying an existing file in a directory does not have to change what is recorded in the directory---what files are in the directory does not change. You only need to be able to write the file, so you can edit a file in a read-only directory. Note that some editors, however, implement an "edit" as "rename the original file as a backup, create a new copy of a file, and edit that new copy". With such an editor, you cannot "edit" a writable file in a read-only directory and the reason would by now be obvious to you once you understand the explanation above. Both renaming the original and creating the new copy would be updating what files are in the directory, requiring you to be able to write to the directory. Hope this helps. ``` ## Daniel Knittl-Frank, 2020-02-20 18:46 Subject: Re: BUG: git clean -d cannot remove files from read-only directories Message-ID: URL: https://gitlist.dev/e/CACx-yZ0e-rAsJbeKr5WO32kG-HChTva4nhpfZ98B7C%2BYiJ4KXw%40mail.gmail.com In-Reply-To: ``` Hi Adam, On Thu, Feb 20, 2020 at 7:27 PM Adam Milazzo wrote: > > Repro steps: > 1. git init > 2. mkdir d > 3. touch d/a > 4. chmod -w d > 5. git clean -fd > > Actual result: > Git doesn't remove anything, saying "warning: failed to remove d/a". > > > Expected result: > Git should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem. this is not a limitation of Git, but of how Linux Filesystems work. When you delete a file, you do not modify the file, but you change the directory which contains the file. In other words, if you have a directory which is write protected (you do not have the write permission), you cannot add, remove, or rename files. You may however still modify the content existing files. Your repro will yield the same result when executed without Git, i.e. plain shell commands: $ mkdir d ; touch d/a ; chmod a-w d $ rm -rf d rm: cannot remove 'd/a': Permission denied If the write permission were removed before touching the file, this would already fail: $ mkdir d ; chmod a-w d $ touch d/a touch: cannot touch 'd/a': Permission denied In summary: this is expected and I doubt Git can do much in such a case. After all, the directory is marked as read-only, so why should Git be able to write it? :) > […] > > * It seems inconsistent for "git clean" to be able to remove read-only files but not files from read-only directories. Again, deleting a file will modify its containing directory, not the file itself. Making a file read-only does not protect it from being renamed or deleted, that's what directory permissions are for. - Daniel -- typed with http://neo-layout.org ``` ## brian m. carlson, 2020-02-21 01:45 Subject: Re: BUG: git clean -d cannot remove files from read-only directories Message-ID: <20200221014535.GC6462@camp.crustytoothpaste.net> URL: https://gitlist.dev/e/20200221014535.GC6462%40camp.crustytoothpaste.net In-Reply-To: ``` On 2020-02-20 at 18:27:22, Adam Milazzo wrote: > Repro steps: > 1. git init > 2. mkdir d > 3. touch d/a > 4. chmod -w d > 5. git clean -fd > > Actual result: > Git doesn't remove anything, saying "warning: failed to remove d/a". > > Expected result: > Git should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem. I don't believe git should depart from rm in this regard. I believe that in general, Unix has stood the test of time, and when in doubt, we should behave as Unix utilities do. Note that other utilities (such as Perl) which have implemented different behavior (usually for compatibility with Windows) have found themselves with security vulnerabilities in that behavior. That seems like a prudent reason not to replicate it. > Why this is important: > * This has a significant impact in real scenarios: the go language, > when using modules, will download referenced modules and place them > in read-only directories inside a cache directory, potentially > inside the git repository. These cached modules can't be cleaned up > by git clean. (Furthermore, git clean then returns a failure status > code, which causes our build system to fail. I don't want to ignore > the failure code.) This sounds like the real problem. Why is Go placing data in cache directories that are read-only? Cache directories are explicitly ephemeral and should be able to be destroyed at any time. You'll probably find more luck convincing the Go maintainers that their caches should be temporary than you will us that git clean should be your automatic destroyer of data. Rust, for example, doesn't do this. You could also just move your cache directories into $TMPDIR/go-cache and then do "chmod -R u+w $TMPDIR/go-cache/* && rm -fr $TMPDIR/go-cache/*". -- brian m. carlson: Houston, Texas, US OpenPGP: https://keybase.io/bk2204 ```