{"thread":{"id":"52849","subject":"BUG: git clean -d cannot remove files from read-only directories","startedAt":"2020-02-20T18:27:31Z","lastAt":"2020-02-21T01:45:44Z","messageCount":5,"participants":["Adam Milazzo","Junio C Hamano","Daniel Knittl-Frank","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"392179","messageId":"BL0PR2101MB10898F2F84B06B1A6B23228380130@BL0PR2101MB1089.namprd21.prod.outlook.com","threadId":"52849","inReplyTo":null,"subject":"BUG: git clean -d cannot remove files from read-only directories","fromName":"Adam Milazzo","fromEmail":"adam.milazzo@microsoft.com","sentAt":"2020-02-20T18:27:22Z","receivedAt":"2020-02-20T18:27:31Z","isPatch":false,"sender":{"key":"adam.milazzo@microsoft.com","avatar":null},"body":"Repro steps:\n1. git init \n2. mkdir d\n3. touch d/a\n4. chmod -w d\n5. git clean -fd\n\nActual result:\nGit doesn't remove anything, saying \"warning: failed to remove d/a\".\n\nExpected result:\nGit should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem.\n\nEnvironment:\ngit version 2.7.4 on GNU/Linux\n\nWhy this is important:\n* This has a significant impact in real scenarios: the go language, when using modules, will download referenced modules and place them in read-only directories inside a cache directory, potentially inside the git repository. These cached modules can't be cleaned up by git clean. (Furthermore, git clean then returns a failure status code, which causes our build system to fail. I don't want to ignore the failure code.)\n\n* There is no simple and reliable way to find out which directories I would need to chmod u+w in order to allow git to delete their contents. I'd have to effectively replicate git's algorithm for determining tracked files. \"git clean\" is invoked at the root of the repository and the repository contains many projects worked on by different teams, so there's no fixed list of things I can manually chmod to make \"git clean\" work reliably.\n\n* It seems inconsistent for \"git clean\" to be able to remove read-only files but not files from read-only directories.\n\n"},{"id":"392180","messageId":"xmqqo8ttaz3p.fsf@gitster-ct.c.googlers.com","threadId":"52849","inReplyTo":"BL0PR2101MB10898F2F84B06B1A6B23228380130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-20T18:32:42Z","receivedAt":"2020-02-20T18:32:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Milazzo <Adam.Milazzo@microsoft.com> writes:\n\n> Repro steps:\n> 1. git init \n> 2. mkdir d\n> 3. touch d/a\n> 4. chmod -w d\n> 5. git clean -fd\n>\n> Actual result:\n> Git doesn't remove anything, saying \"warning: failed to remove d/a\".\n>\n> Expected result:\n> Git should remove the subdirectory 'd' along with its\n> contents. Note that git can remove a read-only file (touch b;\n> chmod -w b; git clean -f) with no problem.\n\nIt is how UNIX-like filesystem works, isn't it?\n\nAs long as a directory D is writable/executable, any regular file\ninside it can be removed regardless of the perm-bits of the file.\n\n\tmkdir d\n\ttouch d/a\n\tchmod -w d\n\trm d/a\n\nwould not let you remove the file d/a from d/, exactly because you\ncannot modify d/ (it is not writable).\n"},{"id":"392181","messageId":"xmqqk14hayea.fsf@gitster-ct.c.googlers.com","threadId":"52849","inReplyTo":"xmqqo8ttaz3p.fsf@gitster-ct.c.googlers.com","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-20T18:47:57Z","receivedAt":"2020-02-20T18:48:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Adam Milazzo <Adam.Milazzo@microsoft.com> writes:\n>\n>> Repro steps:\n>> 1. git init \n>> 2. mkdir d\n>> 3. touch d/a\n>> 4. chmod -w d\n>> 5. git clean -fd\n>>\n>> Actual result:\n>> Git doesn't remove anything, saying \"warning: failed to remove d/a\".\n>>\n>> Expected result:\n>> Git should remove the subdirectory 'd' along with its\n>> contents. Note that git can remove a read-only file (touch b;\n>> chmod -w b; git clean -f) with no problem.\n>\n> It is how UNIX-like filesystem works, isn't it?\n>\n> As long as a directory D is writable/executable, any regular file\n> inside it can be removed regardless of the perm-bits of the file.\n>\n> \tmkdir d\n> \ttouch d/a\n> \tchmod -w d\n> \trm d/a\n>\n> would not let you remove the file d/a from d/, exactly because you\n> cannot modify d/ (it is not writable).\n\nI realize that the above is probably a bit too terse for those who\nare not familiar with the UNIX/POSIX-like filesystems.\n\nA directory is a record of what files (and subdirectories) are in\nit.  If you add a file to or remove a file from a directory, you\nare updating that record---so you need to be able to write to the\ndirectory.  You do not have to be able to write (or even to read for\nthat matter) to the file you are adding or removing to the directory,\nbecause adding or removing a file from a directory involves only\nupdating the record kept in the directory about what files and\nsubdirectories are in that directory (hence you need to be able to\nwrite to the directory---but perm bits on the files you are adding\nor removing do not play a role in this operation).\n\nA collorary is that modifying an existing file in a directory does\nnot have to change what is recorded in the directory---what files\nare in the directory does not change.  You only need to be able to\nwrite the file, so you can edit a file in a read-only directory.  \n\nNote that some editors, however, implement an \"edit\" as \"rename the\noriginal file as a backup, create a new copy of a file, and edit\nthat new copy\".  With such an editor, you cannot \"edit\" a writable\nfile in a read-only directory and the reason would by now be obvious\nto you once you understand the explanation above.  Both renaming the\noriginal and creating the new copy would be updating what files are\nin the directory, requiring you to be able to write to the directory.\n\nHope this helps.\n\n"},{"id":"392184","messageId":"CACx-yZ0e-rAsJbeKr5WO32kG-HChTva4nhpfZ98B7C+YiJ4KXw@mail.gmail.com","threadId":"52849","inReplyTo":"BL0PR2101MB10898F2F84B06B1A6B23228380130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Daniel Knittl-Frank","fromEmail":"knittl89@googlemail.com","sentAt":"2020-02-20T18:46:32Z","receivedAt":"2020-02-20T18:54:56Z","isPatch":false,"sender":{"key":"knittl89@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/2149217?v=4"},"body":"Hi Adam,\n\nOn Thu, Feb 20, 2020 at 7:27 PM Adam Milazzo <Adam.Milazzo@microsoft.com> wrote:\n>\n> Repro steps:\n> 1. git init\n> 2. mkdir d\n> 3. touch d/a\n> 4. chmod -w d\n> 5. git clean -fd\n>\n> Actual result:\n> Git doesn't remove anything, saying \"warning: failed to remove d/a\".\n>\n>\n> Expected result:\n> Git should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem.\n\nthis is not a limitation of Git, but of how Linux Filesystems work.\nWhen you delete a file, you do not modify the file, but you change the\ndirectory which contains the file. In other words, if you have a\ndirectory which is write protected (you do not have the write\npermission), you cannot add, remove, or rename files. You may however\nstill modify the content existing files.\n\nYour repro will yield the same result when executed without Git, i.e.\nplain shell commands:\n\n$ mkdir d ; touch d/a ; chmod a-w d\n$ rm -rf d\nrm: cannot remove 'd/a': Permission denied\n\nIf the write permission were removed before touching the file, this\nwould already fail:\n\n$ mkdir d ; chmod a-w d\n$ touch d/a\ntouch: cannot touch 'd/a': Permission denied\n\nIn summary: this is expected and I doubt Git can do much in such a\ncase. After all, the directory is marked as read-only, so why should\nGit be able to write it? :)\n\n\n> […]\n>\n> * It seems inconsistent for \"git clean\" to be able to remove read-only files but not files from read-only directories.\n\nAgain, deleting a file will modify its containing directory, not the\nfile itself. Making a file read-only does not protect it from being\nrenamed or deleted, that's what directory permissions are for.\n\n- Daniel\n\n-- \ntyped with http://neo-layout.org\n"},{"id":"392238","messageId":"20200221014535.GC6462@camp.crustytoothpaste.net","threadId":"52849","inReplyTo":"BL0PR2101MB10898F2F84B06B1A6B23228380130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-02-21T01:45:35Z","receivedAt":"2020-02-21T01:45:44Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-02-20 at 18:27:22, Adam Milazzo wrote:\n> Repro steps:\n> 1. git init\n> 2. mkdir d\n> 3. touch d/a\n> 4. chmod -w d\n> 5. git clean -fd\n> \n> Actual result:\n> Git doesn't remove anything, saying \"warning: failed to remove d/a\".\n> \n> Expected result:\n> Git should remove the subdirectory 'd' along with its contents. Note that git can remove a read-only file (touch b; chmod -w b; git clean -f) with no problem.\n\nI don't believe git should depart from rm in this regard.  I believe\nthat in general, Unix has stood the test of time, and when in doubt, we\nshould behave as Unix utilities do.\n\nNote that other utilities (such as Perl) which have implemented\ndifferent behavior (usually for compatibility with Windows) have found\nthemselves with security vulnerabilities in that behavior.  That seems\nlike a prudent reason not to replicate it.\n\n> Why this is important:\n> * This has a significant impact in real scenarios: the go language,\n>   when using modules, will download referenced modules and place them\n>   in read-only directories inside a cache directory, potentially\n>   inside the git repository. These cached modules can't be cleaned up\n>   by git clean. (Furthermore, git clean then returns a failure status\n>   code, which causes our build system to fail. I don't want to ignore\n>   the failure code.)\n\nThis sounds like the real problem.  Why is Go placing data in cache\ndirectories that are read-only?  Cache directories are explicitly\nephemeral and should be able to be destroyed at any time.  You'll\nprobably find more luck convincing the Go maintainers that their caches\nshould be temporary than you will us that git clean should be your\nautomatic destroyer of data.  Rust, for example, doesn't do this.\n\nYou could also just move your cache directories into $TMPDIR/go-cache\nand then do \"chmod -R u+w $TMPDIR/go-cache/* && rm -fr $TMPDIR/go-cache/*\".\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}