{"thread":{"id":"52850","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","startedAt":"2020-02-20T18:55:52Z","lastAt":"2020-02-21T01:48:45Z","messageCount":8,"participants":["Adam Milazzo","Elijah Newren","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"392185","messageId":"BL0PR2101MB108917C204868FA653C2948680130@BL0PR2101MB1089.namprd21.prod.outlook.com","threadId":"52850","inReplyTo":null,"subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Adam Milazzo","fromEmail":"adam.milazzo@microsoft.com","sentAt":"2020-02-20T18:55:48Z","receivedAt":"2020-02-20T18:55:52Z","isPatch":false,"sender":{"key":"adam.milazzo@microsoft.com","avatar":null},"body":"> It is how UNIX-like filesystem works, isn't it?\n\nSure, but it doesn't have to be how \"git clean -d\" works. In particular, \"git clean -ffd\" could be more forceful, or there could be another option or a third level of force beyond the current two levels.\n\nIt really comes back to this question: \"How can I avoid the failure, given that I am running 'git clean' from a script and not interactively?\" The only answer I could find that's not unreasonable is to parse the text output of 'git status -s' to find the untracked directories and then run 'chmod -R u+w' on each of those directories before running 'git clean -d'.\" I can do that, but I still think a \"force\" (or other) flag that really forces the cleanup would be preferable, especially given that this isn't a completely idiosyncratic scenario but one that will happen more and more as go modules are adopted (unless go is changed to stop putting them in read-only directories).\n"},{"id":"392189","messageId":"CABPp-BE8t+M5A8BpkYzs-WkoKDTCR_4jiT-vqwpLriuxhx46eQ@mail.gmail.com","threadId":"52850","inReplyTo":"BL0PR2101MB108917C204868FA653C2948680130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-02-20T19:45:12Z","receivedAt":"2020-02-20T19:45:25Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 20, 2020 at 10:58 AM Adam Milazzo\n<Adam.Milazzo@microsoft.com> wrote:\n>\n> > It is how UNIX-like filesystem works, isn't it?\n>\n> Sure, but it doesn't have to be how \"git clean -d\" works. In particular, \"git clean -ffd\" could be more forceful, or there could be another option or a third level of force beyond the current two levels.\n>\n> It really comes back to this question: \"How can I avoid the failure, given that I am running 'git clean' from a script and not interactively?\" The only answer I could find that's not unreasonable is to parse the text output of 'git status -s' to find the untracked directories and then run 'chmod -R u+w' on each of those directories before running 'git clean -d'.\" I can do that, but I still think a \"force\" (or other) flag that really forces the cleanup would be preferable, especially given that this isn't a completely idiosyncratic scenario but one that will happen more and more as go modules are adopted (unless go is changed to stop putting them in read-only directories).\n\n\nTying permission override to e.g -ffd would be a really bad idea.\nPeople would start adopting that for the permission override reason,\nthen someone is going to accidentally nuke a git submodule with\nunpushed changes.\n\nTying it to -fd doesn't have that particular problem, but I'm worried\na bit about a slippery slope if we do it.  If we say that \"git clean\n-fd\" should check and modify directory permission bits, should it also\ncheck and modify any necessary ACLs?  What if the user in question\ndoesn't have perms, but the user in question is in sudoers -- should\ngit try to see if they can run chown or chattr under sudo too?\n\nAlso, another way to look at this; currently 'git clean -fd' behaves\nthe same on untracked directories as 'rm -rf' does and I think that's\na good model for how to behave.  Why should they be different?\n\n$ git clean -fd unwritable-dir/\nwarning: failed to remove unwritable-dir/a: Permission denied\nwarning: failed to remove unwritable-dir/b: Permission denied\nwarning: failed to remove unwritable-dir/c: Permission denied\n$ rm -rf unwritable-dir/\nrm: cannot remove 'unwritable-dir/a': Permission denied\nrm: cannot remove 'unwritable-dir/b': Permission denied\nrm: cannot remove 'unwritable-dir/c': Permission denied\n"},{"id":"392190","messageId":"BL0PR2101MB1089E3CE7738BB70B65C17B680130@BL0PR2101MB1089.namprd21.prod.outlook.com","threadId":"52850","inReplyTo":"CABPp-BE8t+M5A8BpkYzs-WkoKDTCR_4jiT-vqwpLriuxhx46eQ@mail.gmail.com","subject":"RE: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Adam Milazzo","fromEmail":"adam.milazzo@microsoft.com","sentAt":"2020-02-20T20:05:48Z","receivedAt":"2020-02-20T20:05:56Z","isPatch":false,"sender":{"key":"adam.milazzo@microsoft.com","avatar":null},"body":"> Tying permission override to e.g -ffd would be a really bad idea. People would start adopting that for the permission override reason, then someone is going to accidentally nuke a git submodule with unpushed changes.\n\nFair enough. I'm not arguing that it must necessarily be any kind of overloading of -f. It could just as well be a separate option. (Personally I'd say overloading -f to make \"-f -f\" mean \"also delete untracked git repositories\" was the bad idea, since that is not the obvious and intuitive meaning of \"--force\", whereas \"try harder to delete files you would normally delete\" _is_ the intuitive meaning, but the history can't be changed.)\n\n> Also, another way to look at this; currently 'git clean -fd' behaves the same on untracked directories as 'rm -rf' does and I think that's a good model for how to behave.  Why should they be different?\n\nI would ask \"why should they be the same?\", because there's no obvious reason why behavior of \"git clean\", whose purpose is to put a repository back to a pristine state, should always behave the same way as \"rm\", whose purpose is to delete a specified set of files, and more to the point there is no reason why they should have no option to behave differently. I am not arguing for the default behavior of \"git clean\" to change, since that could be seen as a breaking change. Also, they are already different. \"git clean\" will remove a read-only file. \"rm\" will not (without confirmation), although \"rm -f\" will.\n"},{"id":"392210","messageId":"xmqqo8ts99pp.fsf@gitster-ct.c.googlers.com","threadId":"52850","inReplyTo":"BL0PR2101MB1089E3CE7738BB70B65C17B680130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-20T22:26:26Z","receivedAt":"2020-02-20T22:26:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Milazzo <Adam.Milazzo@microsoft.com> writes:\n\n>> Also, another way to look at this; currently 'git clean -fd'\n>> behaves the same on untracked directories as 'rm -rf' does and I\n>> think that's a good model for how to behave.  Why should they be\n>> different?\n>\n> I would ask \"why should they be the same?\"\n\nSimply because that is how users would expect how the world works\n(iow, model things after what they are already familiar with).\n"},{"id":"392217","messageId":"CABPp-BFwNMTRUUonbt3Di8yWDjyF7n2cNQVoLDfHEsk3RVEchQ@mail.gmail.com","threadId":"52850","inReplyTo":"BL0PR2101MB1089E3CE7738BB70B65C17B680130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-02-20T23:13:33Z","receivedAt":"2020-02-20T23:13:47Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Feb 20, 2020 at 12:05 PM Adam Milazzo\n<Adam.Milazzo@microsoft.com> wrote:\n>\n> > Tying permission override to e.g -ffd would be a really bad idea. People would start adopting that for the permission override reason, then someone is going to accidentally nuke a git submodule with unpushed changes.\n>\n> Fair enough. I'm not arguing that it must necessarily be any kind of overloading of -f. It could just as well be a separate option. (Personally I'd say overloading -f to make \"-f -f\" mean \"also delete untracked git repositories\" was the bad idea, since that is not the obvious and intuitive meaning of \"--force\", whereas \"try harder to delete files you would normally delete\" _is_ the intuitive meaning, but the history can't be changed.)\n\nYeah, arguing to change existing behavior (as you started with by\nlabelling the thread with \"BUG\") will probably get you a lot of\npushback, but let me address your new angle of suggesting a new\noption...\n\n> > Also, another way to look at this; currently 'git clean -fd' behaves the same on untracked directories as 'rm -rf' does and I think that's a good model for how to behave.  Why should they be different?\n>\n> I would ask \"why should they be the same?\",\n\nI think it matches user expectation of how the filesystem works.  But\nif we did want to implement a new flag, there's a question of what it\nshould be and what it should do.  Your \"put the repository back to a\npristine state\" isn't enough to really go on from an implementation.\nClearly you don't mean that it should also do a 'reset --hard' --\ndespite the fact that running a reset fits this definition, but even\nin terms of untracked files you've still left out the \"slippery slope\"\nangle from my previous response in what you've addressed here.  Let's\nsay we did implement a new flag that said we should override standard\npermissions.  In such a case, shouldn't ACLs also be overridden?  Can\nwe handle that in any semi-portable fashion?  Assuming we do attempt\nto override ACLs, do we destroy portability of git (I don't know a\nwhole lot about ACLs, so I don't know, but I thought they varied a lot\nbetween systems) -- and initiate a never-ending treadmill of needs to\nhandle more kinds of ACLs?  If we don't try to overrule ACLs, doesn't\nthat defeat your whole argument that \"git clean\" (with various levels\nof forcing) is about trying to do as much as possible to put the\nrepository in a clean state?  Ignoring ACLs for a minute, what if we\nattempt to override the permissions by marking directories as\nwritable, but find out the user isn't the owner of the directory and\nthus can't change its permissions?  Do we try harder by attempting to\ninvoke chmod under sudo to see if we can override the permissions?  At\nwhat point do we give up?  What's the clear line...and why does that\nline not just get drawn at not changing permissions at all?\n\n>  because there's no obvious reason why behavior of \"git clean\", whose purpose is to put a repository back to a pristine state, should always behave the same way as \"rm\", whose purpose is to delete a specified set of files, and more to the point there is no reason why they should have no option to behave differently. I am not arguing for the default behavior of \"git clean\" to change, since that could be seen as a breaking change. Also, they are already different. \"git clean\" will remove a read-only file. \"rm\" will not (without confirmation), although \"rm -f\" will.\n\nHere you've changed the goalposts slightly.  I didn't compare \"git\nclean\" to \"rm\", I compared \"git clean -fd\" to \"rm -rf\" on untracked\ndirectories.\n\nHowever, you are right that even those two aren't the same; \"rm -rf\"\nwill recurse into a git submodule and delete it while \"git clean -fd\"\nwon't.  In other words, \"git clean -fd\" avoids bringing the repository\nback to a pristine state in a certain circumstance; it's actually more\ncautious than rm.\n\n\nI'm sympathetic to \"golang does a dumb thing I don't like\" (the exact\nsituation you hilight has bothered me before too), but it's not clear\nat all to me where or how git might help you cope with that.  To me,\ngolang did a dumb and you have to manually undo the dumb.  If you want\nsomething smarter on the git side instead of the golang side, all the\nstuff I mentioned above are things that would need to be addressed in\nany possible solution that I'm just not seeing.\n\n\nHope that helps,\nElijah\n"},{"id":"392224","messageId":"BL0PR2101MB108926E052DA3B9072AF9F1C80130@BL0PR2101MB1089.namprd21.prod.outlook.com","threadId":"52850","inReplyTo":"xmqqo8ts99pp.fsf@gitster-ct.c.googlers.com","subject":"RE: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Adam Milazzo","fromEmail":"adam.milazzo@microsoft.com","sentAt":"2020-02-20T23:52:40Z","receivedAt":"2020-02-20T23:52:48Z","isPatch":false,"sender":{"key":"adam.milazzo@microsoft.com","avatar":null},"body":"> Simply because that is how users would expect how the world works (iow, model things after what they are already familiar with).\n\nThis seems to be an avoidance of my actual arguments about 1) the purpose of \"git clean\" and what behavior best matches it, and 2) the violation of the general principle that if a tool invoked programmatically can fail, then there should be a reasonable way for users to avoid the failure if possible. But my response is:\n\nFirst, there is no obvious choice for what other tool to model \"git clean\" on, even assuming that it should be so modeled. This goes back to the purpose of \"git clean\". Is it just a recursive delete? Or it bringing the directory tree back to a certain state? I'd argue the latter, and if we want point to existing tools I'd point to rsync, which has no problem deleting files from read-only directories if it's needed to bring a directory tree to the desired state. It doesn't even give a warning about it.\n\nSecond, I doubt anybody here actually knows (i.e. has data demonstrating) that users expect 'git clean' to behave like 'rm'. Also, I am a user, and it is not what _I_ expect. (And since some people here seem keen to dismiss what I say based on an assumption of ignorance, I've been programming for 30 years, using GNU/Linux, BSD, and other UNIX-like systems for almost 20 years, and using various source control systems for about as long. Not that that should carry any intrinsic weight in this discussion.)\n\nComparing to \"rm\" again, there is an easy way for users of \"rm\" to avoid the error. Simply replace \"rm -rf X\" with \"chmod -R u+w X; rm -rf X\". What is the comparable workaround with \"git clean\"? There is none that I'm aware of, and that's perhaps the main reason why it would be useful for \"git clean\" to be able to handle it. If there is a reasonable workaround, what is it? The best simple workarounds I've been able to come up with are:\n\n* For \"git clean -fd\": git status -s -uall | grep -E '^\\?\\?.*/$' | cut -c 4- | xargs -r chmod -R u+w; git clean -fd\n* For \"git clean -fdx\": git status -s -uall --ignored | grep -E '^\\?\\?.*/$' | cut -c 4- | xargs -r chmod -R u+w; git clean -fdx\n* For \"git clean -fX\": ??\n* For \"git clean -f\": ??\n\nThese are not reliable because there are various conditions where they fail (including ours), so I'm not sure they are viable approaches except in certain special cases. It's possible to handle all the possibilities with custom scripting, but the workarounds would become quite complex.\n\nSo I ask again, if \"git clean\" won't have any option to handle it like rsync does, what is the workaround that can be placed in a script to get the same behavior? And if there is no reasonable workaround, perhaps it is a useful feature to have \"git clean\" try a little harder to delete the files, or have an option to do so?\n"},{"id":"392231","messageId":"CABPp-BFyfc=wTnHJgVSCPYqSMdGGb-_1w9OVX0XzOkaZF9UPng@mail.gmail.com","threadId":"52850","inReplyTo":"BL0PR2101MB108926E052DA3B9072AF9F1C80130@BL0PR2101MB1089.namprd21.prod.outlook.com","subject":"Re: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-02-21T00:42:38Z","receivedAt":"2020-02-21T00:42:54Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Adam,\n\nOn Thu, Feb 20, 2020 at 3:52 PM Adam Milazzo <Adam.Milazzo@microsoft.com> wrote:\n>\n> > Simply because that is how users would expect how the world works (iow, model things after what they are already familiar with).\n>\n> This seems to be an avoidance of my actual arguments about 1) the purpose of \"git clean\" and what behavior best matches it, and 2) the violation of the general principle that if a tool invoked programmatically can fail, then there should be a reasonable way for users to avoid the failure if possible. But my response is:\n>\n> First, there is no obvious choice for what other tool to model \"git clean\" on, even assuming that it should be so modeled. This goes back to the purpose of \"git clean\". Is it just a recursive delete? Or it bringing the directory tree back to a certain state? I'd argue the latter, and if we want point to existing tools I'd point to rsync, which has no problem deleting files from read-only directories if it's needed to bring a directory tree to the desired state. It doesn't even give a warning about it.\n\nI'm very sympathetic to the fact that \"git clean\" behavior might not\nbe optimal or even well defined[1][2][3].  I've recently done work in\nthe area, including even changing the existing behavior of some\ncommands, based on arguments about what should be correct behavior.\nSome of that work languished for a year and a half, despite fixing\nknown bugs, because there were edge cases where I couldn't tell what\ncorrect behavior was and no one else seemed to be able to answer\neither.\n\n[1] https://lore.kernel.org/git/20190917163504.14566-1-newren@gmail.com/\n[2] https://lore.kernel.org/git/pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com/\n[3] https://lore.kernel.org/git/pull.692.v3.git.git.1579206117.gitgitgadget@gmail.com/\n\nIf you want to do something similar, I think you need to provide good\nrationale for not only what you are trying to achieve, but explain the\nedge cases and how to address probable future similar requests in a\nway that make sense to someone who might try to implement it.  If I\nwere to try to implement your suggestion, I'm saying that your\ndescriptions are not at all clear to me in terms of how I should\nhandle edge cases and future related improvements that folks ask for,\nnot even if this is implemented as a new option.\n\nFor example, you talk about bringing the directory tree back to a\ncertain state -- does that mean git clean should also run 'git reset\n--hard'?  I need a more precise model/description...\n\n> Second, I doubt anybody here actually knows (i.e. has data demonstrating) that users expect 'git clean' to behave like 'rm'. Also, I am a user, and it is not what _I_ expect. (And since some people here seem keen to dismiss what I say based on an assumption of ignorance, I've been programming for 30 years, using GNU/Linux, BSD, and other UNIX-like systems for almost 20 years, and using various source control systems for about as long. Not that that should carry any intrinsic weight in this discussion.)\n>\n> Comparing to \"rm\" again, there is an easy way for users of \"rm\" to avoid the error. Simply replace \"rm -rf X\" with \"chmod -R u+w X; rm -rf X\". What is the comparable workaround with \"git clean\"? There is none that I'm aware of, and that's perhaps the main reason why it would be useful for \"git clean\" to be able to handle it. If there is a reasonable workaround, what is it? The best simple workarounds I've been able to come up with are:\n>\n> * For \"git clean -fd\": git status -s -uall | grep -E '^\\?\\?.*/$' | cut -c 4- | xargs -r chmod -R u+w; git clean -fd\n> * For \"git clean -fdx\": git status -s -uall --ignored | grep -E '^\\?\\?.*/$' | cut -c 4- | xargs -r chmod -R u+w; git clean -fdx\n> * For \"git clean -fX\": ??\n> * For \"git clean -f\": ??\n\nFor every single case, why not just \"chmod -R u+w $toplevel_dir\"\nfollowed by the git clean command in question, much like you did with\nrm?\n\n> These are not reliable because there are various conditions where they fail (including ours), so I'm not sure they are viable approaches except in certain special cases. It's possible to handle all the possibilities with custom scripting, but the workarounds would become quite complex.\n>\n> So I ask again, if \"git clean\" won't have any option to handle it like rsync does, what is the workaround that can be placed in a script to get the same behavior? And if there is no reasonable workaround, perhaps it is a useful feature to have \"git clean\" try a little harder to delete the files, or have an option to do so?\n\nI think the single simple recursive chmod I mentioned above is a\nreasonable workaround.\n\nBut since you are bringing up rsync as a comparison point...rsync also\naffects ACLs, xattrs, devices and other special files, etc.  So, how\nmuch harder is a little harder?  Is there a good mental model for that\nbeing the right amount of harder, or do we just keep extending it\nevery time the command fails to clean something that users think we\ncould have wiped out?\n\n\nHope that helps,\nElijah\n"},{"id":"392239","messageId":"xmqq36b490cn.fsf@gitster-ct.c.googlers.com","threadId":"52850","inReplyTo":"CABPp-BFwNMTRUUonbt3Di8yWDjyF7n2cNQVoLDfHEsk3RVEchQ@mail.gmail.com","subject":"Re: [EXTERNAL] Re: BUG: git clean -d cannot remove files from read-only directories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-21T01:48:40Z","receivedAt":"2020-02-21T01:48:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren <newren@gmail.com> writes:\n\n> should be and what it should do.  Your \"put the repository back to a\n> pristine state\" isn't enough to really go on from an implementation.\n\nYeah, where does this \"pristine state\" come from?  Do we need to fix\nthe \"git clean\" documentation to reduce such a confusion (I didn't\ncheck to see if the current text is prone to be misunderstood as\nsuch yet).  \"git clean\" is to remove crufts, isn't it?  The\ndefinition of \"cruft\" being untracked junk that are marked\nexpendable (aka \"ignored\")?  And it is not \"remove crurfts no matter\nwhat\".\n\n> angle from my previous response in what you've addressed here.  Let's\n> say we did implement a new flag that said we should override standard\n> permissions.  In such a case, shouldn't ACLs also be overridden?  ...\n> what point do we give up?  What's the clear line...and why does that\n> line not just get drawn at not changing permissions at all?\n\nYup.  Elsewhere in the thread you gave \"rm -fr\" not removing an\nunwritable subdirectory, and that shows where others before us drew\nthe line that has been accepted by the end users.\n\n> I'm sympathetic to \"golang does a dumb thing I don't like\" (the exact\n> situation you hilight has bothered me before too), but it's not clear\n> at all to me where or how git might help you cope with that.  To me,\n> golang did a dumb and you have to manually undo the dumb.  If you want\n> something smarter on the git side instead of the golang side, all the\n> stuff I mentioned above are things that would need to be addressed in\n> any possible solution that I'm just not seeing.\n\nAmen.\n"}]}