{"thread":{"id":"55494","subject":"Git clean enumerates ignored directories (since 2.27)","startedAt":"2021-04-14T17:17:54Z","lastAt":"2021-05-07T04:08:34Z","messageCount":10,"participants":["Jason Gore","brian m. carlson","Jeff King","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"421976","messageId":"DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com","threadId":"55494","inReplyTo":null,"subject":"Git clean enumerates ignored directories (since 2.27)","fromName":"Jason Gore","fromEmail":"jason.gore@microsoft.com","sentAt":"2021-04-14T17:17:46Z","receivedAt":"2021-04-14T17:17:54Z","isPatch":false,"sender":{"key":"jason.gore@microsoft.com","avatar":null},"body":"I'm unsure of whether this issue is Windows-specific but at the very least I suspect this is a change of behavior that would affect all platforms.\n\nIn July 2020 I reported an issue I initially encountered in 2.27 that still seems to be present in 2.31.1.windows.1: https://github.com/git-for-windows/git/issues/2732\n\nSince I haven't seen a response on the issue and it still occurs I decided to email this list as well.\n\nPackage managers such as PNPM tend to create very long filenames due to symlinking. Having git not enumerate these ignored directories is essential to leveraging any sort of clean behavior (both for performance and for warning output) as we did before version 2.27. Our repo clean functions went from taking a few seconds to over 10 minutes due to this change in behavior.\n\nI've also had repeated problems sending you this email as your email server seems to reject any email with a URL in it (per the github link above.)\n\nThanks,\nJason\n"},{"id":"421991","messageId":"YHdzDlAfsuZ21HR7@camp.crustytoothpaste.net","threadId":"55494","inReplyTo":"DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-04-14T22:56:14Z","receivedAt":"2021-04-14T22:56:53Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-04-14 at 17:17:46, Jason Gore wrote:\n> I'm unsure of whether this issue is Windows-specific but at the very least I suspect this is a change of behavior that would affect all platforms.\n> \n> In July 2020 I reported an issue I initially encountered in 2.27 that still seems to be present in 2.31.1.windows.1: https://github.com/git-for-windows/git/issues/2732\n> \n> Since I haven't seen a response on the issue and it still occurs I decided to email this list as well.\n> \n> Package managers such as PNPM tend to create very long filenames due to symlinking. Having git not enumerate these ignored directories is essential to leveraging any sort of clean behavior (both for performance and for warning output) as we did before version 2.27. Our repo clean functions went from taking a few seconds to over 10 minutes due to this change in behavior.\n\nIt looks like there were some changes to the dir.c code between 2.26 and\n2.27 which may have impacted clean. I'm CCing Elijah Newren who wrote\nthose patches and in any event would be more familiar with that code\nthan I am.\n\nI've taken the liberty of pulling in your testcase from that issue and\nconverting it to work in POSIX sh on a Linux machine. If the path is\nnot sufficiently long for your needs, you can bump the value of the variable\n\"last\" below.\n\n----\n#!/bin/sh\n\ngit init test-repo\ncd test-repo\nlongname=\"directory\"\ntouch \"$longname.txt\"\nlast=400\nfor x in $(seq 1 $last); do\n mkdir \"x$longname$x\"\n mv directory* \"x$longname$x\"\n mv \"x$longname$x\" \"$longname$x\"\ndone\ngit clean -ffdxn -e directory$last\n----\n\nWhen it fails, it will complain that it wasn't able to open the\ndirectory. It still exits zero, however.\n\nI haven't bisected this, so I don't know if those patches are related to\nthe problem or not. I'm a little short on time today to investigate\nfurther, but hopefully this can get someone on the right path with a\nmodified version and git bisect run if nothing else.\n\n> I've also had repeated problems sending you this email as your email server seems to reject any email with a URL in it (per the github link above.)\n\nI don't think that's the problem, since this email came through just\nfine. It's more likely that your email was being bounced for HTML,\nwhich isn't allowed on the list.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"422008","messageId":"YHf+m7vOQumdJXwX@coredump.intra.peff.net","threadId":"55494","inReplyTo":"YHdzDlAfsuZ21HR7@camp.crustytoothpaste.net","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-15T08:51:39Z","receivedAt":"2021-04-15T08:51:41Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 14, 2021 at 10:56:14PM +0000, brian m. carlson wrote:\n\n> ----\n> #!/bin/sh\n> \n> git init test-repo\n> cd test-repo\n> longname=\"directory\"\n> touch \"$longname.txt\"\n> last=400\n> for x in $(seq 1 $last); do\n> mkdir \"x$longname$x\"\n> mv directory* \"x$longname$x\"\n> mv \"x$longname$x\" \"$longname$x\"\n> done\n> git clean -ffdxn -e directory$last\n> ----\n> \n> When it fails, it will complain that it wasn't able to open the\n> directory. It still exits zero, however.\n> \n> I haven't bisected this, so I don't know if those patches are related to\n> the problem or not. I'm a little short on time today to investigate\n> further, but hopefully this can get someone on the right path with a\n> modified version and git bisect run if nothing else.\n\nIt bisects to 8d92fb2927 (dir: replace exponential algorithm with a\nlinear one, 2020-04-01). I won't pretend to understand everything going\non in that commit, though.\n\n-Peff\n"},{"id":"422695","messageId":"MN2PR00MB068886B8F36DFC645138DA0489469@MN2PR00MB0688.namprd00.prod.outlook.com","threadId":"55494","inReplyTo":"YHf+m7vOQumdJXwX@coredump.intra.peff.net","subject":"RE: Git clean enumerates ignored directories (since 2.27)","fromName":"Jason Gore","fromEmail":"jason.gore@microsoft.com","sentAt":"2021-04-22T17:18:19Z","receivedAt":"2021-04-22T17:18:26Z","isPatch":false,"sender":{"key":"jason.gore@microsoft.com","avatar":null},"body":"Sorry to add noise to this thread but since the github issue was closed I wanted to make sure I'm not missing any updates here. Would any updates come through on this thread? Thanks!\n\n-----Original Message-----\nFrom: Jeff King \nSent: Thursday, April 15, 2021 1:52 AM\nTo: brian m. carlson \nCc: Jason Gore ; git@vger.kernel.org; Elijah Newren \nSubject: Re: Git clean enumerates ignored directories (since 2.27)\n\nOn Wed, Apr 14, 2021 at 10:56:14PM +0000, brian m. carlson wrote:\n\n> ----\n> #!/bin/sh\n> \n> git init test-repo\n> cd test-repo\n> longname=\"directory\"\n> touch \"$longname.txt\"\n> last=400\n> for x in $(seq 1 $last); do\n> mkdir \"x$longname$x\"\n> mv directory* \"x$longname$x\"\n> mv \"x$longname$x\" \"$longname$x\"\n> done\n> git clean -ffdxn -e directory$last\n> ----\n> \n> When it fails, it will complain that it wasn't able to open the \n> directory. It still exits zero, however.\n> \n> I haven't bisected this, so I don't know if those patches are related \n> to the problem or not. I'm a little short on time today to \n> investigate further, but hopefully this can get someone on the right \n> path with a modified version and git bisect run if nothing else.\n\nIt bisects to 8d92fb2927 (dir: replace exponential algorithm with a linear one, 2020-04-01). I won't pretend to understand everything going on in that commit, though.\n\n-Peff\n"},{"id":"422696","messageId":"CABPp-BGWnnMGeK+507Bhm7go-23tVVmT0WfOph7cpHLVEr6snA@mail.gmail.com","threadId":"55494","inReplyTo":"MN2PR00MB068886B8F36DFC645138DA0489469@MN2PR00MB0688.namprd00.prod.outlook.com","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-04-22T17:24:13Z","receivedAt":"2021-04-22T17:24:28Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Yes, sorry, I've just been swamped. It's still on my list to look at.\n\nOn Thu, Apr 22, 2021 at 10:18 AM Jason Gore wrote:\n>\n> Sorry to add noise to this thread but since the github issue was closed I wanted to make sure I'm not missing any updates here. Would any updates come through on this thread? Thanks!\n>\n> -----Original Message-----\n> From: Jeff King \n> Sent: Thursday, April 15, 2021 1:52 AM\n> To: brian m. carlson \n> Cc: Jason Gore ; git@vger.kernel.org; Elijah Newren \n> Subject: Re: Git clean enumerates ignored directories (since 2.27)\n>\n> On Wed, Apr 14, 2021 at 10:56:14PM +0000, brian m. carlson wrote:\n>\n> > ----\n> > #!/bin/sh\n> >\n> > git init test-repo\n> > cd test-repo\n> > longname=\"directory\"\n> > touch \"$longname.txt\"\n> > last=400\n> > for x in $(seq 1 $last); do\n> > mkdir \"x$longname$x\"\n> > mv directory* \"x$longname$x\"\n> > mv \"x$longname$x\" \"$longname$x\"\n> > done\n> > git clean -ffdxn -e directory$last\n> > ----\n> >\n> > When it fails, it will complain that it wasn't able to open the\n> > directory. It still exits zero, however.\n> >\n> > I haven't bisected this, so I don't know if those patches are related\n> > to the problem or not. I'm a little short on time today to\n> > investigate further, but hopefully this can get someone on the right\n> > path with a modified version and git bisect run if nothing else.\n>\n> It bisects to 8d92fb2927 (dir: replace exponential algorithm with a linear one, 2020-04-01). I won't pretend to understand everything going on in that commit, though.\n>\n> -Peff\n"},{"id":"423837","messageId":"CABPp-BH3tbTyA9gj=2vb878d1MQVCTqT7=kS5U+sB6CoDX0Jww@mail.gmail.com","threadId":"55494","inReplyTo":"YHf+m7vOQumdJXwX@coredump.intra.peff.net","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T02:31:40Z","receivedAt":"2021-05-07T02:31:59Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Apr 15, 2021 at 1:51 AM Jeff King wrote:\n>\n> On Wed, Apr 14, 2021 at 10:56:14PM +0000, brian m. carlson wrote:\n>\n> > ----\n> > #!/bin/sh\n> >\n> > git init test-repo\n> > cd test-repo\n> > longname=\"directory\"\n> > touch \"$longname.txt\"\n> > last=400\n> > for x in $(seq 1 $last); do\n> > mkdir \"x$longname$x\"\n> > mv directory* \"x$longname$x\"\n> > mv \"x$longname$x\" \"$longname$x\"\n> > done\n> > git clean -ffdxn -e directory$last\n> > ----\n> >\n> > When it fails, it will complain that it wasn't able to open the\n> > directory. It still exits zero, however.\n> >\n> > I haven't bisected this, so I don't know if those patches are related to\n> > the problem or not. I'm a little short on time today to investigate\n> > further, but hopefully this can get someone on the right path with a\n> > modified version and git bisect run if nothing else.\n>\n> It bisects to 8d92fb2927 (dir: replace exponential algorithm with a\n> linear one, 2020-04-01). I won't pretend to understand everything going\n> on in that commit, though.\n\nI've got a patch series to fix this and another traversal problem, and\nthe tests pass on all platforms. Wahoo!\n\nBUT!\n\nOn alpine linux-musl, I get an \"error: Tests passed but test cleanup\nfailed; aborting\", which makes it report as a failed build. I'm not\nsure how to fix it and am asking for ideas.\n\nApparently the deeply nested directory hierarchy cannot be removed on\nthat platform with a simple \"rm -rf $dirname\". It throws a \"rm: can't\nstat '/__w/git/git/t/trash\ndirectory.t7300-clean/avoid-traversing-deep-hierarchy/directory400/directory399/directory398/.....(you\nget the idea)....': Filename too long\" error message.[1]\n\nAdding a \"test_when_finished find directory400 -delete\" also gives a\n\"Filename too long\" message followed by a lot of \"Directory not empty\"\nmessages.[2]\n\nAnyone have any bright ideas about how to tweak this test? See [3]\nfor the current incarnation of the code, which was basically taken\nfrom Brian's sample testcase.\n\n[1] https://github.com/git/git/pull/1020/checks?check_run_id=2523414561\n[2] https://github.com/git/git/pull/1020/checks?check_run_id=2523594095\n[3] https://github.com/git/git/pull/1020\n"},{"id":"423838","messageId":"YJSzQm2p5bCAd8Fv@coredump.intra.peff.net","threadId":"55494","inReplyTo":"CABPp-BH3tbTyA9gj=2vb878d1MQVCTqT7=kS5U+sB6CoDX0Jww@mail.gmail.com","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-07T03:25:54Z","receivedAt":"2021-05-07T03:25:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 06, 2021 at 07:31:40PM -0700, Elijah Newren wrote:\n\n> On alpine linux-musl, I get an \"error: Tests passed but test cleanup\n> failed; aborting\", which makes it report as a failed build. I'm not\n> sure how to fix it and am asking for ideas.\n> \n> Apparently the deeply nested directory hierarchy cannot be removed on\n> that platform with a simple \"rm -rf $dirname\". It throws a \"rm: can't\n> stat '/__w/git/git/t/trash\n> directory.t7300-clean/avoid-traversing-deep-hierarchy/directory400/directory399/directory398/.....(you\n> get the idea)....': Filename too long\" error message.[1]\n> \n> Adding a \"test_when_finished find directory400 -delete\" also gives a\n> \"Filename too long\" message followed by a lot of \"Directory not empty\"\n> messages.[2]\n> \n> Anyone have any bright ideas about how to tweak this test? See [3]\n> for the current incarnation of the code, which was basically taken\n> from Brian's sample testcase.\n\nMy guess is that that version of \"rm\" is trying to feed the entire\npathname directly to unlink() and rmdir(), and it exceeds PATH_MAX.\n\nEven with GNU tools, for instance, I get:\n\n $ rmdir $(find avoid-traversing-deep-hierarchy -type d | tail -1)\n rmdir: failed to remove 'avoid-traversing-deep-hierarchy/directory400/\n [...and so on...]/directory1': File name too long\n\nbecause it feeds the whole to a single rmdir() call. Whereas stracing\nGNU \"rm -rf\", it uses unlinkat() and openat() to delete each level\nindividually (probably to avoid this exact problem).\n\nIs the actual path length important, or just the depth? If the latter,\nthen calling it \"d400/d399/.../d2/d1\" would likely help, as that's less\nthan 2000 bytes.\n\n-Peff\n"},{"id":"423841","messageId":"YJS3RIEtittWdFSF@coredump.intra.peff.net","threadId":"55494","inReplyTo":"YJSzQm2p5bCAd8Fv@coredump.intra.peff.net","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-07T03:43:00Z","receivedAt":"2021-05-07T03:43:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 06, 2021 at 11:25:54PM -0400, Jeff King wrote:\n\n> > Anyone have any bright ideas about how to tweak this test? See [3]\n> > for the current incarnation of the code, which was basically taken\n> > from Brian's sample testcase.\n> \n> My guess is that that version of \"rm\" is trying to feed the entire\n> pathname directly to unlink() and rmdir(), and it exceeds PATH_MAX.\n> \n> Even with GNU tools, for instance, I get:\n> \n> $ rmdir $(find avoid-traversing-deep-hierarchy -type d | tail -1)\n> rmdir: failed to remove 'avoid-traversing-deep-hierarchy/directory400/\n> [...and so on...]/directory1': File name too long\n> \n> because it feeds the whole to a single rmdir() call. Whereas stracing\n> GNU \"rm -rf\", it uses unlinkat() and openat() to delete each level\n> individually (probably to avoid this exact problem).\n> \n> Is the actual path length important, or just the depth? If the latter,\n> then calling it \"d400/d399/.../d2/d1\" would likely help, as that's less\n> than 2000 bytes.\n\nReading your commit messages a little more carefully, I'm still not\nquite sure of the answer to that question.\n\nBut if you really do need the long length, a workaround is to avoid\ndealing with the full path all at once. E.g., make two strings, one with\n\"directory400/.../directory200\", and one with \"directory199/.../directory1\".\n\nAnd then you can probably:\n\n (cd $one && rm -rf directory199) &&\n rm -rf directory400\n\nto do it in two parts, with each \"rm\" seeing only a half-length path.\n\nI notice you also run O(n) \"mkdir\" and \"mv\" calls to create the\ndirectory. I \"mkdir -p\" would be much more efficient, though it might\nrun afoul of similar path-length problems (especially on non-GNU\nsystems).\n\nIt might be worth turning to perl here:\n\n perl -e '\n for (reverse 1..400) {\n my $d = \"directory$_\";\n mkdir($d) and chdir($d) or die \"mkdir($d): $!\";\n }\n open(my $fh, \">\", \"some-file\");\n '\n\nand you could probably do something similar to remove it. Sadly, I don't\nthink using File::Path makes building it easier, because it hits the\nsame path limit (it builds up the string internally). However, its\nremoval does work (and is in the set of core modules that we can count\non always being available):\n\n perl -MFile::Path=remove_tree -e 'remove_tree(\"directory400\")'\n\n-Peff\n"},{"id":"423842","messageId":"CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com","threadId":"55494","inReplyTo":"YJSzQm2p5bCAd8Fv@coredump.intra.peff.net","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T03:44:57Z","receivedAt":"2021-05-07T03:45:13Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, May 6, 2021 at 8:25 PM Jeff King wrote:\n>\n> On Thu, May 06, 2021 at 07:31:40PM -0700, Elijah Newren wrote:\n>\n> > On alpine linux-musl, I get an \"error: Tests passed but test cleanup\n> > failed; aborting\", which makes it report as a failed build. I'm not\n> > sure how to fix it and am asking for ideas.\n> >\n> > Apparently the deeply nested directory hierarchy cannot be removed on\n> > that platform with a simple \"rm -rf $dirname\". It throws a \"rm: can't\n> > stat '/__w/git/git/t/trash\n> > directory.t7300-clean/avoid-traversing-deep-hierarchy/directory400/directory399/directory398/.....(you\n> > get the idea)....': Filename too long\" error message.[1]\n> >\n> > Adding a \"test_when_finished find directory400 -delete\" also gives a\n> > \"Filename too long\" message followed by a lot of \"Directory not empty\"\n> > messages.[2]\n> >\n> > Anyone have any bright ideas about how to tweak this test? See [3]\n> > for the current incarnation of the code, which was basically taken\n> > from Brian's sample testcase.\n>\n> My guess is that that version of \"rm\" is trying to feed the entire\n> pathname directly to unlink() and rmdir(), and it exceeds PATH_MAX.\n>\n> Even with GNU tools, for instance, I get:\n>\n> $ rmdir $(find avoid-traversing-deep-hierarchy -type d | tail -1)\n> rmdir: failed to remove 'avoid-traversing-deep-hierarchy/directory400/\n> [...and so on...]/directory1': File name too long\n>\n> because it feeds the whole to a single rmdir() call. Whereas stracing\n> GNU \"rm -rf\", it uses unlinkat() and openat() to delete each level\n> individually (probably to avoid this exact problem).\n>\n> Is the actual path length important, or just the depth? If the latter,\n> then calling it \"d400/d399/.../d2/d1\" would likely help, as that's less\n> than 2000 bytes.\n\nI needed some kind of way to notice and test that we had recursed into\nthe directory. Excessively long paths trigger failures in open()\ncalls in git itself, which was the only marker I knew of. So, I was\nessentially depending on the long paths. (The actual depth was not\nimportant.). If there were a different way to determine whether we\nrecursed into the first level subdirectory, that'd be nicer. I\nthought about adding some trace2_region_enter/trace2_region_leave\ncalls and letting them nest up to whatever depth, though that seemed a\nbit ugly. I also thought about trying to determine the maximum\nrecursion level and seeing if that could be emitted via some kind of\ntrace2 magic, but that sounded like another rabbit hole. In the end,\nI gave up and used Brian's modification of Jason's testcase.\n\nHowever, it looks like manually unstacking and deleting the directory\nis going to work. Incredibly ugly and slow, but adding the following\nat the end of the testcase makes it work (at least on that platform,\nstill awaiting results on others):\n\n # alpine-linux-musl fails to \"rm -rf\" a directory with such\n # a deeply nested hierarchy. Help it out by deleting the\n # leading directories ourselves. Super slow, but, what else\n # can we do? Without this, we will hit a\n # error: Tests passed but test cleanup failed; aborting\n # so do this ugly manual cleanup...\n while test ! -f directory-random-file.txt; do\n name=$(ls -d directory*) &&\n mv $name/* . &&\n rmdir $name\n done\n"},{"id":"423850","messageId":"CABPp-BFWunQjgUHKuJBHatNfxdukL=EHGMiS3Tax3wW=eFSHXg@mail.gmail.com","threadId":"55494","inReplyTo":"MN2PR00MB068886B8F36DFC645138DA0489469@MN2PR00MB0688.namprd00.prod.outlook.com","subject":"Re: Git clean enumerates ignored directories (since 2.27)","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T04:08:20Z","receivedAt":"2021-05-07T04:08:34Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Apr 22, 2021 at 10:18 AM Jason Gore wrote:\n>\n> Sorry to add noise to this thread but since the github issue was closed I wanted to make sure I'm not missing any updates here. Would any updates come through on this thread? Thanks!\n\nI have posted some proposed fixes at\nhttps://lore.kernel.org/git/pull.1020.git.git.1620360300.gitgitgadget@gmail.com/T/#t\n(particularly patches 1 & 3); feel free to watch there, or over at\nhttps://github.com/git/git/pull/1020.\n"}]}