Re: Git clean enumerates ignored directories (since 2.27)

From: Jeff King <peff@peff.net>
Date: May 7, 2021, 03:43 UTC
Message-ID: <YJS3RIEtittWdFSF@coredump.intra.peff.net>
In-Reply-To: <YJSzQm2p5bCAd8Fv@coredump.intra.peff.net>

On Thu, May 06, 2021 at 11:25:54PM -0400, Jeff King wrote:

Show 20 quoted lines

> > Anyone have any bright ideas about how to tweak this test?  See [3]
> > for the current incarnation of the code, which was basically taken
> > from Brian's sample testcase.
> 
> My guess is that that version of "rm" is trying to feed the entire
> pathname directly to unlink() and rmdir(), and it exceeds PATH_MAX.
> 
> Even with GNU tools, for instance, I get:
> 
>   $ rmdir $(find avoid-traversing-deep-hierarchy -type d | tail -1)
>   rmdir: failed to remove 'avoid-traversing-deep-hierarchy/directory400/
>     [...and so on...]/directory1': File name too long
> 
> because it feeds the whole to a single rmdir() call. Whereas stracing
> GNU "rm -rf", it uses unlinkat() and openat() to delete each level
> individually (probably to avoid this exact problem).
> 
> Is the actual path length important, or just the depth? If the latter,
> then calling it "d400/d399/.../d2/d1" would likely help, as that's less
> than 2000 bytes.

Reading your commit messages a little more carefully, I'm still not quite sure of the answer to that question.

But if you really do need the long length, a workaround is to avoid dealing with the full path all at once. E.g., make two strings, one with "directory400/.../directory200", and one with "directory199/.../directory1".

And then you can probably:

  (cd $one && rm -rf directory199) &&
  rm -rf directory400

to do it in two parts, with each "rm" seeing only a half-length path.

I notice you also run O(n) "mkdir" and "mv" calls to create the directory. I "mkdir -p" would be much more efficient, though it might run afoul of similar path-length problems (especially on non-GNU systems).

It might be worth turning to perl here:

  perl -e '
    for (reverse 1..400) {
      my $d = "directory$_";
      mkdir($d) and chdir($d) or die "mkdir($d): $!";
    }
    open(my $fh, ">", "some-file");
  '

and you could probably do something similar to remove it. Sadly, I don't think using File::Path makes building it easier, because it hits the same path limit (it builds up the string internally). However, its removal does work (and is in the set of core modules that we can count on always being available):

  perl -MFile::Path=remove_tree -e 'remove_tree("directory400")'

-Peff