Re: git log exclude pathspec from file - supported? plans?

From: Jeff King <peff@peff.net>
Date: Jun 30, 2021, 17:58 UTC
Message-ID: <YNywsEbFcrQFeH91@coredump.intra.peff.net>
In-Reply-To: <CACPiFCLXxwaWOuR6sy8H4hCG-H0ZFvVYma7COfDq3zxoUt=VtA@mail.gmail.com>

On Wed, Jun 30, 2021 at 12:59:43PM -0400, Martin Langhoff wrote:

Show 10 quoted lines

> long time no see! I'm doing some complex git repo spelunking and
> pushing the boundaries of the pathspec magic for excludes.
> 
> Is there a reasonable way to provide a (potentially large) set of
> excludes? something like
> 
>      git log --exclude-pathspec-file paths-to-exclude.txt .
> 
> Has there been discussion / patches / plans related to this? I may
> have some cycles (hopefully!)

You can feed pathspecs via --stdin. So:

  {
	echo "--"
	sed s/^/:^/ paths-to-exclude.txt
  } | git log --stdin

works. Obviously it's not as turn-key if you really do have a list of paths in a file already, but it's much more flexible.

I'll caution you that the pathspec code is not well-optimized to handle a large number of pathspecs. E.g.:

  [no pathspecs]
  $ time git rev-list HEAD /dev/null
  real	0m0.033s
  user	0m0.017s
  sys	0m0.017s

  [trivial pathspec; now we have to actually open up trees]
  $ { echo --; echo .; } >input
  $ time git rev-list HEAD --stdin <input >/dev/null
  real	0m1.338s
  user	0m1.294s
  sys	0m0.045s

  [lots of pathspecs; now we spend loads of time actually matching
   strings; the ^C is when I got bored and killed it]
  $ { echo --; git ls-files; } >input
  $ time git rev-list HEAD --stdin <input >/dev/null
  ^C
  real	1m24.406s
  user	1m24.369s
  sys	0m0.036s

The problem is that we try to linearly match every pathspec against every path we consider, so it's quadratic-ish in the number of files in the repo. I played a long time ago with storing non-wildcard pathspecs in a trie that we could traverse as we talked the individual trees we were matching. It performed well, but IIRC the interface was hacky (I had to bolt it specifically onto the way the tree-walker uses pathspecs, and the other pathspec matchers didn't benefit at all).

I can probably dig it up if anybody's interested in looking at it.

-Peff