Re: [PATCH 0/4] some chainlint fixes and performance improvements

From: Jeff King <peff@peff.net>
Date: Mar 28, 2023, 21:08 UTC
Message-ID: <20230328210814.GA1754178@coredump.intra.peff.net>
In-Reply-To: <20230328202043.GA1241391@coredump.intra.peff.net>

On Tue, Mar 28, 2023 at 04:20:44PM -0400, Jeff King wrote:

Show 6 quoted lines

> The rest are some old performance improvement ideas I had for the
> internal chain-linter. I doubt they make a huge difference overall,
> but they can be measured in certain cases.  The first one to me looks
> like an obvious win. The second one is debatable, as it involves some
> hand-waving. The third one turned out not to make anything faster, but
> makes the code a little simpler.

BTW, I noticed something really funky when timing t3070 for this series.

  $ time ./t3070-wildmatch.sh
  [a bunch of output]
  real	0m4.750s
  user	0m3.665s
  sys	0m0.955s

  $ time ./t3070-wildmatch.sh >/dev/null
  real	0m18.664s
  user	0m9.185s
  sys	0m9.495s

Er, what? It gets way slower when redirected to /dev/null. I can't figure out why. Replacing the actual tests with a simple command shows the same behavior, so it's not a problem in the tested commands. I tried simplifying what the test script was doing, but it really looks like the slowdown scales with the number of subshells/forks.

The problem is also independent of shell (bash vs dash). And here's an even weirder bit. If I pipe the output through cat, it's still fast:

  $ time ./t3070-wildmatch.sh | cat
  [lots of output]
  real	0m4.719s
  user	0m3.636s
  sys	0m0.946s

but when cat goes to /dev/null, it's slow again!

  $ time ./t3070-wildmatch.sh | cat >/dev/null
  real	0m18.383s
  user	0m8.987s
  sys	0m9.450s

So our scripts are seeing the same environment (a pipe), but somehow cat on the other side is slowing things down. Which seems to me like it could be a kernel problem, but it's hard to imagine what. I tried a few versions and couldn't find one that didn't exhibit it. This is a Debian unstable machine (so running kernel 6.1.20). Another machine running stable (so kernel 5.15) did not exhibit the problem, but there are many different variables beyond kernel version there.

So this doesn't seem like a Git problem at all, but it's an interesting mystery, and I wondered if anybody else has run into it.

-Peff