Re: A potential approach to making tests faster on Windows

From: Jeff King <peff@peff.net>
Date: Apr 3, 2018, 13:27 UTC
Message-ID: <20180403132717.GC18824@sigill.intra.peff.net>
In-Reply-To: <nycvar.QRO.7.76.6.1804031150030.5026@qfpub.tvgsbejvaqbjf.bet>

On Tue, Apr 03, 2018 at 01:43:10PM +0200, Johannes Schindelin wrote:

Show 16 quoted lines

> > I don't have time or interest to work on this now, but thought it was
> > interesting to share. This assumes that something in shellscript like:
> > 
> >     while echo foo; do echo bar; done
> > 
> > Is no slower on Windows than *nix, since it's purely using built-ins, as
> > opposed to something that would shell out.
> 
> It is still interpreting stuff. And it still goes through the POSIX
> emulation layer.
> 
> I did see reports on the Git for Windows bug tracker that gave me the
> impression that such loops in Unix shell scripts may not, in fact, be as
> performant in MSYS2's Bash as you would like to believe:
> 
> https://github.com/git-for-windows/git/issues/1533#issuecomment-372025449

The main problem with `read` loops in shell is that the shell makes one read() syscall per character. It has to, because doing otherwise is user-visible in cases where the descriptor may get passed to a different process.

There's unfortunately no portable way to say "please just read this quickly, I promise nobody else is going to read the descriptor". And nor do I know of any shell which is smart enough to know that it's going to consume to EOF anyway (as you would for something like "cmd | while read").

If you know you have bash, you can use "-N" to get a more efficient read:

  $ echo foo | strace -e read bash -c 'read foo'
  [...]
  read(0, "f", 1)                         = 1
  read(0, "o", 1)                         = 1
  read(0, "o", 1)                         = 1
  read(0, "\n", 1)                        = 1

  $ echo foo | strace -e read bash -c 'read -N 10 foo'
  [...]
  read(0, "foo\n", 10)                    = 4
  read(0, "", 6)                          = 0

but then you have another problem: how to split the resulting buffer into lines in shell. ;)

But if we're at the point of creating custom C builtins for busybox/dash/etc, you should be able to create a primitive for "read this using buffered stdio, other processes be damned, and return one line at a time".

-Peff