Re: Anyone know why git ls-remote output might be corrupted?

From: Jeff King <peff@peff.net>
Date: Jun 4, 2023, 06:25 UTC
Message-ID: <20230604062556.GA42964@coredump.intra.peff.net>
In-Reply-To: <20230604060048.GA38176@coredump.intra.peff.net>

On Sun, Jun 04, 2023 at 02:00:48AM -0400, Jeff King wrote:

Show 10 quoted lines

> Now, why the descriptor would be in non-blocking mode, I have no idea.
> But maybe something funny going on in your python script.
> 
> I'd be curious if applying the patch from:
> 
>   https://lore.kernel.org/git/YUTo1BTp7BXOw6K9@coredump.intra.peff.net/
> 
> reports any problems. As well as whether the suggested "sleep" pipeline
> there (triggered via your script in this case) shows the problem more
> reliably.

It does look like glibc's stdio will throw away buffer contents that get EAGAIN. Doing:

  perl -MFcntl -e '
    fcntl(STDOUT, F_GETFL, $flags);
    $flags |= O_NONBLOCK;
    fcntl(STDOUT, F_SETFL, $flags);
    exec @ARGV;
  ' git ls-remote . | (sleep 1; tee output) | sha256sum

does result in some missing writes and broken input that looks like what's going on in this thread (in this case, it's writing to my terminal, which isn't fast enough to keep up; but you could also pipe to something like "tee output | sha256sum" to see that the output changes with each run). And naturally you'll need a big enough output from ls-remote to fill the pipe buffer.

However, Git _does_ eventually produce a non-zero exit code in this case, because we check ferror() after running any builtin. So it eventually ends with:

  fatal: unknown write failure on standard output

So I dunno. Maybe this is not the same thing. I do think running the problematic case under "strace -o foo.out" may yield more information.

-Peff