Re: Anyone know why git ls-remote output might be corrupted?

From: Jeff King <peff@peff.net>
Date: Jun 4, 2023, 06:30 UTC
Message-ID: <20230604063010.GA47137@coredump.intra.peff.net>
In-Reply-To: <20230604062556.GA42964@coredump.intra.peff.net>

On Sun, Jun 04, 2023 at 02:25:57AM -0400, Jeff King wrote:

Show 16 quoted lines

> It does look like glibc's stdio will throw away buffer contents that get
> EAGAIN. Doing:
> 
>   perl -MFcntl -e '
>     fcntl(STDOUT, F_GETFL, $flags);
>     $flags |= O_NONBLOCK;
>     fcntl(STDOUT, F_SETFL, $flags);
>     exec @ARGV;
>   ' git ls-remote . | (sleep 1; tee output) | sha256sum
> 
> does result in some missing writes and broken input that looks like
> what's going on in this thread (in this case, it's writing to my
> terminal, which isn't fast enough to keep up; but you could also pipe to
> something like "tee output | sha256sum" to see that the output changes
> with each run). And naturally you'll need a big enough output from
> ls-remote to fill the pipe buffer.

Sorry for some slight confusion above. I edited my example command to show piping to "sha256sum", but didn't modify the paragraph below it (originally I was not piping anywhere, just sending to the terminal).

Adding the "sleep 1" means that the command will always fail (ls-remote easily writes all of its output and hits EAGAIN before the other side reads anything). But it means that the output is deterministic (the first PIPE_BUF bytes make it through, and nothing else does). Removing the sleep makes the output non-deterministic, but much more interesting (you get missing chunks in the interior of the output).

So perhaps:

  perl ... git ls-remote . | tee output | sha256sum

is the most interesting case, because you'll get one of several possible broken outputs, which you can identify by the changing sha256 output (and then see the actual breakage by peeking at the "output" file).

-Peff