Re: Anyone know why git ls-remote output might be corrupted?

From: Paul Smith <paul@mad-scientist.net>
Date: Jun 9, 2023, 15:33 UTC
Message-ID: <e6f9334dd93c683b97ecdf61eb06bbe28b0a4e30.camel@mad-scientist.net>
In-Reply-To: <CABPp-BF9Xjww=BBkL4qQcENo-UCHd8eEj334ho1iO1EMbGxhZw@mail.gmail.com>

On Fri, 2023-06-02 at 18:12 -0700, Elijah Newren wrote:

Show 6 quoted lines

> Sounds kind of like
> https://lore.kernel.org/git/6786526.72e2EbofS7@devpool47/ which also
> triggered for some other tooling and then was reduced down to some
> shell commands.  Unfortunately, the thread ended without a lot of
> resolution other than "don't mix stdout and stderr" and "if we slow
> down the network connection somehow, that'll avoid the problem".

As I was riding my bike home last week I had this exact same epiphany. So, I examined my script and discovered that indeed, I was using subprocess.PIPE for stderr, which is the Python equivalent of the shell's 2>&1 operation. So, both stdout and stderr are going to the same place.

I also checked and indeed, the git ls-remote command does print to both stdout and stderr as part of its "standard" behavior:

  $ git ls-remote --heads >/dev/null
  From git@git:myrepo

This is unexpected to me, although of course there's nothing inherently wrong with it but usually you don't expect "regular" output to go to stderr. I suppose the idea is that people can run:

  $ git ls-remote --heads 2>/dev/null

if they want just the output without the header.

Anyway, I modified my scripting to keep stdout and stderr separate and I haven't heard anything about things continuing to fail so I guess that was the problem.

The mode of failure is very very bizarre to me: first why is it just one "bogus" character embedded in the output? Second why are 60 lines just dropped? Third where did the REST of the stderr output go?

And finally, why did this not fail 100% of the time on Linux? I can believe that Windows pipe behavior is not POSIX and maybe Python's emulation of it is imperfect, but surely that is not the case on Linux and we would ALWAYS get the full results of both stdout and stderr on Linux, even if mixed together?!?!

The mysteries never end.

Thanks all; hopefully this helps someone else.