{"thread":{"id":"59830","subject":"Anyone know why git ls-remote output might be corrupted?","startedAt":"2023-06-02T18:59:38Z","lastAt":"2023-06-12T20:01:29Z","messageCount":14,"participants":["Paul Smith","rsbecker@nexbridge.com","Elijah Newren","Jeff King","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"477976","messageId":"b6f210da2c3cc7746b984b797ad89687cba2d1f8.camel@mad-scientist.net","threadId":"59830","inReplyTo":null,"subject":"Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-02T18:59:32Z","receivedAt":"2023-06-02T18:59:38Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"I have some scripting on my CI/CD servers that invokes git ls-remote\nand parses the output.  The scripting is in Python.  Sometimes, but not\nalways, the output of this command is corrupted.  I've enhanced the\nerror handling and I see this:\n\n>> git ls-remote --heads origin\n*** INTERNAL: remote branch lookup failed:\nOutput:\n-----\n8431d80571dea5cc8e6d0848f27124f66346dcc4        refs/heads/foo1\naaec1feb1167cf3fbd39a36cdd7736679a9f4fae        refs/heads/foo2\n6167c73fbaded389ff54d52a01878975f4a6d5e5        refs/heads/foo3\n   ...\n3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx1\n3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx2\n795d2ff669041fc91341cf5bf820aibab79dc92bd741e77a7dcf71d94285a6ae494dc0        refs/heads/yyy1\n1496ea0ddab29ae3935754fced4bd5858cff7940        refs/heads/yyy2\n1496ea0ddab29ae3935754fced4bd5858cff7940        refs/heads/yyy3\n-----\n\nAlso a bunch of the heads are missing.  It's pretty clear that right in\nthe middle of printing one of the SHAs we suddenly lost a bunch of\noutput, and started printing stuff from later (in the last instance 66\nout of 131 heads were missing).  Breaking down the output above you can\nsee:\n\n  3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx2\n  795d2ff669041fc91341cf5bf820aibab79dc92bd741e77a7dcf71d94285a6ae494dc0        refs/heads/yyy1\n                               ^\n\nwhere the \"795d2ff669041fc91341cf5bf820a\" before the \"i\" char is a\nvalid start of a SHA for a head (not shown), then the \"i\", then a fully\nvalid SHA for heads/yyy1 which is 66 heads later.\n\nI've seen this happen with different versions of Git on the client side\nand on different OS's (Windows and Linux).  The only real constant here\nis the Git server.\n\nIs it feasible that this is an error on the server side?  Does the\nserver return formatted output for ls-remote?  I would naively have\nexpected it to return a binary representation of the heads, and the\nformatting be done by the client.  But maybe not.\n\nI should point out that subsequent runs in the same client work fine,\nand when I log into the CI/CD client and run the command by hand I\ncan't reproduce it.  It's random.  I also can't find any errors printed\nanywhere on either the server (we are using ssh access only for client\nconnections, managed via gitolite) or the client, except for the above.\n\nHelp, this is causing a few spurious failures in my CI/CD system per\nday!\n"},{"id":"477979","messageId":"7aa2ab6714bd14671ba9cfff611dea2fa088c99e.camel@mad-scientist.net","threadId":"59830","inReplyTo":"b6f210da2c3cc7746b984b797ad89687cba2d1f8.camel@mad-scientist.net","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-02T19:12:52Z","receivedAt":"2023-06-02T19:13:58Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Fri, 2023-06-02 at 14:59 -0400, Paul Smith wrote:\n> Also a bunch of the heads are missing.  It's pretty clear that right\n> in the middle of printing one of the SHAs we suddenly lost a bunch of\n> output, and started printing stuff from later (in the last instance\n> 66 out of 131 heads were missing).\n\nI forgot to mention: git ls-remote does not exit with an error code. \nThe exit code is 0 (success).\n\nThe reason I get this failure is that as I parse the output I notice\nthat the SHA is invalid (contains a non-hex character \"i\") and it\nthrows this error.\n"},{"id":"477983","messageId":"000501d99589$358d4850$a0a7d8f0$@nexbridge.com","threadId":"59830","inReplyTo":"7aa2ab6714bd14671ba9cfff611dea2fa088c99e.camel@mad-scientist.net","subject":"RE: Anyone know why git ls-remote output might be corrupted?","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-06-02T19:34:06Z","receivedAt":"2023-06-02T19:35:10Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Friday, June 2, 2023 3:13 PM, Paul Smith wrote:\n>On Fri, 2023-06-02 at 14:59 -0400, Paul Smith wrote:\n>> Also a bunch of the heads are missing.  It's pretty clear that right\n>> in the middle of printing one of the SHAs we suddenly lost a bunch of\n>> output, and started printing stuff from later (in the last instance\n>> 66 out of 131 heads were missing).\n>\n>I forgot to mention: git ls-remote does not exit with an error code.\n>The exit code is 0 (success).\n>\n>The reason I get this failure is that as I parse the output I notice that the SHA is invalid\n>(contains a non-hex character \"i\") and it throws this error.\n\nDoes your CI/CD system use sparse checkout or depth=1 or some other partial clone?\n\n"},{"id":"477985","messageId":"679863bd1ed8a54b48472ad310c2bae7f274e1ec.camel@mad-scientist.net","threadId":"59830","inReplyTo":"000501d99589$358d4850$a0a7d8f0$@nexbridge.com","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-02T19:53:09Z","receivedAt":"2023-06-02T19:53:15Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Fri, 2023-06-02 at 15:34 -0400, rsbecker@nexbridge.com wrote:\n> On Friday, June 2, 2023 3:13 PM, Paul Smith wrote:\n> > On Fri, 2023-06-02 at 14:59 -0400, Paul Smith wrote:\n> > > Also a bunch of the heads are missing.  It's pretty clear that\n> > > right in the middle of printing one of the SHAs we suddenly lost\n> > > a bunch of output, and started printing stuff from later (in the\n> > > last instance 66 out of 131 heads were missing).\n> > \n> > I forgot to mention: git ls-remote does not exit with an error\n> > code.  The exit code is 0 (success).\n> > \n> > The reason I get this failure is that as I parse the output I\n> > notice that the SHA is invalid (contains a non-hex character \"i\")\n> > and it throws this error.\n> \n> Does your CI/CD system use sparse checkout or depth=1 or some other\n> partial clone?\n\nYes, the local copy of the repo is a sparse checkout.\n\nI'm surprised that matters to ls-remote... I would have expected that\nthe \"sparseness\" of the local repo is irrelevant when listing the state\nof the remote's heads?  Is that the reason for the issue I'm seeing?\n"},{"id":"477986","messageId":"000f01d9958d$347d6220$9d782660$@nexbridge.com","threadId":"59830","inReplyTo":"679863bd1ed8a54b48472ad310c2bae7f274e1ec.camel@mad-scientist.net","subject":"RE: Anyone know why git ls-remote output might be corrupted?","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-06-02T20:02:42Z","receivedAt":"2023-06-02T20:03:06Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Friday, June 2, 2023 3:53 PM, Paul Smith wrote:\n>On Fri, 2023-06-02 at 15:34 -0400, rsbecker@nexbridge.com wrote:\n>> On Friday, June 2, 2023 3:13 PM, Paul Smith wrote:\n>> > On Fri, 2023-06-02 at 14:59 -0400, Paul Smith wrote:\n>> > > Also a bunch of the heads are missing.  It's pretty clear that\n>> > > right in the middle of printing one of the SHAs we suddenly lost a\n>> > > bunch of output, and started printing stuff from later (in the\n>> > > last instance 66 out of 131 heads were missing).\n>> >\n>> > I forgot to mention: git ls-remote does not exit with an error code.\n>> > The exit code is 0 (success).\n>> >\n>> > The reason I get this failure is that as I parse the output I notice\n>> > that the SHA is invalid (contains a non-hex character \"i\") and it\n>> > throws this error.\n>>\n>> Does your CI/CD system use sparse checkout or depth=1 or some other\n>> partial clone?\n>\n>Yes, the local copy of the repo is a sparse checkout.\n>\n>I'm surprised that matters to ls-remote... I would have expected that the \"sparseness\"\n>of the local repo is irrelevant when listing the state of the remote's heads?  Is that\n>the reason for the issue I'm seeing?\n\nI'm just wondering whether this might be an impact somehow and adding info to help the team diagnose. I have seen other commands have some issues in the past with --depth=n\n--Randall\n\n"},{"id":"477987","messageId":"110083dc83b21cad7a9cb143604cd13dba097110.camel@mad-scientist.net","threadId":"59830","inReplyTo":"000f01d9958d$347d6220$9d782660$@nexbridge.com","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-02T20:12:35Z","receivedAt":"2023-06-02T20:12:41Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Fri, 2023-06-02 at 16:02 -0400, rsbecker@nexbridge.com wrote:\n> > > Does your CI/CD system use sparse checkout or depth=1 or some\n> > > other partial clone?\n> > \n> > Yes, the local copy of the repo is a sparse checkout.\n> > \n> > I'm surprised that matters to ls-remote... I would have expected\n> > that the \"sparseness\" of the local repo is irrelevant when listing\n> > the state of the remote's heads?\n> \n> I'm just wondering whether this might be an impact somehow and adding\n> info to help the team diagnose. I have seen other commands have some\n> issues in the past with --depth=n\n\nI see.  Well I can try changing my call to avoid the local repo in any\nway, and run 'git ls-remote --heads user@server:reponame' from a\ntemporary directory outside of any repo, rather than using \"origin\".\n\nI would be surprised if it makes a difference but the behavior is sure\nstrange enough that I wouldn't be THAT surprised :)\n"},{"id":"477997","messageId":"CABPp-BF9Xjww=BBkL4qQcENo-UCHd8eEj334ho1iO1EMbGxhZw@mail.gmail.com","threadId":"59830","inReplyTo":"b6f210da2c3cc7746b984b797ad89687cba2d1f8.camel@mad-scientist.net","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-06-03T01:12:52Z","receivedAt":"2023-06-03T01:14:28Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jun 2, 2023 at 12:27 PM Paul Smith <paul@mad-scientist.net> wrote:\n>\n> I have some scripting on my CI/CD servers that invokes git ls-remote\n> and parses the output.  The scripting is in Python.  Sometimes, but not\n> always, the output of this command is corrupted.  I've enhanced the\n> error handling and I see this:\n>\n> >> git ls-remote --heads origin\n> *** INTERNAL: remote branch lookup failed:\n> Output:\n> -----\n> 8431d80571dea5cc8e6d0848f27124f66346dcc4        refs/heads/foo1\n> aaec1feb1167cf3fbd39a36cdd7736679a9f4fae        refs/heads/foo2\n> 6167c73fbaded389ff54d52a01878975f4a6d5e5        refs/heads/foo3\n>    ...\n> 3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx1\n> 3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx2\n> 795d2ff669041fc91341cf5bf820aibab79dc92bd741e77a7dcf71d94285a6ae494dc0        refs/heads/yyy1\n> 1496ea0ddab29ae3935754fced4bd5858cff7940        refs/heads/yyy2\n> 1496ea0ddab29ae3935754fced4bd5858cff7940        refs/heads/yyy3\n> -----\n\nCan you trigger this problem with just `git ls-remote --heads origin`,\nor do you only see it after processing by your python script?\n\nIf you can trigger with the former, what does\n    GIT_TRACE_PACKET=1 git ls-remote --heads origin\nreport?\n\nIf the latter, can you find a way to minimize your python script or\nfind some equivalent shell commands with the same buggy behavior?\n\n> Also a bunch of the heads are missing.  It's pretty clear that right in\n> the middle of printing one of the SHAs we suddenly lost a bunch of\n> output, and started printing stuff from later (in the last instance 66\n> out of 131 heads were missing).  Breaking down the output above you can\n> see:\n>\n>   3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx2\n>   795d2ff669041fc91341cf5bf820aibab79dc92bd741e77a7dcf71d94285a6ae494dc0        refs/heads/yyy1\n>                                ^\n>\n> where the \"795d2ff669041fc91341cf5bf820a\" before the \"i\" char is a\n> valid start of a SHA for a head (not shown), then the \"i\", then a fully\n> valid SHA for heads/yyy1 which is 66 heads later.\n\nSounds kind of like\nhttps://lore.kernel.org/git/6786526.72e2EbofS7@devpool47/ which also\ntriggered for some other tooling and then was reduced down to some\nshell commands.  Unfortunately, the thread ended without a lot of\nresolution other than \"don't mix stdout and stderr\" and \"if we slow\ndown the network connection somehow, that'll avoid the problem\".\n"},{"id":"477999","messageId":"CABPp-BF0VZzDbB87JNCfCxkCDZk7LGZ=1SfnxCyHwCpQ3ZFPoQ@mail.gmail.com","threadId":"59830","inReplyTo":"000f01d9958d$347d6220$9d782660$@nexbridge.com","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-06-03T01:17:01Z","receivedAt":"2023-06-03T01:17:20Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jun 2, 2023 at 1:26 PM <rsbecker@nexbridge.com> wrote:\n>\n> On Friday, June 2, 2023 3:53 PM, Paul Smith wrote:\n> >On Fri, 2023-06-02 at 15:34 -0400, rsbecker@nexbridge.com wrote:\n> >> On Friday, June 2, 2023 3:13 PM, Paul Smith wrote:\n> >> > On Fri, 2023-06-02 at 14:59 -0400, Paul Smith wrote:\n> >> > > Also a bunch of the heads are missing.  It's pretty clear that\n> >> > > right in the middle of printing one of the SHAs we suddenly lost a\n> >> > > bunch of output, and started printing stuff from later (in the\n> >> > > last instance 66 out of 131 heads were missing).\n> >> >\n> >> > I forgot to mention: git ls-remote does not exit with an error code.\n> >> > The exit code is 0 (success).\n> >> >\n> >> > The reason I get this failure is that as I parse the output I notice\n> >> > that the SHA is invalid (contains a non-hex character \"i\") and it\n> >> > throws this error.\n> >>\n> >> Does your CI/CD system use sparse checkout or depth=1 or some other\n> >> partial clone?\n> >\n> >Yes, the local copy of the repo is a sparse checkout.\n> >\n> >I'm surprised that matters to ls-remote... I would have expected that the \"sparseness\"\n> >of the local repo is irrelevant when listing the state of the remote's heads?  Is that\n> >the reason for the issue I'm seeing?\n>\n> I'm just wondering whether this might be an impact somehow and adding info to help the team diagnose. I have seen other commands have some issues in the past with --depth=n\n> --Randall\n\nI think if shallowness or sparseness affected ls-remote output in any\nway whatsoever, that would itself be a bug.  Granted, I don't know\nmuch about the protocol side of things, but I'd be very surprised if\neither of these conditions mattered.\n"},{"id":"478023","messageId":"20230604060048.GA38176@coredump.intra.peff.net","threadId":"59830","inReplyTo":"CABPp-BF9Xjww=BBkL4qQcENo-UCHd8eEj334ho1iO1EMbGxhZw@mail.gmail.com","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-04T06:00:48Z","receivedAt":"2023-06-04T06:04:48Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 02, 2023 at 06:12:52PM -0700, Elijah Newren wrote:\n\n> > Also a bunch of the heads are missing.  It's pretty clear that right in\n> > the middle of printing one of the SHAs we suddenly lost a bunch of\n> > output, and started printing stuff from later (in the last instance 66\n> > out of 131 heads were missing).  Breaking down the output above you can\n> > see:\n> >\n> >   3a2e8036a6f6605d4dd14c72bd395298bff9d80e        refs/heads/xxx2\n> >   795d2ff669041fc91341cf5bf820aibab79dc92bd741e77a7dcf71d94285a6ae494dc0        refs/heads/yyy1\n> >                                ^\n> >\n> > where the \"795d2ff669041fc91341cf5bf820a\" before the \"i\" char is a\n> > valid start of a SHA for a head (not shown), then the \"i\", then a fully\n> > valid SHA for heads/yyy1 which is 66 heads later.\n> \n> Sounds kind of like\n> https://lore.kernel.org/git/6786526.72e2EbofS7@devpool47/ which also\n> triggered for some other tooling and then was reduced down to some\n> shell commands.  Unfortunately, the thread ended without a lot of\n> resolution other than \"don't mix stdout and stderr\" and \"if we slow\n> down the network connection somehow, that'll avoid the problem\".\n\nThanks for digging up that thread. I had a vague memory of this coming\nup before, but wasn't sure what to search for to find it. :)\n\nFrom that thread, one theory that I think remains unexplored: could\nls-remote's stdout be opened in non-blocking mode?\n\nThe output of ls-remote is written with printf(). We don't bother\nchecking for errors from stdio, since typically a write error would\nresult in us getting EPIPE and being killed. But if stdout were\nunexpectedly in non-blocking mode, we might get EAGAIN. I'm not sure how\nlibc would handle that.\n\nNow, why the descriptor would be in non-blocking mode, I have no idea.\nBut maybe something funny going on in your python script.\n\nI'd be curious if applying the patch from:\n\n  https://lore.kernel.org/git/YUTo1BTp7BXOw6K9@coredump.intra.peff.net/\n\nreports any problems. As well as whether the suggested \"sleep\" pipeline\nthere (triggered via your script in this case) shows the problem more\nreliably.\n\n-Peff\n"},{"id":"478024","messageId":"20230604062556.GA42964@coredump.intra.peff.net","threadId":"59830","inReplyTo":"20230604060048.GA38176@coredump.intra.peff.net","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-04T06:25:56Z","receivedAt":"2023-06-04T06:26:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 04, 2023 at 02:00:48AM -0400, Jeff King wrote:\n\n> Now, why the descriptor would be in non-blocking mode, I have no idea.\n> But maybe something funny going on in your python script.\n> \n> I'd be curious if applying the patch from:\n> \n>   https://lore.kernel.org/git/YUTo1BTp7BXOw6K9@coredump.intra.peff.net/\n> \n> reports any problems. As well as whether the suggested \"sleep\" pipeline\n> there (triggered via your script in this case) shows the problem more\n> reliably.\n\nIt does look like glibc's stdio will throw away buffer contents that get\nEAGAIN. Doing:\n\n  perl -MFcntl -e '\n    fcntl(STDOUT, F_GETFL, $flags);\n    $flags |= O_NONBLOCK;\n    fcntl(STDOUT, F_SETFL, $flags);\n    exec @ARGV;\n  ' git ls-remote . | (sleep 1; tee output) | sha256sum\n\ndoes result in some missing writes and broken input that looks like\nwhat's going on in this thread (in this case, it's writing to my\nterminal, which isn't fast enough to keep up; but you could also pipe to\nsomething like \"tee output | sha256sum\" to see that the output changes\nwith each run). And naturally you'll need a big enough output from\nls-remote to fill the pipe buffer.\n\nHowever, Git _does_ eventually produce a non-zero exit code in this\ncase, because we check ferror() after running any builtin. So it\neventually ends with:\n\n  fatal: unknown write failure on standard output\n\nSo I dunno. Maybe this is not the same thing. I do think running\nthe problematic case under \"strace -o foo.out\" may yield more\ninformation.\n\n-Peff\n"},{"id":"478025","messageId":"20230604063010.GA47137@coredump.intra.peff.net","threadId":"59830","inReplyTo":"20230604062556.GA42964@coredump.intra.peff.net","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-06-04T06:30:10Z","receivedAt":"2023-06-04T06:30:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jun 04, 2023 at 02:25:57AM -0400, Jeff King wrote:\n\n> It does look like glibc's stdio will throw away buffer contents that get\n> EAGAIN. Doing:\n> \n>   perl -MFcntl -e '\n>     fcntl(STDOUT, F_GETFL, $flags);\n>     $flags |= O_NONBLOCK;\n>     fcntl(STDOUT, F_SETFL, $flags);\n>     exec @ARGV;\n>   ' git ls-remote . | (sleep 1; tee output) | sha256sum\n> \n> does result in some missing writes and broken input that looks like\n> what's going on in this thread (in this case, it's writing to my\n> terminal, which isn't fast enough to keep up; but you could also pipe to\n> something like \"tee output | sha256sum\" to see that the output changes\n> with each run). And naturally you'll need a big enough output from\n> ls-remote to fill the pipe buffer.\n\nSorry for some slight confusion above. I edited my example command to\nshow piping to \"sha256sum\", but didn't modify the paragraph below it\n(originally I was not piping anywhere, just sending to the terminal).\n\nAdding the \"sleep 1\" means that the command will always fail (ls-remote\neasily writes all of its output and hits EAGAIN before the other side\nreads anything). But it means that the output is deterministic (the\nfirst PIPE_BUF bytes make it through, and nothing else does). Removing\nthe sleep makes the output non-deterministic, but much more interesting\n(you get missing chunks in the interior of the output).\n\nSo perhaps:\n\n  perl ... git ls-remote . | tee output | sha256sum\n\nis the most interesting case, because you'll get one of several possible\nbroken outputs, which you can identify by the changing sha256 output\n(and then see the actual breakage by peeking at the \"output\" file).\n\n-Peff\n"},{"id":"478182","messageId":"e6f9334dd93c683b97ecdf61eb06bbe28b0a4e30.camel@mad-scientist.net","threadId":"59830","inReplyTo":"CABPp-BF9Xjww=BBkL4qQcENo-UCHd8eEj334ho1iO1EMbGxhZw@mail.gmail.com","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-09T15:33:42Z","receivedAt":"2023-06-09T15:33:52Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Fri, 2023-06-02 at 18:12 -0700, Elijah Newren wrote:\n> Sounds kind of like\n> https://lore.kernel.org/git/6786526.72e2EbofS7@devpool47/ which also\n> triggered for some other tooling and then was reduced down to some\n> shell commands.  Unfortunately, the thread ended without a lot of\n> resolution other than \"don't mix stdout and stderr\" and \"if we slow\n> down the network connection somehow, that'll avoid the problem\".\n\nAs I was riding my bike home last week I had this exact same epiphany.\nSo, I examined my script and discovered that indeed, I was using\nsubprocess.PIPE for stderr, which is the Python equivalent of the\nshell's 2>&1 operation.  So, both stdout and stderr are going to the\nsame place.\n\nI also checked and indeed, the git ls-remote command does print to both\nstdout and stderr as part of its \"standard\" behavior:\n\n  $ git ls-remote --heads >/dev/null\n  From git@git:myrepo\n\nThis is unexpected to me, although of course there's nothing inherently\nwrong with it but usually you don't expect \"regular\" output to go to\nstderr.  I suppose the idea is that people can run:\n\n  $ git ls-remote --heads 2>/dev/null\n\nif they want just the output without the header.\n\nAnyway, I modified my scripting to keep stdout and stderr separate and\nI haven't heard anything about things continuing to fail so I guess\nthat was the problem.\n\nThe mode of failure is very very bizarre to me: first why is it just\none \"bogus\" character embedded in the output?  Second why are 60 lines\njust dropped?  Third where did the REST of the stderr output go?\n\nAnd finally, why did this not fail 100% of the time on Linux?  I can\nbelieve that Windows pipe behavior is not POSIX and maybe Python's\nemulation of it is imperfect, but surely that is not the case on Linux\nand we would ALWAYS get the full results of both stdout and stderr on\nLinux, even if mixed together?!?!\n\nThe mysteries never end.\n\nThanks all; hopefully this helps someone else.\n"},{"id":"478183","messageId":"xmqqo7loo42s.fsf@gitster.g","threadId":"59830","inReplyTo":"e6f9334dd93c683b97ecdf61eb06bbe28b0a4e30.camel@mad-scientist.net","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-09T18:58:35Z","receivedAt":"2023-06-09T18:58:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Paul Smith <paul@mad-scientist.net> writes:\n\n> I also checked and indeed, the git ls-remote command does print to both\n> stdout and stderr as part of its \"standard\" behavior:\n>\n>   $ git ls-remote --heads >/dev/null\n>   From git@git:myrepo\n>\n> This is unexpected to me, although of course there's nothing inherently\n> wrong with it but usually you don't expect \"regular\" output to go to\n> stderr.  I suppose the idea is that people can run:\n>\n>   $ git ls-remote --heads 2>/dev/null\n>\n> if they want just the output without the header.\n\nThe above sounds like a reasonable expectation; then the issue is\nthere are some fflush missing when the command writes to one stream\nand switches to write to another stream?\n\nThanks.\n"},{"id":"478296","messageId":"21193d97d43094a01b4ac9442448135867e22c70.camel@mad-scientist.net","threadId":"59830","inReplyTo":"xmqqo7loo42s.fsf@gitster.g","subject":"Re: Anyone know why git ls-remote output might be corrupted?","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2023-06-12T19:59:09Z","receivedAt":"2023-06-12T20:01:29Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Sat, 2023-06-10 at 03:58 +0900, Junio C Hamano wrote:\n> Paul Smith <paul@mad-scientist.net> writes:\n> \n> > I also checked and indeed, the git ls-remote command does print to\n> > both stdout and stderr as part of its \"standard\" behavior:\n> > \n> >    $ git ls-remote --heads >/dev/null\n> >    From git@git:myrepo\n> > \n> > This is unexpected to me, although of course there's nothing\n> > inherently wrong with it but usually you don't expect \"regular\"\n> > output to go to stderr.  I suppose the idea is that people can run:\n> > \n> >    $ git ls-remote --heads 2>/dev/null\n> > \n> > if they want just the output without the header.\n> \n> The above sounds like a reasonable expectation; then the issue is\n> there are some fflush missing when the command writes to one stream\n> and switches to write to another stream?\n\nIt's not immediately clear to me which \"above\" you refer to as a\nreasonable expectation (the reason for the stdout vs. stderr different\nI suppose?)\n\nBut I agree that forcing flush would be a good idea (or perhaps forcing\nline buffering?  Changing buffering can be annoying to do portably),\nfor situations in which the output is being sent to a non-TTY, because\nthe default in that situation is fully buffered output.\n\nMaybe it would be sufficient to have any output to stderr run\nfflush(stdout) before the output, and fflush(stderr) after the output.\nPresumably output to stderr is relatively rare, and so this wouldn't be\nvery noticeable.\n"}]}